Digital twinborn model construction method and device of hinge object, terminal and medium

Abstract

The invention provides a digital twinborn model construction method and device for a hinge object, a terminal and a medium, and the method comprises the steps: constructing a three-dimensional Gaussian splashing model in an initial state through employing a first multi-view image set of the hinge object in the initial state, and identifying an initial movable Gaussian model based on a deformed three-dimensional Gaussian splashing model predicted by the deformation network, and realizing complete self-supervised optimization of the deformation network based on a second multi-view image set of the hinge object in an end state and the deformed model. And jointly optimizing the three-dimensional Gaussian splashing model, re-segmenting the initial movable Gaussian model to obtain a target movable Gaussian model, and acting the estimated initial hinge motion parameters on the initial movable Gaussian model to optimize the motion parameters to obtain target hinge motion parameters, so as to construct digital twinning of the hinge object. According to the method, digital twinning of the hinge object can be accurately reconstructed through the sparse state of the hinge object in a complete self-supervision mode, and the hinge logic is estimated with high accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of digital twin model construction, and in particular to a method, device, terminal and medium for constructing a digital twin model of a hinged object. Background Art

[0002] At present, building a digital twin model of a hinged object, accurately reconstructing it and estimating the hinge logic often plays an important role in the fields of robotics, simulation, animation production, etc.

[0003] However, traditional hinged object modeling methods mostly rely on prior knowledge, which requires a large amount of annotated 3D data as pre-training material, which is very labor-intensive and resource-intensive. In addition, when the hinged object to be reconstructed is not in the pre-training dataset, it may be impossible to estimate the correct hinge logic. In addition, the existing purely self-supervised algorithms use neural radiance fields (NeRF) for reconstruction. Their unified optimization method makes the correctness of the results strongly dependent on parameter initialization, resulting in a high error rate. In addition, for the traditional 3D Gaussian algorithm for reconstructing dynamic scenes, it is unable to estimate the hinge logic, resulting in the inability to accurately interpolate the motion state of the object in the case of sparse state input.

[0004] For example, in the field of articulated object reconstruction, there is a way to reconstruct a surface from a series of point clouds, but this method has the limitation of representing the articulated object with a unified surface, and the method of reconstructing the articulated object at the component level also has the problem of being highly dependent on the density of frames, where when the frames are sparse, the Gaussian motion process between frames cannot be accurately predicted; or three-dimensional Gaussian splattering (3D-GS) technology is used to simulate organic articulated objects such as humans, body parts, and animals, but the movements allowed by these objects are usually structurally complex, with a relatively large number of joints connecting the topology and degrees of freedom, resulting in increased construction complexity, and most previous work on the construction of digital twins of articulated objects has focused on the regeneration or construction of three-dimensional geometric representations of articulated objects, which involves estimating articulation parameters or functions to connect articulated objects. Animation to different articulated states is usually based on a mesh-based representation of hinged objects, but the mesh-based representation of hinged objects has relatively low geometric fidelity and visual realism. On the one hand, the current reconstruction by supervised learning usually consumes a large amount of 3D data sets annotated with hinge information, which also limits the generalization ability to unseen object categories. In the self-supervised work with RGB observations as input, neural radiance field (NeRF) is used as an intermediate representation, and a mesh model with compromised geometric and appearance accuracy is output, but it exhibits instability and suboptimality problems during the optimization process. On the other hand, the existing reconstruction of deformable 3D Gaussian splashing requires dense state input, or relies on manual interaction for deformation, and does not have the ability to automatically understand the deformation logic.

[0005] In summary, the existing technologies for constructing digital twins of hinged objects cannot accurately determine the motion state of the hinged object under sparse state input, and cannot accurately estimate the correct hinge logic.

[0006] Therefore, how to provide a solution to the above technical problems is a problem that those skilled in the art need to solve at present. Summary of the invention

[0007] The technical problem to be solved by the present invention is that, in response to the above-mentioned defects of the prior art, a method, device, terminal and medium for constructing a digital twin model of a hinged object are provided, which can estimate the hinge logic of the hinged object with high accuracy, and can improve the geometric accuracy, visual accuracy of the reconstruction of the hinged object and the estimation accuracy of the hinge structure.

[0008] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0009] A method for constructing a digital twin model of a hinge object, wherein the method comprises:

[0010] Acquire a first multi-view picture set when a movable part in a hinge object is in a starting state before movement and a second multi-view picture set when the movable part is in a final state after movement, and construct a three-dimensional Gaussian splash model of the hinge object in the starting state using the first multi-view picture set;

[0011] The pre-constructed deformation network is used to predict the rigid transformation of each Gaussian point in the three-dimensional Gaussian splash model relative to the end state to obtain the corresponding deformed three-dimensional Gaussian splash model, and the initial movable Gaussian model corresponding to the movable part in the hinge object is identified based on the deformed three-dimensional Gaussian splash model; wherein the deformation network is a neural network obtained by self-supervised training optimization based on the second multi-view image set and the deformed three-dimensional Gaussian splash model;

[0012] Applying the pre-estimated initial hinge motion parameters to the initial movable Gaussian model to optimize the initial hinge motion parameters to obtain corresponding target hinge motion parameters, and re-segmenting the initial movable Gaussian model to obtain a target movable Gaussian model;

[0013] A joint optimization operation is performed based on the three-dimensional Gaussian splash model, the target movable Gaussian model and the target hinge motion parameters to construct a target three-dimensional Gaussian digital twin model corresponding to the hinge object, having the movable parts and corresponding hinge logic.

[0014] In one implementation, the method of using a pre-constructed deformation network to predict the rigid transformation of each Gaussian point in the three-dimensional Gaussian splash model relative to the end state to obtain the corresponding deformed three-dimensional Gaussian splash model includes:

[0015] Performing a preset gradient truncation operation on the Gaussian point position corresponding to each Gaussian point in the three-dimensional Gaussian splash model to obtain a processed Gaussian point position;

[0016] Inputting the processed Gaussian point positions into a pre-constructed deformation network to obtain deformation parameters of each Gaussian point predicted and output by the deformation network relative to the end state;

[0017] Determining a corresponding deformed three-dimensional Gaussian splash model based on the deformation parameters;

[0018] The identifying, based on the deformed three-dimensional Gaussian splash model, an initial movable Gaussian model corresponding to the movable component in the hinge object comprises:

[0019] Normalizing the deformation parameters corresponding to the deformed three-dimensional Gaussian splash model to obtain processed deformation parameters;

[0020] Determining whether the processed deformation parameter is less than a first preset threshold;

[0021] When the processed deformation parameter is not less than the first preset threshold, the processed deformation parameter is determined as the target deformation parameter, and the Gaussian model composed of Gaussian points corresponding to the target deformation parameter is determined as the initial movable Gaussian model corresponding to the movable part in the hinge object.

[0022] In one implementation, the performing self-supervised training optimization based on the second multi-view picture set and the deformed three-dimensional Gaussian splash model further includes:

[0023] Inputting the deformed three-dimensional Gaussian splash model into a pre-built Gaussian differentiable rendering framework to obtain a corresponding first rendered image;

[0024] Calculate the L1 loss and D_SSIM loss between the first rendered image and the first target image at the corresponding perspective in the second multi-view image set;

[0025] Determine a corresponding first appearance loss based on the L1 loss and the D_SSIM loss, and determine a corresponding ARAP loss;

[0026] A corresponding deformation loss is determined based on the first appearance loss and the ARAP loss so that the deformation network performs self-supervised training optimization according to the deformation loss.

[0027] In one implementation, the step of applying the pre-estimated initial hinge motion parameters to the initial movable Gaussian model to optimize the initial hinge motion parameters to obtain corresponding target hinge motion parameters includes:

[0028] Applying the pre-estimated initial hinge motion parameters to the initial movable Gaussian model to control the initial movable Gaussian model to perform motion deformation, thereby obtaining a new deformed three-dimensional Gaussian splash model;

[0029] Inputting the new deformed three-dimensional Gaussian splash model into the pre-built Gaussian differentiable rendering framework to obtain a corresponding second rendered image;

[0030] Calculating a loss between the second rendered image and a second target image at a corresponding perspective in the second multi-perspective image set to obtain a corresponding second appearance loss;

[0031] Determine the coordinates of each Gaussian point corresponding to the new deformed three-dimensional Gaussian splash model to obtain a first set of Gaussian point coordinates, and determine a second set of Gaussian point coordinates based on the coordinates of each Gaussian point corresponding to the three-dimensional Gaussian splash model and the deformation parameters of each Gaussian point predicted and output by the deformation network relative to the end state;

[0032] Calculating the distance between the coordinates of the first group of Gaussian points and the coordinates of the second group of Gaussian points to obtain corresponding geometric losses;

[0033] The initial hinge motion parameters are optimized based on the appearance loss and the geometric loss to obtain corresponding target hinge motion parameters.

[0034] In one implementation, the performing of a joint optimization operation based on the three-dimensional Gaussian splash model, the target movable Gaussian model, and the target hinge motion parameters to construct a target three-dimensional Gaussian digital twin model corresponding to the hinge object and having the movable component and corresponding hinge logic includes:

[0035] The target hinge motion parameters are applied to the target movable Gaussian model to determine the state of the three-dimensional Gaussian splash model after the target movable Gaussian model moves, and a target three-dimensional Gaussian digital twin model corresponding to the hinge object and having the movable parts and corresponding hinge logic is obtained.

[0036] In one implementation, the method for constructing a digital twin model of a hinge object further includes:

[0037] Inputting the first target three-dimensional Gaussian digital twin model when the hinge object is in the initial state into a pre-built Gaussian differentiable rendering framework to obtain a corresponding third rendered image;

[0038] Calculating a third appearance loss between the third rendered image and a third target image at a corresponding perspective in the first multi-view image set;

[0039] Inputting the second target three-dimensional Gaussian digital twin model when the hinge object is in the end state into the pre-built Gaussian differentiable rendering framework to obtain a corresponding fourth rendered image;

[0040] Calculating a fourth appearance loss between the fourth rendered image and a fourth target image at a corresponding perspective in the second multi-view image set;

[0041] Dynamically setting corresponding weights for the third appearance loss and the fourth appearance loss to obtain a first target model building loss and a second target model building loss;

[0042] The construction of the target three-dimensional Gaussian digital twin model corresponding to the hinge object is iteratively optimized based on the first target model construction loss and the second target model construction loss.

[0043] In one implementation, when the hinge object is a hinge object having multiple movable parts, it further includes:

[0044] Acquire a third multi-view picture set when another movable part in the hinge object is in an ending state after the movement, and determine the constructed target three-dimensional Gaussian digital twin model corresponding to the hinge object as a new three-dimensional Gaussian splash model when the other movable part in the hinge object is in the starting state before the movement;

[0045] Based on the new three-dimensional Gaussian splash model and the third multi-view image set, the step of using the pre-constructed deformation network to predict the rigid transformation of each Gaussian point in the three-dimensional Gaussian splash model relative to the end state to obtain the corresponding deformed three-dimensional Gaussian splash model and its subsequent steps are re-executed to construct a new target three-dimensional Gaussian digital twin model corresponding to the hinge object, having the movable part, the other movable part and the corresponding hinge logic.

[0046] The present invention also discloses a device for constructing a digital twin model of a hinged object, wherein the device comprises:

[0047] An image acquisition module, used to acquire a first multi-view image set when the movable component in the hinge object is in a starting state before moving and a second multi-view image set when the movable component is in a final state after moving;

[0048] A first model building module, configured to build a three-dimensional Gaussian splash model of the hinge object when the hinge object is in the initial state by using the first multi-view picture set;

[0049] A model deformation module, used to use a pre-constructed deformation network to predict the rigid transformation of each Gaussian point in the three-dimensional Gaussian splash model relative to the end state to obtain a corresponding deformed three-dimensional Gaussian splash model;

[0050] A mobile model recognition module, used to identify an initial movable Gaussian model corresponding to the movable part in the hinge object based on the deformed three-dimensional Gaussian splash model; wherein the deformation network is a neural network obtained by self-supervised training optimization based on the second multi-view image set and the deformed three-dimensional Gaussian splash model;

[0051] A motion parameter optimization module, used for applying the pre-estimated initial hinge motion parameters to the initial movable Gaussian model to optimize the initial hinge motion parameters to obtain corresponding target hinge motion parameters;

[0052] A movable model segmentation module, used for re-segmenting the initial movable Gaussian model to obtain a target movable Gaussian model;

[0053] A digital twin construction module is used to perform a joint optimization operation based on the three-dimensional Gaussian splash model, the target movable Gaussian model and the target hinge motion parameters to construct a target three-dimensional Gaussian digital twin model corresponding to the hinge object with the movable parts and corresponding hinge logic.

[0054] The present invention also discloses a terminal, which includes: a memory, a processor, and a digital twin model construction program for a hinge object stored in the memory and runnable on the processor. When the digital twin model construction program for the hinge object is executed by the processor, the steps of the digital twin model construction method for the hinge object as described above are implemented.

[0055] The present invention also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program can be executed to implement the steps of the method for constructing a digital twin model of a hinge object as described above.

[0056] The present invention provides a method, device, terminal and medium for constructing a digital twin model of a hinged object. The method for constructing a digital twin model of a hinged object includes: obtaining a first multi-view picture set when a movable part in the hinged object is in a starting state before movement and a second multi-view picture set when the movable part is in an ending state after movement, and using the first multi-view picture set to construct a three-dimensional Gaussian splash model of the hinged object when it is in the starting state; using a pre-constructed deformation network to predict the rigid transformation of each Gaussian point in the three-dimensional Gaussian splash model relative to the ending state to obtain a corresponding deformed three-dimensional Gaussian splash model, and identifying the movable part in the hinged object based on the deformed three-dimensional Gaussian splash model. An initial movable Gaussian model corresponding to the movable part; wherein the deformation network is a neural network obtained by self-supervised training and optimization based on the second multi-view image set and the deformed three-dimensional Gaussian splash model; the pre-estimated initial hinge motion parameters are applied to the initial movable Gaussian model to optimize the initial hinge motion parameters to obtain the corresponding target hinge motion parameters, and the initial movable Gaussian model is re-segmented to obtain the target movable Gaussian model; a joint optimization operation is performed based on the three-dimensional Gaussian splash model, the target movable Gaussian model and the target hinge motion parameters to construct a target three-dimensional Gaussian digital twin model corresponding to the hinge object with the movable part and the corresponding hinge logic. It can be seen that the present invention, based on the three-dimensional Gaussian splattering technology, only needs multi-perspective images of the hinge object in two states to realize the self-supervised construction of the digital twin of the hinge object. That is, based on the multi-perspective images of the hinge object in a sparse state, a completely self-supervised method is used, and a hierarchical optimization method is used to stably and high-quality reconstruct the hinge object to obtain a digital twin model of the hinge object. At the same time, the hinge logic of the movable parts in the hinge object can be estimated with high accuracy, thereby improving the geometric accuracy and visual accuracy of the reconstruction of the hinge object and the estimation accuracy of the hinge structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a flow chart of a preferred embodiment of the method for constructing a digital twin model of a hinge object in the present invention;

[0058] Figure 2 It is a schematic diagram of a deformation field estimation process disclosed by the present invention;

[0059] Figure 3 is a schematic diagram of self-supervised hinge object reconstruction disclosed in the present invention;

[0060] Figure 4 It is a schematic diagram of a multi-moving component reconstruction disclosed by the present invention;

[0061] Figure 5This is a schematic diagram for comparing the effects disclosed by the present invention;

[0062] Figure 6 It is a functional principle block diagram of a preferred embodiment of a device for constructing a digital twin model of a hinged object in the present invention;

[0063] Figure 7 It is a functional principle block diagram of a preferred embodiment of the terminal in the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0065] See also Figure 1 , Figure 1 : is a flow chart of the method for constructing a digital twin model of a hinge object in the present invention. Figure 1 As shown, the method for constructing a digital twin model of a hinge object according to an embodiment of the present invention includes:

[0066] Step S11, obtaining a first multi-perspective picture set when the movable part in the hinge object is in a starting state before movement and a second multi-perspective picture set when the movable part is in an ending state after movement, and using the first multi-perspective picture set to construct a three-dimensional Gaussian splash model of the hinge object when it is in the starting state.

[0067] In this embodiment, multi-perspective images of the hinged object in two states I0 and I1 are obtained to obtain corresponding image sets, namely, a first multi-perspective image set when a movable part in the hinged object is in a starting state I0 before movement and a second multi-perspective image set when a movable part in the hinged object is in an ending state I1 after movement, and then the first multi-perspective image set is used to reconstruct a static three-dimensional Gaussian splash model in the starting state, wherein the three-dimensional Gaussian splash model is used as the underlying expression and its differentiable rendering and explicit expression are used to improve the training efficiency and the rendering quality under the new perspective.

[0068] It should be pointed out that the self-supervised reconstruction of hinged objects only requires multi-angle images of two states before and after the hinge structure of the hinged object changes, which is in line with the behavioral pattern of reconstructing hinged objects in daily life. Hinge objects mainly include two types of hinged objects, one is translational hinged objects, such as drawers, paper cutters, etc., and the other is rotational hinged objects, such as refrigerators, ovens, laptops, etc., and the hinge principle of the former is that the moving part translates along a certain direction a in a three-dimensional space, and the latter is that the moving part rotates along an axis in space, where the axis is determined by a point p in space.

[0069] Step S12, using a pre-constructed deformation network to predict the rigid transformation of each Gaussian point in the three-dimensional Gaussian splash model relative to the end state to obtain a corresponding deformed three-dimensional Gaussian splash model, and identifying an initial movable Gaussian model corresponding to the movable part in the hinge object based on the deformed three-dimensional Gaussian splash model; wherein the deformation network is a neural network obtained by self-supervised training and optimization based on the second multi-view image set and the deformed three-dimensional Gaussian splash model.

[0070] In this embodiment, a pre-constructed deformation network is used to predict the rigid transformation of each Gaussian point in the three-dimensional Gaussian splash model relative to the end state to obtain the corresponding deformed three-dimensional Gaussian splash model, and then the initial movable Gaussian model corresponding to the movable part in the hinge object is identified based on the deformed three-dimensional Gaussian splash model. Specifically, a preset gradient truncation operation is performed on the Gaussian point position corresponding to each Gaussian point in the three-dimensional Gaussian splash model to obtain the processed Gaussian point position; the processed Gaussian point position is input into the pre-constructed deformation network to obtain the deformation parameters of each Gaussian point predicted and output by the deformation network relative to the end state; the corresponding deformed three-dimensional Gaussian splash model is determined based on the deformation parameters; the deformation parameters corresponding to the deformed three-dimensional Gaussian splash model are normalized to obtain the processed deformation parameters; it is determined whether the processed deformation parameters are less than a first preset threshold; when the processed deformation parameters are not less than the first preset threshold, the processed deformation parameters are determined as target deformation parameters, and the Gaussian model composed of Gaussian points corresponding to the target deformation parameters is determined as the initial movable Gaussian model corresponding to the movable part in the hinge object. It can be understood that the second multi-view image set is used to pre-train the deformation network to obtain a pre-constructed deformation network, and then the deformation network is used to predict the deformation parameters of the relative end state of each Gaussian point in the three-dimensional Gaussian splash model, so as to determine the deformed three-dimensional Gaussian splash model, and based on the deformation parameters, the movable part of the hinge object can also be identified, that is, the movable Gaussian part in the three-dimensional Gaussian splash model is identified to obtain the initial movable Gaussian model. In other words, the deformation network is used to predict the deformation field of the three-dimensional Gaussian splash model, and then the initial movable area on the three-dimensional Gaussian splash model can be roughly identified, wherein the deformation parameters can include translational motion parameters and rotational motion parameters. For the deformed three-dimensional Gaussian splash model, the motion type can also be classified according to the degree of deformation to determine whether the movable part in the deformed three-dimensional Gaussian splash model is a translational hinge structure or a rotational hinge structure.

[0071] It should be pointed out that the roughly identified initial movable region (initial movable Gaussian model) does not need to be very accurate, because the pre-estimated motion parameters can be further refined in the subsequent joint optimization, and the initial movable Gaussian model can provide a good approximation as initialization for the subsequent joint optimization.

[0072] For example, see Figure 2 As shown, the deformation network uses the 3D Gaussian splash model to model the position x of each Gaussian point i As input, and output the corresponding predicted Gaussian point and Right now:

[0073]

[0074] in, represents Gaussian g i The translation of represents Gaussian g i rotation.

[0075] It should be pointed out that in order not to change the original position of each Gaussian point but only predict the deformation field, it can be i A gradient truncation operation sg(·) is added to the deformation network, and the network architecture of the deformation network can be a multi-layer perceptron (MLP), such as a four-layer perceptron, or a neural network with other structures, or a heuristic method such as simulated annealing.

[0076] In the process of identifying the movable Gaussian part in the three-dimensional Gaussian splash model, the displacement ‖δx‖2 and the rotation ‖δr‖2 are first normalized to between 0 and 1, and the Gaussian part with ‖δx‖2≥θ or ‖δr‖2≥θ is determined as the movable Gaussian model G m .

[0077] It should be noted that selecting a lower first preset threshold θ usually includes a stationary Gaussian part, which may lead to suboptimal results when optimizing motion parameters later. Therefore, a higher first preset threshold θ can be set to obtain a Gaussian part with a higher confidence level, and an initial movable Gaussian model can be obtained, so that it can be used for motion parameter prediction of the entire part. For example, the first preset threshold θ is set to 0.3 as a threshold for identifying whether a Gaussian part composed of Gaussian points is a movable Gaussian part.

[0078] Moreover, in the process of realizing self-supervised training optimization of the deformation network supervised by the end state I1, it can specifically include: inputting the deformed three-dimensional Gaussian splash model into a pre-built Gaussian differentiable rendering framework to obtain a corresponding first rendered image; calculating the L1 loss and D_SSIM loss between the first rendered image and the first target image under the corresponding perspective in the second multi-view image set; determining the corresponding first appearance loss based on the L1 loss and the D_SSIM loss, and determining the corresponding ARAP loss; determining the corresponding deformation loss based on the first appearance loss and the ARAP loss so that the deformation network performs self-supervised training optimization according to the deformation loss.

[0079] For example, the deformed three-dimensional Gaussian splash model is input into the Gaussian differentiable rendering framework to obtain the rendered image, and then the L1 loss and D_SSIM loss between the rendered image and the image at the corresponding perspective in the end state I1 are calculated, and then the L1 loss and D_SSIM loss are processed by linear combination to obtain the corresponding first appearance loss (gradient error), that is:

[0080] L app =(1-λ)L1+λL D_SSIM ;

[0081] Among them, L app represents the first appearance loss, L1 represents the L1 loss, L D_SSIM represents D_SSIM loss, λ represents the weight corresponding to D_SSIM loss, and 1-λ represents the weight corresponding to L1 loss.

[0082] It should be pointed out that based on the prior knowledge that the expected deformation should be locally rigid, an additional ARAP (As-Rigid-As-Possible) loss L is added arap , which is a regularized loss function for learning a deformable shape generator to encourage the motion of each Gaussian point to remain locally rigid, where the ARAP loss L arap for:

[0083]

[0084] Among them, S represents the number of Gaussians, knn i,k represents the k nearest Gaussians around Gaussian i, k can be set to 20, represents the ARAP loss function between Gaussian i and Gaussian j, ω i,j represents the weight factor of the Gaussian pair, x j represents the three-dimensional coordinates of Gaussian j, x i represents the three-dimensional coordinates of Gaussian i, R irepresents the rotation matrix of Gaussian i relative to the world coordinates, Represents the three-dimensional coordinates after Gaussian i is deformed to the final state, represents the three-dimensional coordinates after Gauss j is deformed to the final state, represents the rotation matrix of Gauss i relative to the world coordinates after deformation to the final state, λ ω Represents a hyperparameter and can be set to 20.

[0085] Therefore, the final deformation loss is determined based on the first appearance loss and the ARAP loss, and then the deformation loss is passed back to the deformation network to achieve self-supervised training and learning. For example, the deformation loss is obtained by processing the first appearance loss and the ARAP loss through a linear combination, that is:

[0086] L deform =L app +λ arap L arap ;

[0087] Among them, the weight λ corresponding to the ARAP loss arap Can be set to 1.

[0088] Step S13: applying the pre-estimated initial hinge motion parameters to the initial movable Gaussian model to optimize the initial hinge motion parameters to obtain corresponding target hinge motion parameters, and re-segmenting the initial movable Gaussian model to obtain a target movable Gaussian model.

[0089] In this embodiment, the movable Gaussian part in the three-dimensional Gaussian splash model is identified to obtain the initial movable Gaussian model G m Afterwards, the global rigid transformation can be applied to the initial movable Gaussian model G mTo optimize the estimated motion parameters, the movable Gaussian part will be deformed along with the motion parameters, and more accurate motion parameters can be obtained by maintaining the basic principle of visual and geometric consistency between the two states. Specifically, the pre-estimated initial hinge motion parameters are applied to the initial movable Gaussian model to control the initial movable Gaussian model to perform motion deformation, and obtain a new deformed three-dimensional Gaussian splash model; the new deformed three-dimensional Gaussian splash model is input into the pre-constructed Gaussian differentiable rendering framework to obtain a corresponding second rendered image; the loss between the second rendered image and the second target image under the corresponding perspective in the second multi-perspective image set is calculated to obtain a corresponding second appearance loss; the coordinates of each Gaussian point corresponding to the new deformed three-dimensional Gaussian splash model are determined to obtain a first group of Gaussian point coordinates, and a second group of Gaussian point coordinates are determined based on the coordinates of each Gaussian point corresponding to the three-dimensional Gaussian splash model and the deformation parameters of each Gaussian point predicted and output by the deformation network relative to the end state; the distance between the first group of Gaussian point coordinates and the second group of Gaussian point coordinates is calculated to obtain a corresponding geometric loss; the initial hinge motion parameters are optimized based on the appearance loss and the geometric loss to obtain the corresponding target hinge motion parameters.

[0090] It should be noted that when fitting translational motion with rotational motion parameters, the axis of the rotational motion parameters is usually very far away from the object to ensure that the movable Gaussian part moves in a nearly translational manner during the motion. Therefore, the movable Gaussian part in the three-dimensional Gaussian splash model is a rotational change, and the fulcrum ‖p‖ is greater than the spatial radius R. The motion is considered to be translational, and the optimization parameters and motion function will be reinitialized to adapt to the translational motion, where R can be set to 4.

[0091] In this embodiment, the initial movable Gaussian model is re-segmented to obtain the target movable Gaussian model. It is understandable that the Gaussian of the movable component is re-segmented to obtain a more stringent movable component segmentation result on the Gaussian model for joint optimization. For example, a second preset threshold with a smaller value, such as τ=0.1, can be used to re-segment the movable Gaussian part in the three-dimensional Gaussian splash model to obtain a more stringent segmentation of the movable Gaussian part, and obtain a more accurate segmentation result, that is, the target movable Gaussian model.

[0092] Step S14: performing a joint optimization operation based on the three-dimensional Gaussian splash model, the target movable Gaussian model and the target hinge motion parameters to construct a target three-dimensional Gaussian digital twin model corresponding to the hinge object with the movable parts and corresponding hinge logic.

[0093] In this embodiment, a three-dimensional Gaussian splash model that is closer to the real value, a more accurate movable Gaussian part, and a target hinge motion parameter can be obtained by joint optimization to find the optimal value. Specifically, the target hinge motion parameter is applied to the target movable Gaussian model to determine the state of the three-dimensional Gaussian splash model after the target movable Gaussian model is moved, and a target three-dimensional Gaussian digital twin model corresponding to the hinge object with the movable part and the corresponding hinge logic is obtained. It can be understood that the relatively accurate target hinge motion parameters obtained are applied to the target movable Gaussian model to obtain its corresponding state after movement, and for the immovable part on the three-dimensional Gaussian splash model, the starting state and the ending state are the same.

[0094] It should be pointed out that after constructing the target three-dimensional Gaussian digital twin when the hinged object is in the end state, the target three-dimensional Gaussian digital twin can also be equipped with a high-quality colored appearance, so that it can play a greater role in downstream applications, for example, embodied intelligence can be transferred from the simulator to the real world during training.

[0095] It is also important to point out that during the densification and deletion process of the Gaussian, the classification inaccuracies within the two masks can be regarded as the noise of the Gaussian, which will be automatically updated with the densification and deletion process of the Gaussian, thereby achieving accurate segmentation results of the movable part. Moreover, since only using the starting state to supervise the optimization of the Gaussian may cause the problem of overfitting the Gaussian distribution to the starting state, the weights of the loss functions on the starting state and the ending state can be dynamically set to prevent the final Gaussian distribution from overfitting to the starting state. Specifically, the first target three-dimensional Gaussian digital twin model when the hinge object is in the starting state is input into the pre-built Gaussian differentiable rendering framework to obtain the corresponding third rendered image; the third appearance loss between the third rendered image and the third target image at the corresponding perspective in the first multi-perspective image set is calculated; the second target three-dimensional Gaussian digital twin model when the hinge object is in the ending state is input into the pre-built Gaussian differentiable rendering framework to obtain the corresponding fourth rendered image; the fourth appearance loss between the fourth rendered image and the fourth target image at the corresponding perspective in the second multi-perspective image set is calculated; the corresponding weights are dynamically set for the third appearance loss and the fourth appearance loss to obtain the first target model construction loss and the second target model construction loss; based on the first target model construction loss and the second target model construction loss, the construction of the target three-dimensional Gaussian digital twin model corresponding to the hinge object is iteratively optimized.

[0096] For example, define the third appearance loss The weight is ω 0 , define the fourth appearance loss ω1 , where ω 0 and ω 1 can be set to 0.5, that is, ω 0 =ω 1 =0.5.

[0097] Moreover, when the hinge object is a hinge object with multiple movable parts, it can also specifically include: obtaining a third multi-view picture set when another movable part in the hinge object is in an end state after movement, and determining the constructed target three-dimensional Gaussian digital twin model corresponding to the hinge object as a new three-dimensional Gaussian splash model when the other movable part in the hinge object is in the starting state before movement; based on the new three-dimensional Gaussian splash model and the third multi-view picture set, re-executing the step of using a pre-constructed deformation network to predict the rigid transformation of each Gaussian point in the three-dimensional Gaussian splash model relative to the end state to obtain the corresponding deformed three-dimensional Gaussian splash model and its subsequent steps, so as to construct a new target three-dimensional Gaussian digital twin model corresponding to the hinge object with the movable part, the other movable part and the corresponding hinge logic.

[0098] It can be understood that to reconstruct a hinged object with multiple movable parts, the entire reconstruction process is decomposed into multiple sub-processes of single object modeling, and each sub-process is solved in turn, that is, the hinged object with multiple movable parts is gradually reconstructed by dividing the sub-tasks, see Figure 3 As shown, during the entire reconstruction process, the same set of Gaussian distributions is always maintained, and the starting state of the subsequent sub-process is aligned with a certain state of the previous sub-process to ensure that only one new movable part is introduced in the new sub-process. Before entering the next sub-process, the motion parameters derived from the previous sub-process are used to configure the movable parts in the existing Gaussian distribution to the starting state of the subsequent sub-process, and in each sub-process, a new identifier can be used to mark the movable mask, and the immovable Gaussian distribution can be optimized to update the new movable mask. Since the movable mask can be densified and pruned simultaneously with the Gaussian distribution, and the Gaussian distribution marked by the movable mask in the previous sub-process can also be optimized in the next new sub-process, for the new sub-process, the step of pre-training the initial Gaussian distribution can be skipped and the deformation network can be directly entered.

[0099] It can be seen that in the embodiment of the present invention, based on the three-dimensional Gaussian splattering technology, only multi-perspective images of the hinge object in two states are required to realize the self-supervised construction of the digital twin of the hinge object. That is, based on the multi-perspective images of the hinge object in a sparse state, a completely self-supervised manner is used, and a hierarchical optimization method is used to stably and high-quality reconstruct the hinge object to obtain a digital twin model of the hinge object. At the same time, the hinge logic of the movable parts in the hinge object can be estimated with high accuracy, thereby improving the geometric accuracy and visual accuracy of the reconstruction of the hinge object and the estimation accuracy of the hinge structure.

[0100] It should be pointed out that the simultaneous optimization of motion parameters, movable Gaussian models (movable regions) and Gaussian representations (appearance) has the problem of tending to find local optimal solutions or facing computational failures, that is, the movable parts of the object, rigid body transformation according to the motion parameters, can greatly affect the learning of the geometry and visual appearance of the movable parts, and vice versa, the optimization of movable parts and motion parameters may be significantly affected by the rendering quality. Therefore, it becomes very difficult to optimize the geometry, appearance and motion parameters of the object as a whole. For this reason, the overall goal can be processed step by step, and each step aims to obtain an initial parameter estimate close to the true value, thereby increasing the possibility of reaching the optimal solution.

[0101] By using the three-dimensional Gaussian splashing technology as a carrier to construct the digital twin of the hinged object, for the construction of the digital twin of the hinged object, training in a self-supervised manner can avoid the limitation of the method on the types of objects in the prior, that is, obtaining multi-view RGB views of the hinged object in two states, and self-supervisingly estimating the moving parts of the hinged object and the hinge motion parameters without any prior knowledge, and at the same time constructing the three-dimensional Gaussian splashing digital twin of the object to achieve the purpose of synthesizing new perspectives. For example, see Figure 4 As shown, the two states I0 and I1 of the hinge object are determined. First, only the multi-view image in the first state I0 is used to train the static Gaussian model, and then the deformation field estimation is realized, that is, supervised by the multi-view image in the second state I1, and the deformation field of the three-dimensional Gaussian splash model is estimated by the deformation network, that is, based on the multi-view image in the second state I1, the Gaussian model deformation field is fitted by the deformation network, and the motion parameters are not directly optimized, and then the estimation of the hinge structure is entered. That is, once the deformation field of the three-dimensional Gaussian splash model is obtained, the movable part on the three-dimensional Gaussian splash model can be roughly identified, and the hinge motion parameters are estimated according to the deformation, and then the rigid motion is projected onto the movable Gaussian model to optimize the hinge motion parameters to obtain the target hinge motion parameters, and the movable part on the roughly identified three-dimensional Gaussian splash model is re-segmented to obtain a more accurate segmentation result, and finally the representation of the Gaussian model, the segmentation result and the target hinge motion parameters are jointly optimized to obtain the final reconstruction result of the hinge object and the accurate hinge logic.

[0102] We used the open-source algorithm to conduct experiments on the publicly available PARIS data, and also selected multiple objects in the PartNet dataset to create a dataset for experiments. Figure 5 As shown, the results indicate that the technical solution of the present application has significant improvements in the reconstructed geometric accuracy, visual accuracy, and hinge structure estimation accuracy. Real-life objects are constructed as experimental verification to obtain a realistic effect, and the training efficiency is also higher than that of traditional methods.

[0103] Furthermore, the application scenarios of the technical solution of the present application may include, but are not limited to, reconstruction of the hinge structure of a single human joint, reconstruction of the hinge structure of a robotic arm, etc.

[0104] In one embodiment, if Figure 6 As shown, based on the above-mentioned method for constructing a digital twin model of a hinged object, the present invention also provides a device for constructing a digital twin model of a hinged object, including:

[0105] The image acquisition module 11 is used to acquire a first multi-view image set when the movable part in the hinge object is in a starting state before moving and a second multi-view image set when the movable part is in a final state after moving;

[0106] A first model building module 12, configured to build a three-dimensional Gaussian splash model of the hinge object when it is in the initial state by using the first multi-view picture set;

[0107] A model deformation module 13, used to use a pre-constructed deformation network to predict the rigid transformation of each Gaussian point in the three-dimensional Gaussian splash model relative to the end state to obtain a corresponding deformed three-dimensional Gaussian splash model;

[0108] A mobile model recognition module 14, used to recognize an initial movable Gaussian model corresponding to the movable component in the hinge object based on the deformed three-dimensional Gaussian splash model; wherein the deformation network is a neural network obtained by self-supervised training optimization based on the second multi-view image set and the deformed three-dimensional Gaussian splash model;

[0109] A motion parameter optimization module 15, used for applying the pre-estimated initial hinge motion parameters to the initial movable Gaussian model to optimize the initial hinge motion parameters to obtain corresponding target hinge motion parameters;

[0110] A movable model segmentation module 16, configured to re-segment the initial movable Gaussian model to obtain a target movable Gaussian model;

[0111] The digital twin construction module 17 is used to perform a joint optimization operation based on the three-dimensional Gaussian splash model, the target movable Gaussian model and the target hinge motion parameters to construct a target three-dimensional Gaussian digital twin model corresponding to the hinge object with the movable parts and corresponding hinge logic.

[0112] Figure 7 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal may include:

[0113] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0114] When the processor 502 executes the program, the digital twin model construction method of the hinge object provided in the above embodiment is implemented.

[0115] Furthermore, the terminal further includes:

[0116] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0117] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0118] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0119] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0120] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0121] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0122] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for constructing a digital twin model of a hinge object as described above.

[0123] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0124] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0125] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes additional implementations, in which the order shown or discussed may not be followed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0126] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can read instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary and then storing it in a computer memory.

[0127] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0128] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0129] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0130] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

[0131] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for constructing a digital twin model of a hinged object, characterized in that: The method comprises: Acquire a first multi-view picture set when a movable part in a hinge object is in a starting state before movement and a second multi-view picture set when the movable part is in a final state after movement, and construct a three-dimensional Gaussian splash model of the hinge object in the starting state using the first multi-view picture set; The pre-constructed deformation network is used to predict the rigid transformation of each Gaussian point in the three-dimensional Gaussian splash model relative to the end state to obtain the corresponding deformed three-dimensional Gaussian splash model, and the initial movable Gaussian model corresponding to the movable part in the hinge object is identified based on the deformed three-dimensional Gaussian splash model; wherein the deformation network is a neural network obtained by self-supervised training optimization based on the second multi-view image set and the deformed three-dimensional Gaussian splash model; Applying the pre-estimated initial hinge motion parameters to the initial movable Gaussian model to optimize the initial hinge motion parameters to obtain corresponding target hinge motion parameters, and re-segmenting the initial movable Gaussian model to obtain a target movable Gaussian model; A joint optimization operation is performed based on the three-dimensional Gaussian splash model, the target movable Gaussian model and the target hinge motion parameters to construct a target three-dimensional Gaussian digital twin model corresponding to the hinge object, having the movable parts and corresponding hinge logic.

2. The method for constructing a digital twin model of a hinge object according to claim 1, characterized in that: The method of using a pre-built deformation network to predict the rigid transformation of each Gaussian point in the three-dimensional Gaussian splash model relative to the end state to obtain a corresponding deformed three-dimensional Gaussian splash model includes: Performing a preset gradient truncation operation on the Gaussian point position corresponding to each Gaussian point in the three-dimensional Gaussian splash model to obtain a processed Gaussian point position; Inputting the processed Gaussian point positions into a pre-constructed deformation network to obtain deformation parameters of each Gaussian point predicted and output by the deformation network relative to the end state; Determining a corresponding deformed three-dimensional Gaussian splash model based on the deformation parameters; The identifying, based on the deformed three-dimensional Gaussian splash model, an initial movable Gaussian model corresponding to the movable component in the hinge object comprises: Normalizing the deformation parameters corresponding to the deformed three-dimensional Gaussian splash model to obtain processed deformation parameters; Determining whether the processed deformation parameter is less than a first preset threshold; When the processed deformation parameter is not less than the first preset threshold, the processed deformation parameter is determined as the target deformation parameter, and the Gaussian model composed of Gaussian points corresponding to the target deformation parameter is determined as the initial movable Gaussian model corresponding to the movable part in the hinge object.

3. The method for constructing a digital twin model of a hinge object according to claim 2, characterized in that: The self-supervised training optimization based on the second multi-view picture set and the deformed three-dimensional Gaussian splash model also includes: Inputting the deformed three-dimensional Gaussian splash model into a pre-built Gaussian differentiable rendering framework to obtain a corresponding first rendered image; Calculate the L1 loss and D_SSIM loss between the first rendered image and the first target image at the corresponding perspective in the second multi-view image set; Determine a corresponding first appearance loss based on the L1 loss and the D_SSIM loss, and determine a corresponding ARAP loss; A corresponding deformation loss is determined based on the first appearance loss and the ARAP loss so that the deformation network performs self-supervised training optimization according to the deformation loss.

4. The method for constructing a digital twin model of a hinge object according to claim 3, characterized in that: The step of applying the pre-estimated initial hinge motion parameters to the initial movable Gaussian model to optimize the initial hinge motion parameters to obtain corresponding target hinge motion parameters includes: Applying the pre-estimated initial hinge motion parameters to the initial movable Gaussian model to control the initial movable Gaussian model to perform motion deformation, thereby obtaining a new deformed three-dimensional Gaussian splash model; Inputting the new deformed three-dimensional Gaussian splash model into the pre-built Gaussian differentiable rendering framework to obtain a corresponding second rendered image; Calculating a loss between the second rendered image and a second target image at a corresponding perspective in the second multi-perspective image set to obtain a corresponding second appearance loss; Determine the coordinates of each Gaussian point corresponding to the new deformed three-dimensional Gaussian splash model to obtain a first set of Gaussian point coordinates, and determine a second set of Gaussian point coordinates based on the coordinates of each Gaussian point corresponding to the three-dimensional Gaussian splash model and the deformation parameters of each Gaussian point predicted and output by the deformation network relative to the end state; Calculating the distance between the coordinates of the first group of Gaussian points and the coordinates of the second group of Gaussian points to obtain corresponding geometric losses; The initial hinge motion parameters are optimized based on the appearance loss and the geometric loss to obtain corresponding target hinge motion parameters.

5. The method for constructing a digital twin model of a hinge object according to claim 1, characterized in that: The performing of a joint optimization operation based on the three-dimensional Gaussian splash model, the target movable Gaussian model and the target hinge motion parameters to construct a target three-dimensional Gaussian digital twin model corresponding to the hinge object and having the movable component and corresponding hinge logic includes: The target hinge motion parameters are applied to the target movable Gaussian model to determine the state of the three-dimensional Gaussian splash model after the target movable Gaussian model moves, and a target three-dimensional Gaussian digital twin model corresponding to the hinge object and having the movable parts and corresponding hinge logic is obtained.

6. The method for constructing a digital twin model of a hinge object according to claim 1, characterized in that: Also includes: Inputting the first target three-dimensional Gaussian digital twin model when the hinge object is in the initial state into a pre-built Gaussian differentiable rendering framework to obtain a corresponding third rendered image; Calculating a third appearance loss between the third rendered image and a third target image at a corresponding perspective in the first multi-view image set; Inputting the second target three-dimensional Gaussian digital twin model when the hinge object is in the end state into the pre-built Gaussian differentiable rendering framework to obtain a corresponding fourth rendered image; Calculating a fourth appearance loss between the fourth rendered image and a fourth target image at a corresponding perspective in the second multi-view image set; Dynamically setting corresponding weights for the third appearance loss and the fourth appearance loss to obtain a first target model building loss and a second target model building loss; The construction of the target three-dimensional Gaussian digital twin model corresponding to the hinge object is iteratively optimized based on the first target model construction loss and the second target model construction loss.

7. The method for constructing a digital twin model of a hinge object according to any one of claims 1 to 6, characterized in that: When the hinge object is a hinge object with multiple movable parts, it also includes: Acquire a third multi-view picture set when another movable part in the hinge object is in an ending state after the movement, and determine the constructed target three-dimensional Gaussian digital twin model corresponding to the hinge object as a new three-dimensional Gaussian splash model when the other movable part in the hinge object is in the starting state before the movement; Based on the new three-dimensional Gaussian splash model and the third multi-view image set, the step of using the pre-constructed deformation network to predict the rigid transformation of each Gaussian point in the three-dimensional Gaussian splash model relative to the end state to obtain the corresponding deformed three-dimensional Gaussian splash model and its subsequent steps are re-executed to construct a new target three-dimensional Gaussian digital twin model corresponding to the hinge object, having the movable part, the other movable part and the corresponding hinge logic.

8. A device for constructing a digital twin model of a hinged object, characterized in that: The device comprises: An image acquisition module, used to acquire a first multi-view image set when the movable component in the hinge object is in a starting state before moving and a second multi-view image set when the movable component is in a final state after moving; A first model building module, configured to build a three-dimensional Gaussian splash model of the hinge object when the hinge object is in the initial state by using the first multi-view picture set; A model deformation module, used to use a pre-constructed deformation network to predict the rigid transformation of each Gaussian point in the three-dimensional Gaussian splash model relative to the end state to obtain a corresponding deformed three-dimensional Gaussian splash model; A mobile model recognition module, used to identify an initial movable Gaussian model corresponding to the movable part in the hinge object based on the deformed three-dimensional Gaussian splash model; wherein the deformation network is a neural network obtained by self-supervised training optimization based on the second multi-view image set and the deformed three-dimensional Gaussian splash model; A motion parameter optimization module, used for applying the pre-estimated initial hinge motion parameters to the initial movable Gaussian model to optimize the initial hinge motion parameters to obtain corresponding target hinge motion parameters; A movable model segmentation module, used for re-segmenting the initial movable Gaussian model to obtain a target movable Gaussian model; A digital twin construction module is used to perform a joint optimization operation based on the three-dimensional Gaussian splash model, the target movable Gaussian model and the target hinge motion parameters to construct a target three-dimensional Gaussian digital twin model corresponding to the hinge object with the movable parts and corresponding hinge logic.

9. A terminal, characterized in that: include: A memory, a processor, and a program for building a digital twin model of a hinge object stored in the memory and executable on the processor. When the program for building a digital twin model of a hinge object is executed by the processor, the steps of the method for building a digital twin model of a hinge object as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed to implement the steps of the method for constructing a digital twin model of a hinge object as described in any one of claims 1 to 7.

Images