Synthesis method and device of four-dimensional content, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure relate to a method and apparatus for synthesizing four-dimensional content, an electronic device, and a storage medium. The method includes determining a plurality of three-dimensional Gaussian points on a three-dimensional model of a spatial object; dividing the plurality of three-dimensional Gaussian points into a plurality of component units, wherein a component unit is a constituent part of the three-dimensional model; determining a corresponding physical attribute of each component unit in the plurality of component units; and controlling the plurality of three-dimensional Gaussian points to move according to the corresponding physical attribute, to synthesize four-dimensional content of the spatial object. Thus, the physical rationality and control accuracy of the four-dimensional content can be improved.

Description

Methods, apparatus, electronic devices and storage media for synthesizing four-dimensional content Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus, electronic device, and storage medium for synthesizing four-dimensional content. Background Technology

[0002] Currently, research on four-dimensional (4D) content mainly focuses on controlling the consistent dynamic changes of three-dimensional Gaussian points in the temporal domain. For example, Animate124 (a method for animates a single image into a 4D dynamic scene), 4D-Fy (a 4D text generator), and DreamGaussian4D (a technique that uses Gaussian distribution and deep learning to convert static images into 4D dynamic models) employ video diffusion models combined with techniques such as score distillation sampling (SDS) and reference video supervision to guide the dynamic motion of objects. While these methods have achieved some success, they lack explicit modeling of material properties or dynamics and rely on fuzzy text cues or video prior control, thus remaining limited in terms of physical plausibility and precise content control.

[0003] Sync4D (used to generate 4D content by combining 3D modeling with time progress) and Phy124 (a 4D content compositing method) have introduced the Material Point Method (MPM) to enhance physical plausibility, but they still cannot assign different physical properties to different parts of an object, so they still have certain limitations. Summary of the Invention

[0004] In view of this, in order to solve some or all of the above-mentioned technical problems, the present disclosure provides a method, apparatus, electronic device and storage medium for synthesizing four-dimensional content.

[0005] In a first aspect, embodiments of this disclosure provide a method for synthesizing four-dimensional content, the method comprising:

[0006] Determine multiple 3D Gaussian points on a 3D model of a spatial object;

[0007] Multiple 3D Gaussian points are divided into multiple component units, where the component units are the constituent parts of the 3D model;

[0008] Determine the physical properties of each component unit among multiple component units;

[0009] Control the movement of three-dimensional Gaussian points according to their corresponding physical properties to synthesize the four-dimensional content of spatial objects.

[0010] In some possible implementations, determining multiple three-dimensional Gaussian points on a three-dimensional model of a spatial object includes:

[0011] Acquire images of a spatial object from multiple different shooting perspectives to obtain an image set;

[0012] A three-dimensional Gaussian splashing algorithm with a preset regularization term is adopted to reconstruct spatial objects based on an image set, thereby obtaining multiple three-dimensional Gaussian points on the three-dimensional model of the spatial objects.

[0013] The preset regularization term is used to reduce the distance between the 3D Gaussian points and the surface of the 3D model, and to improve the uniformity of the distribution of multiple 3D Gaussian points.

[0014] In some possible implementations, multiple three-dimensional Gaussian points are divided into multiple component units, including:

[0015] Based on a pre-trained 3D mask model, the mask information corresponding to each 3D Gaussian point among multiple 3D Gaussian points is determined; wherein, the 3D mask model is used to determine the mask information corresponding to the 3D Gaussian points.

[0016] Based on the determined mask information, multiple three-dimensional Gaussian points are divided into multiple component units.

[0017] In some possible implementations, the image set is a sequence of images ordered according to the shooting order of the spatial objects; and

[0018] The 3D mask model was trained in the following way:

[0019] Target tracking is performed on the image sequence to obtain a labeled image sequence, wherein the labels of the labeled images in the labeled image sequence are used to distinguish component units;

[0020] The labeled images in the labeled image sequence are input into a pre-trained two-dimensional mask model to generate mask information for the labeled images. The two-dimensional mask model is used to determine the mask information for the labeled images.

[0021] The three-dimensional Gaussian points corresponding to the labeled image are used as the sample input data, and the mask information of the labeled image is used as the sample output data to generate training samples containing sample input data and sample output data.

[0022] A three-dimensional mask model is obtained by training a machine learning algorithm based on training samples.

[0023] In some possible implementations, controlling the movement of a three-dimensional Gaussian point according to its corresponding physical properties includes:

[0024] Multiple 3D Gaussian points on a 3D model are clustered to obtain multiple clusters;

[0025] Multiple three-dimensional Gaussian points belonging to the same cluster are controlled to move at the same speed and according to their respective physical properties.

[0026] In some possible implementations, controlling the movement of a three-dimensional Gaussian point according to its corresponding physical properties includes:

[0027] Construct a surface mesh for a 3D model of a spatial object;

[0028] Establish the association between multiple 3D Gaussian points and the mesh points of the surface mesh;

[0029] Multiple 3D Gaussian points associated with the same grid point are controlled to move at the same speed as the associated grid point and with their respective physical properties.

[0030] In some possible implementations, the association between multiple 3D Gaussian points and the mesh points of the surface mesh is established, including:

[0031] Multiple 3D Gaussian points on a 3D model are clustered to obtain multiple clusters;

[0032] Determine the target 3D Gaussian point located at the center of the target cluster in multiple clusters;

[0033] From multiple grid points in the 3D model, determine the target grid point with the smallest distance to the target 3D Gaussian point;

[0034] Establish the association between multiple 3D Gaussian points in the target cluster and the target grid points.

[0035] In some possible implementations, controlling the movement of a three-dimensional Gaussian point according to its corresponding physical properties includes:

[0036] Construct a surface mesh for a 3D model of a spatial object;

[0037] Based on the defined bone sequence, bones and skin are bound to the surface mesh to obtain 3D Gaussian points driven by the bone sequence.

[0038] Using three-dimensional Gaussian points driven by the skeleton sequence as material points, the material point method is employed to control the movement of the material points according to their corresponding physical properties.

[0039] Secondly, embodiments of this disclosure provide a four-dimensional content synthesis apparatus, the apparatus comprising:

[0040] The first determining unit is configured to: determine multiple three-dimensional Gaussian points on the three-dimensional model of the spatial object;

[0041] The partitioning unit is configured to divide multiple 3D Gaussian points into multiple component units, where the component units are the constituent parts of the 3D model;

[0042] The second determining unit is configured to: determine the physical properties corresponding to each of the multiple component units;

[0043] The control unit is configured to control the movement of three-dimensional Gaussian points according to their corresponding physical properties in order to synthesize the four-dimensional content of a spatial object.

[0044] In some possible implementations, the first determining unit includes:

[0045] The first determining subunit is configured to: acquire images of a spatial object from multiple different shooting angles to obtain an image set;

[0046] The reconstruction subunit is configured to: use a 3D Gaussian splashing algorithm with a preset regularization term to reconstruct the spatial object based on the image set, and obtain multiple 3D Gaussian points on the 3D model of the spatial object;

[0047] The preset regularization term is used to reduce the distance between the 3D Gaussian points and the surface of the 3D model, and to improve the uniformity of the distribution of multiple 3D Gaussian points.

[0048] In some possible implementations, the partitioning unit includes:

[0049] The second determining subunit is configured to: determine the mask information corresponding to each of the multiple three-dimensional Gaussian points based on a pre-trained three-dimensional mask model; wherein the three-dimensional mask model is used to determine the mask information corresponding to the three-dimensional Gaussian points.

[0050] The sub-units are configured to divide multiple 3D Gaussian points into multiple component units based on the determined mask information.

[0051] In some possible implementations, the image set is a sequence of images ordered according to the shooting order of the spatial objects; and

[0052] The 3D mask model was trained in the following way:

[0053] Target tracking is performed on the image sequence to obtain a labeled image sequence, wherein the labels of the labeled images in the labeled image sequence are used to distinguish component units;

[0054] The labeled images in the labeled image sequence are input into a pre-trained two-dimensional mask model to generate mask information for the labeled images. The two-dimensional mask model is used to determine the mask information for the labeled images.

[0055] The three-dimensional Gaussian points corresponding to the labeled image are used as the sample input data, and the mask information of the labeled image is used as the sample output data to generate training samples containing sample input data and sample output data.

[0056] A three-dimensional mask model is obtained by training a machine learning algorithm based on training samples.

[0057] In some possible implementations, the control unit includes:

[0058] The clustering subunit is configured to cluster multiple 3D Gaussian points on the 3D model to obtain multiple clusters.

[0059] The control subunit is configured to control multiple three-dimensional Gaussian points belonging to the same cluster to move at the same speed and according to their respective physical properties.

[0060] In some possible implementations, the control unit includes:

[0061] The first building subunit is configured to: build the surface mesh of a three-dimensional model of a spatial object;

[0062] Sub-units are created and configured to establish relationships between multiple 3D Gaussian points and surface mesh points;

[0063] The first control subunit is configured to control multiple three-dimensional Gaussian points associated with the same grid point to move at the same speed as the associated grid point and with their respective physical properties.

[0064] In some possible implementations, establishing a subunit includes:

[0065] The clustering module is configured to cluster multiple 3D Gaussian points on a 3D model to obtain multiple clusters.

[0066] The first determining module is configured to: determine the target three-dimensional Gaussian point located at the center of the target cluster in multiple clusters;

[0067] The second determining module is configured to: determine the target grid point with the smallest distance from the target 3D Gaussian point from multiple grid points of the 3D model;

[0068] The module is configured to establish the association between multiple 3D Gaussian points in the target cluster and the target grid points.

[0069] In some possible implementations, the control unit includes:

[0070] The second building subunit is configured to: build the surface mesh of a three-dimensional model of a spatial object;

[0071] The bound sub-unit is configured to bind bones and skin to the surface mesh based on a set bone sequence to obtain 3D Gaussian points driven by the bone sequence.

[0072] The second control subunit is configured to use three-dimensional Gaussian points driven by the skeleton sequence as material points, and employ a material point device to control the material points to move according to their corresponding physical properties.

[0073] Thirdly, embodiments of this disclosure provide an electronic device, including:

[0074] Memory, used to store computer programs;

[0075] A processor is configured to execute a computer program stored in the memory, wherein, when the computer program is executed, it implements the method of any embodiment of the method for synthesizing four-dimensional content of the first aspect of this disclosure.

[0076] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of any embodiment of the four-dimensional content synthesis method of the first aspect described above.

[0077] Fifthly, embodiments of this disclosure provide a computer program product comprising computer-readable code that, when executed on a device, causes a processor in the device to implement the method of any embodiment of the four-dimensional content synthesis method of the first aspect described above.

[0078] Based on the embodiments of this disclosure, by dividing three-dimensional Gaussian points into component units, assigning independent physical attributes to each component unit, and controlling its movement according to these physical attributes, the precise allocation of component-level physical attributes is achieved. This solves the problem in the prior art where the physical attributes of different components cannot be controlled individually, ensuring that the movement of each component conforms to its own physical characteristics and improving physical plausibility. Simultaneously, component-level division and control enable precise motion control, avoiding reliance on vague prompts. Therefore, precise control of component-level physical attributes and precise motion control are achieved, improving the physical plausibility and control accuracy of 4D content. Attached Figure Description

[0079] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0080] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0081] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0082] Figure 1 is a flowchart illustrating a method for synthesizing four-dimensional content according to an embodiment of this disclosure;

[0083] Figure 2 is a flowchart illustrating another method for synthesizing four-dimensional content provided in an embodiment of this disclosure;

[0084] Figure 3 is a schematic diagram of an application scenario of a four-dimensional content synthesis method provided in an embodiment of this disclosure;

[0085] Figure 4 is a schematic diagram of the structure of a four-dimensional content synthesis device provided in an embodiment of this disclosure;

[0086] Figure 5 is a schematic diagram of another four-dimensional content synthesis device provided in an embodiment of this disclosure;

[0087] Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0088] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It is obvious that the described embodiments are only a part of, and not all, of the embodiments described herein. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0089] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of this disclosure are only used to distinguish different steps, devices or modules, and do not represent any specific technical meaning, nor do they indicate the logical order between them.

[0090] It should also be understood that in this embodiment, "multiple" can refer to two or more, and "at least one" can refer to one, two or more.

[0091] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0092] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0093] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0094] The following description of at least one exemplary embodiment is merely illustrative and is not intended to limit the scope of this disclosure or its application or use.

[0095] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0096] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0097] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. To facilitate understanding of the embodiments of this disclosure, the disclosure will be described in detail below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0098] To address the technical challenge of improving the physical plausibility and control precision of 4D content in existing technologies, this disclosure provides a method, apparatus, electronic device, and storage medium for 4D content synthesis. This method involves dividing 3D Gaussian points into component units, assigning independent physical attributes to each component unit, and controlling its movement according to these attributes. By achieving precise allocation of component-level physical attributes, the problem of not being able to control the physical attributes of different components individually in existing technologies is solved. This ensures that the movement of each component conforms to its own physical characteristics, improving physical plausibility. Simultaneously, component-level division and control enable precise motion control, avoiding reliance on fuzzy prompts. Therefore, precise control of component-level physical attributes and precise motion control are achieved, enhancing the physical plausibility and control precision of 4D content.

[0099] Figure 1 is a flowchart illustrating a method for synthesizing four-dimensional content according to an embodiment of this disclosure. This method can be applied to one or more electronic devices such as smartphones, laptops, desktop computers, portable computers, and servers. Furthermore, the execution entity of this method can be hardware or software. When the execution entity is hardware, it can be one or more of the aforementioned electronic devices. For example, a single electronic device can execute this method, or multiple electronic devices can cooperate with each other to execute this method. When the execution entity is software, this method can be implemented as multiple software programs or software modules, or as a single software program or software module. No specific limitations are made here.

[0100] As shown in Figure 1, the method specifically includes:

[0101] Step 101: Determine multiple 3D Gaussian points on the 3D model of the spatial object.

[0102] In this embodiment, the spatial object can be the subject whose four-dimensional content is to be synthesized. As an example, the spatial object can be a three-dimensional object, a three-dimensional scene, etc.

[0103] A 3D model is a three-dimensional digital representation of a spatial object, which may include spatial features such as the object's shape and structure. As a carrier of 3D Gaussian points, it provides a basic framework for the subsequent division of component units and the allocation of physical properties. For example, the aforementioned 3D model could be a 3D model of a vehicle (including component units such as the engine and body); or a 3D model of a sculpture (including component units such as the head and torso).

[0104] Here, the three-dimensional Gaussian points can be determined by the three-dimensional Gaussian splashing algorithm before constructing the three-dimensional model of the spatial object; or the three-dimensional Gaussian points can be obtained by selecting on the pre-constructed three-dimensional model after constructing the three-dimensional model of the spatial object.

[0105] 3D Gaussian points are points generated using 3D Gaussian splashing algorithms or other methods to represent surface or internal features of a 3D model. Each point contains information such as position and color. They are the basic units for 4D content synthesis, carrying physical properties and participating in motion simulation.

[0106] Here, three-dimensional Gaussian points can be obtained through the 3D Gaussian splashing algorithm; or three-dimensional Gaussian points can be derived based on preset three-dimensional model parameters (such as vertex coordinates).

[0107] Step 102: Divide multiple three-dimensional Gaussian points into multiple component units, where the component units are the constituent parts of the three-dimensional model.

[0108] In this embodiment, component units are the constituent parts of the 3D model, representing a fine-grained division of the model. Each component unit can be independently assigned physical properties. This enables component-level physical control, allowing different parts to exhibit different physical characteristics.

[0109] Here, a multi-view image segmentation method can be used to map the image segmentation results to three-dimensional space to achieve the division of component units; alternatively, the K-means clustering algorithm can be used to divide component units based on the spatial location, color, and other features of three-dimensional Gaussian points.

[0110] Step 103: Determine the physical properties of each component unit among the multiple component units.

[0111] In this embodiment, physical properties refer to parameters that describe the physical characteristics of an object, such as material type, material density, Young's modulus, Poisson's ratio, and gravity. Their function is to determine the motion law of a three-dimensional Gaussian point to ensure that the motion conforms to physical laws. For example, in fluid simulation, properties such as density and viscosity need to be assigned; in solid simulation, properties such as rigidity and elasticity need to be assigned. When simulating water flow, physical properties include density and viscosity coefficient; when simulating steel, physical properties include Young's modulus and Poisson's ratio.

[0112] Here, physical properties can be manually assigned or automatically determined according to set rules.

[0113] Step 104: Control the three-dimensional Gaussian points to move according to their corresponding physical properties in order to synthesize the four-dimensional content of the spatial object.

[0114] In this embodiment, four-dimensional content refers to dynamic content that includes three-dimensional spatial information (x, y, z) and a time dimension (t), capable of representing the continuous changes of spatial objects over time. In the technical solution, it is the final synthesized result used to present the dynamic process of spatial objects. Four-dimensional content represents the pose state of a three-dimensional model at different times. As an example, four-dimensional content can be, for example, virtual digital human animation or dynamic scene simulation (such as a 4D simulation of trees swaying in a storm).

[0115] Here, motion states can be calculated based on physical properties using a physics engine (such as Bullet); trajectories can also be solved by combining preset motion equations with physical parameters. Furthermore, a sequence of 3D Gaussian point states at different time points can be integrated into a video stream; 4D content can also be generated through real-time rendering using a rendering engine.

[0116] In addition, other methods can be used to control the movement of three-dimensional Gaussian points according to their corresponding physical properties to synthesize the four-dimensional content of spatial objects. Please refer to the descriptions below for details, which will not be elaborated upon here.

[0117] In some optional implementations of this embodiment, multiple three-dimensional Gaussian points on the three-dimensional model of the spatial object can be determined in the following manner:

[0118] The first step is to acquire images of the spatial object from multiple different shooting angles, resulting in an image set.

[0119] An image set can be a collection of images of a spatial object taken from multiple different perspectives, containing multi-view appearance information of the object. It provides reconstruction data for the 3D Gaussian splashing algorithm, ensuring the accuracy of the 3D model.

[0120] The second step involves using a 3D Gaussian Splatting algorithm with a pre-defined regularization term to reconstruct the spatial object based on the image set, thereby obtaining multiple 3D Gaussian points on the 3D model of the spatial object.

[0121] The preset regularization term is used to reduce the distance between the 3D Gaussian points and the surface of the 3D model, and to improve the uniformity of the distribution of multiple 3D Gaussian points.

[0122] The 3D Gaussian splashing algorithm is used for 3D scene reconstruction. By generating 3D Gaussian points to represent the scene, it can efficiently reconstruct 3D content from multi-view images. It reconstructs 3D models and 3D Gaussian points from image sets.

[0123] The pre-defined regularization term is a constraint added to the aforementioned 3D Gaussian splash algorithm optimization. It reduces the distance between the 3D Gaussian points and the model surface and improves distribution uniformity. This ensures that the 3D Gaussian points are close to the model surface and evenly distributed, improving the accuracy of subsequent processing. For example, when reconstructing and optimizing Gaussian parameters using the 3D Gaussian splash algorithm, a regularized loss function can be added to encourage the 3D Gaussian points to align with the scene surface and distribute evenly on the surface.

[0124] As an example, the above-mentioned predefined regularization term can be the following loss function:

[0125]

[0126] Where P represents a set of sampled three-dimensional Gaussian points, Let n be the gradient of the predicted signed distance function (SDF) at point p, and n be the gradient of n at point p. g * is the normal vector of the nearest Gaussian distribution g* that contributes the most.

[0127] The loss function described above corrects the angular deviation between the SDF normalized gradient and the expected surface normal, thereby ensuring that the surface of the 3D model is consistent with the local geometry implied by the Gaussian model.

[0128] It is understandable that the above-mentioned optional implementation methods reconstruct 3D Gaussian points through a multi-view image set and a 3D Gaussian splashing algorithm with a regularization term. The regularization term reduces the distance between the 3D Gaussian points and the model surface, making them fit the 3D model surface better, while improving the distribution uniformity, providing a more accurate basis for subsequent component partitioning and physical attribute allocation. Accurate 3D Gaussian points improve component partitioning accuracy, thereby enhancing the accuracy of physical control. Therefore, improving the accuracy and distribution uniformity of 3D Gaussian points lays a better foundation for component-level physical control and motion control, improving the quality of 4D content synthesis.

[0129] Optionally, in addition to introducing a preset regularization term, an adaptive sampling strategy can be adopted to adjust the sampling density of three-dimensional Gaussian points according to the surface curvature of the three-dimensional model (more sampling in areas with large curvature and less sampling in areas with small curvature) so that the three-dimensional Gaussian points are close to the model surface and distributed reasonably.

[0130] In some optional implementations of this embodiment, the three-dimensional Gaussian point can be controlled to move according to the corresponding physical properties in the following way:

[0131] The first step is to cluster multiple 3D Gaussian points on the 3D model to obtain multiple clusters.

[0132] Clustering clusters are sets formed by grouping similar 3D Gaussian points together using clustering algorithms. This reduces the number of component units whose motion states need to be calculated individually, thus lowering computational costs.

[0133] The second step is to control multiple three-dimensional Gaussian points belonging to the same cluster to move at the same speed and according to their respective physical properties.

[0134] The velocity can be a physical quantity (units such as m / s) used to describe the speed and direction of the movement of a 3D Gaussian point. It can determine the trajectory of the 3D Gaussian point, ensure consistent movement within the same cluster, and reduce computational complexity.

[0135] Here, K-means clustering (with a preset number of clusters) can be used; or hierarchical clustering (based on distance-based stepwise merging) can be used.

[0136] In addition, the motion of three-dimensional Gaussian points within a cluster can be controlled by setting the velocity field function of the cluster; the velocity of three-dimensional Gaussian points within a cluster can also be calculated and controlled based on the trajectory of the cluster center.

[0137] It is understandable that in the above-mentioned optional implementation methods, clusters are obtained by clustering 3D Gaussian points, and the same cluster is controlled to move at the same speed and with the same physical properties. The same cluster shares the movement speed, reducing the number of component units that need to have their speeds calculated independently, thus reducing the computational load of MPM (Multi-Process Management). At the same time, the 3D Gaussian points maintain their own physical properties, ensuring the physical plausibility of the movement and achieving a balance between computational cost and physical realism. Therefore, the computational cost of MPM is reduced while ensuring the physical plausibility of the 3D Gaussian point movement, thereby improving the efficiency of 4D content compositing.

[0138] Alternatively, in addition to clustering, downsampling can be used, where a representative point is selected every certain number of three-dimensional Gaussian points, and its movement speed is calculated, with other points following the representative point.

[0139] In some optional implementations of this embodiment, the three-dimensional Gaussian point can be controlled to move according to the corresponding physical properties in the following way:

[0140] The first step is to construct the surface mesh of the three-dimensional model of the spatial object.

[0141] A surface mesh is a discrete representation of the surface of a 3D model, composed of vertices, edges, and faces, describing the shape of the model's surface. As an intermediate carrier for motion control, it indirectly controls 3D Gaussian points by controlling the movement of mesh points, achieving precise trajectory control.

[0142] Here, the surface mesh of a three-dimensional model of a spatial object can be constructed using modeling software; alternatively, the surface mesh of a three-dimensional model of a spatial object can be generated based on three-dimensional Gaussian points using a Poisson surface reconstruction algorithm.

[0143] The second step is to establish the relationship between multiple 3D Gaussian points and the mesh points of the surface mesh.

[0144] Grid points are the vertices of a surface grid, possessing three-dimensional coordinates, and are the basic units that constitute the surface grid. Serving as motion references for three-dimensional Gaussian points, their movement drives the associated three-dimensional Gaussian points, achieving precise control.

[0145] The correlation is the mapping relationship between 3D Gaussian points and mesh points, indicating that the 3D Gaussian point is affected by the movement of a specific mesh point. Establishing the motion connection between 3D Gaussian points and mesh points allows the motion of mesh points to be transmitted to the 3D Gaussian points, achieving more precise control.

[0146] Here, the Euclidean distance between a 3D Gaussian point and a grid point can be calculated to associate the nearest grid point; associations can also be established through feature matching (color, normal vector).

[0147] The third step is to control multiple 3D Gaussian points associated with the same grid point to move at the same speed as the associated grid point and with their respective physical properties.

[0148] Here, the movement of mesh points can be driven by skeletal animation, thereby controlling the associated 3D Gaussian points; alternatively, the associated 3D Gaussian points can be controlled by preset mesh point trajectories.

[0149] It is understandable that in the above-mentioned optional implementation methods, by constructing a surface mesh, establishing a connection between 3D Gaussian points and mesh points, and controlling multiple 3D Gaussian points associated with the same mesh point to move according to the mesh point's velocity and their own physical properties. The mesh point movement can be precisely controlled (e.g., through skeleton-driven motion), and the associated 3D Gaussian points can follow its movement, achieving precise motion control; at the same time, the 3D Gaussian points maintain their own physical properties, ensuring the physical rationality of the movement. Therefore, precise motion control of 3D Gaussian points can be achieved while taking into account the physical rationality of the movement.

[0150] Optionally, in addition to establishing the association between 3D Gaussian points and mesh points, motion trajectories (such as keyframe animations) can be preset directly for 3D Gaussian points, controlling their movement according to the trajectory and their own physical properties (suitable for simple motion scenarios).

[0151] In some application scenarios of the above optional implementation methods, the following method can be used to establish the association between multiple 3D Gaussian points and the mesh points of the surface mesh:

[0152] The first step is to cluster multiple 3D Gaussian points on the 3D model to obtain multiple clusters.

[0153] Clustering clusters are formed by grouping similar 3D Gaussian points together using clustering algorithms. This reduces the number of component units whose motion states need to be calculated individually, thus lowering computational costs.

[0154] The second step is to determine the target three-dimensional Gaussian point located at the center of the target cluster among multiple clusters.

[0155] The target cluster can be any cluster selected from multiple clusters.

[0156] The target 3D Gaussian point is the 3D Gaussian point at the center of the target cluster, representing the positional characteristics of the entire cluster.

[0157] The third step is to determine the target grid point with the smallest distance from the target 3D Gaussian point among multiple grid points of the 3D model.

[0158] The target grid point is the grid point with the smallest distance to the target 3D Gaussian point, and it is the associated object of the target cluster. As the motion reference point of the cluster, it drives the motion of the entire cluster's 3D Gaussian points.

[0159] The fourth step is to establish the association between multiple three-dimensional Gaussian points in the target cluster and the target grid points.

[0160] Since each cluster (including the target cluster) contains multiple 3D Gaussian points, the association between each 3D Gaussian point in each cluster and the same target grid point can be established here.

[0161] Understandably, in the above application scenarios, clustering 3D Gaussian points can yield clusters, identify target 3D Gaussian points and their corresponding target grid points, and establish associations between each 3D Gaussian point in the cluster and the target grid point. This reduces the amount of data computation (eliminating the need for individual calculation of each 3D Gaussian point) and improves efficiency. Simultaneously, precise control of the target grid point can drive the movement of all 3D Gaussian points in the corresponding cluster, ensuring precise motion control. Therefore, it can improve the efficiency of establishing associations while guaranteeing the precision of motion control.

[0162] Optionally, in addition to the target 3D Gaussian point at the center of the cluster, multiple 3D Gaussian points in the same cluster can be randomly selected, and multiple corresponding grid points can be calculated. The average motion of these grid points can be used as the motion reference for multiple 3D Gaussian points in the cluster to establish a correlation.

[0163] In some optional implementations of this embodiment, the three-dimensional Gaussian point can be controlled to move according to the corresponding physical properties in the following way:

[0164] The first step is to construct the surface mesh of the three-dimensional model of the spatial object.

[0165] A surface mesh is a discrete representation of the surface of a 3D model, composed of vertices, edges, and faces, describing the shape of the model's surface. As an intermediate carrier for motion control, it indirectly controls 3D Gaussian points by controlling the movement of mesh points, achieving precise trajectory control.

[0166] Here, the surface mesh of a three-dimensional model of a spatial object can be constructed using modeling software; alternatively, the surface mesh of a three-dimensional model of a spatial object can be generated based on three-dimensional Gaussian points using a Poisson surface reconstruction algorithm.

[0167] The second step involves binding bones and skin to the surface mesh based on the set bone sequence to obtain 3D Gaussian points driven by the bone sequence.

[0168] A skeletal sequence is a sequence describing the posture (position, rotation angle) of bones at different points in time, used to drive the skin and the motion of 3D Gaussian points. It provides precise motion control signals to make the 3D Gaussian points move according to a preset skeletal trajectory.

[0169] The skeleton is a virtual skeletal structure composed of bone nodes and joints, used to drive the movement of the surface mesh. As the "skeleton" of motion, its movement drives the skin and three-dimensional Gaussian points, achieving human-like motion.

[0170] Skinning is a weighted mapping connecting bones and surface meshes, defining the weights of the influence of bones on mesh vertices. It ensures smooth transfer of bone motion to the surface mesh, avoiding abrupt movements and improving naturalness. For example, the weight of an arm bone on the forearm mesh vertices is 0.8, and on the hindarm it is 0.2; similarly, the weight allocation of a robotic arm joint bone on the vertices of adjacent link meshes.

[0171] The third step involves using the three-dimensional Gaussian points driven by the bone sequence as material points, and employing the material point method to control the material points to move according to their corresponding physical properties.

[0172] A material point is a point in the Material Point Method (MPM) simulation object; in this case, it refers to a skeleton-driven 3D Gaussian point that carries physical properties. In MPM, it serves as the basic simulation unit, achieving physical simulation based on physical properties and skeleton-driven motion.

[0173] The Material Point Method (MPM) is a numerical method for simulating the mechanical behavior of continuums. It combines the advantages of Lagrange and Eulerian methods to handle large deformations and complex physical phenomena. Motion is calculated based on the physical properties of material points and external drivers (such as skeletal motion), ensuring physical plausibility.

[0174] It is understandable that, among the above-mentioned optional implementation methods, a surface mesh can be constructed, and skeletons and skinning can be bound based on a skeleton sequence to obtain skeleton-driven 3D Gaussian points. These points can then be used as material points to control motion using MPM (Material Performance Management). The skeleton sequence precisely controls the motion trajectory, the skinning makes the motion smooth and natural, and the MPM ensures the rationality of motion based on physical properties, thus achieving a balance between precise motion control and physical realism. This enables precise motion control of 3D Gaussian points while ensuring the physical realism of the motion through MPM, making 4D content compositing more controllable and realistic.

[0175] Alternatively, in addition to the material point method, the finite element method (FEM) can be used to divide the region containing the 3D Gaussian point into a finite element mesh, and calculate motion based on skeleton-driven and physical property calculations (suitable for scenarios requiring higher-precision structural mechanics simulations). Besides using skeleton sequences, physical simulation can also be used to drive the synthesis of 4D content. In addition to skinning prediction, vertex animation can also be used to directly drive the synthesis of 4D content.

[0176] It should be noted that, where there is no conflict, the technical features described in different alternative implementations can be included in the same embodiment. For the sake of brevity, they will not be elaborated here.

[0177] The four-dimensional content synthesis method provided in this disclosure divides three-dimensional Gaussian points into component units, assigns independent physical attributes to each component unit, and controls its movement according to these physical attributes. By achieving precise allocation of component-level physical attributes, it solves the problem in existing technologies where physical attributes cannot be controlled individually for different components, ensuring that the movement of each component conforms to its own physical characteristics and improving physical plausibility. Simultaneously, component-level division and control enable precise motion control, avoiding reliance on vague prompts. Therefore, it achieves precise control of component-level physical attributes and precise motion control, improving the physical plausibility and control accuracy of 4D content.

[0178] Figure 2 is a flowchart illustrating another method for synthesizing four-dimensional content provided in this embodiment of the present disclosure. As shown in Figure 2, the method specifically includes:

[0179] Step 201: Determine multiple three-dimensional Gaussian points on the three-dimensional model of the spatial object.

[0180] In this embodiment, step 201 is basically the same as step 101 in the embodiment corresponding to Figure 1, and will not be described again here.

[0181] Step 202: Based on the pre-trained 3D mask model, determine the mask information corresponding to each of the multiple 3D Gaussian points; wherein, the 3D mask model is used to determine the mask information corresponding to the 3D Gaussian points.

[0182] In this embodiment, the 3D mask model can be a model trained using a machine learning algorithm. The training method for the 3D mask model can be found in the following description and will not be repeated here.

[0183] Mask information is used to identify the component unit to which the 3D Gaussian point belongs. For example, the mask information can be a mask image. Different component units can correspond to different mask information to distinguish the 3D Gaussian points of different component units and ensure accurate segmentation.

[0184] Here, three-dimensional Gaussian point features (such as position and color) can be input into a three-dimensional mask model to output mask information.

[0185] Step 203: Based on the determined mask information, the multiple three-dimensional Gaussian points are divided into multiple component units, where the component units are the constituent parts of the three-dimensional model.

[0186] In this embodiment, multiple three-dimensional Gaussian points can be divided into multiple component units according to the range of mask information values; alternatively, multiple three-dimensional Gaussian points can be divided into multiple component units according to category labels.

[0187] Step 204: Determine the physical properties of each component unit among the multiple component units.

[0188] In this embodiment, step 204 is basically the same as step 103 in the embodiment corresponding to Figure 1, and will not be described again here.

[0189] Step 205: Control the three-dimensional Gaussian points to move according to their corresponding physical properties in order to synthesize the four-dimensional content of the spatial object.

[0190] In this embodiment, step 205 is basically the same as step 104 in the embodiment corresponding to Figure 1, and will not be described again here.

[0191] In some optional implementations of this embodiment, the image set is an image sequence ordered according to the shooting order of the spatial objects.

[0192] Based on this, the 3D mask model is trained in the following way:

[0193] The first step is to perform target tracking on the image sequence to obtain the identified image sequence.

[0194] In this context, the identifiers of the identifier images in the identifier image sequence are used to distinguish component units.

[0195] An image sequence is a collection of multi-view images arranged in the order they were captured, containing continuous information about how spatial objects change over time or from a particular viewpoint. It provides continuous data for target tracking and model training, ensuring the spatiotemporal consistency of masking information.

[0196] Target tracking is a technique that tracks a specific target (component unit) across consecutive image frames while maintaining consistent target identification. Ensuring consistent identification of the same component unit within an image sequence avoids mask information confusion and improves the quality of training samples.

[0197] Image sequence identification: After target tracking, an image sequence is generated that adds identifiers (numbers, colors) to the component units of each image frame. These identifiers are used to distinguish the component units. This provides the 2D mask model with clearly distinguishable components, ensuring that the generated mask information corresponds to the components.

[0198] The second step is to input the labeled images from the labeled image sequence into a pre-trained two-dimensional mask model to generate mask information for the labeled images.

[0199] The 2D masking model is used to determine the masking information of the labeled image. It can segment targets in a 2D image, generate masking information, and distinguish different regions (2D projections of components) in the image. Accurate 2D masking information is generated from the labeled image and serves as a supervisory signal for training the 3D masking model.

[0200] The sample data used to train the two-dimensional mask model can include images and their corresponding mask information, which can be determined manually or automatically.

[0201] The third step involves using the 3D Gaussian points corresponding to the labeled image as the sample input data and the mask information of the labeled image as the sample output data to generate training samples containing both sample input and sample output data.

[0202] Training samples: Data pairs consisting of input data (3D Gaussian points) and output data (mask information) are used to train the 3D mask model. They provide the model with the basis for learning, enabling it to map from 3D Gaussian points to the correct mask information.

[0203] The fourth step involves using machine learning algorithms to train a 3D mask model based on training samples.

[0204] Machine learning algorithms are used to train models, optimizing model parameters by learning the input-output relationship of samples. This allows a 3D mask model to learn the mapping relationship between 3D Gaussian points and mask information, improving prediction accuracy.

[0205] Specifically, the three-dimensional Gaussian points corresponding to the labeled images in the training samples can be used as input data, and the mask information of the labeled images can be used as the expected output data to train and obtain a three-dimensional mask model.

[0206] It is understandable that in the above-mentioned optional implementation methods, target tracking is performed on the image sequence to obtain the labeled image sequence. Then, using the mask information generated by the two-dimensional mask model, training samples are constructed to train the three-dimensional mask model. Target tracking can ensure the consistency of component labels, and the two-dimensional mask model can provide relatively accurate two-dimensional masks as supervision. This allows the trained three-dimensional mask model to more accurately assign mask information to three-dimensional Gaussian points, enhancing the accuracy and spatial consistency of mask information. Therefore, the trained three-dimensional mask model has higher mask information prediction accuracy and can provide more accurate and consistent mask information for three-dimensional Gaussian points, thereby improving the accuracy of component unit partitioning.

[0207] Optionally, in addition to using target tracking and two-dimensional mask models to generate training samples, the manual segmentation results of the three-dimensional model can also be used directly as supervision. The component segmentation labels of the three-dimensional model are associated with the three-dimensional Gaussian points to generate training samples to train the three-dimensional mask model.

[0208] It should be noted that, in addition to the contents described above, this embodiment may also include the corresponding technical features described in the embodiment corresponding to FIG1, thereby achieving the technical effect of the four-dimensional content synthesis method shown in FIG1. ​​For details, please refer to the relevant description in FIG1. ​​For the sake of brevity, it will not be elaborated here.

[0209] The four-dimensional content synthesis method provided in this disclosure determines the mask information of each three-dimensional Gaussian point through a three-dimensional mask model, and then divides the component units based on the mask information. The three-dimensional mask model can accurately identify the component unit to which the three-dimensional Gaussian point belongs, making the division more accurate and ensuring that the three-dimensional Gaussian points of different component units are correctly distinguished. This provides a basis for assigning independent physical attributes to each component and enhances the accuracy of component-level physical control. Therefore, it improves the accuracy of dividing three-dimensional Gaussian points into component units, providing a reliable basis for precise control of component-level physical attributes.

[0210] The following describes the embodiments of this disclosure by way of example. However, it should be noted that the following content is only used to understand the technical solutions of the embodiments of this disclosure and does not constitute a limitation on the protection scope of the embodiments of this disclosure.

[0211] Currently, 4D content based on 3D Gaussian splashing mainly focuses on controlling the consistent dynamic changes of 3D Gaussian points in the time domain. Methods such as Animate124, 4D-Fy, and DreamGaussian4D utilize video diffusion models, employing techniques like Score Distillation Sampling (SDS) and reference video supervision to guide the dynamic motion of objects. While these methods have achieved some success, they lack explicit modeling of material properties or dynamics and rely on fuzzy text cues or video prior control, limiting their physical plausibility and precise control over content. Sync4D and Phy124 further enhance physical plausibility by introducing the Material Point Method (MPM), but they cannot assign different physical properties to different parts of an object, still exhibiting certain limitations.

[0212] Existing methods for generating 4D content based on 3D Gaussian splashing have made some progress, but they suffer from the following drawbacks: the generated motion lacks physical plausibility and explicit modeling of material properties or dynamics; control capabilities are limited, mostly relying on vague text prompts or video priors, lacking interpretable control methods, and making it difficult to achieve precise motion trajectory control. This results in the following insurmountable technical problems: how to achieve precise control of component-level physical properties of 4D content based on 3D Gaussian splashing; how to achieve precise motion control of 4D content based on 3D Gaussian splashing; and how to reduce the computational cost of MPM (Material Point Method) while maintaining physical realism.

[0213] In view of this, this scheme achieves fine-grained physical control by assigning independent physical attributes to different components using masks: a mask physics module is proposed by combining 3D masks and MPM. It can achieve component-level physical attribute assignment, thereby enabling precise control over the physical attributes of 4D content and improving physical plausibility. Precise motion control is achieved by driving MPM particles through skeletal skinning binding. A skeletal driving module is proposed by combining skeletal skinning and MPM and using the K-nearest neighbor algorithm for point-to-point binding. It can drive the motion of 3D Gaussian points using a given skeletal sequence while maintaining physical plausibility, thus achieving precise motion control. Furthermore, this scheme reduces the computational cost of MPM: by performing K-means clustering sampling on MPM particles, a balance is achieved between reducing computational cost and maintaining physical realism.

[0214] Specifically, please refer to Figure 3, which is a schematic diagram of an application scenario of a four-dimensional content synthesis method provided by an embodiment of this disclosure.

[0215] This solution proposes MagicPhysics4D, a novel 4D content compositing framework capable of highly controllable 4D content compositing and customized physical effects. As shown in Figure 3, MagicPhysics4D introduces two modules: a mask physics module and a skeleton-driven module. Combining 3D Gaussian segmentation and material point methods, the mask physics module can achieve fine-grained control over the physical properties of different objects and components through masks. To achieve motion control of 4D content, a skeleton-driven module is introduced. By performing surface reconstruction, bone extraction, and skinning prediction on the masked object, the binding of 3D Gaussian points to surface mesh vertices and the mapping from bone motion to 3D Gaussian point motion are achieved. Through the motion control and mask physics simulation of MagicPhysics4D, not only can the physical plausibility of 4D content be achieved, but different artistic effects can also be realized by assigning physical properties.

[0216] In the mask physics module, the object or scene is first reconstructed using a 3D Gaussian splashing algorithm with a regularization term, based on multi-view images of the object, resulting in 3D Gaussian points. Then, a segmentation model (i.e., the aforementioned 2D masking model, such as SegmentAnything or SegmentAnything2) is used to extract the mask information for each frame of the image (i.e., the images in the aforementioned image sequence). During this process, a target tracking method is employed for mask extraction. Using a tracking method effectively extracts the desired target, avoiding inconsistencies in the mask of the same object in consecutive frames, thus achieving better spatial consistency and 3D Gaussian point segmentation. Finally, a matched distillation loss is used to train scale-gated Gaussian similarity features. After completing the 3D Gaussian point segmentation, a material point method is used to assign physical properties to the masked 3D Gaussian points. By setting parameters such as material type, material density, Young's modulus, Poisson's ratio, and gravity, different physical properties can be assigned to the masked 3D Gaussian points.

[0217] The scale-gated Gaussian similarity features can be trained using the following method:

[0218] First, using a pre-trained 3D Gaussian model, each 3D Gaussian point in the scene is assigned a learnable similarity feature. Then, using a masked 2D image as supervision, this similarity feature is trained.

[0219] The training objective is to transfer the 2D segmentation capability of SAM (SegmentAnything Model) to the similarity features of 3D Gaussian points, thereby achieving 3D Gaussian point segmentation.

[0220] Model input and output: The input is a trained 3D Gaussian model and a masked 2D image (i.e., the mask image); the output is the similarity features of the segmented 3D Gaussian points.

[0221] In the skeleton-driven module, a regularization term is first introduced into the 3D Gaussian splash algorithm to encourage the alignment of 3D Gaussian points with the scene surface and their uniform distribution on the surface to reconstruct the surface mesh of the object. After the surface mesh reconstruction of the masked object is completed, UniRig is used to extract the skeleton of the surface mesh and predict the skinning, thereby obtaining the skeleton-driven 3D asset (i.e., the aforementioned 3D Gaussian points). Then, the binding of surface mesh points and 3D Gaussian points is performed. In the process of binding surface mesh points and 3D Gaussian points, two operations are implemented: (1) the 3D Gaussian points are clustered by the K-means clustering algorithm to speed up the MPM simulation process; (2) the cluster center of each cluster is bound to the nearest mesh vertex using the K-nearest neighbor algorithm, thereby achieving the binding of 3D Gaussian points and surface mesh vertices with the same motion trend. These two operations are mainly based on two considerations. On the one hand, since the number of 3D Gaussian points is huge, if the speed is allocated to each point separately in the MPM process, the computation time cost is unacceptable. On the other hand, since the number of 3D Gaussian points is much greater than the number of mesh vertices, the speed of multiple MPM points will eventually correspond to the same mesh vertex. Therefore, the 3D Gaussian points are clustered in advance using the K-means clustering algorithm, and the 3D bounding box of each cluster is calculated. In the actual MPM process, the points within each 3D bounding box are assigned the same velocity, reducing the difficulty of the solution and accelerating the process.

[0222] It should be noted that, in addition to the contents described above, this embodiment may also include the technical features described in the above embodiments, thereby achieving the technical effect of the four-dimensional content synthesis method shown above. Please refer to the above description for details. For the sake of brevity, it will not be elaborated here.

[0223] The four-dimensional content synthesis method provided in this disclosure achieves the binding of three-dimensional Gaussian points to surface mesh vertices and the mapping of bone motion to three-dimensional Gaussian point motion by performing surface reconstruction, bone extraction, and skinning prediction on the mask object. Through the motion control and mask physics simulation of MagicPhysics4D, not only can the physical rationality of 4D content be achieved, but different artistic effects can also be achieved by assigning physical properties.

[0224] Figure 4 is a schematic diagram of the structure of a four-dimensional content synthesis device provided in an embodiment of this disclosure. Specifically, it includes:

[0225] The first determining unit 401 is configured to: determine multiple three-dimensional Gaussian points on the three-dimensional model of the spatial object;

[0226] The partitioning unit 402 is configured to divide multiple three-dimensional Gaussian points into multiple component units, wherein the component units are components of the three-dimensional model;

[0227] The second determining unit 403 is configured to: determine the physical properties corresponding to each of the multiple component units;

[0228] Control unit 404 is configured to control the movement of three-dimensional Gaussian points according to their corresponding physical properties in order to synthesize the four-dimensional content of a spatial object.

[0229] Figure 5 is a schematic diagram of another four-dimensional content synthesis device provided in an embodiment of this disclosure.

[0230] In some possible implementations, the first determining unit 401 includes:

[0231] The first determining subunit 4011 is configured to: acquire images of a spatial object from multiple different shooting angles to obtain an image set;

[0232] The reconstruction subunit 4012 is configured to: use a three-dimensional Gaussian splashing algorithm with a preset regularization term to reconstruct the spatial object based on the image set, and obtain multiple three-dimensional Gaussian points on the three-dimensional model of the spatial object;

[0233] The preset regularization term is used to reduce the distance between the 3D Gaussian points and the surface of the 3D model, and to improve the uniformity of the distribution of multiple 3D Gaussian points.

[0234] In some possible implementations, the partitioning unit 402 includes:

[0235] The second determining subunit 4021 is configured to: determine the mask information corresponding to each of the multiple three-dimensional Gaussian points based on a pre-trained three-dimensional mask model; wherein, the three-dimensional mask model is used to determine the mask information corresponding to the three-dimensional Gaussian points.

[0236] Sub-unit 4022 is configured to divide multiple three-dimensional Gaussian points into multiple component units based on the determined mask information.

[0237] In some possible implementations, the image set is a sequence of images ordered according to the shooting order of the spatial objects; and

[0238] The 3D mask model was trained in the following way:

[0239] Target tracking is performed on the image sequence to obtain a labeled image sequence, wherein the labels of the labeled images in the labeled image sequence are used to distinguish component units;

[0240] The labeled images in the labeled image sequence are input into a pre-trained two-dimensional mask model to generate mask information for the labeled images. The two-dimensional mask model is used to determine the mask information for the labeled images.

[0241] The three-dimensional Gaussian points corresponding to the labeled image are used as the sample input data, and the mask information of the labeled image is used as the sample output data to generate training samples containing sample input data and sample output data.

[0242] A three-dimensional mask model is obtained by training a machine learning algorithm based on training samples.

[0243] In some possible implementations, the control unit 404 includes:

[0244] Clustering subunit 4041 is configured to cluster multiple 3D Gaussian points on a 3D model to obtain multiple clusters.

[0245] The control subunit 4042 is configured to control multiple three-dimensional Gaussian points belonging to the same cluster to move at the same speed and according to their respective physical properties.

[0246] In some possible implementations, the control unit 404 includes:

[0247] The first building subunit 4043 (not shown in the figure) is configured to: build the surface mesh of the three-dimensional model of the spatial object;

[0248] Sub-unit 4044 (not shown in the figure) is configured to establish the association between multiple three-dimensional Gaussian points and the mesh points of the surface mesh;

[0249] The first control subunit 4045 (not shown in the figure) is configured to control multiple three-dimensional Gaussian points associated with the same grid point to move at the same speed as the associated grid point and with their respective physical properties.

[0250] In some possible implementations, establishing subunit 4044 includes:

[0251] Clustering module 40441 (not shown in the figure) is configured to cluster multiple 3D Gaussian points on a 3D model to obtain multiple clusters;

[0252] The first determining module 40442 (not shown in the figure) is configured to: determine the target three-dimensional Gaussian point located at the center of the target cluster in multiple clusters;

[0253] The second determining module 40443 (not shown in the figure) is configured to: determine the target grid point with the smallest distance from the target three-dimensional Gaussian point from multiple grid points of the three-dimensional model;

[0254] Module 40444 (not shown in the figure) is configured to establish the association between multiple three-dimensional Gaussian points in the target cluster and the target grid points.

[0255] In some possible implementations, the control unit 404 includes:

[0256] The second building subunit 4045 (not shown in the figure) is configured to: build the surface mesh of the three-dimensional model of the spatial object;

[0257] The binding subunit 4046 (not shown in the figure) is configured to bind bones and skin to the surface mesh based on a set bone sequence to obtain three-dimensional Gaussian points driven by the bone sequence.

[0258] The second control subunit 4047 (not shown in the figure) is configured to: use the three-dimensional Gaussian points driven by the skeleton sequence as material points, and use a material point device to control the material points to move according to the corresponding physical properties.

[0259] The four-dimensional content synthesis device provided in this embodiment can be the four-dimensional content synthesis device shown in Figure 4. It can execute all the steps of the four-dimensional content synthesis methods described above, thereby achieving the technical effects of the four-dimensional content synthesis methods described above. For details, please refer to the relevant descriptions above. For the sake of brevity, it will not be elaborated here.

[0260] Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. The electronic device 500 shown in Figure 6 includes: at least one processor 501, a memory 502, at least one network interface 504, and other user interfaces 503. The various components in the electronic device 500 are coupled together through a bus system 505. It is understood that the bus system 505 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 505 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 505 in Figure 6.

[0261] The user interface 503 may include a display, keyboard, or clicking device (e.g., mouse, trackball, touchpad, or touchscreen).

[0262] It is understood that the memory 502 in this embodiment can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 502 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0263] In some implementations, memory 502 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 5021 and application program 5022.

[0264] The operating system 5021 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 5022 includes various applications, such as a media player and a browser, used to implement various application functions. The program implementing the method of this embodiment can be included in the application program 5022.

[0265] In this embodiment, by calling the program or instructions stored in memory 502, specifically the program or instructions stored in application program 5022, processor 501 executes the method steps provided in each method embodiment, including, for example:

[0266] Determine multiple 3D Gaussian points on a 3D model of a spatial object;

[0267] Multiple 3D Gaussian points are divided into multiple component units, where the component units are the constituent parts of the 3D model;

[0268] Determine the physical properties of each component unit among multiple component units;

[0269] Control the movement of three-dimensional Gaussian points according to their corresponding physical properties to synthesize the four-dimensional content of spatial objects.

[0270] The methods disclosed in the above embodiments of this disclosure can be applied to or implemented by processor 501. Processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware or by instructions in the form of software in processor 501. The processor 501 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 502. Processor 501 reads the information in memory 502 and, in conjunction with its hardware, completes the steps of the above method.

[0271] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described above, or combinations thereof.

[0272] For software implementation, the techniques described herein can be implemented by units that perform the functions described above. The software code can be stored in memory and executed by a processor. The memory can be implemented within the processor or external to the processor.

[0273] The electronic device provided in this embodiment can be the electronic device shown in Figure 6, which can execute all the steps of the above-described four-dimensional content synthesis methods, thereby achieving the technical effects of the above-described four-dimensional content synthesis methods. For details, please refer to the above-described related descriptions. For the sake of brevity, it will not be elaborated here.

[0274] This disclosure also provides a storage medium (computer-readable storage medium). This storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; the memory may also include combinations of the above types of memory.

[0275] When one or more programs in the storage medium can be executed by one or more processors to achieve the above-mentioned method of synthesizing four-dimensional content executed on the electronic device side.

[0276] The processor described above is used to execute a synthesis program for four-dimensional content stored in memory to implement the following steps of a four-dimensional content synthesis method executed on the electronic device side:

[0277] Determine multiple 3D Gaussian points on a 3D model of a spatial object;

[0278] Multiple 3D Gaussian points are divided into multiple component units, where the component units are the constituent parts of the 3D model;

[0279] Determine the physical properties of each component unit among multiple component units;

[0280] Control the movement of three-dimensional Gaussian points according to their corresponding physical properties to synthesize the four-dimensional content of spatial objects.

[0281] Furthermore, the computer program product provided in this disclosure embodiment may include computer-readable code that, when executed on a device, causes a processor in the device to implement the steps of the following four-dimensional content synthesis method executed on the electronic device side:

[0282] Determine multiple 3D Gaussian points on a 3D model of a spatial object;

[0283] Multiple 3D Gaussian points are divided into multiple component units, where the component units are the constituent parts of the 3D model;

[0284] Determine the physical properties of each component unit among multiple component units;

[0285] Control the movement of three-dimensional Gaussian points according to their corresponding physical properties to synthesize the four-dimensional content of spatial objects.

[0286] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0287] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0288] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also mean including the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0289] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for synthesizing four-dimensional content, characterized in that, The method includes: determining multiple three-dimensional Gaussian points on a three-dimensional model of a spatial object; dividing the multiple three-dimensional Gaussian points into multiple component units, wherein each component unit is a component of the three-dimensional model and each component unit has independent physical properties; determining the physical properties corresponding to each component unit in the multiple component units, wherein the physical properties are parameters used to determine the motion law of the three-dimensional Gaussian points and describe the physical characteristics of the object; controlling the three-dimensional Gaussian points to move according to the corresponding physical properties to synthesize the four-dimensional content of the spatial object; the division of multiple three-dimensional Gaussian points into multiple component units includes: determining the mask information corresponding to each three-dimensional Gaussian point in the multiple three-dimensional Gaussian points based on a pre-trained three-dimensional mask model; wherein the three-dimensional mask model is used to determine the mask information corresponding to the three-dimensional Gaussian points; dividing the multiple three-dimensional Gaussian points into multiple component units based on the determined mask information; the determination of multiple three-dimensional Gaussian points on a three-dimensional model of a spatial object includes: acquiring images of the spatial object from multiple different shooting angles to obtain an image set; determining multiple three-dimensional Gaussian points on a three-dimensional model of the spatial object based on the image set; the image set is a set of images of the spatial object from multiple shooting angles to obtain an image set. The image sequence is ordered according to the order in which the image was captured; and the three-dimensional mask model is trained as follows: target tracking is performed on the image sequence to obtain a sequence of labeled images, wherein the labels of the labeled images in the sequence of labeled images are used to distinguish component units; the labeled images in the sequence of labeled images are input into a pre-trained two-dimensional mask model to generate mask information of the labeled images, wherein the two-dimensional mask model is used to determine the mask information of the labeled images; the three-dimensional Gaussian points corresponding to the labeled images are used as sample input data, and the mask information of the labeled images is used as sample output data to generate training samples containing the sample input data and the sample output data; a machine learning algorithm is used to train the three-dimensional mask model based on the training samples; the control of the three-dimensional Gaussian points to move according to the corresponding physical properties includes: constructing a surface mesh of the three-dimensional model of the spatial object; binding bones and skin to the surface mesh based on a set bone sequence to obtain three-dimensional Gaussian points driven by the bone sequence; using the three-dimensional Gaussian points driven by the bone sequence as material points, the material point method is used to control the material points to move according to the corresponding physical properties.

2. The method according to claim 1, characterized in that, The step of determining multiple three-dimensional Gaussian points on the three-dimensional model of the spatial object based on the image set includes: using a three-dimensional Gaussian splashing algorithm with a preset regularization term to reconstruct the spatial object based on the image set to obtain multiple three-dimensional Gaussian points on the three-dimensional model of the spatial object; wherein, the preset regularization term is used to reduce the distance between the three-dimensional Gaussian points and the surface of the three-dimensional model, and to improve the distribution uniformity of the multiple three-dimensional Gaussian points.

3. The method according to any one of claims 1-2, characterized in that, The control of the three-dimensional Gaussian points to move according to their corresponding physical properties includes: clustering multiple three-dimensional Gaussian points on the three-dimensional model to obtain multiple clusters; and controlling multiple three-dimensional Gaussian points belonging to the same cluster to move at the same speed and according to their respective physical properties.

4. The method according to any one of claims 1-2, characterized in that, The process of controlling the movement of three-dimensional Gaussian points according to their corresponding physical properties includes: constructing a surface mesh of the three-dimensional model of the spatial object; establishing a relationship between multiple three-dimensional Gaussian points and mesh points of the surface mesh; and controlling multiple three-dimensional Gaussian points associated with the same mesh point to move at the same speed as the associated mesh point and according to their respective physical properties.

5. The method according to claim 4, characterized in that, The process of establishing the association between multiple 3D Gaussian points and the mesh points of the surface mesh includes: clustering multiple 3D Gaussian points on the 3D model to obtain multiple clusters; determining a target 3D Gaussian point located at the center of a target cluster in the multiple clusters; determining a target mesh point with the smallest distance from the target 3D Gaussian point from the multiple mesh points of the 3D model; and establishing the association between multiple 3D Gaussian points in the target cluster and the target mesh point.

6. A four-dimensional content synthesis device, characterized in that, The device includes: a first determining unit configured to: determine multiple three-dimensional Gaussian points on a three-dimensional model of a spatial object; a dividing unit configured to: divide the multiple three-dimensional Gaussian points into multiple component units, wherein the component units are components of the three-dimensional model, and each component unit has independent physical properties; a second determining unit configured to: determine the physical properties corresponding to each component unit among the multiple component units, wherein the physical properties are parameters used to determine the motion laws of the three-dimensional Gaussian points and describe the physical characteristics of the object; and a control unit configured to: control the three-dimensional Gaussian points to move according to the corresponding physical properties to synthesize the spatial object. The four-dimensional content; the division of multiple three-dimensional Gaussian points into multiple component units includes: determining the mask information corresponding to each three-dimensional Gaussian point in the multiple three-dimensional Gaussian points based on a pre-trained three-dimensional mask model; wherein, the three-dimensional mask model is used to determine the mask information corresponding to the three-dimensional Gaussian points; based on the determined mask information, the multiple three-dimensional Gaussian points are divided into multiple component units; the determination of multiple three-dimensional Gaussian points on the three-dimensional model of the spatial object includes: acquiring images of the spatial object from multiple different shooting perspectives to obtain an image set; based on the image set, determining multiple three-dimensional Gaussian points on the three-dimensional model of the spatial object; the... The image set is an image sequence ordered according to the shooting order of the spatial object; and the three-dimensional mask model is trained as follows: target tracking is performed on the image sequence to obtain a sequence of labeled images, wherein the labels of the labeled images in the sequence of labeled images are used to distinguish component units; the labeled images in the sequence of labeled images are input into a pre-trained two-dimensional mask model to generate mask information of the labeled images, wherein the two-dimensional mask model is used to determine the mask information of the labeled images; the three-dimensional Gaussian points corresponding to the labeled images are used as sample input data, and the mask information of the labeled images is used as sample output data to generate... The system includes training samples containing the input and output data; a machine learning algorithm is used to train a 3D mask model based on the training samples; the control unit includes: a second construction subunit configured to construct a surface mesh of the 3D model of the spatial object; a binding subunit configured to bind bones and skin to the surface mesh based on a set bone sequence to obtain 3D Gaussian points driven by the bone sequence; and a second control subunit configured to use the 3D Gaussian points driven by the bone sequence as material points and control the material points to move according to their corresponding physical properties using a material point method.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor is configured to execute a computer program stored in the memory, wherein, when the computer program is executed, it implements the method described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-5.

9. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-5.

Images