Dynamic 3D target modeling, method and device for dynamic 3D object modeling
By optimizing sparse and dense 3D Gaussian dynamic 3D target modeling in stages, the problem of low accuracy of dynamic 3D target modeling in the existing technology is solved, and the effect and accuracy of dynamic 3D target modeling is improved.
Patent Information
- Application Number
- CN202410383787.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-29
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-03-29
AI Technical Summary
The existing dynamic 3D target modeling schemes have poor results in the degree of conformity of reference perspectives, the spatial and temporal consistency of dynamic objects and the authenticity of actions, resulting in low accuracy and poor results.
By obtaining reference videos, initializing and optimizing sparse 3D Gaussian and deformation networks, a dynamic 3D model is built in stages, first rationalizing the movement and shape, and then refining the texture, using dense 3D Gaussian motion to be controlled by sparse control points to reduce motion and appearance. Fuzzy.
It improves the accuracy and effect of dynamic 3D target modeling, and performs better in terms of the degree of conformity of reference perspectives, the spatial and temporal consistency of dynamic objects and the authenticity of actions.
Smart Images

Figure CN118314254B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to computer technology, and more particularly to a method and device for dynamic 3D target modeling and dynamic 3D object modeling. Background Art
[0002] Four-dimensional (4D) objects / scenes refer to dynamic three-dimensional (3D) objects / scenes, which allow 3D objects / scenes to perform different actions. With the development of the field of generative modeling, significant progress has been made in 3D generation technology, including generating the shape, texture, and even entire scenes of static objects from text prompts or single images. Compared with static 3D objects / scenes, dynamic 3D (4D) objects / scenes provide greater spatiotemporal flexibility and have greater application potential in augmented reality (AR) / virtual reality (VR), filming, animation, games, and other fields.
[0003] The current dynamic 3D target modeling solutions are difficult to achieve good results in three aspects: the degree of conformity with the reference perspective, the spatiotemporal consistency of dynamic objects, and the authenticity of movements, resulting in low accuracy and poor results in dynamic 3D target modeling. Summary of the invention
[0004] The present application provides a method and device for dynamic 3D target modeling and dynamic 3D object modeling, so as to solve the problems of low accuracy and poor effect of dynamic 3D target modeling.
[0005] In a first aspect, the present application provides a dynamic 3D target modeling method, comprising:
[0006] Obtain a reference video of a first target; in a first stage, initialize the sparse control points of the first target to a sparse 3D Gaussian, obtain an initialized deformation network, and optimize the sparse 3D Gaussian and deformation network of the first target based on the reference video of the first target to obtain the optimized sparse control points and deformation network of the first target; in a second stage, initialize a dense 3D Gaussian of the first target, and based on the reference video of the first target, the optimized sparse control points and deformation network of the first target, optimize the dense 3D Gaussian of the first target to obtain the optimized dense 3D Gaussian of the first target, wherein the optimized sparse control points, deformation network and dense 3D Gaussian of the first target are used to represent the dynamic 3D model of the first target.
[0007] In a second aspect, the present application provides a dynamic 3D object modeling method, comprising:
[0008] A dynamic 3D object modeling request sent by a terminal device is received, and a reference video of the object is obtained; the sparse control points of the object are initialized as sparse 3D Gaussians, an initialized deformation network is obtained, and the sparse 3D Gaussians and deformation networks of the object are optimized based on the reference video of the object to obtain the optimized sparse control points and deformation network of the object; a dense 3D Gaussian of the object is initialized, and the dense 3D Gaussian of the object is optimized based on the reference video of the object, the optimized sparse control points and deformation network of the object to obtain the optimized dense 3D Gaussian of the object; according to the configured time step sequence, the optimized sparse control points and deformation network of the object, the optimized dense 3D Gaussian of the object is driven to deform, and the deformed dense 3D Gaussian of the object at each time step in the time step sequence is obtained; and the dynamic 3D model of the object is returned to the terminal device, and the dynamic 3D model of the object includes the deformed dense 3D Gaussian of the object at each time step in the time step sequence.
[0009] In a third aspect, the present application provides a server, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the server executes a method provided in any of the preceding aspects.
[0010] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method provided in any of the aforementioned aspects is implemented.
[0011] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the method provided in any of the above aspects when executed by a processor.
[0012] The method and device for dynamic 3D target modeling and dynamic 3D object modeling provided by the present application, in the first stage, initializes the sparse control points to the displayed 3D Gaussian, optimizes the parameters of the sparse 3D Gaussian and the deformation network to obtain the reasonable movement and shape of the dynamic 3D target corresponding to the reference video; in the second stage, optimizes the dense 3D Gaussian to achieve a refined simulation of the texture of the dynamic 3D target, and the movement of the dense 3D Gaussian is controlled by the sparse control points obtained in the first stage; by separately modeling the movement and appearance of the dynamic 3D target, the ambiguity of the movement and appearance is reduced, and a dynamic 3D target (i.e., a 4D target) that fits the reference video very well can be generated, which has achieved better results than the current similar solutions in terms of the degree of conformity with the reference perspective, the spatiotemporal consistency of the dynamic object, and the authenticity of the movement, thereby improving the accuracy and effect of dynamic 3D target modeling. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0014] Figure 1 A schematic diagram of an exemplary system architecture applicable to the present application;
[0015] Figure 2 A flowchart of a dynamic 3D target modeling method provided for an exemplary embodiment of the present application;
[0016] Figure 3 A schematic diagram of initializing a dense 3D Gaussian based on sparse control points provided by an exemplary embodiment of the present application;
[0017] Figure 4 A flowchart of the first stage optimization of sparse 3D Gaussian and deformable networks provided for an exemplary embodiment of the present application;
[0018] Figure 5 A flowchart of the second stage optimization of dense 3D Gaussian provided for an exemplary embodiment of the present application;
[0019] Figure 6 An overall framework diagram of two stages of dynamic 3D object modeling provided by an exemplary embodiment of the present application;
[0020] Figure 7 A schematic diagram of the effect of dynamic 3D target modeling and motion migration application provided by an exemplary embodiment of the present application;
[0021] Figure 8 A flowchart of a dynamic 3D target modeling method based on action migration provided by an exemplary embodiment of the present application;
[0022] Fig. 9 A flowchart of the second stage optimization based on action migration provided by an exemplary embodiment of the present application;
[0023] Fig.10 A framework diagram of dynamic 3D target modeling based on motion migration provided by an exemplary embodiment of the present application;
[0024] Fig.11 A flow chart of a method for dynamic 3D object modeling provided by an exemplary embodiment of the present application;
[0025] Fig.12 A schematic diagram of the structure of a server provided in an embodiment of the present application.
[0026] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0027] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0028] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0029] First, the terms involved in this application are explained:
[0030] Video-to-4D generation: The process of generating corresponding dynamic 3D objects based on a single single-view video.
[0031] 3D Gaussian: represents an ellipsoid, which can be expressed as {μ, q, s, σ, sh}, where μ represents the center position of the 3D Gaussian, q represents the orientation of the 3D Gaussian, s represents the size of the 3D Gaussian (including the radius in different directions), σ represents the opacity of the 3D Gaussian, and sh represents the spherical harmonic coefficients of the 3D Gaussian.
[0032] 3D Gaussian Splatting (3DGS): A method of reconstructing a 3D scene using 3D Gaussians, which represents the 3D scene as a set of displayed 3D Gaussians. It can be defined as: Where ∑ represents the 3D covariance matrix, ∑ = RSS T R T (R, S can be calculated from q, s). For the rendered 3D Gaussian at any viewing angle v i For pixel u in the two-dimensional image, use alpha blending to get the color of pixel u: Where i refers to any 3D Gaussian, Color(u) represents the color of pixel u, and α iis the probability opacity, α i =σ i G(u),v i represents the viewing angle, and SH is the spherical harmonic function.
[0033] Dynamic Gaussian Splatting (DGS): The task and method of reconstructing dynamic objects or scenes using 3D Gaussian.
[0034] Multi-layer Perceptron (MLP): A basic network architecture consisting of fully connected layers and activation layers.
[0035] SC-GS (Sparse-Controlled Gaussian Splatting): is a new dynamic scene reconstruction method that decouples the appearance (i.e., surface) and motion of a 4D scene into dense 3D Gaussian and sparse control points, and uses control points to drive the dense 3D Gaussian.
[0036] The present application provides a dynamic 3D target modeling method, which can be applied to AR / VR, shooting, animation, games and other fields with greater application potential. Specifically, it can be widely used in many industry fields such as film and television animation, game design, industrial design, architectural design, interior design, product design, garden landscape design, engineering, medical treatment, science, education, intelligent transportation, etc., and is used to model dynamic 3D targets based on reference videos of the target to be modeled (such as objects or scenes).
[0037] The present application provides a dynamic 3D target modeling method, which represents the dynamic 3D target by using sparse control points, deformation networks and dense 3D Gaussians, that is, decoupling the appearance and motion of the dynamic 3D target, using dense 3D Gaussians to represent the appearance of the target, and using sparse control points and deformation networks to represent the motion of the target. Specifically, in the first stage, a reference video containing a first target is obtained, and sparse control points and deformation networks for initializing the first target are obtained, wherein the sparse control points are initialized to the sparse 3D Gaussians of the first target; based on the reference video of the first target, the sparse 3D Gaussians and deformation networks of the first target are optimized to obtain the optimized sparse control points and deformation networks of the first target; further, in the second stage, the dense 3D Gaussians of the first target are initialized, and based on the reference video of the first target, the sparse control points and deformation networks of the first target after optimization, the dense 3D Gaussians of the first target are optimized to obtain the optimized dense 3D Gaussians of the first target. Among them, the sparse control points, deformation networks and dense 3D Gaussians of the first target after optimization are used to represent the dynamic 3D model of the first target.
[0038] The deformable network may be implemented using a multi-layer perceptron (MLP) or other network structures with similar capabilities, which is not specifically limited in this embodiment.
[0039] The method of the present application, in the first stage, initializes the sparse control points to the displayed 3D Gaussians, optimizes the parameters of these sparse 3D Gaussians and the deformation network to obtain the motion and shape of the dynamic 3D target that is reasonable and corresponds to the reference video; in the second stage, the dense 3D Gaussians are optimized to achieve a refined simulation of the texture of the dynamic 3D target, and the motion of these dense 3D Gaussians is controlled by the sparse control points obtained in the first stage. By separately modeling the motion and appearance of the dynamic 3D target, the ambiguity of the motion and appearance is reduced, and a dynamic 3D target (i.e., a 4D target) that fits the reference video very well can be generated. It has achieved better results than similar solutions in terms of the degree of conformity with the reference perspective, the spatiotemporal consistency of the dynamic object, and the authenticity of the motion, and has improved the accuracy and effect of dynamic 3D target modeling.
[0040] Figure 1 FIG. 1 is a schematic diagram of an exemplary system architecture applicable to the present application. Figure 1 As shown, the system architecture includes a server and a terminal device. There is a communication link between the server and the terminal device, which can realize the communication connection between the server and the terminal device.
[0041] The server is a device with computing power deployed in the cloud or locally, such as a cloud cluster, etc. The server stores a deformable network. The server is responsible for reconstructing a dynamic 3D target (and 4D target) based on a reference video of a given target.
[0042] The end-side device can be an electronic device running downstream application software / system, and specifically can be a hardware device with network communication function, computing function and information display function, including but not limited to smart phones, tablet computers, desktop computers, local servers, cloud servers, etc. For example, the end-side device can be an electronic device running various software applications / systems such as game design, film and television animation production, industrial design, architectural design, interior design, product design, etc. The downstream application software / system provides the function of modeling dynamic 3D targets based on the reference video of a given target, and the subsequent processing logic such as game design, film and television animation production, and dynamic display of products can be completed based on the dynamic 3D target.
[0043] When the function of modeling a dynamic 3D target based on a reference video is needed, the user sends a reference video of the first target to the server through the terminal device. The server receives the reference video of the first target, obtains the sparse control points and deformation network of the initialized first target, wherein the sparse control points are initialized as the sparse 3D Gaussian of the first target; based on the reference video of the first target, the sparse 3D Gaussian and deformation network of the first target are optimized to obtain the optimized sparse control points and deformation network of the first target; initializes the dense 3D Gaussian of the first target, based on the reference video of the first target, the sparse control points and deformation network of the first target after optimization, optimizes the dense 3D Gaussian of the first target, and obtains the optimized dense 3D Gaussian of the first target.
[0044] The optimized sparse control points, deformation network and dense 3D Gaussian of the first target are used to represent the dynamic 3D model of the first target. Based on any given moment, the sparse control points and deformation network of the first target drive the dense 3D Gaussian to obtain the deformed dense 3D Gaussian at the given moment, and the deformed dense 3D Gaussian at the given moment is rendered according to any perspective, so that a two-dimensional image of the dynamic 3D target at any perspective at any given moment can be obtained.
[0045] Furthermore, the server returns the sparse control points, deformable network and dense 3D Gaussian after the first target optimization to the end-side device. The end-side device can execute subsequent processing logic according to the requirements of downstream application software / system based on the sparse control points, deformable network and dense 3D Gaussian after the first target optimization. For example, based on the sparse control points, deformable network and dense 3D Gaussian after the first target optimization, a display video of multiple perspectives of a dynamic 3D target is generated, and a display video of multiple perspectives of a dynamic 3D target is output.
[0046] It should be noted that, in this embodiment, rendering an image of any 3D Gaussian (such as sparse 3D Gaussian, dense 3D Gaussian, deformed sparse 3D Gaussian, deformed dense 3D Gaussian) at any viewing angle can be obtained by 3DGS: using α blending rendering to obtain the color of pixel u: Where Color(u) represents the color of pixel u after rendering, α i is the probability opacity of pixel u, α i =σ i G(u),v i represents the viewing angle, and SH is the spherical harmonic function.
[0047] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0048] Figure 2 This is a flow chart of a dynamic 3D target modeling method provided by an exemplary embodiment of the present application. The execution subject of this embodiment is the server in the aforementioned system architecture. Figure 2 As shown, the specific steps of this method are as follows:
[0049] Step S201: Obtain a reference video of a first target.
[0050] The first target may be an object or a scene, which may be specified by a user. The reference video may be a video of the first target captured by a camera at a fixed camera position, including multiple frames of images of the first target at multiple different times. For example, the reference video of the first target may be obtained by shooting with a fixed camera perpendicular to the first target.
[0051] Step S202: In the first stage, the sparse control points of the first target are initialized as sparse 3D Gaussians to obtain an initialized deformable network.
[0052] In this embodiment, the dynamic 3D model of the first target is represented by sparse control points, a deformable network and a dense 3D Gaussian, that is, the dynamic 3D target is represented by sparse control points, a deformable network and a dense 3D Gaussian. The appearance and operation of the dynamic 3D target are decoupled, the appearance of the target is represented by a dense 3D Gaussian, and the movement of the target is represented by a sparse control point and a deformable network.
[0053] Based on any given moment, the deformed dense 3D Gaussian at the given moment is obtained by driving the dense 3D Gaussian of the first target through the sparse control points and the deformation network. By rendering the deformed dense 3D Gaussian at the given moment according to any perspective, the image of the dynamic 3D target at any given moment and any perspective can be obtained.
[0054] Among them, the "sparseness" of the control points representing the movement of the dynamic 3D target and the "density" of the 3D Gaussian representing the appearance of the dynamic 3D target are relative concepts. The number of control points representing the movement of the dynamic 3D target is much smaller than the 3D Gaussian representing the appearance of the dynamic 3D target. Therefore, the control points representing the movement of the dynamic 3D target are called "sparse control points", and the 3D Gaussian representing the appearance of the dynamic 3D target is called "dense 3D Gaussian".
[0055] The first number represents the number of sparse control points, and the second number represents the number of dense 3D Gaussians, and the first number is much smaller than the second number. The first number and the second number can be configured and adjusted according to the needs of the actual scene. For example, the first number can be hundreds or thousands, and the second number can be tens of times, hundreds of times or even higher than the first number. The first number and the second number are not specifically limited in this embodiment.
[0056] In this step, the initialized sparse control points of the first target can be obtained by randomly initializing sparse (first number or less than the first number) 3D Gaussians in the reference space. Among them, the 3D model can represent the surface of the target as a point in the 3D space. The reference space refers to a pre-set 3D space for modeling dynamic 3D targets. In this embodiment, in order to obtain more evenly distributed sparse control points, the initialized sparse 3D Gaussians are constrained to spheres with the same radius.
[0057] The deformable network can be implemented using a multi-layer perceptron (MLP) or other network structures with similar capabilities, which are not specifically limited in this embodiment. The parameters of the deformable network can be obtained by random initialization, or a set of initial parameters of the deformable network can be selected according to actual use scenarios and experience, which are not specifically limited here.
[0058] In this embodiment, dynamic 3D target modeling includes two stages: in the first stage, based on the reference video of the first target, the sparse 3D Gaussian and deformation network of the first target are optimized to obtain the optimized sparse control points and deformation network of the first target, that is, to obtain the rationalized motion and shape of the first target corresponding to the reference video.
[0059] In the second stage, based on the reference video of the first target, the optimized sparse control points and deformation network of the first target, the dense 3D Gaussian of the first target is optimized to achieve a refined simulation of the texture of the first target. The movement of these dense 3D Gaussian is controlled by the sparse control points obtained in the first stage.
[0060] In an optional embodiment, in order to obtain sparse control points with more uniform distribution, in the first stage of optimization, the sparse 3D Gaussians serving as sparse control points are constrained to spheres with the same radius (represented by S). Exemplarily, based on the configured constraint range of radius S, M 3D Gaussians with radius S are initialized in the reference space, where M represents the first number of sparse control points.
[0061] Step S203: Based on the reference video of the first target, the sparse 3D Gaussian and deformable network of the first target are optimized to obtain the optimized sparse control points and deformable network of the first target.
[0062] This step performs the first stage of optimization. Based on the reference video of the first target, the initial sparse 3D Gaussian and deformation network of the first target is optimized to obtain the rationalized motion and shape of the first target corresponding to the reference video.
[0063] Specifically, at least one frame of the reference video is used as a reference image, and the time step corresponding to the reference image and the center position of the sparse 3D Gaussian of the first target are input into the deformation network. The displacement of the sparse 3D Gaussian in the corresponding time step is predicted by the deformation network; according to the displacement of the sparse 3D Gaussian in the corresponding time step, the deformed sparse 3D Gaussian of the first target in the corresponding time step can be predicted. The images of the deformed sparse 3D Gaussian of the first target in the corresponding time step at the corresponding perspective of the reference image and other perspectives are rendered to obtain reconstructed images from multiple perspectives. The loss is calculated based on the reference image and the reconstructed images from multiple perspectives, and the parameters of the sparse 3D Gaussian of the first target and the parameters of the deformation network are optimized by back propagation.
[0064] Through the optimization of the first stage, the sparse 3D Gaussian after the first target is optimized can model the shape of the first target. Here, the optimized sparse 3D Gaussian gives a rough and fuzzy shape of the first target. The sparse 3D Gaussian after the first target is optimized as a sparse control point, together with the optimized deformation network, determines the movement of the first target.
[0065] Step S204, in the second stage, initialize the dense 3D Gaussian of the first target, optimize the dense 3D Gaussian of the first target based on the reference video of the first target, the sparse control points and the deformation network after the optimization of the first target, and obtain the dense 3D Gaussian after the optimization of the first target; the sparse control points, deformation network and dense 3D Gaussian after the optimization of the first target are used to represent the dynamic 3D model of the first target.
[0066] This step performs the second stage of optimization. Based on the reference video of the first target, the optimized sparse control points and deformation network of the first target, the dense 3D Gaussian of the first target is optimized to achieve a refined simulation of the texture of the first target. The movement of these dense 3D Gaussians is controlled by the sparse control points obtained in the first stage.
[0067] Before the second stage of optimization, the dense 3D Gaussian of the first target is initialized.
[0068] In a preferred embodiment, the initial dense 3D Gaussian of the first target can be initialized as follows: multiple 3D Gaussians are initialized within the sphere of the optimized sparse control points of the first target to obtain the initialized dense 3D Gaussian of the first target. The dense 3D Gaussian initialized in this way effectively inherits the shape and motion modeled in the first stage. The dense 3D Gaussian is distributed near the surface of the first target, which is conducive to subsequent optimization.
[0069] Exemplarily, according to the first number M of sparse control points and the number K of initial dense 3D Gaussians configured in the sphere of each sparse control point, K 3D Gaussians are randomly initialized in the sphere of each sparse control point, and M×K 3D Gaussians can be obtained. Wherein M and K are both positive integers, M×K is much larger than M, and M and K can be configured and adjusted according to the needs of the actual use scene, and are not specifically limited here.
[0070] Exemplarily, according to the second number N of dense 3D Gaussians to be initialized and the first number M of sparse control points, the number of dense 3D Gaussians to be initialized in the sphere of each sparse control point is determined, so that the N dense 3D Gaussians are more evenly distributed in the M sparse control points; a corresponding number of 3D Gaussians are initialized in the sphere of the sparse control points to obtain the initialized dense 3D Gaussians of the first target. The number of 3D Gaussians initialized in different control points may be different, but the difference in number is small.
[0071] In an optional embodiment, the initial dense 3D Gaussian of the first target can be obtained by uniformly initializing the second number of 3D Gaussians in a sphere in the reference space. The dense 3D Gaussian initialized in this way tends to increase the target thickness, and may cause problems such as texture reduction and position drift.
[0072] For example, Figure 3 A schematic diagram of initializing a dense 3D Gaussian based on sparse control points provided in this embodiment, such as Figure 3 As shown, for the sparse control points obtained in the first stage, each of which is a 3D Gaussian with a radius S, multiple 3D Gaussians are initialized in the sphere of each sparse control point to obtain dense (the second number or not less than the second number) 3D Gaussians. Figure 3 The far right side shows a schematic diagram of a dense 3D Gaussian obtained by random initialization in the reference space. Compared with the dense 3D Gaussian obtained by random initialization in the reference space, the dense 3D Gaussian obtained by initializing multiple 3D Gaussians within the sphere of optimized sparse control points of the first target in this embodiment already has the shape of the first target, which can greatly improve the optimization result of the second stage.
[0073] After initializing the dense 3D Gaussian of the first target, at least one frame of the reference video is used as the reference image, and based on the time step corresponding to the reference image, the optimized sparse control points of the first target, and the deformation network, the dense 3D Gaussian of the first target is driven to deform, and the deformed dense 3D Gaussian of the first target at the corresponding time step is obtained; the deformed sparse 3D Gaussian of the first target at the corresponding time step is rendered at the corresponding view of the reference image and other viewpoints to obtain reconstructed images from multiple viewpoints. The loss is calculated based on the reference image and the reconstructed images from multiple viewpoints, and the parameters of the dense 3D Gaussian of the first target are optimized by back propagation.
[0074] By optimizing the dense 3D Gaussian of the first target in the second stage, a refined simulation of the texture of the first target can be achieved.
[0075] The method of this embodiment, in the first stage, initializes sparse control points to displayed 3D Gaussians, optimizes the parameters of these sparse 3D Gaussians and the deformation network to obtain the reasonable movement and shape of the dynamic 3D target corresponding to the reference video, and in the second stage, optimizes the dense 3D Gaussians to achieve a refined simulation of the texture of the dynamic 3D target; by separately modeling the motion and appearance of the dynamic 3D target, the ambiguity of the motion and appearance is reduced, and a dynamic 3D target (i.e., a 4D target) that fits the reference video very well can be generated, which achieves better results than current similar solutions in terms of the degree of conformity with the reference perspective, the spatiotemporal consistency of the dynamic object, and the authenticity of the movement, thereby improving the accuracy and effect of dynamic 3D target modeling.
[0076] After obtaining the optimized sparse control points, deformation network and dense 3D Gaussian of the first target, based on any given moment, the sparse control points and deformation network of the first target can be used to drive the dense 3D Gaussian to obtain the deformed dense 3D Gaussian at the given moment. By rendering the deformed dense 3D Gaussian at the given moment according to any perspective, the image of the dynamic 3D target at any perspective at any given moment can be obtained.
[0077] In an optional embodiment, after obtaining the optimized sparse control points, deformation network and dense 3D Gaussian of the first target, the server can generate a dynamic 3D model of the first target based on at least one configured time step, and the dynamic 3D model of the first target includes the dense 3D Gaussian of the first target at each time step.
[0078] Specifically, the server uses the sparse control points and deformation network optimized by the first target to drive the dense 3D Gaussian optimized by the first target to deform according to at least one configured time step, obtains the deformed dense 3D Gaussian of the first target in at least one time step, and obtains the dynamic 3D model of the first target. Further, the image of the deformed dense 3D Gaussian of the second target at each time step at any viewing angle can be rendered.
[0079] Among them, based on any time step, the sparse control points and deformation network optimized by the first target are used to drive the dense 3D Gaussian optimized by the first target to make deformation, so as to obtain the implementation principle and technical effect of the deformed dense 3D Gaussian of the first target at the corresponding time step, which will be described in detail in the subsequent embodiments.
[0080] Furthermore, the server can also render the deformation dense 3D Gaussian of the first target at each time step, generate a display video of the dynamic 3D model of the first target, and output the display video of the dynamic 3D model of the first target to demonstrate the effect of generating the dynamic 3D model of the first target.
[0081] Specifically, the server renders images of the first target at each time step in multiple perspectives of the deformed dense 3D Gaussian, and obtains images of the first target at each time step in multiple perspectives; based on the images of the first target at each time step in multiple perspectives, a display video of the dynamic 3D model of the first target is generated, and a display video of the dynamic 3D model of the first target is output. The multiple perspectives displayed in the display video can be configured according to the needs of the actual application scenario, or can be customized by the user through the front-end device, which is not specifically limited here.
[0082] Figure 4 The flowchart of optimizing the sparse 3D Gaussian and deformable network of the first target based on the reference video of the first target provided in this embodiment. In an optional embodiment, as Figure 4 As shown, in the aforementioned step S202, based on the reference video of the first target, the sparse 3D Gaussian and deformation network of the first target is optimized, which can be specifically implemented by the following steps:
[0083] Step S2021: use at least one frame image in the reference video as a first reference image, input the time step corresponding to the first reference image and the center position of the sparse 3D Gaussian of the first target into the deformation network, and predict the displacement of the sparse 3D Gaussian in the corresponding time step through the deformation network.
[0084] In this embodiment, at least one frame of image can be randomly selected from the reference video as a reference image to perform the first stage optimization. In order to distinguish it from the reference image used in the second stage optimization, the reference image sampled in the first stage is called the first reference image, and the reference image sampled in the second stage is called the second reference image.
[0085] The time step corresponding to each frame image in the reference video can be determined by random sampling according to the time sequence of each frame image. For example, the reference video is represented as an image sequence Where F represents the number of frames of the reference video, then the time step t corresponding to the f-th frame image in the reference video can be obtained by sampling in the following manner: t = (f-1) / F. Of course, the time step sampling method in the existing dynamic 3D object reconstruction (i.e., 4D reconstruction) scheme can also be used to determine the time step corresponding to each frame image in the reference video, which is not specifically limited here.
[0086] In this embodiment, in order to obtain sparse control points with more uniform distribution, in the first stage of optimization, the sparse 3D Gaussians as sparse control points are constrained to spheres with the same radius (represented by S). In the first stage of optimization, for any first reference image, the corresponding time step t and the center position of the sparse 3D Gaussian of the first target (i.e., the sparse control point) are input into the deformation network, and the displacement of the sparse 3D Gaussian in the corresponding time step t (i.e., the displacement of the center position of the 3D Gaussian) is predicted by the deformation network.
[0087] Step S2022: Determine the deformed sparse 3D Gaussian of the first target at the corresponding time step according to the sparse 3D Gaussian and the displacement of the sparse 3D Gaussian in the corresponding time step.
[0088] In this step, the position of each sparse 3D Gaussian is adjusted according to the displacement of the predicted sparse 3D Gaussian in the corresponding time step to obtain the deformed sparse 3D Gaussian of the first target in the corresponding time step.
[0089] In order to distinguish it from the dense 3D Gaussian showing the appearance of the first object, the sparse 3D Gaussian as the sparse control point is represented as C i :(p i ,r i ,s,σ i ,sh i ), where p i represents the center position of the sparse 3D Gaussian, r i represents the orientation of the sparse 3D Gaussian, s represents the radius of the sparse 3D Gaussian, σ i Indicates the opacity of the sparse 3D Gaussian, sh i Represents the spherical harmonic coefficients of the sparse 3D Gaussian. Since the sparse 3D Gaussian is a sphere with a radius of S, in the first stage of optimization, the rotation of the sparse 3D Gaussian (i.e., the change in orientation) can be ignored, and the focus is on the displacement of the center position.
[0090] use Represents a sparse 3D Gaussian C i The displacement in time step t, then the center position of the deformed sparse 3D Gaussian of the first target at the corresponding time step is:
[0091] Step S2023: Render an image of the deformed sparse 3D Gaussian at the first viewing angle according to the first reference image to obtain a first reconstructed image.
[0092] In this embodiment, the first viewing angle is used to represent the viewing angle corresponding to the reference image. By rendering a two-dimensional image of the deformed sparse 3D Gaussian at the first viewing angle, a reconstructed image of the first viewing angle of the first target reconstructed based on the sparse control points and the deformable network at the corresponding time step can be obtained, which is used as the reconstructed image of the first reference image at the corresponding time step, and is recorded as the first reconstructed image.
[0093] Step S2024: Calculate a first reconstruction loss according to the first reference image and the first reconstructed image.
[0094] In this step, a first reconstruction loss is calculated according to the difference between the first reference image and the first reconstructed image.
[0095] Exemplarily, according to the first reference image and the first reconstructed image, the 2-norm between the matrices of the first reference image and the first reconstructed image can be calculated by the following formula (1) to obtain the first reconstruction loss:
[0096]
[0097] Among them, L ref represents the first reconstruction loss, I t represents the first reconstructed image, represents the f-th frame reference image in the reference video, and t represents the f-th frame reference image The corresponding time step.
[0098] Optionally, this step can also be implemented by using other methods of calculating the reconstruction loss based on the reference image and the reconstructed image, such as calculating the 1-norm of the reference image and the reconstructed image to obtain the reconstruction loss. This embodiment does not specifically limit the method of calculating the reconstruction loss in this step.
[0099] Step S2025: Optimize the sparse 3D Gaussian and deformable network according to the first reconstruction loss.
[0100] In this step, based on the first reconstruction loss, the parameters of the sparse 3D Gaussian and the deformation network are updated by back propagation.
[0101] In an optional embodiment, the process of optimizing the sparse 3D Gaussian and deformable network of the first target based on the reference video of the first target in the aforementioned step S202 also includes: calculating the first mask loss according to the mask map of the first reference image and the mask map of the first reconstructed image, and optimizing the sparse 3D Gaussian and deformable network according to the first mask loss.
[0102] Exemplarily, according to the mask map of the first reference image and the mask map of the first reconstructed image, the first mask loss can be calculated by formula (2):
[0103]
[0104] Among them, L mask represents the first mask loss, M t represents the mask map of the first reconstructed image, Represents the reference image of the fth frame in the reference video The mask map of , t represents the reference image of the fth frame The corresponding time step.
[0105] Optionally, this step can also be implemented by using other methods of calculating the mask loss based on the mask maps of both the reference image and the reconstructed image, for example, calculating the 1-norm of the mask map of the reference image and the mask map of the reconstructed image to obtain the mask loss. This embodiment does not specifically limit the method of calculating the reconstruction loss in this step.
[0106] Further, according to the weighted sum of the first reconstruction loss and the first mask loss, the parameters of the sparse 3D Gaussian and the parameters of the deformable network are optimized through back propagation. The weight coefficients of the first reconstruction loss and the first mask loss can be configured and adjusted based on experience, and are not specifically limited here.
[0107] In an optional embodiment, the process of optimizing the sparse 3D Gaussian and deformable network of the first target based on the reference video of the first target in the aforementioned step S202 also includes: rendering an image of the deformed sparse 3D Gaussian at the second perspective to obtain a second reconstructed image; calculating the first fractional distillation sampling loss according to the first reference image, the second reconstructed image and the relative angle transformation matrix between the second perspective and the first perspective; optimizing the sparse 3D Gaussian and deformable network according to the first fractional distillation sampling loss. The relative angle transformation matrix includes a displacement matrix and a rotation matrix.
[0108] The second viewing angle is a viewing angle obtained by random sampling and different from the first viewing angle. A plurality of viewing angles are randomly sampled within a given pitch angle range and a yaw angle range (usually -180 to 180) as the second viewing angle. The pitch angle range can be configured according to information of the actual application scenario, and the yaw angle range is usually -180 to 180, and can also be customized, which is not specifically limited in this embodiment.
[0109] Specifically, the perspective diffusion model is used to predict the noise in the noisy image of the second reconstructed image based on the first reference image of the first perspective and the camera extrinsic parameters (displacement matrix and rotation matrix), and the fractional distillation sampling loss is determined based on the difference between the predicted noise and the added original noise.
[0110] Among them, in the application of dynamic 3D object modeling, the perspective diffusion model can be implemented using the Zero123 model or other models with type functions, and in the application of dynamic 3D scene modeling, the perspective diffusion model can be implemented using a scene-based diffusion model with type functions, which is not specifically limited here. Zero123 is a new perspective image generation model based on a diffusion model.
[0111] Exemplarily, according to the first reference image and the second reconstructed image, the first fractional distillation sampling loss can be calculated using formula (3):
[0112]
[0113] in, represents the viewing diffusion model, z t′ represents the noisy image obtained by adding random noise ∈ to the second reconstructed image, t′ represents the random sampling time step of the random noise ∈ added to the second reconstructed image, and ω(t′) is a weighting function related to t′. r represents the first reference image. R and T are the displacement matrix and rotation matrix of the second perspective relative to the first perspective, respectively. Represents the predicted noise, that is, predicting the noisy image z through the extended model t′ The prediction result of the noise in . x represents the rendered image. * represents the stop gradient operation.
[0114] Furthermore, according to the first fractional distillation sampling loss, the parameters of the sparse 3D Gaussian and the parameters of the deformable network are optimized by back-propagation.
[0115] Optionally, the sparse 3D Gaussian and deformable network is optimized by back propagation according to the weighted sum of the first reconstruction loss and the first fractional distillation sampling loss. The weight coefficients of the first reconstruction loss and the first fractional distillation sampling loss can be configured and adjusted based on experience and are not specifically limited here.
[0116] Optionally, the sparse 3D Gaussian and deformable network is optimized by back propagation according to the weighted sum of the first reconstruction loss, the first fractional distillation sampling loss and the first mask loss. The weight coefficients of the first reconstruction loss, the first mask loss and the first fractional distillation sampling loss can be configured and adjusted based on experience, and are not specifically limited here.
[0117] Exemplarily, formula (4) is used to obtain the comprehensive loss by weighted summing the first reconstruction loss, the first fractional distillation sampling loss and the first mask loss:
[0118] L=λ ref L ref +λ mask L mask +λSDS L SDS (4)
[0119] Among them, λ ref , mask and λ SDS They are the first reconstruction loss L ref , the first mask loss L mask and the first fraction distillation sampling loss L SDS The weight coefficient, λ ref , mask and λ SDS The value of can be configured and adjusted based on experience, for example, ref , mask and λ SDS The default values are set to 5000.0, 500.0 and 1.0 respectively, and no specific limitation is given here.
[0120] The method of this embodiment, in the first stage optimization, by taking at least one frame image in the reference video as the first reference image, inputting the time step corresponding to the first reference image and the center position of the sparse 3D Gaussian of the first target into the deformation network, predicting the displacement of the sparse 3D Gaussian in the corresponding time step through the deformation network, determining the deformed sparse 3D Gaussian of the first target in the corresponding time step according to the sparse 3D Gaussian and the displacement of the sparse 3D Gaussian in the corresponding time step, optimizing the sparse 3D Gaussian and the deformation network according to the first reconstructed image and the first reference image rendered to the first perspective according to the deformed sparse 3D Gaussian, and obtaining the action and rough shape of the first target that is rationalized and corresponds to the reference video. The sparse 3D Gaussian with the rough shape of the first target is used as the sparse control point for the second stage optimization, which can inherit the shape and movement of the sparse control point in the first stage.
[0121] In some embodiments, in the second stage, the optimized sparse control points and parameters of the deformation network of the first target can be fixed, and only the dense 3D Gaussian of the first target can be optimized, which can improve the effect of dynamic 3D target modeling and have better modeling effect.
[0122] In other embodiments, in the second stage, based on the reference video of the first target, the optimized sparse control points and the deformation network of the first target, when optimizing the dense 3D Gaussian of the first target, the sparse control points and the deformation network can also be optimized together to optimize the position and movement of the sparse control points, while optimizing the texture details of the first target to better match the reference video, and further refine the motion and shape of the first target, which can effectively ensure the authenticity of the motion and shape of the dynamic 3D target finally learned, and achieve a more stable modeling effect.
[0123] Figure 5The flowchart of optimizing the dense 3D Gaussian of the first target based on the reference video of the first target, the sparse control points and the deformable network provided in this embodiment. In an optional embodiment, as Figure 5 As shown, in the aforementioned step S203, based on the reference video of the first target, the sparse control points and the deformable network after the first target is optimized, the dense 3D Gaussian of the first target is optimized, and the sparse control points and the deformable network after the first target is optimized are optimized, which can be specifically implemented by the following steps:
[0124] Step S2031: Use at least one frame image in the reference video as the second reference image, and based on the time step corresponding to the second reference image, the sparse control points after the optimization of the first target, and the deformation network, drive the dense 3D Gaussian after the optimization of the first target to deform, so as to obtain the deformed dense 3D Gaussian of the first target at the corresponding time step.
[0125] In this embodiment, at least one frame image can be randomly selected from the reference video as the second reference image to perform the second stage optimization. In order to distinguish the reference image used in the second stage optimization, the reference image sampled in the first stage is referred to as the first reference image, and the reference image sampled in the second stage is referred to as the second reference image. The time step corresponding to each frame image in the reference video is consistent with that in the first stage optimization, and will not be repeated here.
[0126] In the second stage of optimization, the sparse 3D Gaussian optimized in the first stage is used as an implicit sparse control point. The optimized sparse control point can be expressed as C i :(μ i ,o i ), where o i is the control radius of the sparse control point, initialized with the size s of the sparse 3D Gaussian after the first stage optimization, μ i is the center position of the sparse control point, which is also the center position of the sparse 3D Gaussian after the first stage optimization.
[0127] For any second reference image, the time step corresponding to the second reference image and the sparse control points obtained in the first stage are input into the deformation network obtained in the first stage, and the deformation network is used to predict the deformation information of the sparse control points in the corresponding time step. The deformation information here includes displacement and rotation information, where the displacement in the deformation information is represented by the displacement matrix T t Indicated by the rotation matrix R t express.
[0128] Furthermore, based on a linear blend skinning (LBS) method, a deformed dense 3D Gaussian of the first target at a corresponding time step is determined according to deformation information of sparse control points adjacent to the dense 3D Gaussian of the first target in a corresponding time step.
[0129] Specifically, use G j :(μ j ,q j ,s j ,σ j ,sh j ) represents any initialized dense 3D Gaussian of the first target, where μ j ,q j ,s j ,σ j ,sh j are the center position, orientation, size, opacity and spherical harmonic coefficients of the dense 3D Gaussian. j The distance between them is chosen to be the same as the dense 3D Gaussian G j The neighboring K sparse control points serve as the dense 3D Gaussian G j Adjacent sparse control points to drive the dense 3D Gaussian G j Make a deformation and determine the deformation dense 3D Gaussian of the first target at the corresponding time step.
[0130] Among them, choose the dense 3D Gaussian C j The K sparse control points of the nearest neighbors can be determined by the K-NearestNeighbor (KNN) algorithm, where K is a positive integer, which can be configured and adjusted according to the actual application scenario and experience. For example, K can be set to 4, and the dense 3D Gaussian G is driven based on the 4 sparse control points of the nearest neighbors. j The present embodiment does not specifically limit K.
[0131] Dense 3D Gaussian G j The deformed dense 3D Gaussian at the corresponding time step t It can be expressed as The deformed dense 3D Gaussian Central location The deformation dense 3D Gaussian is determined according to the center position of the K nearest neighbor sparse control points and the deformation information in the corresponding time step t. Direction It is determined based on the orientation of the K nearest neighbor sparse control points and the deformation information in the corresponding time step t.
[0132] Specifically, the deformed dense 3D Gaussian Central location It can be determined by the following formula (5):
[0133]
[0134] Among them, Mj Represents a dense 3D Gaussian G j The set of K nearest neighbor sparse control points, and Respectively represent M j Any sparse control point M in jk The displacement matrix and rotation matrix in the deformation information, p k Denotes any sparse control point M jk The center position of μ j represents a dense 3D Gaussian G j The center position before deformation. jk For weighted information, according to the dense 3D Gaussian G j The center position of and its adjacent sparse control points M jk The distance between jk and sparse control points M jk The control radius o k Sure.
[0135] Deformed Dense 3D Gaussian Direction It can be determined by the following formula (6):
[0136]
[0137] in, is a dense 3D Gaussian G j The neighboring sparse control points M jk The orientation after deformation at time step t, q j is a dense 3D Gaussian G j The orientation before deformation. jk For the meaning of , see formula (5).
[0138] Step S2032: Render an image of the deformable dense 3D Gaussian at the first viewing angle according to the first viewing angle corresponding to the second reference image to obtain a third reconstructed image of the second reference image.
[0139] In this embodiment, the first perspective represents the perspective corresponding to the reference image. By rendering a two-dimensional image of the deformed dense 3D Gaussian at the first perspective, a reconstructed image of the first target at the corresponding time step after the dense 3D Gaussian is deformed based on the sparse control points and the deformation network is obtained, and the image is used as the reconstructed image of the second reference image at the corresponding time step, which is recorded as the third reconstructed image.
[0140] Step S2033: Calculate a second reconstruction loss according to the second reference image and the third reconstructed image.
[0141] In this step, the second reconstruction loss is calculated according to the difference between the second reference image and the third reconstructed image. The second reconstruction loss in this step is obtained by calculating the 2-norm between the matrices of the second reference image and the third reconstructed image, or calculating the 1-norm between the matrices of the second reference image and the third reconstructed image. The specific calculation principle is consistent with the calculation principle of the first reconstruction loss in step S2024. For details, refer to the relevant contents of the aforementioned embodiment, which will not be repeated here.
[0142] Step S2034: optimize the dense 3D Gaussian of the first target according to the second reconstruction loss, and optimize the sparse control points and the deformable network after the optimization of the first target.
[0143] In this step, based on the second reconstruction loss, the parameters of the dense 3D Gaussian and the parameters of the sparse control points and deformable network optimized in the first stage are updated by back propagation.
[0144] In another optional embodiment, in this step, the sparse control points and parameters of the deformable network after the optimization of the first target may be fixed, and the dense 3D Gaussian of the first target may be optimized according to the second reconstruction loss.
[0145] In an optional embodiment, in the aforementioned step S203, based on the reference video of the first target, the sparse control points and the deformable network after the optimization of the first target, the dense 3D Gaussian of the first target is optimized, and the sparse control points and the deformable network after the optimization of the first target are optimized, and the second mask loss is calculated according to the mask map of the second reference image and the mask map of the third reconstructed image; the dense 3D Gaussian of the first target is optimized according to the second mask loss, and the sparse control points and the deformable network after the optimization of the first target are optimized.
[0146] Specifically, the second mask loss is obtained by calculating the 2-norm of the mask map of the second reference image and the mask map of the third reconstructed image, or calculating the 1-norm of the mask map of the second reference image and the mask map of the third reconstructed image. The specific calculation principle is consistent with the calculation principle of the aforementioned first mask loss. Please refer to the relevant content of the aforementioned embodiment for details, which will not be repeated here.
[0147] Furthermore, according to the weighted sum of the second reconstruction loss and the second mask loss, the parameters of the dense 3D Gaussian of the first target and the parameters of the sparse control points and the deformable network after the first stage optimization are optimized through back propagation. Among them, the weight coefficients of the second reconstruction loss and the second mask loss can be configured and adjusted according to experience, and are not specifically limited here.
[0148] In an optional embodiment, in the aforementioned step S203, based on the reference video of the first target, the sparse control points and the deformable network after the first target is optimized, the dense 3D Gaussian of the first target is optimized, and the sparse control points and the deformable network after the first target is optimized are optimized, and the following further includes:
[0149] Render an image of the deformed dense 3D Gaussian at the third perspective to obtain a fourth reconstructed image; calculate the second fractional distillation sampling loss based on the second reference image, the fourth reconstructed image, and the relative angle transformation matrix between the third perspective and the first perspective; optimize the dense 3D Gaussian of the first target based on the second fractional distillation sampling loss, and optimize the sparse control points and deformation network after the optimization of the first target.
[0150] The third perspective is a perspective obtained by random sampling and different from the first perspective. Multiple perspectives are randomly sampled within a given pitch angle range and yaw angle range (usually -180 to 180) as the second perspective. The pitch angle range can be configured according to information of the actual application scenario, and the yaw angle range is usually -180 to 180, which can also be customized, and is not specifically limited in this embodiment.
[0151] Specifically, the perspective diffusion model is used to predict the noise in the noisy image of the fourth reconstructed image based on the second reference image of the first perspective and the camera extrinsic parameters (displacement matrix and rotation matrix), and the fractional distillation sampling loss is determined based on the difference between the predicted noise and the added original noise.
[0152] In the application of dynamic 3D object modeling, the view diffusion model can be implemented using the Zero123 model or other models with type functions. In the application of dynamic 3D scene modeling, the view diffusion model can be implemented using a scene-based diffusion model with type functions, which is not specifically limited here. Zero123 is a new view image generation model based on an extended model.
[0153] The calculation principle of the second fractional distillation sampling loss is the same as the calculation principle of the first fractional distillation sampling loss. For details, please refer to the relevant content of the aforementioned embodiment, which will not be repeated here.
[0154] Furthermore, according to the second fractional distillation sampling loss, the parameters of the dense 3D Gaussian and the parameters of the sparse control points and deformable network after the first stage optimization are optimized by back propagation.
[0155] Optionally, the parameters of the dense 3D Gaussian of the first target and the parameters of the sparse control points and the deformable network after the first stage optimization can be optimized by back propagation according to the second reconstruction loss and the second fractional distillation sampling loss. The weight coefficients of the second reconstruction loss and the second fractional distillation sampling loss can be configured and adjusted based on experience, and are not specifically limited here.
[0156] Optionally, the parameters of the dense 3D Gaussian of the first target and the parameters of the sparse control points and the deformable network after the first stage optimization can be optimized by back propagation according to the second reconstruction loss, the second fractional distillation sampling loss and the second mask loss. Among them, the weight coefficients of the second reconstruction loss, the second mask loss and the second fractional distillation sampling loss can be configured and adjusted according to experience, and are not specifically limited here.
[0157] In an optional embodiment, even with a good initialization of dense 3D Gaussians, the shape and motion of dynamic 3D objects are still prone to degradation in the later stages of training. The main reason is that when using fractional distillation sampling (SDS) to extract prior knowledge of new perspectives from a perspective diffusion model (such as Zero123), the larger noise time step makes SDS biased towards ensuring the rationality of the overall shape. On the contrary, when the time step is small, SDS tends to focus more on optimizing texture, while its ability to maintain shape is weakened. In order to refine the texture without degrading the motion and shape, Gaussian alignment (GA) loss is added as an additional constraint in this embodiment.
[0158] The second stage optimization in the aforementioned step S203 also includes:
[0159] The Gaussian alignment loss is calculated based on the center position of the deformed sparse 3D Gaussian of the first target at each time step in the first stage and the center position of the deformed sparse control point of the first target at the corresponding time step in the second stage; the sparse control points and deformation network after the optimization of the first target are optimized based on the Gaussian alignment loss.
[0160] Specifically, the center position of the deformed sparse 3D Gaussian of the first target at each time step in the first stage (denoted as ), and the center position of the deformed sparse control point of the first target at the corresponding time step in the second stage (denoted as p t ) to calculate the Gaussian alignment loss.
[0161] For example, the following formula (7) can be used to calculate the Gaussian alignment loss:
[0162]
[0163] Among them, L GA represents the Gaussian alignment loss, p t represents the center position of the deformed sparse control points of the first target at time step t in the second stage, that is, the center position of the sparse control points of the first target after deformation at time step t in the second stage, It represents the center position of the deformed sparse 3D Gaussian of the first target in the first stage at time step t, that is, the center position of the sparse control point of the first target in the first stage after deformation at time step t.
[0164] Furthermore, according to the second reconstruction loss, the second mask loss, the second fractional distillation sampling loss and the Gaussian alignment loss, the parameters of the dense 3D Gaussian of the first target and the parameters of the sparse control points and the deformable network after the first stage optimization are optimized by back propagation.
[0165] For example, formula (8) can be used to weight the second reconstruction loss, the second mask loss, the second fractional distillation sampling loss and the Gaussian alignment loss to obtain the comprehensive loss of the second stage:
[0166] L′=λ′ ref L′ ref +λ′ mask L′ mask +λ′ SDS L′ SDS +λ′ GA L′ GA (8)
[0167] Among them, λ′ ref ,λ′ mask ,λ′ SDS and are the second reconstruction loss L′ ref , the second mask loss L′ mask , the second fractional distillation sampling loss L′ SDS and Gaussian alignment loss L′ GA The weight coefficient, λ′ ref ,λ′ mask ,λ′ SDS and λ′ GA The value of can be configured and adjusted based on experience, for example, ref ,λ′ mask ,λ′ SDS and λ′ GA The default values can be set to 5000.0, 500.0, 1.0 and 10000.0 respectively, and are not specifically limited here.
[0168] In an optional embodiment, during the second stage optimization process, a second reconstruction loss, a second fractional distillation sampling loss and a Gaussian alignment loss can be calculated, and based on the second reconstruction loss, the second fractional distillation sampling loss and the Gaussian alignment loss, the parameters of the dense 3D Gaussian of the first target and the parameters of the sparse control points and the deformable network after the first stage optimization are optimized by back propagation.
[0169] In the method of this embodiment, in the second stage optimization, multiple 3D Gaussians are initialized within the sphere of the optimized sparse control points of the first target to obtain a dense 3D Gaussian of the initialized first target. The dense 3D Gaussian initialized in this way effectively inherits the shape and motion modeled in the first stage; further, at least one frame of the reference video is used as the second reference image, based on the time step corresponding to the second reference image, the sparse control points of the first target after optimization and the deformation network, the dense 3D Gaussian optimized by the first target is driven to deform, and the deformed dense 3D Gaussian of the first target at the corresponding time step is obtained, and the deformed dense 3D Gaussian is rendered. D Gaussian image at the first perspective is used to obtain the third reconstructed image of the second reference image; according to the second reference image and the third reconstructed image, the reconstruction loss, mask loss, SDS loss and Gaussian alignment loss are calculated, and the dense 3D Gaussian of the first target is optimized by back propagation to achieve a refined simulation of the texture of the dynamic 3D target to better match the reference video, and the sparse control points and deformation network after the optimization of the first target are optimized in two stages, which can further refine the motion and shape of the first target, and can effectively ensure the authenticity of the motion and shape of the dynamic 3D target finally learned, and achieve a more stable modeling effect.
[0170] Figure 6 The overall framework diagram of the two stages of dynamic 3D object modeling provided by an exemplary embodiment of the present application. Figure 6 As shown in the figure, the process of dynamic 3D target modeling includes two stages. In the first stage, the reference image corresponding to time step t and the center position of the sparse 3D Gaussian are input into the deformation network MLP, and the displacement T of the sparse 3D Gaussian in time step t is obtained by the deformation network MLP prediction. t , based on sparse 3D Gaussian and displacement T t The deformed sparse 3D Gaussian can be determined. Further, based on the deformed sparse 3D Gaussian and the reference image, the reconstruction loss L of the reference image corresponding to the first perspective is calculated. ref and the mask loss L mask , and combined with the perspective diffusion model to calculate the random new perspective fractional distillation sampling loss L SDS . Further, according to the reconstruction loss L ref , mask loss L mask and fractional distillation sampling loss L SDS Optimizing sparse 3D Gaussian and deformable network MLPs.
[0171] like Figure 6As shown in the figure, in the second stage, the sparse 3D Gaussian optimized in the first stage is used as an implicit sparse control point, and the dense 3D Gaussian is deformed based on the sparse control point and the deformation network. Specifically, the reference image corresponding to time step t and the sparse control point are input into the deformation network MLP, and the deformation information of the sparse control point in time step t (including the displacement matrix T t and the rotation matrix R t ); Based on the linear skinning LBS method, the dense 3D Gaussian deformation is driven based on the sparse control points and deformation information, and the deformed dense 3D Gaussian is determined by the deformation information of the nearest K sparse control points. Furthermore, based on the deformed dense 3D Gaussian and the reference image, the reconstruction loss L′ of the reference image corresponding to the first perspective is calculated ref and mask loss L′ mask , combined with the perspective diffusion model to calculate the random new perspective fractional distillation sampling loss L′ SDS , and calculate the Gaussian alignment loss L′ based on the position of the sparse control point before optimization and the current position GA . Further, according to the reconstruction loss L′ ref , mask loss L′ mask , fractional distillation sampling loss L′ SDS and Gaussian alignment loss L′ GA By optimizing the parameters of dense 3D Gaussian and further optimizing the parameters of sparse control points and deformable network MLP, a more stable effect has been achieved.
[0172] Figure 6 In the figure, the flame mark indicates the part that needs to be optimized, and the snowflake mark indicates the part that does not need to be optimized (i.e., the parameters are frozen). In the first stage, the parameters of the view diffusion model are frozen, and the parameters of the sparse 3D Gaussian and the deformable network MLP are optimized. In the second stage, the parameters of the view diffusion model are frozen, and the parameters of the dense 3D Gaussian, the sparse control points, and the deformable network MLP are optimized. In some embodiments, the second stage can also freeze the parameters of the sparse control points and the deformable network MLP, and optimize the parameters of the dense 3D Gaussian.
[0173] The method of this embodiment, in the first stage, initializes sparse control points to displayed 3D Gaussians, optimizes the parameters of these sparse 3D Gaussians and the deformation network to obtain the reasonable movement and shape of the dynamic 3D target corresponding to the reference video, and in the second stage, optimizes the dense 3D Gaussians to achieve a refined simulation of the texture of the dynamic 3D target; by separately modeling the motion and appearance of the dynamic 3D target, the ambiguity of the motion and appearance is reduced, and a dynamic 3D target (i.e., a 4D target) that fits the reference video very well can be generated, which achieves better results than current similar solutions in terms of the degree of conformity with the reference perspective, the spatiotemporal consistency of the dynamic object, and the authenticity of the movement, thereby improving the accuracy and effect of dynamic 3D target modeling.
[0174] The present application also provides a method for dynamic 3D target modeling of motion migration, which can model a dynamic 3D model of a second target based on a given reference video of a first target and a description text of the second target. Specifically, after dynamic 3D modeling of the first target is performed to obtain the optimized sparse control points, deformation network and dense 3D Gaussian of the first target, the optimized sparse 3D Gaussian and deformation network of the first target are migrated to the second target as the sparse control points and deformation network of the second target; the dense 3D Gaussian of the second target is initialized, and the dense 3D Gaussian of the second target is optimized based on the optimized dense 3D Gaussian of the first target, the description text of the second target, the sparse control points and deformation network of the second target. The sparse control points, deformation network and optimized dense 3D Gaussian of the second target are used to represent the dynamic 3D model of the second target.
[0175] The first target may be an object or scene, and the second target may be an object or scene similar to the first target, which may be specified by the user. For example, the first target may be a cat, and the reference video may be a video of a cat running; the second target may be a lion, a cheetah, etc. The method of this embodiment may model a dynamic 3D model of the second target that matches the reference video, and a video of the second target (such as a lion, a cheetah, etc.) running may be generated based on the modeled dynamic 3D model of the second target.
[0176] For example, Figure 7 The effect diagram of the dynamic 3D target modeling and motion migration application provided in this embodiment is as follows: Figure 7 As shown, taking the first target as "cat", the given reference video is a video of a cat walking, and the second target as "lion" and "cheetah" as examples, according to the reference image of the "cat" in the reference video, dynamic 3D target modeling is used to obtain the dense 3D Gaussian and sparse control points of the "cat" that match the reference video. Figure 7The figure shows the rendering effect of the dense 3D Gaussian and sparse control points of the "cat" at two different perspectives. In the motion transfer application, the sparse control points of the "cat" are transferred to the "lion" and "cheetah". Through the dynamic 3D target modeling of motion transfer, the dense 3D Gaussian of the "lion" and "cheetah" that matches the motion of the cat in the reference video is obtained. Figure 7 The figure shows the rendering effect of the dense 3D Gaussian of the "lion" and "cheetah" modeled by the transfer application under two corresponding perspectives. Figure 7 It can be seen from the display effect that the dynamic 3D models of "lion" and "cheetah" modeled by the motion transfer application and the dynamic 3D model of "cat" modeled by dynamic 3D modeling all match the movements of the "cat" in the reference video.
[0177] Figure 8 The flowchart of a dynamic 3D target modeling method based on motion migration provided by an exemplary embodiment of the present application. The execution subject of this embodiment is the server in the aforementioned system architecture. This embodiment provides a dynamic 3D target modeling method based on motion migration, such as Figure 8 As shown, the specific steps of this method are as follows:
[0178] Step S801: Obtain a reference video containing a first target and a description text of a second target.
[0179] In this embodiment, the first target may be an object or a scene, and the second target may be an object or a scene similar to the first target, which may be specifically specified by a user. The reference video may be a video of the first target captured by a camera at a fixed camera position, including multiple frames of images of the first target at multiple different times. For example, the reference video of the first target may be obtained by shooting with a fixed camera perpendicular to the first target.
[0180] The description text of the second target is used to briefly describe the characteristics such as the type of the second target, and the description text of the second target is used to model the appearance of the second target.
[0181] For example, the first target may be a cat, and the reference video may be a video of a cat running; the second target may be a lion, a cheetah, etc., and the description text of the second target may be "a lion", "a cheetah". Through the motion transfer application, a dynamic 3D model of the second target matching the reference video may be modeled, so that the motion of the modeled second target matches the reference video, and a video of the second target (such as a lion, a cheetah, etc.) running may be generated based on the modeled dynamic 3D model of the second target.
[0182] For another example, the first target may be character A, and the reference video may be an action video of character A. The second target may be a game character B or character C, and the description text of the second target may be the name of game character B or the name of character C. Through the action transfer application, a dynamic 3D model of game character B or character C that matches the action of character A in the reference video may be modeled. An action video of game character B or character C may be generated based on the modeled dynamic 3D model of game character B or character C.
[0183] In this embodiment, the dynamic 3D target modeling based on motion migration includes three stages: the first two stages are based on the reference video of the first target, and the dynamic 3D modeling (i.e., 4D modeling) of the first target is performed to obtain the optimized sparse control points, deformation network, and dense 3D Gaussian of the first target. The specific implementation principle and technical effects refer to the relevant contents of the aforementioned embodiments, which will not be repeated here.
[0184] In the third stage, the sparse 3D Gaussian and deformation network optimized for the first target are transferred to the second target as the sparse control points and deformation network of the second target; the dense 3D Gaussian of the second target is initialized, and the dense 3D Gaussian of the second target is optimized based on the dense 3D Gaussian optimized for the first target, the description text of the second target, the sparse control points and deformation network of the second target. In this stage, the motion of the first target is transferred to the second target, and the texture of the second target is finely simulated by optimizing the dense 3D Gaussian of the second target. The motion of these dense 3D Gaussians is controlled by the sparse control points obtained in the first stage.
[0185] In another optional embodiment, the dynamic 3D target modeling based on motion migration can also include two stages. In the first stage, the sparse control points of the first target are initialized as sparse 3D Gaussians, the initialized deformation network is obtained, and the sparse 3D Gaussians and deformation networks of the first target are optimized based on the reference video of the first target to obtain the optimized sparse control points and deformation network of the first target. In the second stage, the optimized sparse control points and deformation network of the first target obtained in the first stage are migrated to the second target as the sparse control points and deformation network of the second target; the dense 3D Gaussians of the second target are initialized, and the dense 3D Gaussians of the second target are optimized based on the optimized sparse control points of the first target, the description text of the second target, the sparse control points and deformation network of the second target. In this stage, the motion of the first target is migrated to the second target, and the refined simulation of the texture of the second target is achieved by optimizing the dense 3D Gaussians of the second target, and the motion of these dense 3D Gaussians is controlled by the sparse control points obtained in the first stage.
[0186] Step S802: In the first stage, the sparse control points of the first target are initialized to sparse 3D Gaussians, an initialized deformable network is obtained, and based on the reference video of the first target, the sparse 3D Gaussians and the deformable network of the first target are optimized to obtain the optimized sparse control points and deformable network of the first target.
[0187] The implementation principle and technical effect of this step refer to the relevant contents of the aforementioned steps S202-S203, which will not be repeated in this embodiment.
[0188] Step S803, in the second stage, initialize the dense 3D Gaussian of the first target, optimize the dense 3D Gaussian of the first target based on the reference video of the first target, the sparse control points and the deformation network after the optimization of the first target, and obtain the dense 3D Gaussian after the optimization of the first target.
[0189] The implementation principle and technical effect of this step refer to the relevant content of the aforementioned step S204, which will not be repeated in this embodiment.
[0190] Step S804: Migrate the optimized sparse 3D Gaussian and deformable network of the first target to the second target as the sparse control points and deformable network of the second target.
[0191] In the third stage, the sparse 3D Gaussian and deformable network optimized for the first target are transferred to the second target as the sparse control points and deformable network of the second target.
[0192] Step S805, initialize the dense 3D Gaussian of the second target, and optimize the dense 3D Gaussian of the second target based on the optimized dense 3D Gaussian of the first target, the description text of the second target, the sparse control points and the deformation network of the second target; the sparse control points, deformation network and optimized dense 3D Gaussian of the second target are used to represent the dynamic 3D model of the second target.
[0193] This step performs the second stage of optimization, fixes the sparse control points and deformation network parameters of the second target, optimizes the dense 3D Gaussian of the second target based on the optimized dense 3D Gaussian of the first target, the description text of the second target, the sparse control points and deformation network of the second target, to obtain the texture of the second target.
[0194] Before the third stage of optimization, the innovative adaptive Gaussian initialization method of this case is first used to initialize the dense 3D Gaussian of the second target.
[0195] In a preferred embodiment, the initial dense 3D Gaussian of the second target can be initialized by an adaptive Gaussian initialization method: multiple 3D Gaussians are initialized within the sphere of the sparse control points of the second target as the dense 3D Gaussian initialized for the second target. The dense 3D Gaussian initialized in this way effectively inherits the shape and motion modeled in the first stage, so that the dense 3D Gaussian is distributed near the target surface, which is beneficial to subsequent optimization. For the specific implementation principle and technical effects, please refer to the relevant contents of the aforementioned embodiments, which will not be repeated in this embodiment.
[0196] In an optional embodiment, the initial dense 3D Gaussian of the second target can be obtained by uniformly initializing a second number of 3D Gaussians in a sphere in the reference space. The dense 3D Gaussian initialized in this way tends to increase the target thickness, and may cause problems such as texture reduction and position drift.
[0197] After initializing the dense 3D Gaussian of the second target, in the third stage, based on the optimized dense 3D Gaussian of the first target, the description text of the second target, the sparse control points and the deformation network of the second target, the dense 3D Gaussian of the second target is optimized to achieve fine modeling of the texture of the second target and obtain the appearance of the second target.
[0198] The method of this embodiment transfers the sparse 3D Gaussian and deformation network after optimization of the first target to the second target as the sparse control points and deformation network of the second target, drives the deformation of the dense 3D Gaussian through the sparse control points and deformation network, and optimizes the dense 3D Gaussian of the second target in combination with the dense 3D Gaussian after optimization of the first target and the description text of the second target, thereby realizing the modeling of the texture of the second target, and can realize dynamic 3D modeling of different targets that inherit the same motion, and the appearance of different targets is controlled by the description text. The second target can be dynamically 3D modeled based on the reference video of the first target, and the dynamic 3D model of the second target is very consistent with the reference video, and a good modeling effect is achieved in three aspects: the degree of conformity with the reference perspective, the spatiotemporal consistency of the dynamic object, and the authenticity of the action.
[0199] In this embodiment, after obtaining the sparse control points, deformation network and dense 3D Gaussian of the second target, based on any given moment, the sparse control points and deformation network of the second target can be used to drive the dense 3D Gaussian of the second target to obtain the deformed dense 3D Gaussian of the second target at a given moment. By rendering the deformed dense 3D Gaussian at a given moment according to any perspective, the appearance image of the dynamic 3D model of the second target at any perspective at any given moment can be obtained.
[0200] In an optional embodiment, after obtaining the sparse control points, deformation network and optimized dense 3D Gaussian of the second target, the server can generate a dynamic 3D model of the second target based on at least one configured time step, and the dynamic 3D model of the second target includes the 3D model of the second target at each time step.
[0201] Specifically, the server uses the sparse control points and deformation network of the second target to drive the optimized dense 3D Gaussian of the second target to deform according to the configured time step sequence, obtains the deformed dense 3D Gaussian of the second target at each time step, and obtains the dynamic 3D model of the second target. Furthermore, the image of the deformed dense 3D Gaussian of the second target at each time step at any viewing angle can be rendered.
[0202] Among them, based on any time step, the sparse control points and deformation network of the second target are used to drive the optimized dense 3D Gaussian of the second target to make a deformation, and the deformed dense 3D Gaussian of the second target at the corresponding time step is obtained. For the implementation principle and technical effect of the deformed dense 3D Gaussian of the second target at the corresponding time step, refer to the relevant content of using the sparse control points and deformation network after the optimization of the first target to drive the optimized dense 3D Gaussian of the first target to make a deformation, and the deformed dense 3D Gaussian of the first target at the corresponding time step is obtained in the aforementioned embodiment. This embodiment will not be repeated here.
[0203] Furthermore, the server can also render the deformation dense 3D Gaussian of the second target at each time step, generate a display video of the dynamic 3D model of the second target, and output the display video of the dynamic 3D model of the second target to demonstrate the effect of generating the dynamic 3D model of the second target.
[0204] Specifically, the server renders images of the second target at multiple perspectives of the deformed dense 3D Gaussian at each time step, and obtains images of the second target at multiple perspectives at each time step; based on the images of the second target at multiple perspectives at each time step, generates a display video of the dynamic 3D model of the second target; and outputs a display video of the dynamic 3D model of the second target. The multiple perspectives displayed in the display video can be configured according to the needs of the actual application scenario, or can be customized by the user through the front-end device, which is not specifically limited here.
[0205] Fig. 9 A flowchart of optimizing a dense 3D Gaussian of a second target based on a dense 3D Gaussian optimized for a first target, a description text of a second target, sparse control points, and a deformable network for optimizing a dense 3D Gaussian of a second target is provided for an exemplary embodiment of the present application. In an optional embodiment, as Fig. 9 As shown, in the aforementioned step S805, based on the optimized dense 3D Gaussian of the first target, the description text of the second target, the sparse control points and the deformation network of the second target, the dense 3D Gaussian of the second target is optimized, and the specific implementation steps are as follows:
[0206] Step S901: According to at least one time step, the sparse control points of the second target and the deformation network, the dense 3D Gaussian of the second target and the dense 3D Gaussian of the first target are driven to deform, so as to obtain the deformed dense 3D Gaussian of the second target at each time step and the deformed dense 3D Gaussian of the first target at each time step.
[0207] Among them, in the motion migration application, the time step used for optimization in the third stage can be randomly sampled from the time step corresponding to each frame image in the reference video of the first stage; or it can be unrelated to the reference video of the first stage and obtained by random sampling or sampling based on a pre-configured rule, which is not specifically limited in this embodiment.
[0208] In the third stage of optimization, the sparse 3D Gaussian after the optimization of the first target is transferred to the second target as the implicit sparse control point of the second target. The sparse control point can be expressed as C i :(μ i ,o i ), where o i is the control radius of the sparse control point, initialized with the size s of the sparse 3D Gaussian after the first stage optimization, μ i is the center position of the sparse control point, that is, the center position of the sparse 3D Gaussian after the first stage optimization. The deformable network after the first stage optimization is used as the deformable network of the second target.
[0209] The time step and the sparse control points of the second target are input into the deformation network of the second target, and the deformation network is used to predict the deformation information of the sparse control points in the corresponding time step. The deformation information here includes displacement and rotation information, where the displacement in the deformation information is represented by the displacement matrix T t Indicated by the rotation matrix R t express.
[0210] Furthermore, based on a linear skinning (LBS) method, a deformed dense 3D Gaussian of the second target at a corresponding time step is determined according to deformation information of sparse control points adjacent to the dense 3D Gaussian of the second target in a corresponding time step.
[0211] Specifically, use G j :(μ j ,q j ,s j ,σ j ,sh j ) represents any initialized dense 3D Gaussian of the second target, where μ j ,q j ,s j ,σ j ,sh jare the center position, orientation, size, opacity and spherical harmonic coefficients of the dense 3D Gaussian. j The distance between them is chosen to be the same as the dense 3D Gaussian G j The neighboring K sparse control points serve as the dense 3D Gaussian G j Adjacent sparse control points to drive the dense 3D Gaussian G j Make a deformation and determine the deformation dense 3D Gaussian of the second target at the corresponding time step.
[0212] Among them, choose the dense 3D Gaussian C j The K sparse control points of the nearest neighbors can be determined by the K nearest neighbor (KNN) algorithm, where K is a positive integer, which can be configured and adjusted according to the actual application scenario and experience. For example, K can be set to 4, and the dense 3D Gaussian G is driven based on the 4 sparse control points of the nearest neighbors. j The present embodiment does not specifically limit K.
[0213] Dense 3D Gaussian G j The deformed dense 3D Gaussian at the corresponding time step t It can be expressed as The deformed dense 3D Gaussian Central location The deformation dense 3D Gaussian is determined according to the center position of the K nearest neighbor sparse control points and the deformation information in the corresponding time step t. Direction It is determined based on the orientation of the K nearest sparse control points and the deformation information in the corresponding time step t. Specifically, the deformation dense 3D Gaussian of the second target Central location and orientation For the determination principle of , please refer to the relevant contents of formulas (5) and (6), which will not be repeated here.
[0214] Step S902: Render the deformed dense 3D Gaussian image of the second target at each time step at the sixth viewing angle to obtain the seventh reconstructed image at each time step.
[0215] The sixth viewing angle is a randomly sampled viewing angle, which is obtained by randomly sampling multiple viewing angles within a given pitch angle range and a yaw angle range (usually -180 to 180) as the sixth viewing angle. The pitch angle range can be configured according to information of the actual application scenario, and the yaw angle range is usually -180 to 180, which can also be customized, and is not specifically limited in this embodiment.
[0216] By rendering the two-dimensional image of the deformed dense 3D Gaussian of the second target at each time step at the sixth perspective, a reconstructed image of the second target at the sixth perspective at the corresponding time step after the dense 3D Gaussian is deformed based on the sparse control points and the deformation network can be obtained, which is recorded as the seventh reconstructed image.
[0217] Step S903: Render the depth information of the deformed dense 3D Gaussian of the first target at each time step at the sixth viewing angle.
[0218] In this step, the depth information of the first target at each time step is rendered, and the depth information of the first target at the sixth perspective at the corresponding time step is obtained. The depth information of a certain perspective can be obtained based on dense 3D Gaussian rendering using existing methods, which will not be repeated here.
[0219] Among them, rendering the depth information of the deformable dense 3D Gaussian at any perspective can be obtained through the 3DGS method: using alpha blending rendering to obtain the depth information of pixel u: Where i refers to any 3D Gaussian, depth(u) represents the depth information of pixel u, and v i represents the viewing angle, d(v i ) indicates v i The depth of the 3D Gaussian under the viewing angle (i.e., the distance between the center position of the 3D Gaussian and the camera position under the viewing angle), α i is the probability opacity. Step S904: Calculate the fourth fractional distillation sampling loss according to the seventh reconstructed image, the description text and the depth information of the second target.
[0220] In this step, a depth-condition-based diffusion model from text to image is used to predict the noise in the noisy image of the seventh reconstructed image at the corresponding time step based on the description text of the second target and the depth information of the first target at the sixth perspective at the corresponding time step, and the fractional distillation sampling loss is determined based on the difference between the predicted noise and the added original noise.
[0221] Among them, the image diffusion model based on depth conditions can be implemented using ControlNet based on depth conditions, or using other models with type functions, which are not specifically limited here. ControlNet (Adding conditional control to text-to-image diffusion models) is an end-to-end neural network structure that controls a large image diffusion model to learn the input conditions of a specific task. In this embodiment, ControlNet based on depth conditions is used.
[0222] Exemplarily, according to the seventh reconstructed image, the description text and depth information of the second target, the fourth fractional distillation sampling loss can be calculated using formula (9):
[0223]
[0224] in, represents the image diffusion model based on depth condition, z′ t″ represents the noisy image obtained by adding random noise ∈′ to the seventh reconstructed image, t″ represents the random sampling time step of the random noise ∈′ added to the seventh reconstructed image, and ω(t″) is a weighting function related to t″. text represents the description text of the second target. d represents the depth information of the rendered deformation dense 3D Gaussian of the first target at each time step at the sixth perspective. Represents the predicted noise, that is, the noisy image z′ is predicted by the image diffusion model based on the depth condition t″ The prediction result of the noise in . x represents the rendered image. * represents the stop gradient operation.
[0225] Step S905: optimize the dense 3D Gaussian of the second target according to the fourth fractional distillation sampling loss to obtain the dense 3D Gaussian after the second target is optimized.
[0226] In this step, the parameters of the dense 3D Gaussian of the second objective are optimized by back-propagation according to the fourth fractional distillation sampling loss.
[0227] In this embodiment, in order to prevent motion and shape degradation due to the displacement of dense 3D Gaussians away from control points during the optimization process, a depth-conditioned image diffusion model (such as ControlNet) is used as supervision, and depth information is obtained from the dense 3D Gaussian rendering of the dynamic 3D model modeled based on the reference video in the first stage. Specifically, in the first stage, based on the reference video of the first target, the sparse 3D Gaussian and deformation network of the first target are optimized; in the third stage, the sparse 3D Gaussian and deformation network of the first target after optimization in the first stage are migrated to the second target, used as the sparse control points and deformation network of the second target, driving the dense 3D Gaussian deformation of the second target, and obtaining the dense 3D Gaussian of the second target at a random time step, and rendering the dense 3D Gaussian based on a random perspective to obtain a reconstructed image of a random perspective at a random time step; based on the depth information of the rendered dense 3D Gaussian of the first target at the sixth perspective, the description text of the second target and the reconstructed image, the fractional distillation sampling loss is calculated using the depth-conditional image diffusion model, and the dense 3D Gaussian of the second target is optimized by back propagation to achieve a fine simulation of the texture of the second target.
[0228] Fig.10A framework diagram of a dynamic 3D target modeling based on motion migration provided by an exemplary embodiment of the present application. The dynamic 3D target modeling based on motion migration includes two stages. Through the first stage and the second stage, the first target (e.g., Fig.10 Dynamic 3D object modeling of the "cat" shown in the figure is performed to obtain dense 3D Gaussian, sparse 3D Gaussian and deformation networks after the first target optimization.
[0229] In the third stage, if Fig.10 As shown, the sparse 3D Gaussian and deformable network MLP optimized for the first target are transferred to the second target (e.g. Fig.10 The “lion” shown in the figure is used as the sparse control points and deformation network MLP of the second target. The dense 3D Gaussian of the second target is initialized based on the sparse control points of the second target. Based on the dense 3D Gaussian optimized for the first target, the description text of the second target (such as “a lion”), the sparse control points of the second target and the deformation network, the dense 3D Gaussian of the second target is optimized. Specifically, the sparse 3D Gaussian optimized in the first stage is used as an implicit sparse control point, and the randomly sampled time step and the center position of the sparse control point of the second target are input into the deformation network MLP. The deformation information of the sparse control point within time step t (including the displacement matrix T) is obtained through the deformation network MLP prediction. t and the rotation matrix R t ), based on the sparse control points and deformation information, drive the dense 3D Gaussian deformation of the first target and the second target, and determine the deformed dense 3D Gaussian of the first target and the second target at time step t respectively through the deformation information of the nearest K sparse control points. Further, based on the reconstructed image of the deformed dense 3D Gaussian of the second target obtained by rendering at a random perspective, the description text of the second target, and the depth information of the rendered deformed dense 3D Gaussian of the first target at the corresponding perspective, calculate the fractional distillation sampling loss L SDS , and back-propagate to update the dense 3D Gaussian of the second target to achieve a refined simulation of the texture of the second target.
[0230] The method of this embodiment transfers the optimized sparse 3D Gaussian and deformation network of the first target to the second target as the sparse control points and deformation network of the second target, drives the deformation of the dense 3D Gaussian through the sparse control points and deformation network, optimizes the dense 3D Gaussian of the second target, and realizes the texture modeling of the second target. It can realize dynamic 3D modeling of different targets that inherit the same motion. The appearance of different targets is controlled by describing text. It can perform dynamic 3D modeling of the second target based on the reference video of the first target, and make the dynamic 3D model of the second target fit the reference video very well, and achieve good modeling effect in three aspects: the degree of conformity with the reference perspective, the spatiotemporal consistency of the dynamic object, and the authenticity of the action.
[0231] The present application provides a dynamic 3D object modeling method. Fig.11 A flow chart of a method for dynamic 3D object modeling provided by an exemplary embodiment of the present application. Fig.11 As shown, the method of any of the above embodiments can be applied to the scene of dynamic 3D object modeling. Fig.11 As shown, the specific steps of this method are as follows:
[0232] Step S1101: Receive a dynamic 3D object modeling request sent by a terminal device and obtain a reference video of the object.
[0233] In this embodiment, the first target to be modeled is an object. The reference video may be a video of the object to be modeled captured by a camera at a fixed camera position, including multiple frames of images of the object to be modeled at multiple different times. For example, the reference video of the object to be modeled may be obtained by shooting with a fixed camera perpendicular to the object to be modeled.
[0234] Step S1102: Initialize the sparse control points of the object as sparse 3D Gaussians, obtain the initialized deformation network, optimize the sparse 3D Gaussians and deformation network of the object based on the reference video of the object, and obtain the optimized sparse control points and deformation network of the object.
[0235] The implementation principle and technical effect of this step refer to the relevant contents of steps S202-S203 in the aforementioned embodiment, which will not be repeated here.
[0236] Step S1103 , initializing a dense 3D Gaussian of the object, optimizing the dense 3D Gaussian of the object based on a reference video of the object, the optimized sparse control points and the deformation network of the object, and obtaining an optimized dense 3D Gaussian of the object.
[0237] The implementation principle and technical effect of this step refer to the relevant content of step S204 in the above embodiment, which will not be repeated here.
[0238] Step S1104: According to the configured time step sequence, the optimized sparse control points and the deformation network of the object, the optimized dense 3D Gaussian deformation of the object is driven to obtain the deformation dense 3D Gaussian of the object at each time step in the time step sequence.
[0239] The time step sequence can be specified by the user; or it can be obtained by sampling according to the number of frames F' specified by the user: t = (f-1) / F', where f = 1, 2, ..., F, which is not specifically limited here. For example, assuming that the number of frames specified by the user is 32, the following time step sequence can be sampled:
[0240] This step can be implemented based on the linear skinning (LBS) method. Please refer to the above embodiment for details, which will not be repeated here.
[0241] Step S1105: Return the dynamic 3D model of the object to the end-side device, where the dynamic 3D model of the object includes a dense 3D Gaussian of the deformation of the object at each time step in the time step sequence.
[0242] The method of this embodiment initializes the sparse control points to displayed 3D Gaussians in the first stage, optimizes the parameters of these sparse 3D Gaussians and the deformation network to obtain the reasonable movement and shape of the dynamic 3D object corresponding to the reference video, and optimizes the dense 3D Gaussians in the second stage to achieve a refined simulation of the texture of the dynamic 3D object; by separately modeling the movement and appearance of the dynamic 3D object, the ambiguity of the movement and appearance is reduced, and a dynamic 3D object (i.e., a 4D object) that fits the reference video very well can be generated, which achieves better results than current similar solutions in terms of the degree of conformity with the reference perspective, the spatiotemporal consistency of the dynamic object, and the authenticity of the movement, thereby improving the accuracy and effect of dynamic 3D object modeling.
[0243] Fig.12 A schematic diagram of the structure of a server provided in an embodiment of the present application. Fig.12 As shown, the server includes: a memory 1201 and a processor 1202. The memory 1201 is used to store computer-executable instructions and can be configured to store various other data to support operations on the server. The processor 1202 is connected to the memory 1201 in communication and is used to execute the computer-executable instructions stored in the memory 1201 to implement the technical solution provided by any of the above method embodiments. The specific functions and technical effects that can be achieved are similar and will not be repeated here.
[0244] Optional, such as Fig.12 As shown, the server also includes: a firewall 1203, a load balancer 1204, a communication component 1205, a power supply component 1206 and other components. Fig.12 Only some components are shown schematically, which does not mean that the server only includes Fig.12 Components shown. Fig.12 In the description, the server is only taken as a cloud server deployed in the cloud as an example for exemplary description. The server can also be deployed locally, and this embodiment is not specifically limited here.
[0245] An embodiment of the present application also provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the method of any of the aforementioned embodiments is implemented. The specific functions and technical effects that can be achieved are not repeated here.
[0246] The present application also provides a computer program product, including a computer program, which implements the method of any of the above embodiments when executed by a processor. The computer program is stored in a readable storage medium, and at least one processor of the server can read the computer program from the readable storage medium. At least one processor executes the computer program so that the server executes the technical solution provided by any of the above method embodiments. The specific functions and technical effects that can be achieved are not repeated here.
[0247] The embodiment of the present application provides a chip, including: a processing module and a communication interface, the processing module can execute the technical solution of the server in the aforementioned method embodiment. Optionally, the chip also includes a storage module (such as a memory), the storage module is used to store instructions, the processing module is used to execute the instructions stored in the storage module, and the execution of the instructions stored in the storage module enables the processing module to execute the technical solution provided by any of the aforementioned method embodiments.
[0248] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.
[0249] It should be understood that the above processor can be a central processing unit (CPU), a graphics processing unit (GPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in at least one processor.
[0250] The memory may include high-speed random access memory (RAM), and may also include non-volatile storage, such as at least one disk storage, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.
[0251] The above storage may be an object storage service (OSS).
[0252] The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.
[0253] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as a mobile hotspot (WiFi), a second-generation mobile communication system (2G), a third-generation mobile communication system (3G), a fourth-generation mobile communication system (4G) / Long Term Evolution (LongTerm Evolution, referred to as LTE), a fifth-generation mobile communication system (5G) and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (Near Field Communication, referred to as NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (Radio Frequency Identification, referred to as RFID) technology, infrared technology, ultra-wideband (Ultra Wide Band, referred to as UWB) technology, Bluetooth technology and other technologies.
[0254] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.
[0255] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.
[0256] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in a dedicated integrated circuit. Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.
[0257] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0258] The order of the above-mentioned embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. In addition, in some of the processes described in the above-mentioned embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed or executed in parallel in the order in which they appear in this article, and are only used to distinguish between different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or less operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit "first" and "second" to different types. The meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.
[0259] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.
[0260] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application.
[0261] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A dynamic 3D target modeling method, characterized in that: include: Obtain a reference video of the first target; In the first stage, the sparse control points of the first target are initialized as sparse 3D Gaussians, an initialized deformable network is obtained, and the sparse 3D Gaussians and the deformable network of the first target are optimized based on a reference video of the first target to obtain the optimized sparse control points and the deformable network of the first target; In the second stage, a dense 3D Gaussian of the first target is initialized, and based on a reference video of the first target, the optimized sparse control points and the deformable network of the first target, the dense 3D Gaussian of the first target is optimized to obtain the optimized dense 3D Gaussian of the first target, wherein the optimized sparse control points, the deformable network and the dense 3D Gaussian of the first target are used to represent the dynamic 3D model of the first target; The step of optimizing the sparse 3D Gaussian and deformable network of the first target based on the reference video of the first target to obtain the optimized sparse control points and deformable network of the first target includes: Taking at least one frame of the reference video as a first reference image, inputting the time step corresponding to the first reference image and the center position of the sparse 3D Gaussian of the first target into a deformation network, and predicting the displacement of the sparse 3D Gaussian in the corresponding time step through the deformation network; Determining a deformed sparse 3D Gaussian of the first target at the corresponding time step according to the sparse 3D Gaussian and the displacement of the sparse 3D Gaussian in the corresponding time step; Rendering an image of the deformed sparse 3D Gaussian at the first viewing angle according to the first reference image to obtain a first reconstructed image; A first reconstruction loss is calculated according to the first reference image and the first reconstructed image, and the sparse 3D Gaussian and deformable network is optimized according to the first reconstruction loss.
2. The method according to claim 1, characterized in that Also includes: Rendering an image of the deformed sparse 3D Gaussian at a second viewing angle to obtain a second reconstructed image; Calculating a first fractional distillation sampling loss according to the first reference image, the second reconstructed image, and a relative angle transformation matrix between the second viewing angle and the first viewing angle; The sparse 3D Gaussian and deformable networks are optimized according to the first fractional distillation sampling loss.
3. The method according to claim 1 or 2, characterized in that: Also includes: A first mask loss is calculated according to the mask map of the first reference image and the mask map of the first reconstructed image, and the sparse 3D Gaussian and deformation network is optimized according to the first mask loss.
4. The method according to claim 1, characterized in that: The initializing the dense 3D Gaussian of the first target comprises: Initialize multiple 3D Gaussians within the sphere of the optimized sparse control points of the first target to obtain an initialized dense 3D Gaussian of the first target.
5. The method according to claim 1, characterized in that: The reference video based on the first target, the sparse control points and the deformation network after the optimization of the first target, and the dense 3D Gaussian of the first target are optimized, including: Based on the reference video of the first target, the optimized sparse control points and the deformable network of the first target, the dense 3D Gaussian of the first target is optimized, and the sparse control points and the deformable network after the optimization of the first target are optimized.
6. The method according to claim 5, characterized in that The reference video based on the first target, the sparse control points and the deformable network after the optimization of the first target, optimizing the dense 3D Gaussian of the first target, and optimizing the sparse control points and the deformable network after the optimization of the first target, including: Using at least one frame of the reference video as a second reference image, and based on the time step corresponding to the second reference image, the sparse control points after optimization of the first target, and the deformation network, driving the optimized dense 3D Gaussian of the first target to deform, so as to obtain the deformed dense 3D Gaussian of the first target at the corresponding time step; Rendering an image of the deformable dense 3D Gaussian at the first viewing angle according to the first viewing angle corresponding to the second reference image to obtain a third reconstructed image; Calculating a second reconstruction loss according to the second reference image and the third reconstructed image; The dense 3D Gaussian of the first target is optimized according to the second reconstruction loss, and the sparse control points and the deformable network after the optimization of the first target are optimized.
7. The method according to claim 6, characterized in that Also includes: Rendering an image of the deformed dense 3D Gaussian at a third viewing angle to obtain a fourth reconstructed image; Calculating a second fractional distillation sampling loss according to the second reference image, the fourth reconstructed image, and a relative angle transformation matrix between the third perspective and the first perspective; According to the second fractional distillation sampling loss, the dense 3D Gaussian of the first target is optimized, and the sparse control points and the deformable network after the optimization of the first target are optimized.
8. The method according to claim 6, characterized in that Also includes: Calculating a second mask loss according to the mask map of the second reference image and the mask map of the third reconstructed image; The dense 3D Gaussian of the first target is optimized according to the second mask loss, and the sparse control points and the deformable network after the optimization of the first target are optimized.
9. The method according to any one of claims 6 to 8, characterized in that: Also includes: Calculate the Gaussian alignment loss according to the center position of the deformed sparse 3D Gaussian of the first target at the corresponding time step in the first stage and the center position of the deformed sparse control point of the first target at the corresponding time step in the second stage; According to the Gaussian alignment loss, the sparse control points and the deformable network after the first objective optimization are optimized.
10. The method according to claim 1, characterized in that Also includes: According to at least one configured time step, using the sparse control points and deformation network optimized by the first target to drive the dense 3D Gaussian optimized by the first target to deform, obtaining the deformed dense 3D Gaussian of the first target in the at least one time step, and obtaining a dynamic 3D model of the first target; The deformation dense 3D Gaussian of the first target at each time step is rendered to generate a display video of the dynamic 3D model of the first target.
11. The method according to claim 1, characterized in that: Also includes: Migrating the sparse 3D Gaussian and deformable network optimized for the first target to the second target as the sparse control points and deformable network of the second target; Initialize multiple 3D Gaussians within the sphere of the sparse control points of the second target as dense 3D Gaussians initialized for the second target; Optimizing the dense 3D Gaussian of the second target based on the optimized dense 3D Gaussian of the first target, the description text of the second target, the sparse control points and the deformable network of the second target; The sparse control points, the deformation network and the optimized dense 3D Gaussian of the second object are used to represent the dynamic 3D model of the second object.
12. The method according to claim 11, characterized in that The step of optimizing the dense 3D Gaussian of the second target based on the optimized dense 3D Gaussian of the first target, the description text of the second target, the sparse control points and the deformation network of the second target includes: According to at least one time step, the sparse control points of the second target and the deformation network, a dense 3D Gaussian of the second target and a dense 3D Gaussian of the first target are driven to deform, so as to obtain a deformed dense 3D Gaussian of the second target at each time step and a deformed dense 3D Gaussian of the first target at each time step; Rendering an image of the deformed dense 3D Gaussian of the second target at each time step at a sixth viewing angle to obtain a seventh reconstructed image at each time step, and rendering depth information of the deformed dense 3D Gaussian of the first target at each time step at the sixth viewing angle; According to the seventh reconstructed image, the description text of the second target and the depth information, a fourth fractional distillation sampling loss is calculated, and based on the fourth fractional distillation sampling loss, a dense 3D Gaussian of the second target is optimized to obtain an optimized dense 3D Gaussian of the second target.
13. The method according to claim 11, characterized in that Also includes: According to the configured time step sequence, the sparse control points and deformation network of the second target are used to drive the optimized dense 3D Gaussian of the second target to deform, so as to obtain the deformed dense 3D Gaussian of the second target at each time step, and obtain the dynamic 3D model of the second target; The deformation dense 3D Gaussian of the second target at each time step is rendered to generate a display video of the dynamic 3D model of the second target.
14. A dynamic 3D object modeling method, characterized in that: include: Receive a dynamic 3D object modeling request sent by a terminal device and obtain a reference video of the object; Initializing the sparse control points of the object as sparse 3D Gaussians, obtaining an initialized deformation network, optimizing the sparse 3D Gaussians and deformation network of the object based on a reference video of the object, and obtaining optimized sparse control points and deformation network of the object; Initializing a dense 3D Gaussian of the object, optimizing the dense 3D Gaussian of the object based on a reference video of the object, the optimized sparse control points and the deformation network of the object, and obtaining an optimized dense 3D Gaussian of the object; According to the configured time step sequence, the optimized sparse control points of the object and the deformation network, the optimized dense 3D Gaussian deformation of the object is driven to obtain the deformation dense 3D Gaussian of the object at each time step in the time step sequence; Returning a dynamic 3D model of the object to the client device, wherein the dynamic 3D model of the object includes a dense 3D Gaussian of a deformation of the object at each time step in the time step sequence; The method of optimizing the sparse 3D Gaussian and deformation network of the object based on the reference video of the object to obtain the optimized sparse control points and deformation network of the object includes: Taking at least one frame of the reference video as a reference image, inputting the time step corresponding to the reference image and the center position of the sparse 3D Gaussian of the object into a deformation network, and predicting the displacement of the sparse 3D Gaussian in the corresponding time step through the deformation network; Determine a deformation sparse 3D Gaussian of the object at the corresponding time step according to the sparse 3D Gaussian and the displacement of the sparse 3D Gaussian in the corresponding time step; Rendering the image of the deformed sparse 3D Gaussian at the viewing angle according to the viewing angle corresponding to the reference image to obtain a reconstructed image; A reconstruction loss is calculated according to the reference image and the reconstructed image, and the sparse 3D Gaussian and deformable network is optimized according to the reconstruction loss.
15. A server, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the server to execute the method described in any one of claims 1-14.
16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 14 is implemented.
17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.
