A dynamic object reconstruction method and device based on motion decomposition

Abstract

The application discloses a dynamic object reconstruction method and device based on motion decomposition, which comprises the following steps: segmenting an initial dynamic video set V containing a motion process of a target object frame by frame to obtain a foreground dynamic area of each frame of image in V; using a multi-view reconstruction method to obtain camera extrinsic parameters and intrinsic parameters of each frame of image in V; constructing a neighbor set for each frame of V and estimating optical flow of selected picture pairs in the neighbor set by means of a pre-trained optical flow method to obtain optical flow between two frames; establishing a non-rigid neural body rendering model according to the foreground dynamic area of each frame of image in V, the camera extrinsic parameters and the intrinsic parameters; obtaining an optimized non-rigid neural body rendering model according to the non-rigid neural body rendering model and the optical flow combined with a time sequence consistency constraint; and reconstructing complete geometry and color of each frame of the target object according to the optimized non-rigid neural body rendering model.

Description

Technical Field

[0001] This invention belongs to the field of inferring non-rigid structures from 2D video motion and neural body rendering, and particularly relates to a method and apparatus for reconstructing dynamic objects based on motion decomposition. Background Technology

[0002] Reconstructing non-rigid 3D geometry from monocular dynamic video is a crucial problem in computer vision, with significant applications in dynamic object reconstruction, VR / AR, and other technologies. Monocular dynamic video reconstruction refers to reconstructing the geometry of a dynamic object and obtaining its camera pose from a series of consecutive frames with unknown camera pose.

[0003] Traditional methods for reconstructing dynamic objects from monocular video mainly fall into two categories: template-based surface reconstruction and non-template-based reconstruction. Template-based reconstruction typically involves creating a template model from a single frame of the input image and then learning the deformation field for other frames. This approach is generally suitable for dynamic objects with prior models, such as the human body and faces. However, for general types of dynamic objects, it is difficult to establish a template model beforehand. The other method is non-rigid structure from motion (NRSfM), which does not establish a template model. To find matching relationships between multiple images, it usually involves extracting sparse keypoints for matching or using optical flow to match dense corresponding points for tracking and reconstruction. Because this problem is ill-posed, constraints are typically added to improve optimization and facilitate reconstruction. Some methods use constraints such as point motion trajectories, temporal consistency, local rigidity assumptions, and shape priors to constrain deformation. Other methods assume that the reconstructed shape space is low-rank and obtain the optimal solution by reducing the dimensionality of the solution space. However, existing methods still suffer from inadequacy with real-world data (which may contain noise, missing data, or significant deformation).

[0004] Recently, neural network-based reconstruction and rendering techniques have gained increasing popularity due to their higher reconstruction accuracy, more realistic rendering quality, and lower data storage requirements compared to traditional methods. Some existing methods combine non-rigid deformation fields and neural radiation fields to model non-rigid scenes, optimizing the deformation field and neural radiation field by making the rendered image as consistent as possible with the input real image. However, these methods still suffer from low geometric accuracy in reconstruction. To better combine reconstructed geometry with color information, some methods set up a shape space as a module for predicting geometry and a module for predicting color, estimating transparency using the predicted geometric information, and combining the two through volumetric rendering. Compared to methods directly based on neural radiation fields, this method can reconstruct more accurate geometry. However, this method is only suitable for modeling rigid objects or scenes. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this application is to provide a method and apparatus for dynamic object reconstruction based on motion decomposition.

[0006] According to a first aspect of the embodiments of this application, a method for reconstructing dynamic objects based on motion decomposition is provided, comprising:

[0007] (1) The initial dynamic video set V containing the motion process of the target object is segmented frame by frame to obtain the foreground dynamic region of each frame in the initial dynamic video set V.

[0008] (2) Using a multi-view reconstruction method, the camera extrinsic and intrinsic parameters of each frame in the initial dynamic video set V are obtained;

[0009] (3) For each frame of the initial dynamic video set V, construct a neighbor set, and use a pre-trained optical flow method to estimate the optical flow of the selected image pairs in the neighbor set to obtain the optical flow between the two frames.

[0010] (4) Based on the foreground dynamic region, camera extrinsic parameters and intrinsic parameters of each frame image in the initial dynamic video set V, establish a non-rigid neural body rendering model;

[0011] (5) Based on the non-rigid neural body rendering model and the optical flow, combined with the temporal consistency constraint, an optimized non-rigid neural body rendering model is obtained.

[0012] (6) Based on the optimized non-rigid neural body rendering model, reconstruct the geometry and color of the target object for each frame.

[0013] Furthermore, the non-rigid neural body rendering model includes a standard space geometry module, a standard space color module, and other frame-shifted deformation field modules to standard space.

[0014] Furthermore, the geometric modules of the standard space are:

[0015] x→σ:f θ (x)

[0016] Where θ is the geometric model parameter of the standard space, which decodes a point in the standard space into a signed distance function of the standard space geometry, σ is the output scalar attribute, and f θ (x) is a parameterized implicit function that models the mapping between x and σ.

[0017] The color module of the standard space is:

[0018] (p, v, c) appe (t))→c:c ψ (p, v, c) appe(I))

[0019] Where θ is the geometric model parameter of the standard space, which decodes a point in the standard space into a symbolic distance function of the standard space geometry; ψ is the color model parameter of the standard frame; p is the position in the standard space; v is the ray direction of point p in the standard space; and c... appe (I) is the appearance condition vector corresponding to I images, where c is the color attribute, c ψ For modeling (p, v, c) appe (t) is a parameterized implicit function, where t represents the t-th frame.

[0020] The other frame-shifted deformation field modules to standard space are:

[0021] (p(t), c def (t))→p′(t):D η (p(t), c def (t))

[0022] Where η is the parameter of the deformation field that transforms a point in frame t to standard space, p(t) represents the position of the point in frame t, p'(t) represents the position transformed to the standard frame, and c def (t) is the deformation condition vector corresponding to frame t.

[0023] Furthermore, the non-rigid neural body rendering model for frame t is as follows:

[0024]

[0025] Where o is the camera center in frame t, s near With s far Let v be the near and far planes corresponding to the camera in frame t, ω(s) be the point density after deformation along ray v to the corresponding point p(s) in standard space, v(u,t) represent the viewing direction corresponding to pixel u for frame t, and J is the Jacobian matrix corresponding to the deformation at point p.

[0026]

[0027]

[0028] Furthermore, the non-rigid neural body rendering model is obtained through other frame-shifted deformation field modules to standard space and standard space rendering modules C(o,v), which are obtained based on the standard space geometry module and standard space color module.

[0029]

[0030]

[0031] φ γ (x)=γe -γx / (1+e -γx ) -2

[0032] Where o is the camera center corresponding to image I, v is the ray connecting pixel u to the camera center, and s near With s far These are the near and far planes of the camera corresponding to image I, where ω(s) is the density at point p(s) along ray v, obtained by transforming it using the signed distance function at that point, and φ... γ (x) is a non-linear activation function that acts on the density function. Under the activation of this function, the density can achieve unbiased properties. γ is a hyperparameter used to control the sparsity of geometry and rendering.

[0033] Furthermore, in step (5), when the model's loss function converges, the model optimization is complete, wherein the loss function is:

[0034]

[0035] The specific losses are as follows:

[0036] L indicates that the rendered color is the same as the color of the current frame image. color :

[0037]

[0038] Regarding the geometric rules, the constrained geometry should, as far as possible, satisfy the constraints L of the Eikonal equation. reg :

[0039]

[0040] Regarding the mask constraint L in the foreground region mask Using the cross-entropy loss function:

[0041]

[0042] The mask estimate is obtained by integrating density through volume rendering:

[0043]

[0044] Constraints regarding geometric consistency in time:

[0045]

[0046] In the above model, θ, η, Let I represent the learnable parameters of the multilayer perceptron. The input parameters of this system include: the position p(t) of frame t, the retrieval direction v(u, t) corresponding to pixel u in frame t, the deformation condition vector c_{def}(t) of frame t, and the appearance condition vector c_{appe}(t) of frame t. The output is the symbolic distance value and color information of the point. g (u, t) represents the RGB value of the u-th pixel in the t-th true color image, M g (u, t) represents the mask value of the u-th pixel in the t-th ground truth mask image, and Δu(t, t′) represents the optical flow values ​​at pixel u in the t-th and t′-th frames.

[0047] According to a second aspect of the embodiments of this application, a dynamic object reconstruction apparatus based on motion decomposition is provided, comprising:

[0048] The segmentation module is used to segment the initial dynamic video set V containing the motion process of the target object frame by frame to obtain the foreground dynamic region of each frame in the initial dynamic video set V.

[0049] The multi-view reconstruction module is used to obtain the camera extrinsic and intrinsic parameters of each frame in the initial dynamic video set V using a multi-view reconstruction method.

[0050] The construction module is used to construct a neighbor set for each frame of the initial dynamic video set V, and to estimate the optical flow of selected image pairs in the neighbor set by means of a pre-trained optical flow method to obtain the optical flow between two frames.

[0051] The modeling module is used to establish a non-rigid neural body rendering model based on the foreground dynamic region, camera extrinsic parameters and intrinsic parameters of each frame image in the initial dynamic video set V.

[0052] The model optimization module is used to obtain an optimized non-rigid neural body rendering model based on the non-rigid neural body rendering model and the optical flow, combined with temporal consistency constraints.

[0053] The reconstruction module is used to reconstruct the geometry and color of the target object for each frame based on the optimized non-rigid neural body rendering model.

[0054] According to a third aspect of the embodiments of this application, an electronic device is provided, comprising:

[0055] One or more processors;

[0056] Memory, used to store one or more programs;

[0057] When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in the first aspect.

[0058] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0059] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0060] As can be seen from the above embodiments, this application 1) uses a dynamic implicit function model based on volume rendering to infer non-rigid structures from motion, avoiding the need to consider the template topology of moving objects; 2) decomposes the motion of the dynamic implicit function and increases optical flow to eliminate two degrees of freedom, which can make the motion estimation more accurate.

[0061] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0062] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0063] Figure 1 This is a schematic diagram illustrating the modeling process in a dynamic object reconstruction method based on motion decomposition, according to an exemplary embodiment.

[0064] Figure 2 This is a block diagram illustrating a dynamic object reconstruction apparatus based on motion decomposition according to an exemplary embodiment.

[0065] Figure 3 This is a schematic diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0066] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0067] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0068] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0069] Figure 1 This is a flowchart illustrating a dynamic implicit function reconstruction method based on motion decomposition according to an exemplary embodiment, such as... Figure 1 As shown, the method may include the following steps:

[0070] (1) The initial dynamic video set V containing the motion process of the target object is segmented frame by frame to obtain the foreground dynamic region of each frame in the initial dynamic video set V.

[0071] (2) Using a multi-view reconstruction method, the camera extrinsic and intrinsic parameters of each frame in the initial dynamic video set V are obtained;

[0072] (3) For each frame of the initial dynamic video set V, construct a neighbor set, and use a pre-trained optical flow method to estimate the optical flow of the selected image pairs in the neighbor set to obtain the optical flow between the two frames.

[0073] (4) Based on the foreground dynamic region, camera extrinsic parameters and intrinsic parameters of each frame image in the initial dynamic video set V, establish a non-rigid neural body rendering model;

[0074] (5) Based on the non-rigid neural body rendering model and the optical flow, combined with the temporal consistency constraint, the model is trained to obtain an optimized non-rigid neural body rendering model.

[0075] (6) Based on the optimized non-rigid neural body rendering model, reconstruct the geometry and color of the target object for each frame.

[0076] As can be seen from the above embodiments, this application 1) uses a dynamic implicit function model based on volume rendering to infer non-rigid structures from motion, avoiding the need to consider the template topology of moving objects; 2) decomposes the motion of the dynamic implicit function and increases optical flow to eliminate two degrees of freedom, which can make the motion estimation more accurate.

[0077] In the specific implementation of step (1), the initial dynamic video set V is segmented frame by frame. After segmentation, each frame image includes the following elements: a foreground dynamic region and a background static region M. g(u, t). In the initial dynamic video set V, each frame of color image is an RGB three-channel image. For the segmentation of the foreground and background in the image, any automated image segmentation technique can be used to obtain a segmented image that conforms to the corresponding semantics, which will not be elaborated here.

[0078] In the specific implementation of step (2), the multi-view reconstruction method for estimating camera intrinsic and extrinsic parameter information includes various multi-view reconstruction methods based on traditional multi-view models and neural networks, such as COLMAP, OpenMVG, and other classic SFM algorithms. Using such classic algorithms to obtain camera intrinsic and extrinsic parameters is relatively accurate and robust, and can also improve our reconstruction results. The camera intrinsic and extrinsic parameters estimated in this step are mainly obtained during parameter training in step (5), which requires obtaining the 3D rays passing through the center of the camera from the pixels in the 2D image, thereby completing the volume rendering process.

[0079] In the specific implementation of step (3), for the t-th frame of the dynamic video set V, we select the N frames before and after as its neighbor set, so the neighbor set of frame t is [tN, t+N]. The purpose of this construction is to eliminate the problem that if the difference in the number of frames is too large, the motion difference will be too large, which will ultimately make the optimization not robust enough. The optical flow estimation of the neighbor set is for the optical flow between two frames. Any optical flow estimation technique can be used, such as the RAFT or SuperGlue algorithm of deep learning. The selection of optical flow estimation technique is a conventional setting in this field and will not be elaborated here.

[0080] In the specific implementation of step (4), the non-rigid neural body rendering model includes a geometric module in standard space, a color module in standard space, and other deformation field modules that are frame-shifted to standard space.

[0081] Specifically, the geometric module (1) of the standard space is:

[0082] x→σ:f θ (x)

[0083] Where θ is the geometric model parameter of the standard space, which decodes a point in the standard space into a signed distance function of the standard space geometry, σ is the output scalar attribute, and f θ (x) is a parameterized implicit function that models the mapping between x and σ.

[0084] The color module (2) of the standard space is:

[0085] (p, v, c) appe (t))→c:c ψ (p, v, c) appe (I))

[0086] Where θ is the geometric model parameter of the standard space, which decodes a point in the standard space into a symbolic distance function of the standard space geometry; ψ is the color model parameter of the standard frame; p is the position in the standard space; v is the ray direction of point p in the standard space; and c... appe (I) is the appearance condition vector corresponding to I images, where c is the color attribute (RGB). ψ For modeling (p, v, c) appe (t) is a parameterized implicit function, where t represents the t-th frame.

[0087] The other frame-shifted deformation field module (3) to standard space is:

[0088] (p(t), c def (t))→p′(t):D η (p(t), c def (t))

[0089] Where η is the parameter of the deformation field that transforms a point in frame t to standard space, p(t) represents the position of the point in frame t, p'(t) represents the position transformed to the standard frame, and c def (t) is the deformation condition vector corresponding to frame t.

[0090] For the standard space rendering module C(o, v):

[0091]

[0092]

[0093] φ γ (x)=γe -γx / (1+e -γx ) -2

[0094] Where o is the camera center corresponding to image I, v is the ray connecting pixel u to the camera center, and s near With s far These are the near and far planes of the camera corresponding to image I, where ω(s) is the density at point p(s) along ray v, obtained by transforming it using the signed distance function at that point, and φ... γ (x) is a non-linear activation function that acts on the density function. Under the activation of this function, the density can achieve unbiased properties. γ is a hyperparameter used to control the sparsity of geometry and rendering.

[0095] Therefore, by combining the deformation field module and the standard frame rendering module, and by transforming the information of frame t to the standard frame module through the deformation field module, we can obtain the non-rigid neural body rendering model of frame t as follows:

[0096]

[0097] Where o is the camera center in frame t, s near With s far Let v be the near and far planes corresponding to the camera in frame t, ω(s) be the point density after deformation along ray v to the corresponding point p(s) in standard space, v(u,t) represent the viewing direction corresponding to pixel u for frame t, and J is the Jacobian matrix corresponding to the deformation at point p.

[0098]

[0099]

[0100] It should be noted that the parameterized implicit function model learned for the target image is a fully connected neural network model, which includes a geometry module, a color module, and a deformation field module. It consists of, in sequence: an input layer, a fully connected layer, a nonlinear layer, a pooling layer, another fully connected layer, and an output layer.

[0101] In the specific implementation of step (5), in order to increase the temporal consistency of the reconstructed dynamic function, a two-dimensional decomposition of the dynamic function is considered: along the ray direction of the camera center and within the plane of the camera:

[0102]

[0103] Where x(u, t) is the visible intersection of the ray formed by pixel u and the center of the camera with the geometry of frame t, u(t) is the position of pixel u in frame t, and du / dt is the motion speed of pixel u in frame t.

[0104] The location of the intersection point can be obtained by rendering the distance volume of the ray:

[0105]

[0106] First, we solve the 6-dimensional linear equations after motion decomposition.

[0107]

[0108] The motion of the spatial position with respect to the phase plane can be obtained:

[0109]

[0110] Furthermore, the solution process is differentiable, where x'(u,t) is the point in standard space that x(u,t) is transformed through the deformation field:

[0111] x′(u,t)=D η (x(u,t),c) def (t)),

[0112] It is the gradient of the geometric module in standard space with respect to its spatial position; furthermore... The correspondence between the geometric projection of frame t and the image can be obtained. Since the solution of the linear field equation is differentiable, the solution of the above linear equation can guarantee that the geometric relationship is consistent in time and the gradient is transferred to the model.

[0113] Based on the dynamic motion modeling process above, the relationship between color and each frame of the image is established in terms of geometry and rendering. In addition, the temporal geometric relationship between frames can be established through optical flow. After discretizing them, the following loss function is constructed to learn the implicit functions of the modules (1, 2, 3) established above. When each loss converges, the model optimization is completed.

[0114]

[0115] The specific losses are as follows:

[0116] L indicates that the rendered color is the same as the color of the current frame image. color :

[0117]

[0118] Regarding the geometric rules, the constrained geometry should, as far as possible, satisfy the constraints L of the Eikonal equation. reg :

[0119]

[0120] Regarding the mask constraint L in the foreground region mask Using the cross-entropy loss function:

[0121]

[0122] The mask estimate is obtained by integrating density through volume rendering:

[0123]

[0124] Constraints regarding geometric consistency in time:

[0125]

[0126] In the above model, θ, η, The system's input parameters include: the position p(t) of frame t, the retrieval direction v(u,t) corresponding to pixel u in frame t, the deformation condition vector c_{def}(t) of frame t, and the appearance condition vector c_{appe}(t) of frame t. The output is the symbolic distance value and color information of that point. g (u, t) represents the RGB value of the u-th pixel in the t-th ground truth color image, BCE represents the cross-entropy function, used to constrain the generated mask to be consistent with the ground truth mask, M g (u, t) represents the mask value of the u-th pixel in the t-th ground truth mask image, and Δu(t, t′) represents the optical flow values ​​at pixel u in the t-th and t′-th frames.

[0127] Therefore, after the above optimization is completed, we can obtain the optimized standard frame geometry module, rendering module, and deformation field module that deforms from other frames to the standard frame.

[0128] In the specific implementation of step (6), for the extraction of geometry for each frame of the dynamic video, for a certain frame, we first establish a uniform 3D bounded grid G ​​with a resolution of (m1, m2, m3) for that frame, and then deform the grid G ​​to a standard frame through the deformation field module. From the standard frame, we obtain the geometric and color attributes of each grid point on the grid G ​​through the geometry module and color module of the standard frame. The geometric attribute is the sign distance value of each grid point. We can obtain the corresponding isosurface S of the grid G ​​through the classic Marching Cubes algorithm. The isosurface S corresponds to the geometry of this frame.

[0129] For color extraction of each frame of a dynamic video, for a given frame, we use a virtual camera C with camera parameters π to synthesize a new image I from that camera's perspective using our color module. The method for obtaining the color value of each pixel u in the synthesized image I is as follows: a ray r(u) is generated along pixel u through the camera center of camera C. Within the near and far planes of camera C, the ray is uniformly sampled to obtain the sampling point sequence {r(u)}. i Then, the deformation field module is used to process the sampling point set {r(u)}; i The process is transformed to a standard frame, and the geometric and color attributes of the sampling points are obtained from the standard frame through the geometric and color modules of the standard frame. Finally, with the help of the volume rendering method, the geometric and color attributes sampled on the ray are accumulated and summed to obtain the color value of pixel u.

[0130] Corresponding to the aforementioned embodiments of the dynamic object reconstruction method based on motion decomposition, this application also provides embodiments of a dynamic object reconstruction apparatus based on motion decomposition.

[0131] Figure 2 This is a block diagram illustrating a dynamic object reconstruction apparatus based on motion decomposition, according to an exemplary embodiment. (Refer to...) Figure 2 The device may include:

[0132] Segmentation module 21 is used to segment the initial dynamic video set V containing the motion process of the target object frame by frame to obtain the foreground dynamic region of each frame in the initial dynamic video set V.

[0133] The multi-view reconstruction module 22 is used to obtain the camera extrinsic and intrinsic parameters of each frame image in the initial dynamic video set V using a multi-view reconstruction method.

[0134] Construction module 23 is used to construct a neighbor set for each frame of the initial dynamic video set V, and to estimate the optical flow of selected image pairs in the neighbor set by means of a pre-trained optical flow method to obtain the optical flow between two frames.

[0135] Modeling module 24 is used to establish a non-rigid neural body rendering model based on the foreground dynamic region, camera extrinsic parameters and intrinsic parameters of each frame image in the initial dynamic video set V;

[0136] The model optimization module 25 is used to obtain an optimized non-rigid neural body rendering model based on the non-rigid neural body rendering model and the optical flow, combined with the temporal consistency constraints.

[0137] Reconstruction module 26 is used to reconstruct the geometry and color of the target object for each frame based on the optimized non-rigid neural body rendering model.

[0138] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0139] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0140] Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the motion decomposition-based dynamic object reconstruction method as described above. Figure 3 The diagram shown is a hardware structure diagram of any device with data processing capabilities, used in an embodiment of the present invention to provide a dynamic object reconstruction method based on motion decomposition. (Except for...) Figure 3 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0141] Accordingly, this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the motion decomposition-based dynamic object reconstruction method described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of a wind turbine, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0142] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0143] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for reconstructing dynamic objects based on motion decomposition, characterized in that, include: (1) The initial dynamic video set V containing the motion process of the target object is segmented frame by frame to obtain the foreground dynamic region of each frame in the initial dynamic video set V; (2) Using a multi-view reconstruction method, the camera extrinsic and intrinsic parameters of each frame in the initial dynamic video set V are obtained; (3) For each frame of the initial dynamic video set V, construct a neighbor set, and use a pre-trained optical flow method to estimate the optical flow of the selected image pairs in the neighbor set to obtain the optical flow between the two frames. (4) Based on the foreground dynamic region, camera extrinsic parameters and intrinsic parameters of each frame in the initial dynamic video set V, establish a non-rigid neural body rendering model; (5) Based on the non-rigid neural body rendering model and the optical flow, combined with the temporal consistency constraint, an optimized non-rigid neural body rendering model is obtained; (6) Reconstruct the geometry and color of the target object for each frame based on the optimized non-rigid neural body rendering model.

2. The method according to claim 1, characterized in that, The non-rigid neural body rendering model includes a standard space geometry module, a standard space color module, and other frame-shifted deformation field modules to standard space.

3. The method according to claim 2, characterized in that, The geometric modules of the standard space are: ， in, These are the geometric model parameters of the standard space. This model decodes a point in the standard space into a signed distance function of the standard space geometry. For the output scalar properties, To model x and The parameterized implicit function of the mapping between them; The color module of the standard space is: ， in These are the geometric model parameters of standard space. This model decodes a point in standard space into a signed distance function of standard space geometry. These are the color model parameters for the standard frame, where p is the position in standard space, v is the ray direction of point p in standard space, and c... appe (I) represents the appearance condition vector for I images, where c is the color attribute. For modeling (p, v, c) appe (t) is a parameterized implicit function, where t represents the t-th frame; The other frame-shifted deformation field modules to standard space are: ， in, These are the parameters of the deformation field used to transform a point in frame t to standard space. p(t) represents the position of the point in frame t, p'(t) represents the position transformed to the standard frame, and c def (t) is the deformation condition vector corresponding to frame t.

4. The method according to claim 1, characterized in that, The non-rigid neural body rendering model for frame t is: ， Where o is the camera center in frame t, and v is the ray connecting pixel u to the camera center. and It is the near and far planes corresponding to the camera in frame t. It is the point density along ray v at the corresponding point p(s) in standard space after deformation. For modeling (p, v, c) appe The parameterized implicit function of (t) is c. appe (t) is the appearance condition vector for frame t. These are the parameters of the deformation field that transforms a point in frame t to standard space. v(u, t) represents the viewpoint direction corresponding to pixel u for frame t. def (t) is the deformation condition vector of the t-th frame, and J corresponds to the Jacobian matrix of the deformation at point p; ， ， This is a nonlinear activation function that acts on the density function. Activation by this function can make the density achieve unbiased properties. It is a hyperparameter used to control the sparsity of geometry and rendering.

5. The method according to claim 2, characterized in that, The non-rigid neural body rendering model uses other frame-shifted deformation field modules and standard space rendering modules. The rendering module of the standard space obtains the following based on the geometry module and color module of the standard space: ， ， ， Where o is the camera center corresponding to image I, and v is the ray connecting pixel u to the camera center. and These are the near and far planes of the camera corresponding to image I, where... It is the density along ray v at the corresponding point p(s), obtained by transforming the sign distance function of point p(s). For modeling (p, v, c) appe The parameterized implicit function of (t) is c. appe (I) is the appearance condition vector corresponding to I images. This is a nonlinear activation function that acts on the density function. Activation by this function can make the density achieve unbiased properties. It is a hyperparameter used to control the sparsity of geometry and rendering. These are the geometric model parameters of the standard space. This model decodes a point in the standard space into a signed distance function of the standard space geometry. For the output scalar properties, To model x and The parameterized implicit function that maps between them.

6. The method according to claim 1, characterized in that, In step (5), the model optimization is complete when the model's loss function converges, wherein the loss function is: ， The various losses are as follows: L indicates that the rendered color is the same as the color of the current frame image. color : ， Regarding the geometric regularities, the constraint geometry satisfies the constraint L of the Eikonal equation. reg : ， Regarding the mask constraint L in the foreground region mask Using the cross-entropy loss function: ， The mask estimate is obtained by integrating density through volume rendering: ， Constraints regarding geometric consistency in time: ， In the above model The learnable parameters of the multilayer perceptron are represented by: the position p(t) of the t-th frame, the retrieval direction v(u, t) corresponding to the pixel u in the t-th frame, and the deformation condition vector c of the t-th frame. def (t), the appearance condition vector c of frame t appe (t), outputting the sign distance value of point p(s) and the color information of that point. This represents the non-rigid neural body rendering model for frame t. Let represent the RGB value of the u-th pixel in the t-th ground truth color image, where N is the number of frames selected. and It is the near and far planes corresponding to the camera in frame t. These are the parameters of the deformation field that transforms a point in frame t to standard space. Let x(u(t), t) represent the mask value of the u-th pixel in the t-th ground truth mask image, where x(u(t), t) is the visible intersection point of the ray formed by pixel u and the camera center in the t-th frame with the geometry of the t-th frame. Indicates the relationship between frame t and frame t. The optical flow value at pixel u of the frame.

7. A dynamic object reconstruction device based on motion decomposition, characterized in that, include: The segmentation module is used to segment the initial dynamic video set V containing the motion process of the target object frame by frame to obtain the foreground dynamic region of each frame in the initial dynamic video set V. The multi-view reconstruction module is used to obtain the camera extrinsic and intrinsic parameters of each frame in the initial dynamic video set V using a multi-view reconstruction method. The construction module is used to construct a neighbor set for each frame of the initial dynamic video set V, and to estimate the optical flow of selected image pairs in the neighbor set by means of a pre-trained optical flow method to obtain the optical flow between two frames. The modeling module is used to establish a non-rigid neural body rendering model based on the foreground dynamic region, camera extrinsic parameters and intrinsic parameters of each frame image in the initial dynamic video set V. The model optimization module is used to obtain an optimized non-rigid neural body rendering model based on the non-rigid neural body rendering model and the optical flow, combined with temporal consistency constraints. The reconstruction module is used to reconstruct the geometry and color of the target object for each frame based on the optimized non-rigid neural body rendering model.

8. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-6.

Images