A method, system and device for 3D reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression

Abstract

The present invention discloses a method, system and device for three-dimensional reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression. The method designs a new decomposable wavelet transform (FDWT) method, which mainly roughly estimates the scene flow between implicit point clouds in the video in the frequency domain, embeds and encodes the four-dimensional point cloud coordinates and sends them to the decomposable wavelet transform module to predict the point cloud motion, thereby achieving a preliminary prediction of the scene flow. An implicit scene flow error filter (ISEF) model focusing on the spatiotemporal domain is designed for scene flow correction, and the preliminary predicted scene flow output by the decomposition wavelet transform is corrected in the spatiotemporal domain to improve the prediction accuracy of the model. Finally, the modified neural implicit expression is used to aggregate the video information of adjacent frames to perform three-dimensional reconstruction of dynamic video. The present invention can be used in fields such as three-dimensional modeling of mobile terminals (mobile phones, etc.).

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method, system and device for three-dimensional reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression. Background Art

[0002] In the classic task of synthesizing new perspectives in videos, the primary goal is to reconstruct the 3D scene from a given monocular video and synthesize a new perspective for each frame. To achieve this, images from different camera poses are often used to supplement the features of the current frame. Traditional multi-view geometry theory requires multiple viewpoints at the same moment; when combined with image-based rendering methods, high-fidelity scene synthesis from new perspectives can be achieved using representations such as multi-layer perceptrons, point clouds, and voxels.

[0003] Aggregating features from adjacent frames is crucial for single-frame novel view synthesis, because occluded information of the target frame may appear in adjacent frames and if each frame is rendered independently, the occluded information may appear in the adjacent frames. Figure 1 However, neural implicit representations pose obstacles to dynamic scene modeling because scenes are modeled as static continuous distributions, which cannot aggregate image and motion information from adjacent frames. This often results in artifacts and other poor image quality issues for moving parts. Summary of the Invention

[0004] The present invention aims to address the deficiencies in the prior art and to provide a method, system and device for 3D reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression.

[0005] As mentioned in the background technology, the existing technology still has a lot of shortcomings in solving the synthesis of new perspectives for dynamic scenes in monocular videos, and it becomes crucial to integrate signals from the spatiotemporal and frequency domains. The present invention proposes a new decomposable wavelet transform, mainly in the frequency domain, to roughly estimate the scene flow between implicit point clouds in the video, and proposes an implicit scene flow error filtering model focusing on the spatiotemporal domain for scene flow correction. In addition, shape matching and point cloud registration are proposed as rigid constraints to improve the predicted implicit scene flow and achieve high-quality new perspective synthesis. New views can be synthesized by aggregating the features of adjacent frames into the current frame through the predicted corresponding point clouds.

[0006] The object of the present invention is achieved through the following technical solution: a method for 3D reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression, the method comprising the following steps:

[0007] (1) Decomposition and prediction of wavelet transform coefficients: obtain the monocular video shot in the dynamic scene and sample the sampling points (x, y, z, t), and use the four-dimensional decomposition wavelet method to divide the wavelet coefficients and wavelet basis functions into three dimensions: (x, t), (y, t), (z, t), where x, y, z represent the three spatial dimensions, and t represents the time dimension, which represents the frame number in the video. For a given point p t , decompose it into three-dimensional p t (m), where m∈U=x,y,z; then three independent multilayer perceptrons are used to predict the wavelet transform coefficients, in each dimension m: Where K is the number of wavelet basis functions;

[0008] (2) Decomposition of wavelet transform basis function optimization: the wavelet basis function Θ t Decomposed into three orthogonal coordinates {Θ t (m)} m∈U , and use a multilayer perceptron W Θ (m) In each frame of monocular video I t They are optimized in the iteration of ; the basis functions are initialized as:

[0009]

[0010] Where T is the number of video frames, parameters i∈[1,M],j∈[0,N],t describes a wavelet basis function in the t-th frame, where k i Controls the frequency of the wavelet basis function, j controls the center offset of the wavelet basis function, and N and M are hyperparameters;

[0011] (3) Decomposition of implicit scene flow prediction based on wavelet transform: predict the implicit scene flow in the dynamic scene expressed by the neural implicit expression and establish the position between the adjacent frames of the monocular video and the current frame I t The correspondence between them includes using a multi-layer perceptron to predict and update the learnable wavelet coefficients and wavelet basis functions of the wavelet transform, and then calculating the implicit scene flow; after decomposing the learnable wavelet transform coefficients and optimizing the basis functions obtained from steps (1) and (2), the motion of the sampling points in adjacent frames is predicted, thereby aggregating the video information of adjacent frames, and performing trajectory prediction (x ′ ,y ′ ,z ′ ,t ′ ) is input into the multi-layer perceptron predicted wavelet transform coefficient in step (1), and the coefficient is used to perform point multiplication with the optimizable wavelet basis function in step (2) to obtain a preliminary estimate of the implicit scene flow;

[0012] (4) Implicit scene flow error correction: The implicit scene flow estimated in step (3) is corrected to obtain the error-corrected scene flow.

[0013]

[0014] in, represents the implicit scene flow obtained by preliminary estimation, Represents the point cloud at time t+1 of the preliminary prediction: H t+1 Represents the point cloud from time t Estimates The credibility weight, K t+1 Indicates the significance of the difference between the measurement and the prediction, ∈ indicates the deviation between the true value and the predicted value;

[0015] Finally, the corrected scene flow is used to predict the point cloud at time t+1

[0016]

[0017] (5) Implicit scene flow adjustment: For the error-corrected scene flow obtained in step (4), shape matching and chamfer distance constraints are introduced to obtain the 3D reconstruction result of the monocular dynamic video.

[0018] Furthermore, after obtaining the monocular video shot in a dynamic scene, preprocessing is required, including: depth prediction and segmentation of the dynamic and static parts of the video frame.

[0019] Furthermore, the chamfer distance constraint is specifically: given the point cloud of frame t and its adjacent frame t′ frame point cloud and the predicted scene flow between the two Recalculate Then calculate the chamfer distance between the two point clouds:

[0020]

[0021] Where dist and index represent For each point in a point cloud, the distance to the nearest point in another point cloud and the number of the nearest point are the sets, so

[0022]

[0023] where dist i It represents the distance between the i-th point in a point cloud and the nearest point in another point cloud, and N is the number of point clouds.

[0024] Furthermore, the shape matching constraint is specifically: for the point cloud P0 and P, the goal is to find the rotation matrix R to minimize the objective function:

[0025]

[0026] where c and c 0 Represent the corresponding point cloud P 0 and the centroid of the P point cloud, p i and They represent the points in the corresponding point cloud, A is the variable to be optimized, represents the rotation and translation matrix, m i Representing the mass of the point, the above formula is simplified to:

[0027]

[0028] in, q i =p i -c, and find the optimal value by taking the derivative. i m i u i q i T )(∑ i m i q i q i T ) -1 =A uq A qq , at this time the objective function is the minimum, where the second term is the pure movement of the symmetric matrix without the rotation component, and we get and then:

[0029]

[0030] Where N is the number of point clouds, and p i denote the points in the estimated and true point clouds, respectively.

[0031] In a second aspect, the present invention further provides a monocular dynamic video 3D reconstruction system based on scene flow prediction and neural implicit expression, the system comprising: a decomposable microwave transform module, an implicit scene flow error correction module, and an implicit scene flow bundle adjustment module;

[0032] The decomposable wavelet transform module is used to preliminarily predict the scene flow, including using a multi-layer perceptron to predict and update the learnable wavelet coefficients and wavelet basis functions related to the traditional wavelet transform, and embedding and encoding the four-dimensional point cloud coordinates to perform a preliminary prediction of the scene flow;

[0033] The implicit scene flow error correction module is used to correct the scene flow prediction error, and uses the Kalman filter method to perform error correction on the preliminary predicted scene flow output by the decomposition and microwave transform module;

[0034] The implicit scene flow bundle adjustment module is used to introduce shape matching and chamfer distance into the corrected scene flow prediction output by the implicit scene flow error correction module to perform physical constraints.

[0035] In the third aspect, the present invention also provides a monocular dynamic video three-dimensional reconstruction device based on scene flow prediction and neural implicit expression, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the monocular dynamic video three-dimensional reconstruction method based on scene flow prediction and neural implicit expression.

[0036] In a fourth aspect, the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for three-dimensional reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression is implemented.

[0037] In a fifth aspect, the present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method for three-dimensional reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression.

[0038] The beneficial effects of the present invention are:

[0039] 1. It integrates the advantages of wavelet transform and neural implicit field, improves the ability to aggregate video information of adjacent frames of monocular video and the feature extraction ability, and effectively improves the training stability and convergence speed of the model.

[0040] 2. A four-dimensional decomposable wavelet method is employed, which involves using a multilayer perceptron to predict and update the learnable wavelet coefficients and wavelet basis functions associated with traditional wavelet transforms, and then calculating the implicit scene flow. Separating the wavelet coefficients and wavelet basis functions into three dimensions enhances model capabilities and improves performance.

[0041] 3. Using wavelet transform instead of cosine transform commonly used in existing technologies to predict implicit scene flow improves the accuracy of implicit scene flow prediction and also speeds up training convergence.

[0042] 4. The Kalman filter method is introduced to correct the error of the implicit scene flow initially predicted by the decomposition microwave transform module, thereby improving the accuracy of scene flow prediction.

[0043] 5. Relying solely on the original RGB rendering constraints may not be sufficient to achieve optimal performance. In order to ensure rigidity and shape consistency, two geometric physical constraints, shape matching and chamfer distance, are introduced to enhance the prediction ability of the model.

[0044] In summary, this paper provides a method and system for 3D reconstruction of monocular dynamic video based on scene flow prediction and neural implicit representation, which can be widely applied in fields such as video surveillance, intelligent driving, and sports analysis. This method can effectively improve the accuracy and robustness of monocular dynamic scene reconstruction, and has promising application prospects and commercial value. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 The present invention provides a flowchart of a method for 3D reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression.

[0047] Figure 2 The overall structural diagram of the model of the present invention.

[0048] Figure 3 This is a schematic diagram of the calculation process of the decomposition microwave transform of the present invention.

[0049] Figure 4 Schematic diagram of the implicit scene flow error correction module of the present invention.

[0050] Figure 5 This is a schematic diagram of a module for aggregating adjacent frame image information and synthesizing a new perspective image in the present invention.

[0051] Figure 6 This is a structural diagram of a monocular dynamic video 3D reconstruction device based on scene flow prediction and neural implicit expression provided by the present invention. DETAILED DESCRIPTION

[0052] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0053] like Figure 1 and Figure 2As shown, the present invention provides a method for three-dimensional reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression, which adopts a four-dimensional decomposition-based wavelet method. This includes using a multi-layer perceptron to predict and update the learnable wavelet coefficients and wavelet basis functions related to the traditional wavelet transform, and then calculating the implicit scene flow. In order to enhance the model capability and improve the performance, the wavelet coefficients and wavelet basis functions are divided into three dimensions; the Kalman filter is used to correct the error of the implicit scene flow, and the accuracy of the implicit scene flow and the training convergence speed are increased; the implicit scene flow beam adjustment is added, and shape matching and chamfer distance constraints are introduced to ensure shape consistency and rigidity to enhance the prediction ability of the model. The method of the present invention specifically includes the following five steps:

[0054] (1) Obtain a monocular video captured in a dynamic scene and perform preprocessing. Based on the input new perspective camera pose, ray sampling is performed on the rays corresponding to the pixels of the output synthesized new perspective, thereby obtaining the coordinates of the sampling points. Preprocessing includes depth prediction and video frame segmentation. Specifically, depth prediction uses the method described in the paper Consistent Depth of Moving Objects in Video, and video frame segmentation is achieved using Segment Anything segmentation.

[0055] (2) Decomposition can predict implicit scene flow using wavelet transform: Figure 3 As shown, in order to predict the implicit scene flow in the neural implicit expression dynamic scene and establish the position in the adjacent frame and the current frame I t The method adopts a four-dimensional decomposition of learnable wavelet method. This involves using a multi-layer perceptron to predict and update the learnable wavelet coefficients and learnable wavelet basis functions related to the traditional wavelet transform, and then calculating the implicit scene flow. In order to enhance the model capability and improve the performance, the wavelet coefficients and wavelet basis functions are divided into three dimensions: (x, t), (y, t), (z, t). For a given point p t , we decompose it into three-dimensional p t (m), where m∈U={x,y,z}. We then use three independent multilayer perceptrons to predict the transformation coefficients, where in each dimension m we have: Where K is the number of wavelet basis functions. The present invention also realizes the decomposition of wavelet transform basis function optimization, and the wavelet basis function Θ t Decomposed into three orthogonal coordinates {Θ t (m)} m∈U , and use a multilayer perceptron W Θ (m) In each frame of monocular video I t They are optimized in the iteration of ; the basis functions are initialized as:

[0056]

[0057] Where T is the number of video frames, parameters i∈[1,M],j∈[0,N],t describes a wavelet basis function in the t-th frame, where k i Controls the frequency of the wavelet basis function, j controls the center offset of the wavelet basis function, and N and M are hyperparameters;

[0058] After obtaining the decomposable wavelet transform coefficients and optimized basis functions, the motion of the sampling points in adjacent frames is predicted, thereby gathering the video information of adjacent frames and performing trajectory prediction (x ′ ,y ′ ,z ′ ,t ′ ) is input into the multi-layer perceptron to predict the wavelet transform coefficient, and the coefficient is used to perform point multiplication with the optimizable wavelet basis function to obtain a preliminary estimate of the implicit scene flow;

[0059] (3) Use Kalman filtering to correct the implicit scene flow error: Figure 4 As shown, assuming that the estimated value and the measured value m t+1 There is a linear mapping between the deviation b, the formula is Among them, W m Represents the linear mapping weight, and Y represents the Boolean value "True". Since it involves matrix multiplication, an unbiased multilayer perceptron W is used. conf Mapping the confidence H of the estimated value to obtain Where F represents the Boolean value "False". The parameter K is a weight matrix whose values ​​range from 0 to 1, indicating the degree of influence of the difference between the measured value and the predicted value. Similarly, using the linear projection W K To map this difference, denoted as K t+1 ~W K In addition, considering the accumulation of errors over time, the learnable state update equation is modified as follows: So we get the error correction of the initial predicted implicit scene flow, and get the scene flow after error correction

[0060]

[0061] in, represents the implicit scene flow obtained by preliminary estimation, Represents the point cloud at time t+1 of the preliminary prediction: H t+1 Represents the point cloud from time t Estimates The credibility weight, K t+1Indicates the significance of the difference between the measurement and the prediction, ∈ indicates the deviation between the true value and the predicted value;

[0062] Finally, the corrected scene flow is used to predict the point cloud at time t+1

[0063]

[0064] (4) Adding Implicit Scene Flow Adjustment: For the error-corrected scene flow obtained in the previous module, there is a lack of physical constraints to ensure rigidity and shape consistency. Relying solely on the original RGB rendering constraints may not be sufficient to achieve optimal performance. Therefore, two geometric constraints, shape matching and chamfer distance, are introduced to enhance the model's prediction ability.

[0065] (5) Figure 5 As shown in the figure, the input video frame is passed through the feature extraction network to obtain a feature vector map. According to the predicted implicit scene flow, features are aggregated from the feature vectors of adjacent frames to obtain image information of adjacent frames in the video data. The feature vectors obtained by aggregating all sampling points on a ray are output using a Ray Transform module to obtain color and density outputs, and the new perspective image is synthesized using the volume rendering module to obtain the 3D reconstruction result of the monocular dynamic video.

[0066] In this paper, the model parameters are iteratively adjusted using a stochastic gradient descent optimization algorithm to improve model accuracy and generalization. As shown in Table 1, the present invention outperforms existing technologies in all parameters of novel perspective synthesis. Compared to the cosine-variant implicit scene flow prediction implemented by DynIBaR, the present invention improves the implicit scene flow prediction accuracy and accelerates the convergence rate during training.

[0067] Table 1 Test index record table

[0068]

[0069]

[0070] On the other hand, the present invention also provides a monocular dynamic video three-dimensional reconstruction system based on scene flow prediction and neural implicit expression, the system including: a decomposition-dependent microwave transform module, an implicit scene flow error correction module, and an implicit scene flow bundle adjustment module; the specific implementation process of each module refers to the embodiment steps of the method for three-dimensional reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression provided above by the present invention.

[0071] The decomposable wavelet transform module is used to preliminarily predict the scene flow, which includes using a multi-layer perceptron to predict and update the learnable wavelet coefficients and wavelet basis functions related to the traditional wavelet transform, and embedding and encoding the four-dimensional point cloud coordinates to perform a preliminary prediction of the scene flow;

[0072] The implicit scene flow error correction module is used to correct the scene flow prediction error, and uses the Kalman filter method to perform error correction on the preliminary predicted scene flow output by the wavelet transform module;

[0073] The implicit scene flow bundle adjustment module is used to impose physical constraints on the corrected scene flow prediction output by the implicit scene flow error correction module, and introduces two constraints: shape matching and chamfer distance.

[0074] Corresponding to the aforementioned embodiment of monocular dynamic video three-dimensional reconstruction based on scene flow prediction and neural implicit expression, the present invention also provides an embodiment of a monocular dynamic video three-dimensional reconstruction device based on scene flow prediction and neural implicit expression.

[0075] See also Figure 6 An embodiment of the present invention provides a monocular dynamic video 3D reconstruction device based on scene flow prediction and neural implicit expression, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a monocular dynamic video 3D reconstruction method based on scene flow prediction and neural implicit expression in the above embodiment.

[0076] The embodiment of the monocular dynamic video three-dimensional reconstruction device based on scene flow prediction and neural implicit expression provided by the present invention can be applied to any device with data processing capabilities, and the any device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 6 As shown in FIG, a hardware structure diagram of a monocular dynamic video 3D reconstruction device based on scene flow prediction and neural implicit expression provided by the present invention is located in any device with data processing capabilities, except Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0077] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0078] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0079] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a method for three-dimensional reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression in the above embodiment is implemented.

[0080] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0081] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modifications and changes made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A method for 3D reconstruction of monocular dynamic video based on scene flow prediction and neural implicit representation, characterized in that: The method comprises the following steps: (1) Decomposition and prediction of wavelet transform coefficients: obtain the monocular video shot in the dynamic scene and sample the sampling points (x, y, z, t), and use the four-dimensional decomposition wavelet method to divide the wavelet coefficients and wavelet basis functions into three dimensions: (x, t), (y, t), (z, t), where x, y, z represent the three spatial dimensions, and t represents the time dimension, which represents the frame number in the video. For a given point p t , decompose it into three-dimensional p t (m), where m∈U=x,y,z; then three independent multilayer perceptrons are used to predict the wavelet transform coefficients, in each dimension m: Where K is the number of wavelet basis functions; (2) Decomposition of wavelet transform basis function optimization: the wavelet basis function Θ t Decomposed into three orthogonal coordinates {Θ t (m)} m∈U , and use a multilayer perceptron W Θ (m) In each frame of monocular video I t They are optimized in the iteration of ; the basis functions are initialized as: Where T is the number of video frames, parameters i∈[1,M],j∈[0,N],t describes a wavelet basis function in the t-th frame, where k i Controls the frequency of the wavelet basis function, j controls the center offset of the wavelet basis function, and N and M are hyperparameters; (3) Decomposition of implicit scene flow prediction based on wavelet transform: predict the implicit scene flow in the dynamic scene expressed by the neural implicit expression and establish the position between the adjacent frames of the monocular video and the current frame I t The correspondence between them includes using a multi-layer perceptron to predict and update the learnable wavelet coefficients and wavelet basis functions of the wavelet transform, and then calculating the implicit scene flow; after decomposing the learnable wavelet transform coefficients and optimizing the basis functions obtained from steps (1) and (2), the motion of the sampling points in adjacent frames is predicted, thereby aggregating the video information of adjacent frames, and performing trajectory prediction (x ′ ,y ′ ,z ′ ,t ′ ) is input into the multi-layer perceptron predicted wavelet transform coefficient in step (1), and the coefficient is used to perform point multiplication with the optimizable wavelet basis function in step (2) to obtain a preliminary estimate of the implicit scene flow; (4) Implicit scene flow error correction: The implicit scene flow estimated in step (3) is corrected to obtain the error-corrected scene flow. in, represents the implicit scene flow obtained by preliminary estimation, Represents the point cloud at time t+1 of the preliminary prediction: H t+1 Represents the point cloud from time t Estimates The credibility weight, K t+1 Indicates the significance of the difference between the measurement and the prediction, ∈ indicates the deviation between the true value and the predicted value; Finally, the corrected scene flow is used to predict the point cloud at time t+1 (5) Implicit scene flow adjustment: For the error-corrected scene flow obtained in step (4), shape matching and chamfer distance constraints are introduced to obtain the 3D reconstruction result of the monocular dynamic video.

2. The method for 3D reconstruction of monocular dynamic video based on scene flow prediction and neural implicit representation according to claim 1, characterized in that: After obtaining the monocular video shot in a dynamic scene, preprocessing is required, including depth prediction and segmentation of the dynamic and static parts of the video frame.

3. The method for 3D reconstruction of monocular dynamic video based on scene flow prediction and neural implicit representation according to claim 1, characterized in that: The chamfer distance constraint is specifically: given the t-th frame point cloud and its adjacent frame t′ frame point cloud and the predicted scene flow between the two Recalculate Then calculate the chamfer distance between the two point clouds: Where dist and index represent For each point in a point cloud, the distance to the nearest point in another point cloud and the number of the nearest point are the sets, so where dist i It represents the distance between the i-th point in a point cloud and the nearest point in another point cloud, and N is the number of point clouds.

4. The method for 3D reconstruction of monocular dynamic video based on scene flow prediction and neural implicit representation according to claim 1, characterized in that: The shape matching constraint is specifically: for the point cloud P 0 and P, the goal is to find the rotation matrix R to minimize the objective function: where c and c 0 Represent the corresponding point cloud P 0 and the centroid of the P point cloud, p i and They represent the points in the corresponding point cloud, A is the variable to be optimized, represents the rotation and translation matrix, m i Representing the mass of the point, the above formula is simplified to: in, q i =p i -c, and find the optimal value by taking the derivative. i m i u i q i T )(∑ i m i q i q i T ) -1 =A uq A qq , at this time the objective function is the minimum, where the second term is the pure movement of the symmetric matrix without the rotation component, and we get and then: Where N is the number of point clouds, and p i denote the points in the estimated and true point clouds, respectively.

5. A monocular dynamic video 3D reconstruction system based on scene flow prediction and neural implicit representation that implements the method according to any one of claims 1 to 4, characterized in that: The system includes: a decomposable microwave transform module, an implicit scene flow error correction module and an implicit scene flow bundle adjustment module; The decomposable wavelet transform module is used to preliminarily predict the scene flow, including using a multi-layer perceptron to predict and update the learnable wavelet coefficients and wavelet basis functions related to the traditional wavelet transform, and embedding and encoding the four-dimensional point cloud coordinates to perform a preliminary prediction of the scene flow; The implicit scene flow error correction module is used to correct the scene flow prediction error, and uses the Kalman filter method to perform error correction on the preliminary predicted scene flow output by the decomposition and microwave transform module; The implicit scene flow bundle adjustment module is used to introduce shape matching and chamfer distance into the corrected scene flow prediction output by the implicit scene flow error correction module to perform physical constraints.

6. A monocular dynamic video 3D reconstruction device based on scene flow prediction and neural implicit representation, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it implements a monocular dynamic video three-dimensional reconstruction method based on scene flow prediction and neural implicit expression according to any one of claims 1 to 4.

7. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a monocular dynamic video three-dimensional reconstruction method based on scene flow prediction and neural implicit expression is implemented as described in any one of claims 1 to 4.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements a monocular dynamic video three-dimensional reconstruction method based on scene flow prediction and neural implicit expression as described in any one of claims 1 to 4.

Images