Monocular dynamic video three-dimensional reconstruction method, system and device based on scene flow prediction and neural implicit expression

Abstract

The invention discloses a monocular dynamic video three-dimensional reconstruction method, a monocular dynamic video three-dimensional reconstruction system and a monocular dynamic video three-dimensional reconstruction device based on scene flow prediction and neural implicit expression. According to the method, a new decomposition wavelet transform (FDWT) method is designed, a scene stream between implicit point clouds in a video is roughly estimated in a frequency domain, four-dimensional point cloud coordinates are subjected to embedded coding and then are sent to a decomposition wavelet transform module for point cloud motion prediction, and preliminary prediction of the scene stream is realized. An implicit scene flow error filtering (ISEF) model focusing on a time-space domain is designed and used for scene flow correction, initial prediction scene flow output by decomposition wavelet transform is corrected in the time-space domain, and the prediction accuracy of the model is improved. And finally, converging video information of adjacent frames by using the modified neural implicit expression, and performing three-dimensional reconstruction of a dynamic video. The method can be applied to the fields of three-dimensional modeling of mobile terminals (mobile phones and the like) and the like.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method, system and device for three-dimensional reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression. Background Art

[0002] In the classic video new view synthesis task, the main goal is to reconstruct the 3D scene given a monocular video and synthesize a new view for each frame. To achieve this, images with different camera poses are usually used to supplement the features of the current frame. Traditional multi-view geometry theory requires multi-view input at the same time; when combined with image-based rendering methods, high-fidelity scene new view synthesis can be achieved using representations such as multi-layer perceptrons, point clouds, and voxels.

[0003] Aggregating features from adjacent frames is crucial for single-frame novel view synthesis, because occluded information of the target frame may appear in adjacent frames and if each frame is rendered independently, the occluded information may appear in the adjacent frames. Figure 1 However, neural implicit expression brings obstacles to dynamic scene modeling, because the scene is modeled as a static continuous distribution, which cannot aggregate the image information and motion information of adjacent frames. Usually, artifacts and other poor image quality problems will occur in the moving parts. Summary of the invention

[0004] The present invention aims to address the deficiencies in the prior art and to provide a method, system and device for 3D reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression.

[0005] As mentioned in the background technology, the existing technology still has a lot of shortcomings in solving the synthesis of new perspectives for dynamic scenes in monocular videos, and it becomes crucial to integrate signals from the spatiotemporal and frequency domains. The present invention proposes a new decomposable wavelet transform, mainly in the frequency domain, to roughly estimate the scene flow between implicit point clouds in the video, and proposes an implicit scene flow error filter model focusing on the spatiotemporal domain for scene flow correction. In addition, shape matching and point cloud registration are proposed as rigid constraints to improve the predicted implicit scene flow and achieve high-quality new perspective synthesis. New views can be synthesized by aggregating adjacent frame features into the current frame through the predicted corresponding point clouds.

[0006] The object of the present invention is achieved through the following technical solution: a method for 3D reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression, the method comprising the following steps:

[0007] (1) Decomposition and prediction of wavelet transform coefficients: The monocular video shot in the dynamic scene is sampled to obtain the sampling points (x, y, z, t), and the wavelet coefficients and wavelet basis functions are divided into three dimensions using the four-dimensional decomposition wavelet method: (x, t), (y, t), (z, t), where x, y, z represent the three spatial dimensions, and t represents the time dimension, which represents the frame number in the video. For a given point p t , decompose it into three-dimensional p t (m), where m∈U=x,y,z; then three independent multilayer perceptrons are used to predict the wavelet transform coefficients, in each dimension m: Where K is the number of wavelet basis functions;

[0008] (2) Decomposition can be used to optimize the wavelet transform basis function: t Decomposed into three orthogonal coordinates {Θ t (m)} m∈U , and use a multi-layer perceptron W Θ (m) In each frame of monocular video I t They are optimized in iterations of ; the basis functions are initialized as:

[0009]

[0010] Where T is the number of video frames, the parameters i∈[1,M],j∈[0,N],t describe a wavelet basis function in the tth frame, where k i Controls the frequency of the wavelet basis function, j controls the center offset of the wavelet basis function, and N and M are hyperparameters;

[0011] (3) Decomposition of implicit scene flow prediction based on wavelet transform: predicting the implicit scene flow in the dynamic scene expressed by the neural implicit expression and establishing the position between the adjacent frames of the monocular video and the current frame I t The correspondence between the two includes using a multi-layer perceptron to predict and update the learnable wavelet coefficients and wavelet basis functions of the wavelet transform, and then calculating the implicit scene flow; after decomposing the learnable wavelet transform coefficients and optimizing the basis functions obtained from steps (1) and (2), predicting the motion of the sampling points in adjacent frames, thereby aggregating the video information of adjacent frames, and performing trajectory prediction (x ′ ,y ′ ,z ′ ,t ′ ) is input into the multi-layer perceptron predicted wavelet transform coefficient in step (1), and the coefficient is used to perform point multiplication with the optimizable wavelet basis function in step (2) to obtain a preliminary estimated implicit scene flow;

[0012] (4) Implicit scene flow error correction: The implicit scene flow estimated initially in step (3) is corrected to obtain the scene flow after error correction.

[0013]

[0014] in, represents the implicit scene flow obtained by preliminary estimation, Represents the point cloud of the initial prediction at time t+1: H t+1 Represents the point cloud from time t Estimates The credibility weight, K t+1 Indicates the significance of the difference between the measurement and the prediction, ∈ indicates the deviation between the true value and the predicted value;

[0015] Finally, the corrected scene flow is used to predict the point cloud at time t+1

[0016]

[0017] (5) Implicit scene flow adjustment: For the error-corrected scene flow obtained in step (4), shape matching and chamfer distance constraints are introduced to obtain the 3D reconstruction result of the monocular dynamic video.

[0018] Furthermore, after obtaining the monocular video shot in the dynamic scene, preprocessing is required, including: depth prediction and segmentation of the dynamic and static parts of the video frame.

[0019] Furthermore, the chamfer distance constraint is specifically: given the point cloud of the tth frame And its adjacent frame t′ frame point cloud and the predicted scene flow between the two Recalculate So calculate the chamfer distance between the two point clouds:

[0020]

[0021] Where dist and index represent For each point in a point cloud, the distance to the nearest point in another point cloud and the number of the nearest point are the set, so

[0022]

[0023] where dist i It represents the distance from the i-th point in a point cloud to the nearest point in another point cloud, and N is the number of point clouds.

[0024] Furthermore, the shape matching constraint is specifically: for the point cloud P0 and P, the goal is to find the rotation matrix R to minimize the objective function:

[0025]

[0026] where c and c 0 Respectively represent the corresponding point cloud P 0 and the centroid of the P point cloud, p i and They represent the points in the corresponding point cloud, A is the variable to be optimized, represents the rotation and translation matrix, m i Representing the mass of the point, the above formula is simplified to:

[0027]

[0028] in, q i =p i -c, and find the optimal value by taking the derivative. i m i u i q i T )(∑ i m i q i q i T ) -1 =A uq A qq , at this time the objective function is the minimum, where the second term is the pure movement of the symmetric matrix without the rotation component, and we get and then:

[0029]

[0030] Where N is the number of point clouds, and p i denote the points in the estimated and true point clouds, respectively.

[0031] In a second aspect, the present invention also provides a monocular dynamic video 3D reconstruction system based on scene flow prediction and neural implicit expression, the system comprising: a decomposable microwavelet transform module, an implicit scene flow error correction module and an implicit scene flow bundle adjustment module;

[0032] The decomposable wavelet transform module is used to preliminarily predict the scene flow, including using a multi-layer perceptron to predict and update learnable wavelet coefficients and wavelet basis functions related to traditional wavelet transform, and embedding and encoding the four-dimensional point cloud coordinates to perform a preliminary prediction of the scene flow;

[0033] The implicit scene flow error correction module is used to correct the scene flow prediction error, and uses the Kalman filter method to perform error correction on the preliminary predicted scene flow output by the decomposition and microwave transform module;

[0034] The implicit scene flow adjustment module is used to introduce shape matching and chamfer distance to perform physical constraints on the corrected scene flow prediction output by the implicit scene flow error correction module.

[0035] In the third aspect, the present invention also provides a monocular dynamic video three-dimensional reconstruction device based on scene flow prediction and neural implicit expression, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the monocular dynamic video three-dimensional reconstruction method based on scene flow prediction and neural implicit expression.

[0036] In a fourth aspect, the present invention further provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the method for three-dimensional reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression.

[0037] In a fifth aspect, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the method for three-dimensional reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression.

[0038] The beneficial effects of the present invention are:

[0039] 1. It combines the advantages of wavelet transform and neural implicit field, improves the ability to aggregate the information of adjacent frames of monocular video and the ability to extract features, and effectively improves the training stability and convergence speed of the model.

[0040] 2. Adopt a four-dimensional decomposable wavelet method, which includes using a multi-layer perceptron to predict and update the learnable wavelet coefficients and wavelet basis functions associated with the traditional wavelet transform, and then calculate the implicit scene flow. The wavelet coefficients and wavelet basis functions are divided into three dimensions to enhance the model capabilities to improve performance.

[0041] 3. Using wavelet transform to replace the cosine transform commonly used in the prior art to predict the implicit scene flow improves the accuracy of implicit scene flow prediction and also improves the speed of training convergence.

[0042] 4. The Kalman filter method is introduced to correct the error of the implicit scene flow preliminarily predicted by the decomposition and microwave transform module, thereby improving the accuracy of scene flow prediction.

[0043] 5. Relying solely on the original RGB rendering constraints may not be sufficient to achieve optimal performance. In order to ensure rigidity and shape consistency, two geometric physical constraints, shape matching and chamfer distance, are introduced to enhance the prediction ability of the model.

[0044] In summary, the present invention provides a method and system for 3D reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression, which can be widely used in video surveillance, intelligent driving, sports analysis and other fields. This method can effectively improve the accuracy and robustness of monocular video dynamic scene reconstruction, and has good application prospects and commercial value. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 The present invention provides a flowchart of a method for 3D reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression.

[0047] Figure 2 The overall structural diagram of the model of the present invention.

[0048] Figure 3 This is a schematic diagram of the calculation process of the decomposition microwave transform of the present invention.

[0049] Figure 4 Schematic diagram of the implicit scene flow error correction module of the present invention.

[0050] Figure 5 This is a schematic diagram of a module for aggregating adjacent frame image information and synthesizing a new viewing angle image in the present invention.

[0051] Figure 6 A structural diagram of a monocular dynamic video 3D reconstruction device based on scene flow prediction and neural implicit expression provided by the present invention. DETAILED DESCRIPTION

[0052] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0053] like Figure 1 and Figure 2As shown, the present invention provides a method for three-dimensional reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression, which adopts a four-dimensional decomposable wavelet method. This includes using a multi-layer perceptron to predict and update the learnable wavelet coefficients and wavelet basis functions related to the traditional wavelet transform, and then calculating the implicit scene flow. In order to enhance the model capability to improve performance, the wavelet coefficients and wavelet basis functions are divided into three dimensions; the Kalman filter is used to correct the error of the implicit scene flow, and the accuracy of the implicit scene flow and the training convergence speed are increased; the implicit scene flow beam adjustment is added, and shape matching and chamfer distance constraints are introduced to ensure shape consistency and rigidity to enhance the prediction ability of the model. The method of the present invention specifically includes the following 5 steps:

[0054] (1) Obtain a monocular video shot in a dynamic scene and perform preprocessing. According to the input new perspective camera pose, perform light sampling on the light corresponding to the pixel of the output synthetic new perspective, and obtain the sampling point coordinates; preprocessing includes depth prediction and video frame dynamic and static segmentation. The specific depth prediction is based on the method in the paper Consistent Depth of Moving Objects in Video, and the video frame dynamic and static segmentation is obtained using Segment Anything segmentation.

[0055] (2) Decomposition can predict implicit scene flow using wavelet transform: Figure 3 As shown, in order to predict the implicit scene flow in the neural implicit expression dynamic scene and establish the position in the adjacent frame and the current frame I t The method adopts a four-dimensional decomposable wavelet method. This involves using a multi-layer perceptron to predict and update the learnable wavelet coefficients and learnable wavelet basis functions associated with the traditional wavelet transform, and then calculating the implicit scene flow. In order to enhance the model capability to improve performance, the wavelet coefficients and wavelet basis functions are divided into three dimensions: (x, t), (y, t), (z, t). For a given point p t , we decompose it into three-dimensional p t (m), where m∈U={x,y,z}. We then use three independent multilayer perceptrons to predict the transformation coefficients, where in each dimension m we have: Where K is the number of wavelet basis functions. The present invention also achieves the optimization of the decomposition of the wavelet transform basis function, and the wavelet basis function Θ t Decomposed into three orthogonal coordinates {Θ t (m)} m∈U , and use a multi-layer perceptron W Θ (m) In each frame of monocular video I t They are optimized in iterations of ; the basis functions are initialized as:

[0056]

[0057] Where T is the number of video frames, the parameters i∈[1,M],j∈[0,N],t describe a wavelet basis function in the tth frame, where k i Controls the frequency of the wavelet basis function, j controls the center offset of the wavelet basis function, and N and M are hyperparameters;

[0058] After obtaining the decomposable wavelet transform coefficients and optimizing the basis functions, the motion of the sampling points in adjacent frames is predicted, thereby gathering the video information of adjacent frames and performing trajectory prediction (x ′ ,y ′ ,z ′ ,t ′ ) is input into a multi-layer perceptron to predict the wavelet transform coefficient, and the coefficient is used to perform a point multiplication with the optimizable wavelet basis function to obtain a preliminary estimate of the implicit scene flow;

[0059] (3) Use Kalman filtering to correct the error of implicit scene flow: Figure 4 As shown, assuming that the estimated value and the measured value m t+1 There is a linear mapping between and deviation b, the formula is Among them, W m Represents the linear mapping weight, and Y represents the Boolean value "True". Since it involves matrix multiplication, an unbiased multilayer perceptron W is used conf As the confidence H of the estimated value, we can get Where F represents the Boolean value "False". The parameter K is a weight matrix whose values ​​range from 0 to 1, indicating the degree of influence of the difference between the measured value and the predicted value. Similarly, using the linear projection W K To map this difference, denoted as K t+1 ~W K In addition, considering the accumulation of errors over time, the learnable state update equation is modified as follows: So we get the error correction of the initial prediction implicit scene flow, and get the scene flow after error correction

[0060]

[0061] in, represents the implicit scene flow obtained by preliminary estimation, Represents the point cloud of the initial prediction at time t+1: H t+1 Represents the point cloud from time t Estimates The credibility weight, Kt+1 Indicates the significance of the difference between the measurement and the prediction, ∈ indicates the deviation between the true value and the predicted value;

[0062] Finally, the corrected scene flow is used to predict the point cloud at time t+1

[0063]

[0064] (4) Adding implicit scene flow adjustment: For the error-corrected scene flow obtained in the previous module, there is a lack of physical constraints to ensure rigidity and shape consistency. Relying solely on the original RGB rendering constraints may not be sufficient to achieve optimal performance. Therefore, two geometric constraints, shape matching and chamfer distance, are introduced to enhance the prediction ability of the model.

[0065] (5) Figure 5 As shown in the figure, the input video frame is passed through the feature extraction network to obtain a feature vector map, and the features are aggregated from the feature vectors of adjacent frames according to the predicted implicit scene flow, so as to obtain the image information of adjacent frames in the video data, and the feature vector obtained by aggregating all the sampling points on a ray is used to obtain the color and density output using a Ray Transform module, so as to use the volume rendering module to synthesize the new perspective image and obtain the 3D reconstruction result of the monocular dynamic video.

[0066] In the present invention, the model parameters are iteratively adjusted through the stochastic gradient descent optimization algorithm to improve the accuracy and generalization ability of the model. As shown in Table 1, it can be seen that the present invention is superior to the existing technology in various index parameters of new perspective synthesis, and compared with the cosine change implicit scene flow prediction implemented by DynIBaR, the prediction accuracy of the implicit scene flow is improved, and the convergence speed in the training process is also accelerated.

[0067] Table 1 Test index record table

[0068]

[0069]

[0070] On the other hand, the present invention also provides a monocular dynamic video three-dimensional reconstruction system based on scene flow prediction and neural implicit expression, the system comprising: a decomposable microwave transform module, an implicit scene flow error correction module, and an implicit scene flow bundle adjustment module; the specific implementation process of each module refers to the embodiment steps of the method for three-dimensional reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression provided by the present invention.

[0071] The decomposable wavelet transform module is used to preliminarily predict the scene flow, which includes using a multi-layer perceptron to predict and update the learnable wavelet coefficients and wavelet basis functions related to the traditional wavelet transform, and embedding and encoding the four-dimensional point cloud coordinates to perform a preliminary prediction of the scene flow;

[0072] The implicit scene flow error correction module is used to correct the scene flow prediction error, and uses the Kalman filter method to perform error correction on the preliminary predicted scene flow output by the wavelet transform module;

[0073] The implicit scene flow adjustment module is used to impose physical constraints on the corrected scene flow prediction output by the implicit scene flow error correction module, and introduces two constraints: shape matching and chamfer distance.

[0074] Corresponding to the aforementioned embodiment of monocular dynamic video three-dimensional reconstruction based on scene flow prediction and neural implicit expression, the present invention also provides an embodiment of a monocular dynamic video three-dimensional reconstruction device based on scene flow prediction and neural implicit expression.

[0075] See also Figure 6 An embodiment of the present invention provides a monocular dynamic video 3D reconstruction device based on scene flow prediction and neural implicit expression, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a monocular dynamic video 3D reconstruction method based on scene flow prediction and neural implicit expression in the above embodiment.

[0076] An embodiment of a monocular dynamic video 3D reconstruction device based on scene flow prediction and neural implicit expression provided by the present invention can be applied to any device with data processing capability, and the device with data processing capability can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capability in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 6 As shown in FIG. 1 , a hardware structure diagram of a monocular dynamic video 3D reconstruction device based on scene flow prediction and neural implicit expression provided by the present invention is provided in any device with data processing capability, except Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0077] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0078] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0079] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a method for three-dimensional reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression in the above embodiment is implemented.

[0080] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0081] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A method for 3D reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression, characterized in that: The method comprises the following steps: (1) Decomposition and prediction of wavelet transform coefficients: The monocular video shot in the dynamic scene is sampled to obtain the sampling points (x, y, z, t), and the wavelet coefficients and wavelet basis functions are divided into three dimensions using the four-dimensional decomposition wavelet method: (x, t), (y, t), (z, t), where x, y, z represent the three spatial dimensions, and t represents the time dimension, which represents the frame number in the video. For a given point p t , decompose it into three-dimensional p t (m), where m∈U=x,y,z; then three independent multilayer perceptrons are used to predict the wavelet transform coefficients, in each dimension m: Where K is the number of wavelet basis functions; (2) Decomposition can be used to optimize the wavelet transform basis function: t Decomposed into three orthogonal coordinates {Θ t (m)} m∈U , and use a multi-layer perceptron W Θ (m) In each frame of monocular video I t They are optimized in iterations of ; the basis functions are initialized as: Where T is the number of video frames, the parameters i∈[1,M],j∈[0,N],t describe a wavelet basis function in the tth frame, where k i Controls the frequency of the wavelet basis function, j controls the center offset of the wavelet basis function, and N and M are hyperparameters; (3) Decomposition of implicit scene flow prediction based on wavelet transform: predicting the implicit scene flow in the dynamic scene expressed by the neural implicit expression and establishing the position between the adjacent frames of the monocular video and the current frame I t The correspondence between the two includes using a multi-layer perceptron to predict and update the learnable wavelet coefficients and wavelet basis functions of the wavelet transform, and then calculating the implicit scene flow; after decomposing the learnable wavelet transform coefficients and optimizing the basis functions obtained from steps (1) and (2), predicting the motion of the sampling points in adjacent frames, thereby aggregating the video information of adjacent frames, and performing trajectory prediction (x ′ ,y ′ ,z ′ ,t ′ ) is input into the multi-layer perceptron predicted wavelet transform coefficient in step (1), and the coefficient is used to perform point multiplication with the optimizable wavelet basis function in step (2) to obtain a preliminary estimated implicit scene flow; (4) Implicit scene flow error correction: The implicit scene flow estimated initially in step (3) is corrected to obtain the scene flow after error correction. in, represents the implicit scene flow obtained by preliminary estimation, Represents the point cloud of the initial prediction at time t+1: H t+1 Represents the point cloud from time t Estimates The credibility weight, K t+1 Indicates the significance of the difference between the measurement and the prediction, ∈ indicates the deviation between the true value and the predicted value; Finally, the corrected scene flow is used to predict the point cloud at time t+1 (5) Implicit scene flow adjustment: For the error-corrected scene flow obtained in step (4), shape matching and chamfer distance constraints are introduced to obtain the 3D reconstruction result of the monocular dynamic video.

2. The method for 3D reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression according to claim 1, characterized in that: After obtaining the monocular video shot in a dynamic scene, preprocessing is required, including: depth prediction and segmentation of the dynamic and static parts of the video frame.

3. The method for 3D reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression according to claim 1, characterized in that: The chamfer distance constraint is specifically: given the t-th frame point cloud And its adjacent frame t′ frame point cloud and the predicted scene flow between the two Recalculate So calculate the chamfer distance between the two point clouds: Where dist and index represent For each point in a point cloud, the distance to the nearest point in another point cloud and the number of the nearest point are the set, so where dist i It represents the distance from the i-th point in a point cloud to the nearest point in another point cloud, and N is the number of point clouds.

4. The method for 3D reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression according to claim 1, characterized in that: The shape matching constraint is specifically: for the point cloud P 0 and P, the goal is to find the rotation matrix R to minimize the objective function: where c and c 0 Respectively represent the corresponding point cloud P 0 and the centroid of the P point cloud, p i and They represent the points in the corresponding point cloud, A is the variable to be optimized, represents the rotation and translation matrix, m i Represents the mass of the point, simplifying the above formula to: in, q i =p i -c, and find the optimal value by taking the derivative. i m i u i q i T )(∑ i m i q i q i T ) -1 =A uq A qq , at this time the objective function is the minimum, where the second term is the pure movement of the symmetric matrix without the rotation component, and we get and then: Where N is the number of point clouds, and p i denote the points in the point cloud in the estimation and the true value respectively.

5. A monocular dynamic video 3D reconstruction system based on scene flow prediction and neural implicit expression that implements the method according to any one of claims 1 to 4, characterized in that: The system comprises: a decomposable microwave transform module, an implicit scene flow error correction module and an implicit scene flow beam adjustment module; The decomposable wavelet transform module is used to preliminarily predict the scene flow, including using a multi-layer perceptron to predict and update learnable wavelet coefficients and wavelet basis functions related to traditional wavelet transform, and embedding and encoding the four-dimensional point cloud coordinates to perform a preliminary prediction of the scene flow; The implicit scene flow error correction module is used to correct the scene flow prediction error, and uses the Kalman filter method to perform error correction on the preliminary predicted scene flow output by the decomposition and microwave transform module; The implicit scene flow adjustment module is used to introduce shape matching and chamfer distance to perform physical constraints on the corrected scene flow prediction output by the implicit scene flow error correction module.

6. A monocular dynamic video 3D reconstruction device based on scene flow prediction and neural implicit expression, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a monocular dynamic video three-dimensional reconstruction method based on scene flow prediction and neural implicit expression is implemented as described in any one of claims 1-4.

7. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a monocular dynamic video three-dimensional reconstruction method based on scene flow prediction and neural implicit expression is implemented as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, a method for three-dimensional reconstruction of monocular dynamic video based on scene flow prediction and neural implicit expression as described in any one of claims 1 to 4 is implemented.

Images