A single image 3D reconstruction method based on deep learning video supervision

Abstract

The present invention provides a single-image 3D reconstruction method based on deep learning video supervision, comprising the following steps: 1) constructing an object pose prediction module: according to the object in the input image, the position of the camera shooting relative to the object can be obtained; 2) constructing an object 3D shape estimation module: according to the input single object image, its 3D point cloud is obtained by iteratively optimizing the loss prediction; 3) constructing a multi-frame shape fusion module: the video line sequence is input into the above two modules in parallel, and the camera pose and 3D shape prediction of a single frame can be obtained, and more accurate prediction can be obtained through multi-frame weight fusion and consistency constraints and smoothness constraints; 4) overall training framework: including three stages of data preprocessing, model framework training and testing. The method of the present invention realizes end-to-end 3D reconstruction, can use video sequences for training, and can predict its 3D point cloud using only a single image in the test stage.

Description

[0001] Technology Neighborhood

[0002] The present invention belongs to the field of three-dimensional reconstruction technology, and specifically relates to a single-image three-dimensional reconstruction method based on deep learning video supervision. Background Art

[0003] In recent years, with the development of deep learning, the solution of computer vision problems has been greatly developed. Recently, various two-dimensional image processing technologies have been gradually improved and applied to three-dimensional problems, and the reconstruction of the three-dimensional shape of objects has also become one of the hot issues. Many previous methods require complete three-dimensional model data for supervision, but such data is scarce and the acquisition process is complicated and expensive. As a result, reconstruction methods using many multi-images and single images have emerged. Due to the weakening of supervision information, the accuracy of local details will be reduced and the ambiguity of the object's perspective will be ambiguous. At the same time, the multi-image method needs to rely on the correlation between training input images, which is difficult to obtain in large quantities in real scenes. Due to the reduction in the total amount of information, single images not only need to introduce other additional information, but also lead to a significant reduction in reconstruction accuracy.

[0004] To solve the problem of 3D reconstruction, the main technical ideas at present are as follows: Reference 1: JLSch€onbergerandJ.-M.Frahm, "Structure-from-motion revisited," in Proc.IEEEConf.Comput.Vis.Pattern Recognit., 2016, pp.4104–4113, which uses traditional optimization methods for reconstruction. Reference 2: H.Fan, H.Su, and L.Guibas, "A point set generation network for 3D object reconstruction from a single image," in IEEE CVPR, vol.38, 2017, which uses large-scale 3D datasets, is the first to use point cloud representation for 3D reconstruction, and proposes a calculation standard for point cloud distance. Reference 3: CB Choy, D. Xu, J. Gwak, K. Chen, and S. Savarese. 3DR2N2: Aunified approach for single and multi-view 3D object reconstruction. In ECCV 2016, a multi-image 3D reconstruction method using a recurrent neural network was proposed. This method can use the information of the previous input image to complete the reconstruction of the subsequent input. Reference 4: M. Wang, L. Wang, and Y. Fang. 3DensiNet: Arobust neural network architecture toward 3D volumetric object prediction from 2D image. In ACM MM 2017, max pooling is used to aggregate features from the input image for reconstruction. Reference 5:

[38] Multi-view Consistency as Supervisory Signal for Learning Shape and Pose Prediction 2018 proposed a multi-image reconstruction method that does not require real shooting perspective input. Although these reconstruction methods can obtain relatively ideal 3D reconstruction results, they all have shortcomings due to the difficulty and high cost of obtaining 3D data, the input constraints of multi-image correlation, the time-consuming and memory-consuming running process, and the sensitivity to input order. Summary of the invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide a single image 3D reconstruction method based on deep learning video supervision in view of the shortcomings of the prior art, comprising the following steps:

[0006] Step 1: Construct an object pose prediction module: Build a pose prediction network, input the object image into the pose prediction network model, and obtain the pose prediction of the object in the input image;

[0007] Step 2: Build a 3D shape estimation module: Build a 3D point cloud prediction network, input the object image into the 3D point cloud prediction network model, and obtain the 3D point cloud prediction of the object in the input image;

[0008] Step 3, construct a multi-frame shape fusion module: use the object pose prediction and 3D point cloud prediction obtained in step 1 and step 2 to input them into the fusion module, and further process the 3D point cloud prediction results by iteratively optimizing multi-frame consistency constraints, multi-frame smoothness constraints and multi-frame shape fusion;

[0009] Step 4, overall framework training: The input video sequence is obtained through the instance segmentation method. The objects and their mask sequences are input into the model in parallel for each frame to obtain the viewing angle and 3D point cloud prediction of each frame. The fused point cloud is input into the multi-frame fusion module to obtain the fused point cloud. The pose prediction module and the 3D shape estimation module are jointly optimized through reprojection and multi-frame constraints. The trained 3D shape estimation module is used to realize single-image 3D point cloud prediction.

[0010] Step 1 includes the following steps:

[0011] Step 1-1, building a posture prediction network includes building an object posture prediction network model G;

[0012] Step 1-2: Input video frame sequence {I t Generate pose predictions for objects in each frame Where t∈{1,2,…,n}, n is the total length of the frame sequence, I t represents the t-th frame image, P t Represents the pose prediction result of the t-th frame image.

[0013] In step 1-1, the object pose prediction network G includes an encoder and a decoder, and the trainable parameters in each layer of the network are represented by θ G The encoder part contains nine 3×3 convolutional layers, followed by a batch norm pooling layer, with ReLU selected as the activation function, and then two fully connected layers, with ReLU selected as the activation function, to finally obtain the encoding of the input. The decoder part contains three fully connected layers, with ReLU selected as the activation function. Here, the ReLU activation function is f(x)=max(0,x).

[0014] Step 2 includes the following steps:

[0015] Step 2-1, building a 3D point cloud prediction network includes building a 3D shape prediction network model K;

[0016] Step 2-2, the input video frame sequence {I t Generate a 3D point cloud prediction for each frame of the object

[0017] And the confidence of each point {S t}, where t∈{1,2,…,n}, n is the total length of the frame sequence, C t Represents the three-dimensional point cloud prediction result of the t-th frame image, S t Represents the confidence of the 3D point cloud prediction result of the t-th frame image.

[0018] In step 2-1, the 3D shape prediction network K includes an encoder and a decoder, and the trainable parameters in each layer of the network are represented by θ K The encoder part contains nine 3×3 convolutional layers, followed by a batch norm pooling layer, with ReLU selected as the activation function, and then two fully connected layers, with ReLU selected as the activation function, to finally obtain the encoding of the input; the decoder part contains one fully connected layer, with the hyperbolic function tanh selected as the activation function. Here, the ReLU activation function is f(x)=max(0,x).

[0019] Step 3 includes the following steps:

[0020] Step 3-1, fusing the prediction results of multiple frames to obtain a unified prediction C for the input object video sequence;

[0021] Step 3-2, reproject the prediction C under different viewing angles;

[0022] Step 3-3, design the loss function L for the pose prediction module and the 3D shape estimation module;

[0023] Step 3-4: Predict the pose Point cloud prediction And the prepared real image mask Mask corresponding to the input t Input, and calculate the loss L, and continuously iteratively optimize L to jointly update the parameters θ of the pose prediction network model G G and the parameters θ of the 3D point cloud estimation network model K K .

[0024] In step 3-1, the calculation formula for the fused multi-frame prediction result is as follows:

[0025]

[0026] Where f is the total length of the input video sequence, Ci is the prediction confidence of the point cloud of the i-th frame, S i is the point cloud prediction result of the i-th frame, and i ranges from 1 to f.

[0027] Step 3-2 includes the following steps:

[0028] Step 3-2-1: predict the pose {P t}Rotation acts on point cloud C;

[0029] Step 3-2-2, define the volume grid D to which the point cloud is to be mapped 1 ×D 2 ×D 3 , define the tensor in k 1 ∈[1,D 1 ],k 2 ∈[1,D 2 ],k 3 ∈[1,D 3 ], i = {1, 2, 3}, n ∈ [1, N], N represents the number of predicted point clouds, D 1 , D 2 , D 3 They represent the lengths of the mapped volume grid in the xyz directions respectively. The coordinate system is defined as the standard right-hand system. (k 1 , k 2 , k 3 ) represents the point coordinates in the point cloud, and i represents the coordinate dimension;

[0030] Step 3-2-3, calculate the Gaussian distribution of each point in the predicted point cloud in the three directions of the standard right-handed system xyz where x n is the predicted point cloud coordinates of the nth point, σ n is the variance of the Gaussian distribution for the nth point;

[0031] Step 3-2-4, calculate the occupancy probability of each position in the grid Where N is the number of points in the predicted point cloud, Indicates that the nth point is at the coordinate (k 1 ,k 2 ,k 3 )’s probability of occurrence;

[0032] Step 3-2-5, calculate the ray termination probability at each position in the grid

[0033]

[0034] Step 3-2-6, calculate the projection results at all obtained pose prediction angles Then the kth 1 Row k 2 List

[0035] The loss function L described in step 3-3 consists of three parts: reprojection loss L proj and pose smoothing loss L pose and multi-frame consistency loss L f , the calculation formula is as follows:

[0036] L=L proj +λ 1 L pose +λ 2 L f ,

[0037] Among them, the reprojection loss L proj It is used to distinguish whether the object is still in its original shape when viewed from the perspective of the input image. The pose smoothing loss L pose The multi-frame consistency loss L is used to constrain the pose results predicted for continuous input sequences to be continuous. f It is used to constrain the prediction results of inputs from different perspectives to be consistent, 1 Take it as 30, λ 2 Take it as 10; f takes values ​​from 1 to F, where F is the length of the video sequence;

[0038] The reprojection loss L proj , the calculation formula is as follows:

[0039]

[0040] Where N is the number of video sequences, F is the length of the video sequence, hw is the length and width of the image size, Proj is the reprojection result, Mask is the original image mask, n is 1 to N, f is 1 to F, x is 0 to h, and y is 0 to w.

[0041] The pose smoothing loss L pose , the calculation formula is as follows:

[0042]

[0043] Where ″ represents the second-order derivative of the discrete prediction value, N is the number of video sequences, F is the length of the video sequence, P f Indicates the pose prediction result at the fth frame, n ranges from 1 to N, and f ranges from 1 to F;

[0044] The multi-frame consistency loss L f , the calculation formula is as follows:

[0045]

[0046] Where CD(·) represents Chamfer Distance, N is the number of video sequences, F is the length of the video sequence, and C f Indicates the prediction result of the 3D point cloud of the f-th frame, n ranges from 1 to N, and f ranges from 1 to F;

[0047] Step 4 includes the following steps:

[0048] Step 4-1, data preprocessing: Use instance segmentation to segment the input video sequence into a foreground image sequence {I t} and foreground mask sequence {Mask t};

[0049] Step 4-2, model framework training: Use the AdamW gradient descent algorithm to jointly train the pose prediction network and the 3D point cloud prediction network until the loss function no longer decreases within a certain number of iterations or meets the preset number of iterations, then the training is completed. In the present invention, the value is generally 200,000 to 400,000 rounds, and 300,000 rounds are selected in the implementation example;

[0050] Step 4-3, model framework test: input a single image of the object for testing, and obtain the 3D point cloud prediction of it through the trained 3D point cloud prediction module.

[0051] The preprocessing process described in step 4-1 first splits the input video into a single-frame image sequence, then extracts all objects belonging to the target category we want to reconstruct from the existing strength segmentation algorithm for each frame, and then extracts the objects that appear in each frame as the target objects we reconstruct for this video. Finally, we set all pixel values ​​in the image except the object to 0 to obtain the foreground image, set the pixels contained in the object in the image to 255, and set the other pixels to 0 to obtain the foreground mask image.

[0052] The method disclosed by the present invention uses a video sequence to train a network, utilizes the coherence and consistency of information in the video sequence and utilizes a differentiable reprojection process, and simultaneously trains to obtain the position and three-dimensional point cloud of an input object.

[0053] Beneficial effects: The present invention has the following advantages: First, the use of video for supervised training avoids the direct use of three-dimensional information. Compared with multiple images, it avoids the constraints of input image correlation. It can adapt to the input of real scenes and avoid the limitation of rendered image categories. Second, the information in the video is continuous, while the image information is always discrete. The use of video sequences can better predict the pose and three-dimensional shape of the input image. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the present invention and the above and / or other advantages will become more clear.

[0055] Figure 1 It is a schematic diagram of the processing flow of the present invention;

[0056] Figure 2a is an example of an image input for reconstruction;

[0057] Figure 2b is the corresponding reconstruction result example;

[0058] Figure 2c is an example of a real object mask;

[0059] Figure 2d This is an example of a mask result where the reconstruction result is reprojected under the predicted view. DETAILED DESCRIPTION

[0060] like Figure 1 As shown, the present invention discloses a 3D reconstruction method based on deep learning video supervision, which is specifically implemented according to the following steps:

[0061] 1. Construct object pose prediction module

[0062] Input: Video sequence of objects

[0063] Output: Predicted object pose for each frame

[0064] 1.1 Building a pose prediction network includes building an object pose prediction network model G

[0065] The object pose prediction network G consists of an encoder and a decoder. The trainable parameters in each layer of the network are represented by θ G The encoder part contains nine 3×3 convolutional layers, followed by a batch norm pooling layer, with ReLU selected as the activation function, and then two fully connected layers, with ReLU selected as the activation function, to finally obtain the encoding of the input. The decoder part contains three fully connected layers, with ReLU selected as the activation function. Here, the ReLU activation function is f(x)=max(0,x).

[0066] 1.2 Input video frame sequence {I t Generate pose predictions for objects in each frame Where t∈{1,2,…,n}, n is the total length of the frame sequence, I t represents the t-th frame image, P t It represents the pose prediction result of the t-th frame image. The length of the video segment selected during the training process is 60 frames, that is, n=60.

[0067] 2. Building a 3D point cloud prediction network includes building an object 3D shape estimation module

[0068] Input: Video sequence of objects

[0069] Output: Predicted 3D point cloud for each frame

[0070] 2.1 Construction of 3D shape prediction network model K

[0071] The 3D shape prediction network K consists of an encoder and a decoder. The trainable parameters in each layer of the network are represented by θ K ; The encoder part contains nine 3×3 convolutional layers, followed by a batch norm pooling layer, with ReLU selected as the activation function, and then two fully connected layers, with ReLU selected as the activation function, to finally obtain the encoding of the input. The decoder part contains three fully connected layers, with ReLU selected as the activation function; the decoder part contains one fully connected layer, with the hyperbolic function tanh selected as the activation function. Here, the ReLU activation function is f(x)=max(0,x).

[0072] 2.2 Input video frame sequence {I t Generate a 3D point cloud prediction for each frame of the object And the confidence of each point {S t} is used in the subsequent steps, where t∈{1,2,…,n}, n is the total length of the frame sequence, C t Represents the three-dimensional point cloud prediction result of the t-th frame image, S t It represents the confidence of the 3D point cloud prediction result of the t-th frame image. The length of the video segment selected during the training process is 60 frames, that is, n=60.

[0073] 3. Construct multi-frame shape fusion module

[0074] Input: Pose prediction and 3D point cloud prediction for each frame of the input video sequence

[0075] Output: unified 3D point cloud of video sequence and various loss functions

[0076] 3.1 Fusion of multi-frame prediction results to obtain a unified prediction C for the input object video sequence

[0077] The calculation formula for the fusion multi-frame prediction result is as follows:

[0078]

[0079] Where f is the total length of the input video sequence, C i is the prediction confidence of the point cloud of the i-th frame, S i It is the point cloud prediction result of the i-th frame, and i takes the value of 1-f.

[0080] 3.2 Reproject the prediction C at different perspectives

[0081] Step 1: The predicted pose {P t}Rotation acts on point cloud C;

[0082] Step 2: Define the volume grid D to which the point cloud is mapped 1 ×D 2 ×D 3 , define the tensor in k 1 ∈[1,D 1 ],k 2 ∈[1,D 2 ],k 3 ∈[1,D 3 ], i = {1, 2, 3}, n ∈ [1, N], N represents the number of predicted point clouds, D 1 , D 2 , D 3 They represent the lengths of the mapped volume grid in the xyz directions respectively. The coordinate system is defined as the standard right-hand system. (k 1 , k 2 , k 3 ) represents the point coordinates in the point cloud, i represents the coordinate dimension, and the number of points in the point cloud predicted during the training process is 8000, that is, N = 8000;

[0083] Step 3: Calculate the Gaussian distribution of each point in the predicted point cloud in the three directions of the standard right-handed system xyz where x n is the predicted point cloud coordinates of the nth point, σ n is the variance of the Gaussian distribution for the nth point;

[0084] Step 4: Calculate the volume grid D 1 ×D 2 ×D 3 The occupancy probability of each position in Where N is the number of points in the predicted point cloud, which is set to 8000 during the training process.

[0085] Indicates that the nth point is at the coordinate (k 1 ,k 2 ,k 3 )’s probability of occurrence;

[0086] Step 5: Calculate the ray termination probability at each position in the grid

[0087]

[0088] Step 6: Calculate the projection results Then the kth1 Row k 2 List

[0089]

[0090] During the training process, the size of the projection result is set to 128*128.

[0091] 3.3 Design of loss function L for pose prediction module and 3D shape prediction module

[0092] The loss function L consists of three parts: reprojection loss L proj and pose smoothing loss L pose and multi-frame consistency loss L f , the calculation formula is as follows:

[0093] L=L proj +λ 1 L pose +λ 2 L f (4)

[0094] Among them, the reprojection loss L proj It is used to distinguish whether the object is still in its original shape when viewed from the perspective of the input image. The pose smoothing loss L pose The multi-frame consistency loss L is used to constrain the pose results predicted for continuous input sequences to be continuous. f It is used to constrain the prediction results of inputs from different perspectives to be consistent, 1 Take it as 30, λ 2 Take it as 10.

[0095] The reprojection loss L proj , the calculation formula is as follows:

[0096]

[0097] Where N is the number of video sequences, F is the length of the video sequence, hw is the length and width of the image size, Proj is the reprojection result, Mask is the original image mask, n is 1-N, f is 1-f, x is 0-h, and y is 0-w.

[0098] The pose smoothing loss L pose , the calculation formula is as follows:

[0099]

[0100] Where ″ represents the second-order derivative of the discrete prediction value, N is the number of video sequences, F is the length of the video sequence, and P f Represents the pose prediction result at the f-th frame, n takes the value of 1-N, and f takes the value of 1-f.

[0101] The multi-frame consistency loss L f , the calculation formula is as follows:

[0102]

[0103] Where CD(·) represents Chamfer Distance, N is the number of video sequences, and F is the length of the video sequence.

[0104] 3.4 Pose Prediction Point cloud prediction And the real image mask Mask t Input, and calculate the loss L, and continuously iteratively optimize L to jointly update the parameters θ of the pose prediction network model G G and the parameters θ of the 3D point cloud estimation network model K K

[0105] pass The process of minimizing the loss to further predict the obtained pose and 3D point cloud.

[0106] 4. Overall framework training

[0107] The input video sequence is obtained through instance segmentation and other methods, and each frame of the object and its mask sequence is input into the model in parallel to obtain the viewing angle and 3D point cloud prediction of each frame, which is then input into the multi-frame fusion module to obtain the fused point cloud. The pose prediction module and the 3D shape estimation module are jointly optimized through reprojection and multi-frame constraints, and the trained 3D shape estimation module is used to realize the single-image 3D point cloud prediction.

[0108] 4.1 Data Preprocessing

[0109] Input: Video sequence of objects

[0110] Output: A sequence of foreground images of the object and a sequence of images of the object mask

[0111] The preprocessing process described in step 4-1 first splits the input video into a single-frame image sequence, then extracts all objects belonging to the target category we want to reconstruct from the existing strength segmentation algorithm for each frame, then extracts the objects that appear in each frame as the target objects we reconstruct for this video, and finally sets all pixel values ​​in the image except the object to 0 to obtain the foreground image, sets the pixels contained in the object in the image to 255, and sets the other pixels to 0 to obtain the foreground mask image. And unify the size of all images to 256×256.

[0112] 4.2 Model Framework Training

[0113] Input: A sequence of foreground images of the object and a sequence of images of the object mask

[0114] Output: Unified 3D point cloud of video sequence

[0115] Step 1: Input the foreground image sequence of the object into the object pose prediction module and the object 3D shape prediction module in parallel, and obtain the pose prediction result and 3D point cloud prediction result of each frame respectively.

[0116] Step 2 Input the pose prediction results and 3D point cloud prediction results of each frame into the multi-frame fusion module to obtain the loss functions L proj ,L pose ,L f .

[0117] Step 3: Jointly optimize the object pose prediction module and the object 3D shape prediction module according to the optimization objective function of the overall model framework. When training the model, set the pose smoothing loss weight to 30, the reprojection loss weight to 1, and the multi-frame consistency loss to 10. And use the Adam optimizer with a momentum term of 0.9. Train the pose prediction network G and the 3D shape prediction network K from scratch, with the value of each layer initialized to a zero-mean Gaussian distribution, a standard deviation of 0.02, and a deviation initialized to 0. The model uses 24 batches for training, 300,000 iterations, the initial learning rate is set to 0.0001, and the weight decay is 0.001.

[0118] 4.3 Model framework test: Input a single image of the object for testing, and obtain its 3D point cloud prediction through the trained 3D point cloud prediction module.

[0119] Input: Single object image

[0120] Output: Reconstructed 3D point cloud result

[0121] Step 1: Segment the foreground object from the input image and input it into the trained network.

[0122] Step 2 obtains the predicted three-dimensional point cloud result of the object through the object three-dimensional shape prediction module.

[0123] In the present invention, Figure 2a The figure shows an example of an input image. Through the single image 3D reconstruction method based on deep learning video supervision described in the present invention, the existing instance segmentation method is used to obtain the following Figure 2c The object foreground mask and object foreground image shown in the figure are then input into the trained 3D point cloud prediction network to obtain the following Figure 2b The reconstruction result shown ( Figure 2bThe clarity of the point cloud has nothing to do with the invention point and does not affect the content expressed by the technical solution of this application), and is input into the trained posture prediction network to obtain an estimate of the object posture, and the obtained three-dimensional point cloud prediction is projected under this posture. Figure 2d The reprojection result is shown.

[0124] The present invention provides a single image 3D reconstruction method based on deep learning video supervision. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. A single image 3D reconstruction method based on deep learning video supervision, characterized in that: include Follow these steps: Step 1: Construct an object pose prediction module: Build a pose prediction network, input the object image into the pose prediction network model, and obtain the pose prediction of the object in the input image; Step 2: Build a 3D shape estimation module: Build a 3D point cloud prediction network, input the object image into the 3D point cloud prediction network model, and obtain the 3D point cloud prediction of the object in the input image; Step 3, construct a multi-frame shape fusion module: use the object pose prediction and 3D point cloud prediction obtained in step 1 and step 2 to input them into the fusion module, and further process the 3D point cloud prediction results by iteratively optimizing multi-frame consistency constraints, multi-frame smoothness constraints and multi-frame shape fusion; Step 4, overall framework training: The input video sequence is segmented by an instance method to obtain the object and its mask sequence, and each frame is input into the model in parallel to obtain the view angle and 3D point cloud prediction of each frame, which is then input into the multi-frame fusion module to obtain the fused point cloud. The pose prediction module and the 3D shape estimation module are jointly optimized through reprojection and multi-frame constraints, and the trained 3D shape estimation module is used to realize single-image 3D point cloud prediction; Step 1 includes the following steps: Step 1-1, building a posture prediction network module includes building an object posture prediction network model G; Step 1-2: Input video frame sequence {I t Generate pose predictions for objects in each frame Where t∈{1,2,…,n}, n is the total length of the video frame sequence, I t represents the t-th frame image, P t Represents the pose prediction result of the t-th frame image; In step 1-1, the object pose prediction network G includes an encoder and a decoder, and the trainable parameters in each layer of the network G are represented by θ G ; The encoder part contains nine 3×3 convolutional layers, followed by a batch norm pooling layer, with ReLU selected as the activation function, and then two fully connected layers, with ReLU selected as the activation function, to finally obtain the encoding of the input; the decoder part contains three fully connected layers, with ReLU selected as the activation function; Step 2 includes the following steps: Step 2-1, building a 3D point cloud prediction network includes building a 3D shape prediction network model K. The trainable parameters in each layer of the network K are represented by θ K ; Step 2-2, the input video frame sequence {I t Generate a 3D point cloud prediction for each frame of the object And the confidence of each point {S t }, where t∈{1,2,…,n}, n is the total length of the video frame sequence, C t Represents the three-dimensional point cloud prediction result of the t-th frame image, S t Indicates the confidence of the 3D point cloud prediction result of the t-th frame image; In step 2-1, the 3D shape prediction network K includes an encoder and a decoder, and the trainable parameters in each layer of the network K are represented by θ K ; The encoder part contains nine 3×3 convolutional layers, followed by a batch norm pooling layer, with ReLU selected as the activation function, and then two fully connected layers, with ReLU selected as the activation function, to finally obtain the encoding of the input; the decoder part contains one fully connected layer, with the hyperbolic function tanh selected as the activation function; Step 3 includes the following steps: Step 3-1, fusing the prediction results of multiple frames to obtain a unified prediction C for the input object video sequence; Step 3-2, reproject the prediction C under different viewing angles; Step 3-3, design the loss function L for the pose prediction module and the 3D shape estimation module; Step 3-4: Predict the pose Point cloud prediction And the prepared real image mask Mask corresponding to the input t Input, and calculate the loss L, and continuously iteratively optimize L to jointly update the parameters θ of the pose prediction network model G G and the parameters θ of the 3D point cloud estimation network model K K ; In step 3-1, the calculation formula of the fused multi-frame prediction result C is as follows: Where f is the total length of the input video sequence, C i is the prediction confidence of the point cloud of the i-th frame, S i is the point cloud prediction result of the i-th frame, i ranges from 1 to f; Step 3-2 includes the following steps: Step 3-2-1: predict the pose {P t }Rotation acts on point cloud prediction C; Step 3-2-2, define the volume grid D1×D2×D3 to which the point cloud is to be mapped, and define the tensor in k1∈[1,D1],k2∈[1,D2],k3∈[1,D3],i={1,2,3},n∈[1,N],N represents the number of predicted point clouds, D1, D2, D3 represent the lengths of the mapped volume grid in the xyz directions respectively, (k1, k2, k3) represent the point coordinates in the point cloud, i represents the coordinate dimension; Step 3-2-3, calculate the Gaussian distribution of each point in the predicted point cloud in the xyz directions where x n is the predicted point cloud coordinates of the nth point, σ n is the variance of the Gaussian distribution for the nth point; Step 3-2-4, calculate the occupancy probability of each position in the volume grid D1×D2×D3 Where N is the number of points in the predicted point cloud, represents the probability of the nth point at the coordinates (k1, k2, k3); Step 3-2-5, calculate the ray termination probability at each position in the volume grid D1×D2×D3 Step 3-2-6, calculate the projection results Then the k1th row and k2th column in Proj The loss function L described in step 3-3 consists of three parts: reprojection loss L proj and pose smoothing loss L pose and multi-frame consistency loss L f , the calculation formula is as follows: L=L proj +λ1L pose +λ2L f ; Among them, the reprojection loss L proj It is used to distinguish whether the object is still in its original shape when viewed from the perspective of the input image. The pose smoothing loss L pose The multi-frame consistency loss L is used to constrain the pose results predicted for continuous input sequences to be continuous. f It is used to constrain the prediction results of inputs from different perspectives to be consistent. λ1 and λ2 are the weights of the corresponding losses. f ranges from 1 to F, where F is the length of the video sequence. The reprojection loss L proj , the calculation formula is as follows: Where N is the number of video sequences, F is the length of the video sequence, hw is the length and width of the image, Proj is the reprojection result, Mask is the original image mask, n is 1 to N, f is 1 to F, x is 0 to h, and y is 0 to w; The pose smoothing loss L pose , the calculation formula is as follows: Where ″ represents the second-order derivative of the discrete prediction value, N is the number of video sequences, F is the length of the video sequence, P f Indicates the pose prediction result at the fth frame, n ranges from 1 to N, and f ranges from 1 to F; The multi-frame consistency loss L f , the calculation formula is as follows: Where CD(·) represents Chamfer Distance, N is the number of video sequences, F is the length of the video sequence, and C f Represents the 3D point cloud prediction result of the f-th frame, n ranges from 1 to N, and f ranges from 1 to F.

2. The method according to claim 1, characterized in that: Step 4 includes the following steps: Step 4-1, data preprocessing: Segment the input video sequence to obtain a foreground image sequence {I t } and foreground mask sequence {Mask t }; Step 4-2, model framework training: Use the AdamW gradient descent algorithm to jointly train the pose prediction network and the 3D point cloud prediction network until the loss function no longer decreases within a certain number of iterations or meets the preset number of iterations, then the training is completed; Step 4-3, model framework test: input a single image of the object for testing, and obtain the 3D point cloud prediction of it through the trained 3D point cloud prediction module.

Images