A three-dimensional reconstruction method, device, electronic device and storage medium
By calculating point cloud information of the three-primary color images in depth and updating the implicit voxel, the geometric uncertainty problems caused by sensor noise and view occlusion in three-dimensional reconstruction are solved, and high-quality three-dimensional scene reconstruction is achieved.
Patent Information
- Application Number
- CN202110546636.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-05-19
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-05-19
AI Technical Summary
The prior art is difficult to effectively solve the problem of geometric uncertainty caused by sensor noise or scanning blur in three-dimensional reconstruction, especially incomplete areas and low geometric quality caused by view occlusion.
By using point cloud information of the three primary color images in depth, point cloud information under the world coordinate system is calculated and stored in a preset voxel grid, the hidden vector is calculated to obtain the probability local implicit voxel, and the historical implicit is updated. Voxels realize implicit three-dimensional scene representation, and use the encoder-decoder network model to solve the camera position and generate a three-dimensional color grid model.
Effectively eliminate geometric uncertainty caused by sensor noise and view occlusion, improve the quality and integrity of three-dimensional scene reconstruction, and achieve efficient surface reconstruction and scene completion.
Smart Images

Figure CN113487739B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and in particular to a three-dimensional reconstruction method, device, electronic equipment and storage medium. Background Art
[0002] Thanks to the popularity of commercial depth sensors, real-time dense 3D reconstruction has made great progress in the past decade. Most deep fusion algorithms focus on using bundle adjustment or loop closure detection to achieve global continuous 3D reconstruction. However, since then, there has been little development in the underlying representation of the 3D scene itself. At the same time, due to geometric uncertainty caused by sensor noise or scanning blur such as view occlusion, the reconstruction result is likely to have many incomplete areas, and its geometric quality is often unsatisfactory.
[0003] Therefore, how to solve the geometric uncertainty caused by sensor noise or scanning blur such as view occlusion and better realize three-dimensional scene reconstruction has become a research focus of the industry. Summary of the invention
[0004] The present invention provides a three-dimensional reconstruction method, device, electronic device and storage medium, which are used to solve the geometric uncertainty caused by sensor noise or scanning blur such as view occlusion, and better realize three-dimensional scene reconstruction.
[0005] The present invention provides a three-dimensional reconstruction method, comprising:
[0006] Calculate, according to the first point cloud information of the first depth three-primary-color image, second point cloud information of the first depth three-primary-color image in a world coordinate system;
[0007] The first depth three-primary color image is any frame image among the acquired continuous multiple frames of depth three-primary color images;
[0008] The second point cloud information is stored in a preset voxel grid set, and a latent vector of the second point cloud information in each preset voxel grid is calculated to obtain a first probabilistic local latent voxel corresponding to the first depth three-primary color image;
[0009] According to the latent vector in the first probability local latent voxel, the latent vector in the historical probability local latent voxel is updated to obtain an implicit three-dimensional scene representation corresponding to the first depth three-primary color image;
[0010] The latent vector in the historical probability local latent voxel is obtained based on the point cloud information of each frame image before the first depth three-primary color image.
[0011] According to the three-dimensional reconstruction method provided by the present invention, the second point cloud information of the first depth three-primary color image in the world coordinate system is calculated according to the first point cloud information of the first depth three-primary color image, specifically:
[0012] According to the first point cloud information of the first depth three-primary color image, solving the camera pose corresponding to the first depth three-primary color image;
[0013] According to the camera pose, second point cloud information of the first depth three-primary color image in a world coordinate system is calculated.
[0014] According to the 3D reconstruction method provided by the present invention, the second point cloud information is stored in a preset voxel grid set, and the latent vector of the second point cloud information in each preset voxel grid is calculated, specifically:
[0015] According to the coordinate information of the second point cloud information, each point in the second point cloud information is stored in a preset voxel grid set to obtain second point cloud sub-information in each grid in the preset voxel grid set;
[0016] The latent vector of each point in the second point cloud sub-information in each grid is calculated, and the latent vector of each point in each grid is weighted averaged to obtain the latent vector of the second point cloud information in each preset voxel grid.
[0017] According to the 3D reconstruction method provided by the present invention, according to the latent vector in the first probability local implicit voxel, the latent vector in the historical probability local implicit voxel is updated to obtain the implicit 3D scene representation corresponding to the first depth three-primary color image, specifically:
[0018] Taking a weighted average of the latent vector in the first probability local latent voxel and the latent vector in the historical probability local latent voxel to obtain an updated latent vector in the probability local latent voxel;
[0019] According to the latent vector in the updated probabilistic local implicit voxel, an implicit three-dimensional scene representation corresponding to the first depth three-primary color image is obtained.
[0020] According to the three-dimensional reconstruction method provided by the present invention, solving the camera pose corresponding to the first depth three-primary color image according to the first point cloud information of the first depth three-primary color image is specifically as follows:
[0021] Inputting the first point cloud information of the first depth three-primary color image into a decoder in a trained encoder-decoder network model to obtain a signed distance probability distribution of the first point cloud information;
[0022] According to the signed distance probability distribution, solving the camera pose corresponding to the first depth three-primary color image;
[0023] The trained encoder-decoder network model is obtained by training based on deep three-primary-color image point cloud information samples carrying signed distance probability distribution labels.
[0024] According to the three-dimensional reconstruction method provided by the present invention, the camera pose corresponding to the first depth three-primary color image is solved according to the signed distance probability distribution, specifically:
[0025] Calculating the sum of the signed distance values of each point in the first point cloud information according to the signed distance probability distribution;
[0026] According to the coordinate information of the first point cloud information, find each point in the third point cloud information corresponding to each point in the first point cloud information to obtain a plurality of point pairs, and calculate the sum of the color differences of the three primary colors of each point pair;
[0027] Wherein, the third point cloud information is the point cloud information of the previous frame image of the first depth three-primary color image;
[0028] The camera pose corresponding to the first depth three-primary color image is obtained by performing a minimization error function calculation based on the sum of the signed distance values of each point in the first point cloud information and the sum of the three-primary color differences of each point pair.
[0029] According to the 3D reconstruction method provided by the present invention, after obtaining the implicit 3D scene representation corresponding to the first depth three-primary color image, the method further includes:
[0030] Calculate texture based on preset multi-frame colored point cloud information;
[0031] Calculating a signed distance value corresponding to each probabilistic local implicit voxel in the implicit three-dimensional scene representation according to an implicit vector in the implicit three-dimensional scene representation;
[0032] A three-dimensional color mesh model corresponding to the first depth three-primary color image is generated according to the symbol distance value and the texture.
[0033] According to the three-dimensional reconstruction method provided by the present invention, before inputting the first point cloud information of the first depth three-primary color image into a decoder in a trained encoder-decoder network model, the method further includes:
[0034] Acquire multiple point cloud information samples of deep three-primary color images carrying signed distance probability distribution labels; use each point cloud information sample of the deep three-primary color image carrying the signed distance probability distribution label as a group of training samples to obtain multiple groups of training samples, and use the multiple groups of training samples to train the encoder-decoder network model.
[0035] According to the three-dimensional reconstruction method provided by the present invention, the encoder-decoder network model is trained using multiple sets of training samples, specifically:
[0036] For any set of training samples, input the training samples into the encoder-decoder network model, and output the prediction probability corresponding to the training samples;
[0037] Calculating a loss value using a preset loss function according to the predicted probability corresponding to the training sample and the signed distance probability distribution label in the training sample;
[0038] If the loss value converges, the encoder-decoder network model training is completed.
[0039] The present invention also provides a three-dimensional reconstruction device, comprising:
[0040] A world coordinate system point cloud information acquisition module, used to calculate second point cloud information of the first depth three-primary color image in the world coordinate system according to the first point cloud information of the first depth three-primary color image;
[0041] The first depth three-primary color image is any frame image of the acquired continuous multiple frames of depth three-primary color images;
[0042] A probabilistic local implicit voxel acquisition module, used to store the second point cloud information in a preset voxel grid set, calculate the implicit vector of the second point cloud information in each preset voxel grid, and obtain the first probabilistic local implicit voxel corresponding to the first depth three-primary color image;
[0043] An implicit three-dimensional scene representation generation module, used to update the latent vector in the historical probability local implicit voxel according to the latent vector in the first probability local implicit voxel, so as to obtain the implicit three-dimensional scene representation corresponding to the first depth three-primary color image;
[0044] The latent vector in the historical probability local latent voxel is obtained based on the point cloud information of each frame image before the first depth three-primary color image.
[0045] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned three-dimensional reconstruction methods are implemented.
[0046] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-mentioned three-dimensional reconstruction methods are implemented.
[0047] The present invention provides a 3D reconstruction method, device, electronic device and storage medium, which obtains any frame image in multiple consecutive frames of deep three-primary color images, calculates the point cloud information of the frame image in the world coordinate system, and thereby calculates the implicit vector of the point cloud information in each preset voxel grid, and obtains the probabilistic local implicit voxel corresponding to the frame to eliminate the influence of geometric uncertainty caused by sensor noise or view occlusion; and then updates the historical implicit vectors in the grid corresponding to the probabilistic local implicit voxel according to the implicit vector in the grid corresponding to the probabilistic local implicit voxel, and obtains the implicit 3D scene representation corresponding to the frame, thereby realizing high-quality 3D scene reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0049] Figure 1 It is a schematic flow chart of the three-dimensional reconstruction method provided by the present invention;
[0050] Figure 2 It is a schematic diagram of the overall framework of the neural network model in the three-dimensional reconstruction method provided by the present invention;
[0051] Figure 3 It is a schematic diagram of the processing flow of the real-time three-dimensional reconstruction algorithm in the three-dimensional reconstruction method provided by the present invention;
[0052] Figure 4 is a schematic structural diagram of a three-dimensional reconstruction device provided by the present invention;
[0053] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0054] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0055] It should be noted that the advantages of implicit geometric representation parameterized by neural networks have been widely demonstrated. This method represents geometry as a continuous implicit function, so that the underlying shape can be extracted at arbitrary resolution, which effectively improves the quality of surface reconstruction. In addition, since the network structure of the implicit function consists of only simple fully connected layers, such methods are also very efficient. Another important feature of this deep implicit representation is the ability to encode geometric prior knowledge, which makes it suitable for many application scenarios, such as shape interpolation or reconstruction. By decomposing and encoding implicit domains in local voxels, this ability can be extended to scene priors, thereby achieving high-quality scene completion or reconstruction.
[0056] Figure 1 is a flow chart of the three-dimensional reconstruction method provided by the present invention, such as Figure 1 As shown, including:
[0057] Step S1, calculating second point cloud information of a first depth three-primary-color image in a world coordinate system according to first point cloud information of the first depth three-primary-color image;
[0058] The first depth three-primary color image is any frame image among the acquired continuous multiple frames of depth three-primary color images;
[0059] Specifically, the continuous multi-frame deep three-primary color images described in the present invention refer to selecting multiple frames of images that are continuous in time, and each frame of the image is a three-primary color (Red Green Blue; RGB) image with depth information.
[0060] The first point cloud information described in the present invention refers to point cloud data obtained by performing camera internal parameter back projection processing based on the image information of the first depth three-primary color image, wherein the camera refers to the camera used to shoot the first depth three-primary color image.
[0061] In an embodiment of the present invention, the second point cloud information refers to point cloud data of the first depth three-primary color image in a world coordinate system.
[0062] Further, according to the first point cloud information of the first depth three-primary-color image, second point cloud information of the first depth three-primary-color image in the world coordinate system may be calculated.
[0063] Step S2, storing the second point cloud information into a preset voxel grid set, calculating the implicit vector of the second point cloud information in each preset voxel grid, and obtaining a first probabilistic local implicit voxel (Probabilistic Local Implicit Voxel; PLIVox) corresponding to the first depth three-primary color image;
[0064] Specifically, the preset voxel grid set described in the present invention refers to a set of all voxel grids obtained by gridding a preset three-dimensional scene.
[0065] The first probabilistic local implicit voxel described in the present invention refers to an implicit representation of the second point cloud information of the first depth three-primary color image in each preset voxel grid.
[0066] Furthermore, by storing the second point cloud information into a corresponding grid in a preset voxel grid set, the implicit vector of the second point cloud information in each preset voxel grid is calculated to obtain a first probabilistic local implicit voxel.
[0067] Step S3, updating the latent vector in the historical probability local implicit voxel according to the latent vector in the first probability local implicit voxel, and obtaining the implicit three-dimensional scene representation corresponding to the first depth three-primary color image;
[0068] The latent vector in the historical probability local latent voxel is obtained based on the point cloud information of each frame image before the first depth three-primary color image.
[0069] Specifically, the latent vector in the historical probability local latent voxel described in the present invention refers to the latent vector of the point cloud information of each frame of image in each voxel grid calculated based on the point cloud information of each frame of image before the first depth three-primary color image, and then taking each voxel grid as a unit, through weighted averaging, the latent vector in each voxel grid corresponding to each frame and the latent vector in the same voxel grid corresponding to the previous frame are fused to obtain the accumulated latent vector through continuous fusion.
[0070] The implicit three-dimensional scene representation corresponding to the first depth three-primary color image described in the present invention refers to obtaining the accumulated latent vectors in the probabilistic local implicit voxels corresponding to the first depth three-primary color image by fusing the latent vectors in the first probabilistic local implicit voxels into the latent vectors of the historical probabilistic local implicit voxels, thereby obtaining the corresponding implicit three-dimensional scene representation.
[0071] In an embodiment of the present application, a latent vector is stored in each probabilistic local implicit voxel. By continuously updating the latent vector in each probabilistic local implicit voxel according to the point cloud information of each frame of the three-primary-color depth image in the world coordinate system, the fusion of the surface model in the three-dimensional reconstruction process is efficiently completed.
[0072] The method of the embodiment of the present invention obtains any frame image in multiple consecutive frames of deep three-primary color images, calculates the point cloud information of the frame image in the world coordinate system, thereby calculating the latent vector of the point cloud information in each preset voxel grid, and obtains the probabilistic local implicit voxel corresponding to the frame to eliminate the influence of geometric uncertainty caused by sensor noise or view occlusion; then, according to the latent vector in the grid corresponding to the probabilistic local implicit voxel, the historical latent vector in the grid corresponding to the probabilistic local implicit voxel is updated to obtain the implicit three-dimensional scene representation corresponding to the frame, thereby realizing high-quality three-dimensional scene reconstruction.
[0073] Optionally, the calculating, according to the first point cloud information of the first depth three-primary-color image, second point cloud information of the first depth three-primary-color image in a world coordinate system is specifically:
[0074] According to the first point cloud information of the first depth three-primary color image, solving the camera pose corresponding to the first depth three-primary color image;
[0075] According to the camera pose, second point cloud information of the first depth three-primary color image in a world coordinate system is calculated.
[0076] Specifically, the camera pose described in the present invention refers to calculating the estimated position and pose of the camera motion used to shoot these images based on a given number of images.
[0077] Furthermore, according to the first point cloud information of the first depth three-primary color image, the camera pose corresponding to the first depth three-primary color image can be solved through model calculation and data analysis;
[0078] Furthermore, according to the camera posture, after coordinate system conversion, second point cloud information of the first depth three-primary color image in the world coordinate system can be calculated.
[0079] Through the method of the embodiment of the present invention, the corresponding camera pose is estimated based on the first point cloud information of the first depth three-primary color image, so as to solve the second point cloud information of the first depth three-primary color image in the world coordinate system based on the camera pose.
[0080] Optionally, the second point cloud information is stored in a preset voxel grid set, and the latent vector of the second point cloud information in each preset voxel grid is calculated, specifically as follows:
[0081] According to the coordinate information of the second point cloud information, each point in the second point cloud information is stored in a preset voxel grid set to obtain second point cloud sub-information in each grid in the preset voxel grid set;
[0082] The latent vector of each point in the second point cloud sub-information in each grid is calculated, and the latent vector of each point in each grid is weighted averaged to obtain the latent vector of the second point cloud information in each preset voxel grid.
[0083] Specifically, the coordinate information of the second point cloud information described in the present invention refers to the coordinate position of each point in the second point cloud information.
[0084] The second point cloud sub-information described in the present invention refers to storing each point in the second point cloud information in each grid of the corresponding coordinates in a preset voxel grid set according to the coordinate position of each point in the second point cloud information, thereby obtaining each point in each grid.
[0085] Furthermore, after obtaining the second point cloud sub-information in each grid in the preset voxel grid set, the latent vector of each point in the second point cloud sub-information in each grid can be obtained by inputting the second point cloud sub-information in each grid into the encoder in the trained encoder-decoder network model, thereby performing weighted averaging on the latent vectors of each point in each grid to obtain the latent vector of the second point cloud information in each preset voxel grid.
[0086] Through the method of the embodiment of the present invention, based on the coordinate information of the second point cloud information, the second point cloud sub-information in each grid in the preset voxel grid set is obtained, and through model calculation, the latent vector of each point in the second point cloud sub-information is obtained, and the latent vector of the second point cloud information in each preset voxel grid is calculated based on weighted average.
[0087] Optionally, according to the latent vector in the first probability local implicit voxel, the latent vector in the historical probability local implicit voxel is updated to obtain the implicit three-dimensional scene representation corresponding to the first depth three-primary color image, specifically as follows:
[0088] Taking a weighted average of the latent vector in the first probability local latent voxel and the latent vector in the historical probability local latent voxel, to obtain an updated latent vector in the probability local latent voxel;
[0089] According to the latent vector in the updated probabilistic local implicit voxel, an implicit three-dimensional scene representation corresponding to the first depth three-primary color image is obtained.
[0090] Specifically, the latent vector in the updated probabilistic local implicit voxel described in the present invention refers to the weighted average of the latent vector in the first probabilistic local implicit voxel and the latent vector of the historical probabilistic local implicit voxel, so as to achieve the fusion of the latent vector in the first probabilistic local implicit voxel and the latent vector of the historical probabilistic local implicit voxel, and finally obtain the accumulated latent vector in each probabilistic local implicit voxel.
[0091] Furthermore, according to the latent vector in the updated probabilistic local implicit voxel, an implicit three-dimensional scene representation corresponding to the first depth three-primary color image can be obtained.
[0092] Through the method of the embodiment of the present invention, the latent vector in the first probability local implicit voxel and the latent vector in the historical probability local implicit voxel are weighted averaged to obtain the latent vector in the updated probability local implicit voxel, and the implicit three-dimensional scene representation corresponding to the first depth three-primary color image is obtained.
[0093] Optionally, solving the camera pose corresponding to the first depth three-primary-color image according to the first point cloud information of the first depth three-primary-color image is specifically:
[0094] Inputting the first point cloud information of the first depth three-primary color image into a decoder in a trained encoder-decoder network model to obtain a signed distance probability distribution of the first point cloud information;
[0095] According to the signed distance probability distribution, solving the camera pose corresponding to the first depth three-primary color image;
[0096] The trained encoder-decoder network model is obtained by training based on deep three-primary-color image point cloud information samples carrying signed distance probability distribution labels.
[0097] Specifically, the signed distance probability distribution of the first point cloud information described in the present invention refers to the signed distance probability distribution at the coordinate position of each point in the first point cloud information, wherein the signed distance probability distribution refers to determining the distance probability distribution from a point to the boundary of an area in a finite area in three-dimensional space.
[0098] The trained encoder-decoder network model described in the present invention is obtained after training based on training samples, and is used to encode the input depth three-primary color image point cloud information samples to obtain the latent vector of the point cloud information samples; at the same time, by decoding the latent vector of the point cloud information samples, the signed distance probability distribution of each point in the point cloud information samples is output.
[0099] In an embodiment of the present invention, the training samples are composed of multiple groups of depth three-primary-color image point cloud information samples carrying signed distance probability distribution labels.
[0100] The signed distance probability distribution labels described in the present invention are predetermined based on the depth three-primary color image point cloud information samples and correspond one-to-one to the depth three-primary color image point cloud information samples. That is, each point of the depth three-primary color image point cloud information sample in the training sample is pre-set to carry a corresponding signed distance probability distribution as a label.
[0101] Furthermore, the first point cloud information of the first depth three-primary color image is input into the trained decoder network model for decoding to obtain the signed distance probability distribution of the first point cloud information.
[0102] Specifically, according to the coordinate information of the first point cloud information, the historical PLIVox at the coordinate position corresponding to the coordinate information of the first point cloud information is obtained, and the latent vector in the historical PLIVox is obtained; then, the coordinate information of the first point cloud information and the latent vector in the historical PLIVox are input into the decoder in the trained encoder-decoder network model for decoding to obtain the signed distance probability distribution of the first point cloud information.
[0103] Further, according to the signed distance probability distribution of the first point cloud information, a minimization error function calculation is performed, and then the camera pose corresponding to the first depth three-primary color image is obtained by solving the minimum value of the error function;
[0104] Through the method of the embodiment of the present invention, the first point cloud information of the first depth three-primary color image is input into the decoder in the trained encoder-decoder network model to obtain the signed distance probability distribution of the first point cloud information, and then based on the signed distance probability distribution, the camera pose corresponding to the first depth three-primary color image is solved.
[0105] Optionally, according to the signed distance probability distribution, solving the camera pose corresponding to the first depth three-primary color image is specifically:
[0106] Calculating the sum of the signed distance values of each point in the first point cloud information according to the signed distance probability distribution;
[0107] According to the coordinate information of the first point cloud information, find each point in the third point cloud information corresponding to each point in the first point cloud information to obtain a plurality of point pairs, and calculate the sum of the color differences of the three primary colors of each point pair;
[0108] Wherein, the third point cloud information is the point cloud information of the previous frame image of the first depth three-primary color image;
[0109] The camera pose corresponding to the first depth three-primary color image is obtained by performing a minimization error function calculation based on the sum of the signed distance values of each point in the first point cloud information and the sum of the three-primary color differences of each point pair.
[0110] Specifically, the point pair described in the present invention refers to a combination of two points formed by any point in the first point cloud information and its corresponding point in the third point cloud information of the previous frame image having the same position coordinates. Therefore, multiple point pairs can be obtained based on each point in the first point cloud information and each point corresponding to it in the third point cloud information.
[0111] The three-primary color difference described in the present invention refers to the color difference between two points in a point pair, that is, the difference in RGB information of the two points.
[0112] The error function described in the present invention refers to a function obtained by weighted averaging the sum of the signed distance values of each point in the first point cloud information and the sum of the three primary color differences of each point pair.
[0113] In an embodiment of the present application, a Gauss-Newton solver may be used to solve the minimization error function.
[0114] Furthermore, a Gauss-Newton solver is used to solve the minimum value of the error function to estimate the camera pose corresponding to the first depth three-primary color image.
[0115] The method of the embodiment of the present invention obtains an error function by calculating the sum of the signed distance values of each point in the first point cloud information and the sum of the RGB color differences between each point in the first point cloud information and each corresponding point in the previous frame of point cloud information, and obtains the camera pose corresponding to the first depth three-primary color image based on solving the minimization error function.
[0116] Optionally, after acquiring the implicit three-dimensional scene representation corresponding to the first depth three-primary color image, the method further includes:
[0117] Calculate texture based on preset multi-frame colored point cloud information;
[0118] Calculating a signed distance value corresponding to each probabilistic local implicit voxel in the implicit three-dimensional scene representation according to an implicit vector in the implicit three-dimensional scene representation;
[0119] A three-dimensional color mesh model corresponding to the first depth three-primary color image is generated according to the symbol distance value and the texture.
[0120] Specifically, the preset multi-frame colored point cloud information described in the present invention refers to each point in the preset multi-frame image point cloud information including not only three-dimensional coordinate information but also RGB color information.
[0121] The signed distance value corresponding to each probabilistic local implicit voxel described in the present invention refers to the signed distance value of the grid at the coordinate corresponding to each probabilistic local implicit voxel.
[0122] In an embodiment of the present application, the texture is obtained by averaging the RGB values based on preset multi-frame colored point cloud information.
[0123] In an embodiment of the present application, the latent vector in the implicit three-dimensional scene representation is decoded by the decoder in the trained encoder-decoder network model to obtain the signed distance value of the grid at the coordinate corresponding to each probabilistic local implicit voxel.
[0124] Furthermore, according to the obtained signed distance value and texture, surface geometry reconstruction can be performed based on the marching cube algorithm, and a reconstruction result with texture can be obtained, that is, a three-dimensional color mesh model corresponding to the first depth three-primary color image can be generated.
[0125] The method of the embodiment of the present invention, after obtaining the implicit three-dimensional scene representation, can extract meshes of arbitrary clarity for the implicit three-dimensional scene at any time during the reconstruction process by calculating the texture based on a trained decoder network model and reconstruction algorithm to obtain a mesh representation of the current three-dimensional scene.
[0126] Optionally, before inputting the first point cloud information of the first depth three-primary color image into a decoder in a trained encoder-decoder network model, the method further includes:
[0127] Acquire multiple point cloud information samples of deep three-primary color images carrying signed distance probability distribution labels; use each point cloud information sample of the deep three-primary color image carrying the signed distance probability distribution label as a group of training samples to obtain multiple groups of training samples, and use the multiple groups of training samples to train the encoder-decoder network model.
[0128] Specifically, before inputting the first point cloud information of the first depth three-primary color image into the decoder in the trained encoder-decoder network model, the encoder-decoder network model needs to be trained. The specific training process is as follows:
[0129] The point cloud information samples of the depth three-primary color image and the signed distance probability distribution labels carried by the samples are taken as a group of training samples, that is, each point cloud information sample of the depth three-primary color image with a signed distance probability distribution label is taken as a group of training samples, thereby obtaining multiple groups of training samples.
[0130] In an embodiment of the present invention, the point cloud information samples of the depth three-primary-color image and the signed distance probability distribution labels carried by the point cloud information samples of the depth three-primary-color image are in one-to-one correspondence.
[0131] Then, after obtaining multiple groups of training samples, the multiple groups of training samples are input into the encoder-decoder network model in sequence, that is, the point cloud information samples of the depth three primary color images and the signed distance probability distribution labels in each group of training samples are simultaneously input into the encoder-decoder network model, and according to each output result of the encoder-decoder network model, the parameters of the encoder-decoder network model are adjusted by calculating the loss function value, and finally the training process of the encoder-decoder network model is completed.
[0132] Through the method of the embodiment of the present invention, the point cloud information samples of the depth three-primary color image and the signed distance probability distribution labels carried by the point cloud information samples of the depth three-primary color image are used as a group of training samples, and the encoder-decoder network model is trained using multiple groups of training samples.
[0133] Optionally, the training of the encoder-decoder network model using multiple groups of training samples is specifically as follows:
[0134] For any set of training samples, input the training samples into the encoder-decoder network model, and output the prediction probability corresponding to the training samples;
[0135] Calculating a loss value using a preset loss function according to the predicted probability corresponding to the training sample and the signed distance probability distribution label in the training sample;
[0136] If the loss value converges, the encoder-decoder network model training is completed.
[0137] Specifically, after obtaining multiple groups of training samples, for any group of training samples, the point cloud information samples of the depth three-primary-color images and the signed distance probability distribution labels in the training samples are simultaneously input into the encoder-decoder network model, and the prediction probability corresponding to the training samples is output, where the prediction probability refers to the prediction probability corresponding to the point cloud information of the training samples for different depth three-primary-color images.
[0138] In an embodiment of the present invention, the preset loss function refers to a loss function pre-set in the encoder-decoder network model for model evaluation.
[0139] On this basis, the preset loss function is used to calculate the loss value according to the predicted probability corresponding to the training sample and the signed distance probability distribution label in the training sample.
[0140] Furthermore, after the loss value is calculated, the training process ends, and the model parameters in the encoder-decoder network model are updated using the back propagation algorithm before the next training. During the training process, if the loss value calculated for a certain set of training samples converges, the training of the encoder-decoder network model is completed.
[0141] The method of the embodiment of the present invention controls the loss value of the encoder-decoder network model within a preset range by training the encoder-decoder network model, thereby facilitating improving the accuracy of the probability distribution of the symbol distance output by the encoder-decoder network model.
[0142] Figure 2 is a schematic diagram of the overall framework of the neural network model in the three-dimensional reconstruction method provided by the present invention, such as Figure 2As shown in the figure, the network structure design specifically includes: dividing the three-dimensional scene under a certain frame of deep RGB image into several three-dimensional grids according to the preset size, and for each grid, encoding the coordinates and normal vector of each point in the deep RGB image point cloud inside it through the trained encoder network model to obtain the hidden vector of each point, thereby obtaining the implicit representation of the grid, namely PLIVox. Then the hidden vectors of all points in the PLIVox are weighted averaged to obtain the hidden vector l of the PLIVox in the current frame m The specific definitions are as follows:
[0143]
[0144] in, is the point cloud set in the mth PLIVox, y i and n i Represent the local coordinates and normal vector of the i-th point respectively, Represents the encoder network model.
[0145] In the embodiment of the present application, for any position in each grid, the latent vector of the grid and the local coordinates of the position can be decoded by a trained decoder to obtain the signed distance probability distribution of the position, which is fitted in the form of a Gaussian distribution, that is, the decoder outputs a two-tuple {μ D ,σ D}, the probability distribution of the signed distance at this position is μ D , with variance σ D Gaussian distribution.
[0146] The embodiment of the present invention provides an implicit three-dimensional scene representation with deep prior knowledge based on a neural network, based on probabilistic local implicit voxels PLIVox, and the training method of the encoder-decoder network model thereof is as follows:
[0147] The grid model in the ShapeNet dataset is used to obtain the point cloud information inside the grid in real situations and its corresponding coordinate-sign distance combination.
[0148] For each PLIVox, its internal point cloud information is encoded through the encoder network model and weighted averaged to obtain the latent vector of the entire PLIVox;
[0149] Furthermore, the probability distribution of the signed distance of each coordinate is obtained through the decoder network model;
[0150] Finally, the weighted sum of the inverse of the likelihood sum of the selected coordinate symbol distances in all PLIVox and the square of the modulus length of the latent vector is used as the loss function, and the gradient is calculated for back propagation to minimize the function to train the encoder-decoder network model. The loss function is specifically defined as follows:
[0151]
[0152] Among them, δ is the balance coefficient of the two terms in the loss function, The definition is as follows:
[0153]
[0154] Among them, y i represents the local coordinates of the selected coordinates, Represents the actual signed distance, μ D represents the mean of the signed distance probability distribution of the selected local coordinates, σ D represents the standard deviation of the probability distribution of the signed distance of the selected local coordinate, l m Represents the latent vector of PLIVox corresponding to the selected local coordinates.
[0155] Figure 3 3D reconstruction method provided by the present invention is a processing flow diagram of a real-time 3D reconstruction algorithm. Figure 3 As shown, it is a schematic diagram of the processing flow of the real-time 3D reconstruction method based on PLIVox implicit representation with deep prior knowledge provided by an embodiment of the present invention.
[0156] It is worth noting that by encoding meaningful scene priors with continuous functions, improved surface reconstruction and accurate camera trajectories can be achieved. However, there are still several challenges to be addressed: (1) geometric uncertainty needs to be explicitly encoded in the deep implicit representation to combat the effects of sensor noise or view occlusion; (2) accurate camera tracking algorithms based on such implicit representations have not yet been proposed, which is crucial in the reconstruction process; (3) there is a lack of an efficient surface modeling method that can incrementally integrate new observations based on the implicit representation.
[0157] like Figure 3 As shown, in order to solve the above technical problems, an embodiment of the present invention provides a real-time 3D reconstruction method based on PLIVox implicit representation with deep prior knowledge, and the process steps include:
[0158] Step S310, aligning the current point cloud with the surface model to estimate the camera pose. Specifically:
[0159] For each frame of the input depth RGB or RGB-D sequence, define:
[0160] and are the depth map and RGB map of the tth frame respectively. is the camera pose of frame t-1, T(ξ t )=exp((ξ t )^)(ξ t ∈se(3)) is and The relative position between them, that is, by optimizing T(ξ t ), the camera pose of the tth frame can be obtained by T t =T t-1 T(ξ t )get.
[0161] Specifically, the point cloud of the current frame is first obtained by back-projecting the camera internal parameters, namely:
[0162]
[0163] Wherein, π represents the projection function, and π′ represents the inverse function of π.
[0164] According to the coordinate information of the current frame point cloud information, the historical PLIVox at the corresponding coordinate position is obtained, and then the hidden vector in the historical PLIVox corresponding to the coordinate information of the current frame point cloud information is obtained;
[0165] The latent vector in the above historical PLIVox is decoded by the decoder network model to obtain the surface model information of the current frame point cloud information, that is, μ corresponding to the probability distribution of the signed distance estimated by the decoder for each point D With σ D .
[0166] Furthermore, the sum of the signed distance values of each point in the point cloud is calculated:
[0167]
[0168]
[0169] where ρ(·) represents the Huber loss function.
[0170] At the same time, calculate the sum of the color differences between each point and the corresponding pixel in the previous frame:
[0171]
[0172] Furthermore, the weighted average of the two is used as the error function, that is,
[0173] E(ξ)=E sdf (ξ t )+wE int(ξ t );
[0174] Furthermore, a Gauss-Newton solver is used to solve the above-mentioned minimization error function to estimate the camera pose.
[0175] In optimizing the above E sdf (ξ t ) item, an important step is to calculate r(·) is the decoder This is a highly nonlinear composite function of , which can lead to poor local estimates. Therefore, σ should be used in the local linearization. D This significantly improves the convergence of the optimization. The estimated gradient is calculated as follows:
[0176]
[0177] Among them, R t-1 T t-1 The rotating part, p t is the 3D coordinate of the point in the first point cloud, (p) ⊙ Defined as [Ip ∧ ], where p ∧ Transform p into its antisymmetric matrix form, T(ξ t ) represents the relative position, This can be computed efficiently using back-propagation through the decoder network.
[0178] In step S310 of the embodiment of the present invention, the point cloud of the current frame is obtained by back-projection of the camera intrinsic parameters, the point cloud information of the current frame is encoded by the encoder network model, and the latent vector corresponding to the point cloud information of the current frame is obtained, and then the latent vector is decoded by the decoder network model to obtain the surface model of the point cloud information of the current frame, with the goal of minimizing the sum of the signed distance values of each point in the point cloud and the sum of the color differences between each point and the corresponding pixel in the previous frame, and the approximate gradient when the standard deviation of the signed distance probability distribution is regarded as a constant is used for optimization to estimate the camera pose.
[0179] Step S320, using the obtained camera position to obtain point cloud information in the world coordinate system, storing it in PLIVox, and integrating it into the surface model. Specifically:
[0180] The latent vector obtained by encoding the internal point cloud information of each PLIVox in the current frame through the encoder is weighted by the number of points accumulated in the PLIVox and the overall latent vector corresponding to the PLIVox is weighted averaged to efficiently complete the fusion of the surface model. The specific formula is defined as follows:
[0181]
[0182] Among them, l m represents the latent vector accumulated by the m-th PLIVox before fusion with the t-th frame information, represents the latent vector obtained through the point cloud information of the tth frame, represents the weight of the information of the t-th frame, and its specific value is the number of points in the m-th PLIVox of the point cloud of the t-th frame; w m Represents the weight of the accumulated latent vector, which is obtained by adding the weights of the latent vectors fused each time.
[0183] In step S320 of the embodiment of the present invention, the latent vector obtained by encoding the internal point cloud information of each PLIVox corresponding to the current frame through the encoder is weighted by the number of points accumulated in the PLIVox and the latent vector of the historical population corresponding to the PLIVox is weighted averaged to efficiently complete the fusion of the surface model.
[0184] Step S330: At any time during the reconstruction process, the grid representation of the current scene can be extracted with any definition as needed. Specifically,
[0185] The latent vector corresponding to PLIVox is decoded through the decoder network model to obtain the signed distance value of the corresponding coordinate prescription grid, and the texture is obtained by finding the average of the RGB values of multiple nearest points at the corresponding coordinates in the stored multi-frame colored point cloud. Finally, the surface can be reconstructed by the marching cube algorithm.
[0186] In step S330 of the embodiment of the present invention, by decoding the latent vector corresponding to PLIVox based on the decoder network model, and obtaining the texture through the stored multi-frame colored point cloud information, and then reconstructing the surface through the marching cube algorithm, it is possible to extract the grid representation of the current scene with arbitrary clarity.
[0187] An embodiment of the present invention provides a real-time 3D reconstruction algorithm based on implicit fields with deep prior knowledge. The method constructs an advanced probabilistic local implicit voxel PLIVox, which uses the same deep neural network to simultaneously encode the geometric structure of the scene and an additional uncertainty feature to effectively assist reconstruction; and based on the PLIVox representation, an approximate gradient is designed to efficiently complete camera tracking. In addition, by constructing a deep implicit representation in the encoder-decoder network model of the present invention, geometric integration is achieved in the latent vector field, and a high-quality surface model is achieved more efficiently than the fusion method based on the truncated signed distance function (TSDF). The method of the present invention achieves camera tracking and surface model quality that is superior to classical methods.
[0188] Figure 4 is a schematic diagram of the structure of the three-dimensional reconstruction device provided by the present invention, such as Figure 4 As shown, including:
[0189] A world coordinate system point cloud information acquisition module 410 is used to calculate second point cloud information of the first depth three-primary color image in the world coordinate system according to the first point cloud information of the first depth three-primary color image;
[0190] The first depth three-primary color image is any frame image among the acquired continuous multiple frames of depth three-primary color images;
[0191] A probabilistic local implicit voxel acquisition module 420 is used to store the second point cloud information in a preset voxel grid set, calculate the implicit vector of the second point cloud information in each preset voxel grid, and obtain the first probabilistic local implicit voxel corresponding to the first depth three-primary color image;
[0192] An implicit three-dimensional scene representation generation module 430 is used to update the latent vector in the historical probability local implicit voxel according to the latent vector in the first probability local implicit voxel, so as to obtain the implicit three-dimensional scene representation corresponding to the first depth three-primary color image;
[0193] The latent vector in the historical probability local latent voxel is obtained based on the point cloud information of each frame image before the first depth three-primary color image.
[0194] The device described in this embodiment can be used to execute the above method embodiment, and its principles and technical effects are similar, which will not be repeated here.
[0195] Figure 5 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5As shown, the electronic device may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute the three-dimensional reconstruction method, which includes: calculating the second point cloud information of the first deep three-primary color image in the world coordinate system based on the first point cloud information of the first deep three-primary color image; wherein the first deep three-primary color image is any frame image of the acquired continuous multiple frames of deep three-primary color images; storing the second point cloud information in a preset voxel grid set, calculating the implicit vector of the second point cloud information in each preset voxel grid, and obtaining the first probability local implicit voxel corresponding to the first deep three-primary color image; according to the implicit vector in the first probability local implicit voxel, updating the implicit vector in the historical probability local implicit voxel, and obtaining the implicit three-dimensional scene representation corresponding to the first deep three-primary color image; wherein the implicit vector in the historical probability local implicit voxel is obtained based on the point cloud information of each frame image before the first deep three-primary color image.
[0196] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0197] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the three-dimensional reconstruction method provided by the above methods, and the method includes: according to the first point cloud information of the first depth three-primary color image, calculating the second point cloud information of the first depth three-primary color image in the world coordinate system; wherein the first depth three-primary color image is any frame image in the acquired continuous multiple frames of depth three-primary color images; storing the second point cloud information in a preset voxel grid set, calculating the implicit vector of the second point cloud information in each preset voxel grid, and obtaining the first probability local implicit voxel corresponding to the first depth three-primary color image; according to the implicit vector in the first probability local implicit voxel, updating the implicit vector in the historical probability local implicit voxel, and obtaining the implicit three-dimensional scene representation corresponding to the first depth three-primary color image; wherein the implicit vector in the historical probability local implicit voxel is obtained based on the point cloud information of each frame image before the first depth three-primary color image.
[0198] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented by a processor to execute the three-dimensional reconstruction method provided by the above-mentioned methods when the computer program is executed, the method comprising: calculating the second point cloud information of the first depth three-primary color image in the world coordinate system based on the first point cloud information of the first depth three-primary color image; wherein the first depth three-primary color image is any frame image of the acquired continuous multiple frames of depth three-primary color images; storing the second point cloud information in a preset voxel grid set, calculating the latent vector of the second point cloud information in each preset voxel grid, and obtaining the first probability local implicit voxel corresponding to the first depth three-primary color image; updating the latent vector in the historical probability local implicit voxel based on the latent vector in the first probability local implicit voxel, and obtaining the implicit three-dimensional scene representation corresponding to the first depth three-primary color image; wherein the latent vector in the historical probability local implicit voxel is obtained based on the point cloud information of each frame image before the first depth three-primary color image.
[0199] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0200] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0201] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A three-dimensional reconstruction method, It is characterized in that include: Calculate, according to the first point cloud information of the first depth three-primary-color image, second point cloud information of the first depth three-primary-color image in a world coordinate system; The first depth three-primary color image is any frame image of the acquired continuous multiple frames of depth three-primary color images; The second point cloud information is stored in a preset voxel grid set, and a latent vector of the second point cloud information in each preset voxel grid is calculated to obtain a first probabilistic local latent voxel corresponding to the first depth three-primary color image; According to the latent vector in the first probability local latent voxel, the latent vector in the historical probability local latent voxel is updated to obtain an implicit three-dimensional scene representation corresponding to the first depth three-primary color image; The latent vector in the historical probability local latent voxel is obtained according to the point cloud information of each frame image before the first depth three-primary color image; The first probabilistic local implicit voxel refers to an implicit representation of the second point cloud information of the first depth three-primary color image in each preset voxel grid.
2. The three-dimensional reconstruction method according to claim 1, It is characterized in that The calculating, according to the first point cloud information of the first depth three-primary-color image, second point cloud information of the first depth three-primary-color image in the world coordinate system is specifically: According to the first point cloud information of the first depth three-primary color image, solving the camera pose corresponding to the first depth three-primary color image; According to the camera pose, second point cloud information of the first depth three-primary color image in a world coordinate system is calculated.
3. The three-dimensional reconstruction method according to claim 1, It is characterized in that The second point cloud information is stored in a preset voxel grid set, and the latent vector of the second point cloud information in each preset voxel grid is calculated, specifically: According to the coordinate information of the second point cloud information, each point in the second point cloud information is stored in a preset voxel grid set to obtain second point cloud sub-information in each grid in the preset voxel grid set; The latent vector of each point in the second point cloud sub-information in each grid is calculated, and the latent vector of each point in each grid is weighted averaged to obtain the latent vector of the second point cloud information in each preset voxel grid.
4. The three-dimensional reconstruction method according to claim 1, It is characterized in that According to the latent vector in the first probability local latent voxel, the latent vector in the historical probability local latent voxel is updated to obtain the implicit three-dimensional scene representation corresponding to the first depth three-primary color image, specifically: Taking a weighted average of the latent vector in the first probability local latent voxel and the latent vector in the historical probability local latent voxel to obtain an updated latent vector in the probability local latent voxel; According to the latent vector in the updated probabilistic local implicit voxel, an implicit three-dimensional scene representation corresponding to the first depth three-primary color image is obtained.
5. The three-dimensional reconstruction method according to claim 2, It is characterized in that The step of solving the camera pose corresponding to the first depth three-primary-color image according to the first point cloud information of the first depth three-primary-color image is specifically as follows: Inputting the first point cloud information of the first depth three-primary color image into a decoder in a trained encoder-decoder network model to obtain a signed distance probability distribution of the first point cloud information; According to the signed distance probability distribution, solving the camera pose corresponding to the first depth three-primary color image; The trained encoder-decoder network model is obtained by training based on deep three-primary-color image point cloud information samples carrying signed distance probability distribution labels.
6. The three-dimensional reconstruction method according to claim 5, It is characterized in that According to the signed distance probability distribution, the camera pose corresponding to the first depth three-primary color image is solved, specifically: Calculating the sum of the signed distance values of each point in the first point cloud information according to the signed distance probability distribution; According to the coordinate information of the first point cloud information, find each point in the third point cloud information corresponding to each point in the first point cloud information to obtain a plurality of point pairs, and calculate the sum of the color differences of the three primary colors of each point pair; Wherein, the third point cloud information is the point cloud information of the previous frame image of the first depth three-primary color image; The camera pose corresponding to the first depth three-primary color image is obtained by performing a minimization error function calculation based on the sum of the signed distance values of each point in the first point cloud information and the sum of the three-primary color differences of each point pair.
7. The three-dimensional reconstruction method according to claim 1, It is characterized in that After acquiring the implicit three-dimensional scene representation corresponding to the first depth three-primary color image, the method further includes: Calculate texture based on preset multi-frame colored point cloud information; Calculating a signed distance value corresponding to each probabilistic local implicit voxel in the implicit three-dimensional scene representation according to an implicit vector in the implicit three-dimensional scene representation; Generate a three-dimensional color grid model corresponding to the first depth three-primary color image according to the symbol distance value and the texture; The first probabilistic local implicit voxel refers to an implicit representation of the second point cloud information of the first depth three-primary color image in each preset voxel grid.
8. The three-dimensional reconstruction method according to claim 5, It is characterized in that Before inputting the first point cloud information of the first depth three-primary color image into a decoder in the trained encoder-decoder network model, the method further includes: Obtaining multiple point cloud information samples of three-primary-color depth images carrying signed distance probability distribution labels; Each point cloud information sample of the deep three-primary color image carrying a signed distance probability distribution label is used as a group of training samples to obtain multiple groups of training samples, and the encoder-decoder network model is trained using the multiple groups of training samples.
9. The three-dimensional reconstruction method according to claim 8, It is characterized in that The encoder-decoder network model is trained using multiple sets of training samples, specifically: For any set of training samples, input the training samples into the encoder-decoder network model, and output the prediction probability corresponding to the training samples; Calculating a loss value using a preset loss function according to the predicted probability corresponding to the training sample and the signed distance probability distribution label in the training sample; If the loss value converges, the encoder-decoder network model training is completed.
10. A three-dimensional reconstruction device, It is characterized in that include: A world coordinate system point cloud information acquisition module, used to calculate second point cloud information of the first depth three-primary color image in the world coordinate system according to the first point cloud information of the first depth three-primary color image; The first depth three-primary color image is any frame image of the acquired continuous multiple frames of depth three-primary color images; A probabilistic local implicit voxel acquisition module, used to store the second point cloud information in a preset voxel grid set, calculate the implicit vector of the second point cloud information in each preset voxel grid, and obtain the first probabilistic local implicit voxel corresponding to the first depth three-primary color image; An implicit three-dimensional scene representation generation module, used to update the latent vector in the historical probability local implicit voxel according to the latent vector in the first probability local implicit voxel, so as to obtain the implicit three-dimensional scene representation corresponding to the first depth three-primary color image; The latent vector in the historical probability local latent voxel is obtained according to the point cloud information of each frame image before the first depth three-primary color image; The first probabilistic local implicit voxel refers to an implicit representation of the second point cloud information of the first depth three-primary color image in each preset voxel grid.
11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the steps of the three-dimensional reconstruction method according to any one of claims 1 to 9 are implemented.
12. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the three-dimensional reconstruction method according to any one of claims 1 to 9 are implemented.
Citation Information
Patent Citations
Real-time 3D reconstruction method based on depth map
CN108961390A
Three-dimensional model reconstruction technology based on deep learning
CN110223387A
