Three-dimensional reconstruction method, device, electronic equipment and storage medium

Abstract

The present disclosure relates to a three-dimensional reconstruction method, device, electronic device and storage medium, wherein the method obtains a depth map and a segmentation map of each frame of the image to be reconstructed estimated based on a position prediction network, and obtains the position and posture information of the image acquisition device that acquires the image to be reconstructed, and projects the pixel points of the image to be reconstructed to a three-dimensional space that has been divided into spatial voxel blocks according to the position and posture information and the depth map of each frame of the image to be reconstructed, thereby obtaining a plurality of voxel blocks of each frame of the image to be reconstructed, and obtaining the voxel feature value of each voxel block in each frame of the image to be reconstructed, and then performing face element extraction to obtain a three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed. In the above-mentioned three-dimensional reconstruction process, the depth map and segmentation map of each frame of the image to be reconstructed are estimated by the position prediction network, thereby not relying on the depth sensor, and face element extraction is performed by spatial voxel division, so that high-quality surface modeling can be achieved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a three-dimensional reconstruction method, device, electronic device and storage medium. Background Art

[0002] 3D Reconstruction refers to the establishment of a mathematical model suitable for computer representation and processing of three-dimensional objects. It is the basis for processing, operating and analyzing its properties in a computer environment. It is also a key technology for establishing virtual reality in computers to express the objective world.

[0003] With the popularity of manufacturing and augmented reality, gaming, and robotics applications, the demand for large-scale real-time 3D reconstruction has greatly increased. In these tasks, users or robots scan the space in real time, and the accumulated 3D information is instantly and continuously integrated into the required application, thereby providing immediate user feedback in the process of mapping the physical object to the virtual world. Although there are many existing 3D reconstruction systems, most of the 3D reconstruction technologies require large-scale parallel computing operations using GPUs on the PC side to meet real-time requirements. In the few 3D reconstruction technologies for mobile terminals, it is necessary to rely on the depth sensor of the mobile terminal to collect the depth map to achieve 3D reconstruction. Therefore, there is an urgent need for a 3D reconstruction technology that does not rely on the depth sensor of the mobile terminal and can achieve high-quality 3D reconstruction. Summary of the invention

[0004] The present disclosure provides a 3D reconstruction method, device, electronic device and storage medium to at least solve the problem of limited development of 3D reconstruction technology on mobile terminals in related technologies. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a three-dimensional reconstruction method is provided, comprising:

[0006] Obtaining a depth map and a segmentation map of each frame of the image to be reconstructed based on the estimation of the position prediction network, and obtaining the position and posture information of the image acquisition device that acquires the image to be reconstructed;

[0007] Projecting the pixel points of the image to be reconstructed into a three-dimensional space divided into spatial voxel blocks according to the posture information and the depth map of each frame of the image to be reconstructed, to obtain a plurality of voxel blocks of each frame of the image to be reconstructed;

[0008] Based on the depth value of the pixel point corresponding to each voxel block of each frame of the image to be reconstructed and the projection depth of the voxel block, a voxel feature value of each voxel block in each frame of the image to be reconstructed is obtained;

[0009] Facet extraction is performed according to the segmentation map of each frame of the image to be reconstructed and the voxel feature value of each voxel block in each frame of the image to be reconstructed, so as to obtain a three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed.

[0010] In one embodiment, the voxel feature value includes an initial voxel value and a preset initial weight value; the voxel feature value of each voxel block in each frame of the image to be reconstructed is obtained based on the depth value of the pixel point corresponding to each voxel block in each frame of the image to be reconstructed and the projection depth of the voxel block, including: determining the initial voxel value of the corresponding voxel block based on the difference between the depth value of the pixel point corresponding to each voxel block in each frame of the image to be reconstructed and the projection depth of the voxel block; updating the voxel feature value of the corresponding voxel block in the image to be reconstructed according to the initial voxel value of each voxel block in each frame of the image to be reconstructed and the preset initial weight value.

[0011] In one of the embodiments, updating the voxel feature value of the corresponding voxel block in each frame of the image to be reconstructed according to the initial voxel value and the preset initial weight value of each voxel block in the image to be reconstructed includes: obtaining the voxel update value of each voxel block in the image to be reconstructed by using a truncated signed distance function based on the initial voxel value and the preset initial weight value of each voxel block in the image to be reconstructed, and the voxel feature value of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed; and obtaining the weight update value of each voxel block in the image to be reconstructed based on the preset initial weight value of each voxel block in the image to be reconstructed and the weight value of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed.

[0012] In one embodiment, if the previous frame image adjacent to the image to be reconstructed is the first frame image, the voxel feature value of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed includes the initial voxel value of the voxel block and a preset initial weight value, and the weight value of the corresponding voxel block is the preset initial weight value; if the previous frame image adjacent to the image to be reconstructed is not the first frame image, the voxel feature value of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed includes the voxel update value and the weight update value of the voxel block, and the weight value of the corresponding voxel block is the corresponding weight update value.

[0013] In one of the embodiments, after obtaining the three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed, the method further includes: based on the three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed, using a spatial hash function to encode and store the position of the voxel block corresponding to the three-dimensional reconstructed image.

[0014] In one of the embodiments, the position prediction network includes a depth estimation network and a plane segmentation network; the acquisition of the depth map and segmentation map of each frame of the image to be reconstructed based on the estimation of the position prediction network includes: using the depth estimation network to perform depth detection on each frame of the image to be reconstructed to obtain a depth map and normal corresponding to the image to be reconstructed; inputting the depth map and the corresponding normal of each frame of the image to be reconstructed into the plane segmentation network to obtain a segmentation map corresponding to each frame of the image to be reconstructed.

[0015] In one of the embodiments, the step of obtaining the posture information of the image acquisition device that acquires the image to be reconstructed includes: extracting feature points from a previous frame image adjacent to the image to be reconstructed, tracking the extracted feature points in the image to be reconstructed based on the initial posture information, and obtaining first coordinate information of the feature points in the image to be reconstructed; determining the spatial coordinates corresponding to the feature points extracted from the previous frame image according to a depth map corresponding to the previous frame image, and reprojecting the spatial coordinates into the image to be reconstructed to obtain second coordinate information of the feature points in the image to be reconstructed; and performing posture optimization solution based on the first coordinate information and the second coordinate information to obtain the posture information.

[0016] In one of the embodiments, the face element extraction is performed according to the segmentation map of each frame of the image to be reconstructed and the voxel feature values ​​of each voxel block in each frame of the image to be reconstructed to obtain the three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed, including: obtaining the face element intersecting with the voxel block according to the segmentation map of each frame of the image to be reconstructed and the voxel feature values ​​of each voxel block in each frame of the image to be reconstructed; determining the normal of the face element based on the face element and the intersection point of the face element with the voxel block; and obtaining the three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed according to the face element and the normal of the face element.

[0017] In one of the embodiments, before obtaining the face element intersecting with the voxel block, the method also includes: based on the depth map of each frame of the image to be reconstructed, projecting several adjacent voxel blocks with the same depth value into a plane to obtain the plane projection coordinates of the several adjacent voxel blocks; if the plane projection coordinates are all located in a corresponding plane in the segmentation map, merging the several adjacent voxel blocks.

[0018] In one of the embodiments, the obtaining of the face elements intersecting with the voxel block includes: obtaining corresponding voxel feature values ​​based on the merged voxel block, and using a marching cubes algorithm to obtain the face elements intersecting with the merged voxel block.

[0019] According to a second aspect of an embodiment of the present disclosure, there is provided a three-dimensional reconstruction device, comprising:

[0020] An acquisition module is configured to acquire a depth map and a segmentation map of each frame of the image to be reconstructed estimated by the position prediction network, and to acquire the position and posture information of the image acquisition device that acquires the image to be reconstructed;

[0021] A voxel projection module is configured to project the pixel points of the image to be reconstructed into a three-dimensional space divided into spatial voxel blocks according to the posture information and the depth map of each frame of the image to be reconstructed, so as to obtain a plurality of voxel blocks of each frame of the image to be reconstructed;

[0022] A voxel feature value determination module is configured to obtain a voxel feature value of each voxel block in each frame of the image to be reconstructed based on a depth value of a pixel point corresponding to each voxel block in each frame of the image to be reconstructed and a projection depth of the voxel block;

[0023] The facet extraction module is configured to perform facet extraction according to the segmentation map of each frame of the image to be reconstructed and the voxel feature value of each voxel block in each frame of the image to be reconstructed, so as to obtain a three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed.

[0024] In one of the embodiments, the voxel feature value includes an initial voxel value and a preset initial weight value; the voxel feature value determination module includes: an initial voxel value calculation unit, configured to determine the initial voxel value of the corresponding voxel block based on the difference between the depth value of the pixel point corresponding to each voxel block in each frame of the image to be reconstructed and the projection depth of the voxel block; a voxel feature value update unit, configured to update the voxel feature value of the corresponding voxel block in the image to be reconstructed according to the initial voxel value of each voxel block in each frame of the image to be reconstructed and the preset initial weight value.

[0025] In one embodiment, the voxel feature value updating unit is configured to perform: based on the initial voxel value and the preset initial weight value of each voxel block in the image to be reconstructed, and the voxel feature value of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed, using a truncated signed distance function to obtain the voxel update value of each voxel block in the image to be reconstructed; and, based on the preset initial weight value of each voxel block in the image to be reconstructed and the weight value of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed, obtain the weight update value of each voxel block in the image to be reconstructed.

[0026] In one embodiment, if the previous frame image adjacent to the image to be reconstructed is the first frame image, the voxel feature value of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed includes the initial voxel value of the voxel block and a preset initial weight value, and the weight value of the corresponding voxel block is the preset initial weight value; if the previous frame image adjacent to the image to be reconstructed is not the first frame image, the voxel feature value of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed includes the voxel update value and the weight update value of the voxel block, and the weight value of the corresponding voxel block is the corresponding weight update value.

[0027] In one of the embodiments, the device further includes a coding storage module configured to perform: based on the three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed, encoding and storing the position of the voxel block corresponding to the three-dimensional reconstructed image using a spatial hash function.

[0028] In one embodiment, the position prediction network includes a depth estimation network and a plane segmentation network; the acquisition module is configured to execute: using the depth estimation network to perform depth detection on each frame of the image to be reconstructed to obtain a depth map and normal corresponding to the image to be reconstructed; inputting the depth map and corresponding normal of each frame of the image to be reconstructed into the plane segmentation network to obtain a segmentation map corresponding to each frame of the image to be reconstructed.

[0029] In one of the embodiments, the acquisition module is further configured to perform: extracting feature points from a previous frame image adjacent to the image to be reconstructed, tracking the extracted feature points in the image to be reconstructed based on initial pose information, and obtaining first coordinate information of the feature points in the image to be reconstructed; determining spatial coordinates corresponding to the feature points extracted from the previous frame image according to a depth map corresponding to the previous frame image, and reprojecting the spatial coordinates into the image to be reconstructed to obtain second coordinate information of the feature points in the image to be reconstructed; and performing pose optimization solution based on the first coordinate information and the second coordinate information to obtain the pose information.

[0030] In one embodiment, the face element extraction module is configured to perform: based on the segmentation map of each frame of the image to be reconstructed and the voxel feature value of each voxel block in each frame of the image to be reconstructed, obtain the face element intersecting with the voxel block; based on the face element and the intersection point between the face element and the voxel block, determine the normal of the face element; based on the face element and the normal of the face element, obtain the three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed.

[0031] In one of the embodiments, the face element extraction module is further configured to perform: based on the depth map of each frame of the image to be reconstructed, projecting several adjacent voxel blocks with the same depth value into a plane to obtain the plane projection coordinates of several adjacent voxel blocks; if the plane projection coordinates are all located in a corresponding plane in the segmentation map, merging the several adjacent voxel blocks; based on the merged voxel block, obtaining the corresponding voxel feature value, and using the moving cube algorithm to obtain the face element intersecting with the merged voxel block.

[0032] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions so that the electronic device executes the three-dimensional reconstruction method described in any one of the embodiments of the first aspect.

[0033] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the three-dimensional reconstruction method described in any one of the embodiments of the first aspect.

[0034] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, the program product comprising a computer program, the computer program being stored in a readable storage medium, at least one processor of a device reading and executing the computer program from the readable storage medium, so that the device performs the three-dimensional reconstruction method described in any one of the embodiments of the first aspect.

[0035] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: by obtaining the depth map and segmentation map of each frame of the image to be reconstructed estimated based on the position prediction network, and obtaining the position and posture information of the image acquisition device that acquires the image to be reconstructed, and projecting the pixel points of the image to be reconstructed to the three-dimensional space that has been divided into spatial voxel blocks according to the position and posture information and the depth map of each frame of the image to be reconstructed, a number of voxel blocks of each frame of the image to be reconstructed are obtained, and based on the depth value of the pixel points corresponding to each voxel block of each frame of the image to be reconstructed and the projection depth of the voxel block, the voxel feature value of each voxel block in each frame of the image to be reconstructed is obtained, and then the face element extraction is performed according to the segmentation map of each frame of the image to be reconstructed and the voxel feature value of each voxel block in each frame of the image to be reconstructed, so as to obtain the three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed. Since in the above-mentioned three-dimensional reconstruction process, the depth map and segmentation map of each frame of the image to be reconstructed are estimated by the position prediction network, it is not dependent on the depth sensor, and the face element extraction is performed by spatial voxel division, which can achieve high-quality surface modeling, thereby facilitating the development of three-dimensional reconstruction technology in mobile terminals.

[0036] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0038] Figure 1 The figure is a flow chart of a 3D reconstruction method according to an exemplary embodiment.

[0039] Figure 2 The figure is a schematic diagram of steps for obtaining voxel feature values ​​of a voxel block according to an exemplary embodiment.

[0040] Figure 3 is a flow chart of another 3D reconstruction method according to an exemplary embodiment.

[0041] Figure 4 The figure is a schematic diagram of steps for obtaining a depth map and a segmentation map according to an exemplary embodiment.

[0042] Figure 5 The figure is a schematic diagram of a specific scenario for obtaining a depth map and a segmentation map according to an exemplary embodiment.

[0043] Figure 6 The figure is a schematic diagram of a specific scenario for obtaining posture information according to an exemplary embodiment.

[0044] Figure 7 The figure is a schematic diagram of the steps of facet extraction according to an exemplary embodiment.

[0045] Figure 8 is a schematic diagram of a basic mode of a facet according to an exemplary embodiment.

[0046] Fig. 9 is a flow chart of yet another 3D reconstruction method according to an exemplary embodiment.

[0047] Fig.10 The figure is a block diagram of a 3D reconstruction device according to an exemplary embodiment.

[0048] Fig.11 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0049] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0050] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0051] The three-dimensional reconstruction method provided by the present disclosure can be applied to a terminal, a server, or a system including a terminal and a server, and can be implemented through the interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, and the server can be implemented as an independent server or a server cluster composed of multiple servers.

[0052] Figure 1 is a flow chart of a three-dimensional reconstruction method according to an exemplary embodiment. Figure 1 As shown, the following steps are included.

[0053] In step S110, a depth map and a segmentation map of each frame of the image to be reconstructed estimated based on the position prediction network are obtained, and the position and posture information of the image acquisition device that acquires the image to be reconstructed is obtained.

[0054] Among them, the position prediction network is a deep learning network capable of forward detection, for example, it can be a convolutional neural network. The image to be reconstructed refers to the image to be three-dimensionally reconstructed, which can be an image shot in real time, or an image frame obtained from video data shot in real time, or a static image stored locally or in a server, which is not limited in this embodiment. The depth map means that each pixel value of the image represents the distance between a certain point in the scene and the corresponding image acquisition device. The segmentation map is a planar binary map determined based on the projection state of each pixel in the image. For example, if the projection of a certain pixel is in the plane, its corresponding value in the segmentation map is represented by 1, and if the projection of a certain pixel is not in the plane, its corresponding value in the segmentation map is represented by 0. The posture information of the image acquisition device is the motion information of the image acquisition device estimated based on several images acquired by the image acquisition device. Since real-time dense depth estimation is a prerequisite for MR (Mixed Reality) to achieve immersive interaction, and the method of obtaining depth maps by adding depth sensors limits the development of three-dimensional reconstruction technology in mobile terminals, based on this, in this embodiment, when three-dimensional reconstruction is to be performed, the depth map and segmentation map of each frame of the image to be reconstructed are first obtained based on the position prediction network, and the posture information of the image acquisition device that acquires the image to be reconstructed is obtained, and the subsequent steps are used to complete the three-dimensional reconstruction.

[0055] In step S120, the pixel points of the image to be reconstructed are projected into a three-dimensional space divided into spatial voxel blocks according to the posture information and the depth map of each frame of the image to be reconstructed, so as to obtain a plurality of voxel blocks of each frame of the image to be reconstructed.

[0056] The three-dimensional space that has been divided into spatial voxel blocks refers to the three-dimensional space being divided into N*N*N grids per cubic meter. Specifically, in this embodiment, according to the above-obtained posture information and the depth map of each frame of the image to be reconstructed, the pixel points of the image to be reconstructed are projected into the three-dimensional space that has been divided into spatial voxel blocks, so as to obtain a plurality of voxel blocks corresponding to each frame of the image to be reconstructed after projection.

[0057] In step S130 , based on the depth value of the pixel point corresponding to each voxel block in each frame of the image to be reconstructed and the projection depth of the voxel block, the voxel feature value of each voxel block in each frame of the image to be reconstructed is obtained.

[0058] Among them, the depth value of a pixel point refers to the distance between the pixel point and the corresponding image acquisition device in the depth map. The projection depth of a voxel block refers to the distance between the voxel block and the corresponding image acquisition device after projection. The voxel eigenvalue is a parameter that expresses the surface area of ​​the real scene of the image based on the TSDF (truncated signed distance function) algorithm. In this embodiment, the voxel eigenvalue of each voxel block in each frame of the image to be reconstructed can be obtained by mathematical calculation based on the depth value of the pixel point corresponding to each voxel block of each frame of the image to be reconstructed and the projection depth of the voxel block.

[0059] In step S140 , facet extraction is performed according to the segmentation map of each frame of the image to be reconstructed and the voxel feature values ​​of each voxel block in each frame of the image to be reconstructed, so as to obtain a three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed.

[0060] A facet is a feature surface such as an object surface or a curved surface in a wing that is discretized. After generating a grid, a plane or a curved surface is used to replace the original object surface for each grid. In this embodiment, a facet is a plane or a curved surface that describes the surface area of ​​the image to be reconstructed based on a real scene. Specifically, facet extraction is performed based on the segmentation map of each frame of the image to be reconstructed and the voxel feature value of each voxel block in each frame of the image to be reconstructed, so as to obtain a 3D reconstructed image corresponding to each frame of the image to be reconstructed.

[0061] In the above three-dimensional reconstruction method, by obtaining the depth map and segmentation map of each frame of the image to be reconstructed estimated based on the position prediction network, and obtaining the position information of the image acquisition device that acquires the image to be reconstructed, and projecting the pixel points of the image to be reconstructed to the three-dimensional space that has been divided into spatial voxel blocks according to the position information and the depth map of each frame of the image to be reconstructed, a number of voxel blocks of each frame of the image to be reconstructed are obtained, and based on the depth value of the pixel points corresponding to each voxel block of each frame of the image to be reconstructed and the projection depth of the voxel block, the voxel feature value of each voxel block in each frame of the image to be reconstructed is obtained, and then according to the segmentation map of each frame of the image to be reconstructed and the voxel feature value of each voxel block in each frame of the image to be reconstructed, face element extraction is performed to obtain the three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed. Since in the above three-dimensional reconstruction process, the depth map and segmentation map of each frame of the image to be reconstructed are estimated by the position prediction network, it does not rely on the depth sensor, and face element extraction is performed by spatial voxel division, which can achieve high-quality surface modeling.

[0062] In an exemplary embodiment, the voxel feature value includes an initial voxel value and a preset initial weight value, such as Figure 2As shown, in step S130, based on the depth value of the pixel point corresponding to each voxel block of each frame of the image to be reconstructed and the projection depth of the voxel block, the voxel feature value of each voxel block in each frame of the image to be reconstructed is obtained, which can be specifically achieved by the following steps:

[0063] In step S210 , an initial voxel value of a corresponding voxel block is determined based on a difference between a depth value of a pixel corresponding to each voxel block of each frame of the image to be reconstructed and a projection depth of the voxel block.

[0064] The preset initial weight value may be a preset fixed value, which is usually set to 1. The initial voxel value is an initial TSDF value determined based on the characteristics of the image to be reconstructed. Specifically, in this embodiment, the difference between the depth value of the pixel point corresponding to each voxel block of each frame of the image to be reconstructed and the projection depth of the voxel block is calculated, so that the difference is determined as the initial voxel value of the corresponding voxel block.

[0065] In step S220 , the voxel feature value of the corresponding voxel block in the image to be reconstructed is updated according to the initial voxel value of each voxel block in each frame of the image to be reconstructed and the preset initial weight value.

[0066] Among them, updating the voxel feature value of the corresponding voxel block in the image to be reconstructed is a process of determining the fused TSDF value of each voxel block in the image to be reconstructed by comprehensively considering the features of the image to be reconstructed and the features of the previous frame image adjacent to the image to be reconstructed. In this embodiment, by updating the voxel feature value of the corresponding voxel block in the image to be reconstructed, the voxel update value and weight update value of the corresponding voxel block in the image to be reconstructed can be obtained.

[0067] In the above embodiment, based on the difference between the depth value of the pixel point corresponding to each voxel block of each frame of the image to be reconstructed and the projection depth of the voxel block, the initial voxel value of the corresponding voxel block is determined, and then according to the initial voxel value of each voxel block in each frame of the image to be reconstructed and the preset initial weight value, the voxel feature value of the corresponding voxel block in the image to be reconstructed is updated, that is, the TSDF value of each voxel block in the image to be reconstructed is fused and updated, so as to obtain a high-quality dense surface, which is conducive to the modeling of continuous surfaces.

[0068] In an exemplary embodiment, in step S220, the voxel feature value of the corresponding voxel block in the image to be reconstructed is updated according to the initial voxel value and the preset initial weight value of each voxel block in each frame of the image to be reconstructed, specifically including: based on the initial voxel value and the preset initial weight value of each voxel block in the image to be reconstructed, as well as the voxel feature value of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed, using a truncated signed distance function to obtain the voxel update value of each voxel block in the image to be reconstructed; and, based on the preset initial weight value of each voxel block in the image to be reconstructed and the weight value of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed, obtaining the weight update value of each voxel block in the image to be reconstructed. Among them, the voxel feature value of the corresponding voxel block in the previous frame image includes the voxel value and weight value of the corresponding voxel block. It should be noted that if the previous frame image adjacent to the image to be reconstructed is the first frame image (that is, the previous frame image is the first frame image), then the voxel value in the voxel feature value of the corresponding voxel block in the previous frame image is the corresponding initial voxel value, and its weight value is the corresponding initial weight value; if the previous frame image adjacent to the image to be reconstructed is not the first frame image, then the voxel value in the voxel feature value of the corresponding voxel block in the previous frame image is the corresponding voxel update value, and its weight value is the corresponding weight update value.

[0069] For example, for a voxel block x in the image i to be reconstructed, if the initial voxel value of the voxel block x is tsdf i (x), the preset initial weight value is w i (x), the voxel value of the corresponding voxel block x in the previous frame image (i-1) adjacent to the image to be reconstructed i is TSDF i-1 (x), the corresponding weight value is W i-1 (x), the following truncated signed distance function is used to calculate the voxel update value TSDF of the voxel block x in the image to be reconstructed i: i (x):

[0070]

[0071] The weight update value W of the voxel block x in the image i to be reconstructed is calculated using the following formula: i (x):

[0072] W i (x) = W i-1 (x)+w i (x)

[0073] In the above embodiment, based on the initial voxel value and the preset initial weight value of each voxel block in the image to be reconstructed, as well as the voxel feature value of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed, a truncated signed distance function is used to calculate the voxel update value of each voxel block in the image to be reconstructed, thereby realizing a fusion update of the voxel value of each voxel block in the image to be reconstructed.

[0074] Then, based on the preset initial weight value of each voxel block in the image to be reconstructed and the weight value of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed, the weight update value of each voxel block in the image to be reconstructed is obtained, so as to realize the fusion update of the weight value of each voxel block in the image to be reconstructed. The voxel features of each voxel block in the image to be reconstructed are fused with the voxel features of the corresponding voxel block in the previous frame image, that is, the information at different angles is fused, so that the noise can be reduced, which is conducive to improving the accuracy of three-dimensional reconstruction.

[0075] In an exemplary embodiment, if Figure 3 As shown, in step S140, after obtaining the 3D reconstructed image corresponding to each frame of the image to be reconstructed, the above method may further include:

[0076] In step S150 , based on the three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed, a spatial hash function is used to encode and store the position of the voxel block corresponding to the three-dimensional reconstructed image.

[0077] The position of the voxel block refers to the coordinate position of the voxel block, specifically, it can be the coordinate position of a vertex of the voxel block, for example, the coordinate position of the vertex of the lower left corner of the voxel block. In this embodiment, if the coordinate position of the vertex of the lower left corner of a voxel block in the 3D reconstructed image corresponding to the image to be reconstructed is (x, y, z), the spatial hash function f(x) is used to encode it as follows:

[0078] f(x)=(x×73856093^y×19349669^z×83492791)%N

[0079] Wherein, N is the length of the hash table, and the constant multiplied by the coordinate position is an empirical value. In this embodiment, the coordinate position of each voxel block in the 3D reconstructed image corresponding to the image to be reconstructed is encoded and stored by using a spatial hash function, thereby optimizing storage and computing memory.

[0080] In an exemplary embodiment, the position prediction network includes a depth estimation network and a plane segmentation network, then Figure 4 As shown, in step S110, a depth map and a segmentation map of each frame of the image to be reconstructed estimated based on the position prediction network are obtained, specifically including:

[0081] In step S410, a depth estimation network is used to perform depth detection on each frame of the image to be reconstructed to obtain a depth map and normal corresponding to the image to be reconstructed.

[0082] Among them, the depth estimation network is a deep learning network capable of forward detection, specifically, it can be a convolutional neural network. The depth map means that each pixel value of the image represents the distance between a certain point in the scene and the corresponding image acquisition device. The normal line refers to the line corresponding to the tangent line of each pixel point in the depth map. In this embodiment, the depth estimation network is used to perform depth detection on each frame of the image to be reconstructed, so as to obtain the depth map and normal line corresponding to the image to be reconstructed.

[0083] In step S420, the depth map and the corresponding normal of each frame of the image to be reconstructed are input into a plane segmentation network to obtain a segmentation map corresponding to each frame of the image to be reconstructed.

[0084] The plane segmentation network is also a deep learning network capable of forward detection, and specifically, it can also be a convolutional neural network. In this embodiment, the depth map and normal estimated by the depth estimation network are input into the plane segmentation network to obtain a segmentation map corresponding to each frame of the image to be reconstructed.

[0085] Specifically, Figure 5 As shown, since the depth map refers to each pixel value of the image representing the distance between a certain point in the scene and the corresponding image acquisition device, the depth value of the pixel in the depth map can be supervised based on the depth loss calculated by the depth predicted by the depth estimation network and the depth sample map (ie, the depth gt in the figure). The segmentation map is a plane binary map determined based on the projection state of each pixel in the image. Therefore, for any position in the depth map, the corresponding depth and normal are input into the plane segmentation network to obtain the corresponding plane segmentation information. For example, if the projection of a pixel is in the plane, the corresponding value in the segmentation map is represented by 1, and if the projection of a pixel is not in the plane, the corresponding value in the segmentation map is represented by 0. In this embodiment, the plane segmentation information in the segmentation map can also be supervised based on the plane segmentation loss calculated by the segmentation information predicted by the plane segmentation network and the plane sample map (ie, the plane gt in the figure).

[0086] In the above embodiment, a depth estimation network is used to perform depth detection on each frame of the image to be reconstructed to obtain a depth map and normal corresponding to the image to be reconstructed, and the depth map and the corresponding normal of each frame of the image to be reconstructed are input into a plane segmentation network to obtain a segmentation map corresponding to each frame of the image to be reconstructed. This method does not rely on a depth sensor and does not require the additional introduction of high-end image acquisition equipment, thereby improving the application scope of three-dimensional reconstruction.

[0087] In an exemplary embodiment, obtaining the posture information of an image acquisition device that acquires an image to be reconstructed specifically includes: extracting feature points from a previous frame of image adjacent to the image to be reconstructed, tracking the extracted feature points in the image to be reconstructed based on the initial posture information, and obtaining first coordinate information of the feature points in the image to be reconstructed. Determine the spatial coordinates corresponding to the feature points extracted from the previous frame of image based on the depth map corresponding to the previous frame of image, reproject the spatial coordinates into the image to be reconstructed, and obtain second coordinate information of the feature points in the image to be reconstructed. Perform posture optimization and solution based on the first coordinate information and the second coordinate information to obtain the posture information of the image acquisition device. The initial posture information is obtained based on the IMU (Inertial measurement unit) of the image acquisition device. Usually, there is a corresponding IMU reading for each frame of image to be reconstructed.

[0088] Specifically, Figure 6 As shown, I j is the image to be reconstructed, I i is the image to be reconstructed I j The adjacent previous frame image, then for image I i Perform feature extraction. If the extracted feature points include a1, a2, and a3, then based on the initial pose information, the extracted feature points a1, a2, and a3 are extracted in the image to be reconstructed I j The feature points are tracked in order to obtain the image to be reconstructed I j The first coordinate information of the corresponding feature point in I j a1', a2' and a3' in the previous frame image I i The corresponding depth map I id Determine image I i The spatial coordinates a1 corresponding to the extracted feature points a1, a2, and a3 t 、a2 t and a3 t , the spatial coordinate a1 t 、a2 t and a3 t Reproject to the image to be reconstructed I j In order to obtain the image I to be reconstructed j The second coordinate information of the corresponding feature point in I ja1", a2" and a3". Then, according to the first coordinate information a1', a2' and a3' and the second coordinate information a1", a2" and a3", the pose optimization solution is performed. Specifically, the PnP (Perspective-n-Point, a method for solving 3D to 2D point pair motion) algorithm can be used for optimization solution, that is, by minimizing the distance between a1", a2" and a3" and the corresponding a1', a2' and a3', the pose information of the image acquisition device is obtained, which realizes a robust repositioning method without relying on time coherence.

[0089] In an exemplary embodiment, if Figure 7 As shown, in step S140, facet extraction is performed according to the segmentation map of each frame of the image to be reconstructed and the voxel feature value of each voxel block in each frame of the image to be reconstructed to obtain a three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed, which specifically includes:

[0090] In step S710, based on the segmentation map of each frame of the image to be reconstructed and the voxel feature value of each voxel block in each frame of the image to be reconstructed, the facets intersecting with the voxel block are obtained.

[0091] In this embodiment, a marching cube algorithm can be used to obtain the facets intersecting with the voxel block in the image to be reconstructed. Specifically, for each voxel block, each vertex has two cases, that is, the vertex is higher than the equal value and is inside the surface, or the vertex is lower than the equal value and is outside the surface. A voxel block has 8 vertices, so there are 256 cases. Considering the rotational symmetry, after reclassifying these 256 cases, the following can be obtained: Figure 8 For example, in this embodiment, the corresponding mode is determined according to the positive and negative states of the voxel feature values ​​of each vertex of the voxel block, such as Figure 8 The first mode indicates that the voxel eigenvalues ​​of each vertex in the voxel block are all negative, that is, there is no isosurface inside the voxel block; Figure 8 The second mode in the figure indicates that there is a vertex among the vertices of the voxel block (that is, the vertex where the solid circle is located) whose voxel eigenvalue is positive, and the voxel eigenvalues ​​of other vertices are all negative, indicating that in this mode there is at least one isosurface inside the voxel block (that is, the corresponding small triangle in the figure, that is, the face element intersecting with the voxel block). The other modes are similar and will not be described here one by one.

[0092] In this embodiment, in order to quickly determine which mode a voxel block in the image to be reconstructed belongs to, the following can be used: Figure 8The state of each vertex in each mode shown in the figure is encoded into a voxel state table in advance, so that the position relationship between the isosurface and the edge inside the voxel can be quickly calculated according to the vertex state index of the current voxel block, that is, the intersection of the edge is obtained, that is, which edges have intersection points. Then, based on connecting the intersection points, the approximate surface element of the isosurface is obtained, that is, the surface element intersecting with the voxel block is obtained (such as Figure 8 The shaded small triangle inside the voxel block in the figure).

[0093] In an exemplary embodiment, based on the depth map of each frame of the image to be reconstructed, several adjacent voxel blocks with the same depth value can be projected into a plane to obtain the plane projection coordinates of several adjacent voxel blocks. If the plane projection coordinates of the obtained several adjacent voxel blocks are all located in a corresponding plane in the segmentation map, the several adjacent voxel blocks are merged, and then the corresponding voxel feature values ​​are obtained based on the merged voxel blocks, and the above-mentioned marching cube algorithm is used to obtain the face element intersecting with the merged voxel block. In this embodiment, by merging the voxel blocks, fine-grained voxel features can be deleted, and triangular face elements of different sizes can be output to further improve storage and computing efficiency.

[0094] In step S720 , a normal of the bin is determined based on the bin and the intersection point where the bin intersects with the voxel block.

[0095] Since there are at most 1 to 4 triangular face elements in each voxel block, the normal of each face element can be obtained by the normal calculation formula based on each face element in the voxel block and the intersection point between each face element and the voxel block (that is, the vertex of each face element).

[0096] In step S730, a 3D reconstructed image corresponding to each frame of the image to be reconstructed is obtained according to the surface elements and the normal lines of the surface elements.

[0097] Specifically, based on the obtained facets and facet normals, a dense surface that can represent the global image to be reconstructed is obtained. Therefore, by performing three-dimensional reconstruction based on the dense surface, a three-dimensional reconstructed image corresponding to the image to be reconstructed can be obtained. Since the facets are extracted using the method of the above embodiment, not only the global structure can be retained, but also high local accuracy can be achieved. Therefore, the corresponding three-dimensional reconstructed image can well reflect the global features and local features, thereby improving the quality of the three-dimensional reconstructed image.

[0098] In an exemplary embodiment, the following is combined Fig. 9 The three-dimensional reconstruction method of the present application is further described, and its specific implementation is as follows:

[0099] 1) Data acquisition part: Based on the depth estimation network, the depth map of each frame of the image to be reconstructed is obtained, and the depth map includes the depth value corresponding to each pixel; based on the plane segmentation network, the plane segmentation map corresponding to each frame of the image to be reconstructed is obtained; based on each frame of the image to be reconstructed and the IMU of the image acquisition device, the pose estimation of the image acquisition device is obtained. For details, please refer to Figure 4 , Figure 5 as well as Figure 6 The corresponding embodiments will not be described in detail in this embodiment.

[0100] 2) Based on the depth value and pose estimation obtained above, point cloud pre-processing is performed, such as statistical filtering, confidence screening, and point cloud smoothing, so as to obtain a point cloud map mapped to three-dimensional space.

[0101] 3) Based on the voxel division of the three-dimensional space, several voxel blocks of the point cloud image are obtained. For each voxel block, the TSDF fusion algorithm is used to calculate the corresponding TSDF value and weight value, and the spatial hash function is used to encode the position of the voxel block, and the corresponding voxel information is stored in blocks in the CPU.

[0102] 4) Multi-scale processing is performed based on depth values, such as the deletion of general dynamic objects. Voxel blocks that meet the conditions are merged based on plane segmentation, so as to achieve the deletion of fine-grained voxel blocks. Based on the processed depth map and plane segmentation map, the voxel truncation value of the voxel block is updated in the GPU, and the updated result is saved in the CPU.

[0103] 5) Based on the voxel cutoff value update result, the face element extraction and normal calculation are performed (normal smoothing can be performed to improve the accuracy of the normal), and then the face element map is updated. At the same time, the extracted face elements are output. The output face elements can also be used as a reference for the next frame of the image to be reconstructed in the 3D reconstruction process, such as TSDF fusion calculation.

[0104] In the above 3D reconstruction process, the depth map and segmentation map of each frame of the image to be reconstructed are estimated through the depth estimation network and the plane segmentation network, and the pose estimation is performed based on the IMU of the image acquisition device, so that it does not rely on the depth sensor, which is conducive to the development of 3D reconstruction technology in mobile terminals. Based on the deletion and merging of fine-grained voxel blocks, triangular face elements of different sizes are output, which improves the storage and computing efficiency.

[0105] It should be understood that although Figure 1-Figure 9 The steps in the flowchart are shown in sequence as indicated by the arrows, but the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of the steps, and the steps can be executed in other orders. Moreover, Figure 1-Figure 9At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0106] Fig.10 is a block diagram of a 3D reconstruction device according to an exemplary embodiment. Fig.10 The device includes an acquisition module 1002, a voxel projection module 1004, a voxel feature value determination module 1006 and a face extraction module 1008.

[0107] The acquisition module 1002 is configured to execute acquisition of a depth map and a segmentation map of each frame of the image to be reconstructed estimated based on the position prediction network, and to acquire the position and posture information of the image acquisition device that acquires the image to be reconstructed;

[0108] A voxel projection module 1004 is configured to project the pixel points of the image to be reconstructed to a three-dimensional space divided into spatial voxel blocks according to the posture information and the depth map of each frame of the image to be reconstructed, so as to obtain a plurality of voxel blocks of each frame of the image to be reconstructed;

[0109] The voxel feature value determination module 1006 is configured to obtain the voxel feature value of each voxel block in each frame of the image to be reconstructed based on the depth value of the pixel point corresponding to each voxel block in each frame of the image to be reconstructed and the projection depth of the voxel block;

[0110] The facet extraction module 1008 is configured to perform facet extraction according to the segmentation map of each frame of the image to be reconstructed and the voxel feature value of each voxel block in each frame of the image to be reconstructed, so as to obtain a 3D reconstructed image corresponding to each frame of the image to be reconstructed.

[0111] In an exemplary embodiment, the voxel feature value includes an initial voxel value and a preset initial weight value; the voxel feature value determination module includes: an initial voxel value calculation unit, configured to determine the initial voxel value of the corresponding voxel block based on the difference between the depth value of the pixel point corresponding to each voxel block in each frame of the image to be reconstructed and the projection depth of the voxel block; a voxel feature value update unit, configured to update the voxel feature value of the corresponding voxel block in the image to be reconstructed according to the initial voxel value of each voxel block in each frame of the image to be reconstructed and the preset initial weight value.

[0112] In an exemplary embodiment, the voxel feature value updating unit is configured to perform: based on the initial voxel value and the preset initial weight value of each voxel block in the image to be reconstructed, and the voxel feature value of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed, using a truncated signed distance function to obtain the voxel update value of each voxel block in the image to be reconstructed; and, based on the preset initial weight value of each voxel block in the image to be reconstructed and the weight value of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed, obtain the weight update value of each voxel block in the image to be reconstructed.

[0113] In an exemplary embodiment, if the previous frame image adjacent to the image to be reconstructed is the first frame image, the voxel feature values ​​of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed include the initial voxel value of the voxel block and a preset initial weight value, and the weight value of the corresponding voxel block is the preset initial weight value; if the previous frame image adjacent to the image to be reconstructed is not the first frame image, the voxel feature values ​​of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed include the voxel update value and the weight update value of the voxel block, and the weight value of the corresponding voxel block is the corresponding weight update value.

[0114] In an exemplary embodiment, the device further includes a coding storage module configured to perform: based on the three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed, encoding and storing the position of the voxel block corresponding to the three-dimensional reconstructed image using a spatial hash function.

[0115] In an exemplary embodiment, the position prediction network includes a depth estimation network and a plane segmentation network; the acquisition module is configured to execute: using the depth estimation network to perform depth detection on each frame of the image to be reconstructed to obtain a depth map and normal corresponding to the image to be reconstructed; inputting the depth map and corresponding normal of each frame of the image to be reconstructed into the plane segmentation network to obtain a segmentation map corresponding to each frame of the image to be reconstructed.

[0116] In an exemplary embodiment, the acquisition module is further configured to perform: extracting feature points from a previous frame image adjacent to the image to be reconstructed, tracking the extracted feature points in the image to be reconstructed based on initial pose information, and obtaining first coordinate information of the feature points in the image to be reconstructed; determining spatial coordinates corresponding to the feature points extracted from the previous frame image according to a depth map corresponding to the previous frame image, and reprojecting the spatial coordinates into the image to be reconstructed to obtain second coordinate information of the feature points in the image to be reconstructed; performing pose optimization solution based on the first coordinate information and the second coordinate information to obtain the pose information.

[0117] In an exemplary embodiment, the face element extraction module is configured to perform: based on the segmentation map of each frame of the image to be reconstructed and the voxel feature value of each voxel block in each frame of the image to be reconstructed, obtain the face element intersecting with the voxel block; based on the face element and the intersection point of the face element with the voxel block, determine the normal of the face element; based on the face element and the normal of the face element, obtain the three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed.

[0118] In an exemplary embodiment, the face element extraction module is further configured to perform: based on the depth map of each frame of the image to be reconstructed, projecting several adjacent voxel blocks with the same depth value into a plane to obtain the plane projection coordinates of the several adjacent voxel blocks; if the plane projection coordinates are all located in a corresponding plane in the segmentation map, merging the several adjacent voxel blocks; based on the merged voxel blocks, obtaining the corresponding voxel feature values, and using the moving cube algorithm to obtain the face elements intersecting with the merged voxel blocks.

[0119] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0120] Fig.11 1 is a block diagram of a device Z00 for three-dimensional reconstruction according to an exemplary embodiment. For example, the device Z00 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0121] Reference Fig.11 , the device Z00 may include one or more of the following components: a processing component Z02, a memory Z04, a power component Z06, a multimedia component Z08, an audio component Z10, an input / output (I / O) interface Z12, a sensor component Z14, and a communication component Z16.

[0122] The processing component Z02 generally controls the overall operation of the device Z00, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component Z02 may include one or more processors Z20 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component Z02 may include one or more modules to facilitate interaction between the processing component Z02 and other components. For example, the processing component Z02 may include a multimedia module to facilitate interaction between the multimedia component Z08 and the processing component Z02.

[0123] The memory Z04 is configured to store various types of data to support operations on the device Z00. Examples of such data include instructions for any application or method operating on the device Z00, contact data, phone book data, messages, pictures, videos, etc. The memory Z04 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0124] The power supply assembly Z06 provides power to the various components of the device Z00. The power supply assembly Z06 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device Z00.

[0125] The multimedia component Z08 includes a screen that provides an output interface between the device Z00 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component Z08 includes a front camera and / or a rear camera. When the device Z00 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0126] The audio component Z10 is configured to output and / or input audio signals. For example, the audio component Z10 includes a microphone (MIC), which is configured to receive external audio signals when the device Z00 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory Z04 or sent via the communication component Z16. In some embodiments, the audio component Z10 also includes a speaker for outputting audio signals.

[0127] I / O interface Z12 provides an interface between processing component Z02 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0128] The sensor assembly Z14 includes one or more sensors for providing various aspects of status assessment for the device Z00. For example, the sensor assembly Z14 can detect the open / closed state of the device Z00, the relative positioning of components, such as the display and keypad of the device Z00, and the sensor assembly Z14 can also detect the position change of the device Z00 or a component of the device Z00, the presence or absence of user contact with the device Z00, the orientation or acceleration / deceleration of the device Z00, and the temperature change of the device Z00. The sensor assembly Z14 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly Z14 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly Z14 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0129] The communication component Z16 is configured to facilitate wired or wireless communication between the device Z00 and other devices. The device Z00 can access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component Z16 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component Z16 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0130] In an exemplary embodiment, the device Z00 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0131] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory Z04 including instructions, and the above instructions can be executed by a processor Z20 of the device Z00 to perform the above method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0132] In an exemplary embodiment, a computer program product is also provided, comprising a computer program, wherein when the computer program is executed by a processor, the three-dimensional reconstruction method as described above is implemented.

[0133] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0134] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A three-dimensional reconstruction method, characterized in that: include: Obtaining a depth map and a segmentation map of each frame of the image to be reconstructed based on the estimation of the position prediction network, and obtaining the position and posture information of the image acquisition device that acquires the image to be reconstructed; Projecting the pixel points of the image to be reconstructed into a three-dimensional space divided into spatial voxel blocks according to the posture information and the depth map of each frame of the image to be reconstructed, to obtain a plurality of voxel blocks of each frame of the image to be reconstructed; Based on the depth value of the pixel point corresponding to each voxel block of each frame of the image to be reconstructed and the projection depth of the voxel block, a voxel feature value of each voxel block in each frame of the image to be reconstructed is obtained; Performing facet extraction according to the segmentation map of each frame of the image to be reconstructed and the voxel feature value of each voxel block in each frame of the image to be reconstructed, so as to obtain a three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed; The position prediction network includes a depth estimation network and a plane segmentation network; The method of obtaining a depth map and a segmentation map of each frame of the image to be reconstructed estimated based on the position prediction network includes: using the depth estimation network to perform depth detection on each frame of the image to be reconstructed to obtain a depth map and a normal corresponding to the image to be reconstructed; inputting the depth map and the corresponding normal of each frame of the image to be reconstructed into the plane segmentation network to obtain a segmentation map corresponding to each frame of the image to be reconstructed; The method of performing facet extraction according to the segmentation map of each frame of the image to be reconstructed and the voxel feature values ​​of each voxel block in each frame of the image to be reconstructed to obtain a three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed includes: obtaining facets intersecting with the voxel blocks according to the segmentation map of each frame of the image to be reconstructed and the voxel feature values ​​of each voxel block in each frame of the image to be reconstructed; determining the normal of the facet based on the facet and the intersection point between the facet and the voxel block; and obtaining the three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed according to the facet and the normal of the facet.

2. The method according to claim 1, characterized in that The voxel feature value includes an initial voxel value and a preset initial weight value; the voxel feature value of each voxel block in each frame of the image to be reconstructed is obtained based on the depth value of the pixel point corresponding to each voxel block of each frame of the image to be reconstructed and the projection depth of the voxel block, including: Determine an initial voxel value of the corresponding voxel block based on a difference between a depth value of a pixel point corresponding to each voxel block of each frame of the image to be reconstructed and a projection depth of the voxel block; According to the initial voxel value of each voxel block in each frame of the image to be reconstructed and the preset initial weight value, the voxel feature value of the corresponding voxel block in the image to be reconstructed is updated.

3. The method according to claim 2, characterized in that The updating of the voxel feature value of the corresponding voxel block in the image to be reconstructed according to the initial voxel value of each voxel block in each frame of the image to be reconstructed and the preset initial weight value includes: Based on the initial voxel value and the preset initial weight value of each voxel block in the image to be reconstructed, and the voxel feature value of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed, a voxel update value of each voxel block in the image to be reconstructed is obtained by using a truncated signed distance function; and Based on the preset initial weight value of each voxel block in the image to be reconstructed and the weight value of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed, the weight update value of each voxel block in the image to be reconstructed is obtained.

4. The method according to claim 3, characterized in that If the previous frame image adjacent to the image to be reconstructed is the first frame image, the voxel feature value of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed includes the initial voxel value of the voxel block and a preset initial weight value, and the weight value of the corresponding voxel block is the preset initial weight value; If the previous frame image adjacent to the image to be reconstructed is not the first frame image, the voxel feature value of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed includes the voxel update value and weight update value of the voxel block, and the weight value of the corresponding voxel block is the corresponding weight update value.

5. The method according to claim 1, characterized in that After obtaining the three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed, the method further includes: Based on the three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed, a spatial hash function is used to encode and store the position of the voxel block corresponding to the three-dimensional reconstructed image.

6. The method according to claim 1, characterized in that The step of acquiring the position and posture information of the image acquisition device that acquires the image to be reconstructed includes: Extracting feature points from a previous frame of image adjacent to the image to be reconstructed, and tracking the extracted feature points in the image to be reconstructed based on the initial posture information to obtain first coordinate information of the feature points in the image to be reconstructed; Determine the spatial coordinates corresponding to the feature points extracted from the previous frame image according to the depth map corresponding to the previous frame image, and reproject the spatial coordinates into the image to be reconstructed to obtain second coordinate information of the feature points in the image to be reconstructed; Perform posture optimization solution according to the first coordinate information and the second coordinate information to obtain the posture information.

7. The method according to claim 1, characterized in that Before obtaining the face element intersecting with the voxel block, the method further includes: Based on the depth map of each frame of the image to be reconstructed, a number of adjacent voxel blocks with the same depth value are projected into a plane to obtain the plane projection coordinates of the several adjacent voxel blocks; If the plane projection coordinates are all located in a corresponding plane in the segmentation map, the plurality of adjacent voxel blocks are merged.

8. The method according to claim 7, characterized in that The obtaining of the face element intersecting with the voxel block comprises: The corresponding voxel feature value is obtained based on the merged voxel block, and the surface element intersecting with the merged voxel block is obtained by using the marching cube algorithm.

9. A three-dimensional reconstruction device, characterized in that: include: An acquisition module is configured to acquire a depth map and a segmentation map of each frame of the image to be reconstructed estimated by the position prediction network, and to acquire the position and posture information of the image acquisition device that acquires the image to be reconstructed; A voxel projection module is configured to project the pixel points of the image to be reconstructed into a three-dimensional space divided into spatial voxel blocks according to the posture information and the depth map of each frame of the image to be reconstructed, so as to obtain a plurality of voxel blocks of each frame of the image to be reconstructed; A voxel feature value determination module is configured to obtain a voxel feature value of each voxel block in each frame of the image to be reconstructed based on a depth value of a pixel point corresponding to each voxel block in each frame of the image to be reconstructed and a projection depth of the voxel block; A facet extraction module is configured to perform facet extraction according to a segmentation map of each frame of the image to be reconstructed and a voxel feature value of each voxel block in each frame of the image to be reconstructed, so as to obtain a three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed; The position prediction network includes a depth estimation network and a plane segmentation network; the acquisition module is further configured to perform: using the depth estimation network to perform depth detection on each frame of the image to be reconstructed to obtain a depth map and a normal corresponding to the image to be reconstructed; inputting the depth map and the corresponding normal of each frame of the image to be reconstructed into the plane segmentation network to obtain a segmentation map corresponding to each frame of the image to be reconstructed; The face element extraction module is further configured to execute: according to the segmentation map of each frame of the image to be reconstructed and the voxel feature value of each voxel block in each frame of the image to be reconstructed, obtain the face element intersecting with the voxel block; Based on the face element and the intersection point between the face element and the voxel block, the normal of the face element is determined; and according to the face element and the normal of the face element, a three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed is obtained.

10. The device according to claim 9, characterized in that The voxel feature value includes an initial voxel value and a preset initial weight value; The voxel feature value determination module comprises: an initial voxel value calculation unit, configured to determine an initial voxel value of a corresponding voxel block based on a difference between a depth value of a pixel point corresponding to each voxel block of each frame of the image to be reconstructed and a projection depth of the voxel block; The voxel feature value updating unit is configured to update the voxel feature value of the corresponding voxel block in each frame of the image to be reconstructed according to the initial voxel value of each voxel block in the image to be reconstructed and a preset initial weight value.

11. The device according to claim 10, characterized in that The voxel feature value updating unit is configured to perform: Based on the initial voxel value and the preset initial weight value of each voxel block in the image to be reconstructed, and the voxel feature value of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed, a voxel update value of each voxel block in the image to be reconstructed is obtained by using a truncated signed distance function; as well as, Based on the preset initial weight value of each voxel block in the image to be reconstructed and the weight value of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed, the weight update value of each voxel block in the image to be reconstructed is obtained.

12. The device according to claim 11, characterized in that If the previous frame image adjacent to the image to be reconstructed is the first frame image, the voxel feature value of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed includes the initial voxel value of the voxel block and a preset initial weight value, and the weight value of the corresponding voxel block is the preset initial weight value; If the previous frame image adjacent to the image to be reconstructed is not the first frame image, the voxel feature value of the corresponding voxel block in the previous frame image adjacent to the image to be reconstructed includes the voxel update value and weight update value of the voxel block, and the weight value of the corresponding voxel block is the corresponding weight update value.

13. The device according to claim 9, characterized in that The device also includes a coding storage module configured to execute: Based on the three-dimensional reconstructed image corresponding to each frame of the image to be reconstructed, a spatial hash function is used to encode and store the position of the voxel block corresponding to the three-dimensional reconstructed image.

14. The device according to claim 9, characterized in that The acquisition module is further configured to execute: Extracting feature points from a previous frame of image adjacent to the image to be reconstructed, and tracking the extracted feature points in the image to be reconstructed based on the initial posture information to obtain first coordinate information of the feature points in the image to be reconstructed; Determine the spatial coordinates corresponding to the feature points extracted from the previous frame image according to the depth map corresponding to the previous frame image, and reproject the spatial coordinates into the image to be reconstructed to obtain second coordinate information of the feature points in the image to be reconstructed; Perform posture optimization solution according to the first coordinate information and the second coordinate information to obtain the posture information.

15. The device according to claim 9, characterized in that The facet extraction module is further configured to perform: Based on the depth map of each frame of the image to be reconstructed, a number of adjacent voxel blocks with the same depth value are projected into a plane to obtain the plane projection coordinates of the several adjacent voxel blocks; If the plane projection coordinates are all located in a corresponding plane in the segmentation map, the plurality of adjacent voxel blocks are merged.

16. The device according to claim 15, characterized in that The facet extraction module is further configured to perform: The corresponding voxel feature value is obtained based on the merged voxel block, and the surface element intersecting with the merged voxel block is obtained by using the marching cube algorithm.

17. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the three-dimensional reconstruction method according to any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the three-dimensional reconstruction method as claimed in any one of claims 1 to 8.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the three-dimensional reconstruction method according to any one of claims 1 to 8 is implemented.

Images