Object three-dimensional model reconstruction method, equipment, storage medium and device

Abstract

The invention relates to the technical field of image processing, and discloses an object three-dimensional model reconstruction method, equipment, a storage medium and a device, geometric features are stored based on a pyramid nerve body, so that a complex object can be expressed by using a very small multi-layer sensing network, and the training speed and the convergence speed can be improved; the rough depth image is utilized to construct the initial three-dimensional model of the object, the number of body rendering ray sampling points is reduced, and the possibility of falling into local optimum is reduced, so that the precision of the reconstruction model is improved, and the reconstruction time consumption is reduced.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, equipment, storage medium and device for reconstructing a three-dimensional model of an object. Background Art

[0002] Reconstructing accurate 3D model surfaces from multiple views is crucial for computer vision and graphics, for example, it can be applied to various applications in virtual and augmented reality, e-commerce and robotics. Currently, neural surface reconstruction algorithms based on voxel rendering have gradually become an important algorithm for reconstructing 3D models of objects, because of their high reconstruction quality and ability to represent complex objects. However, the above algorithm only uses a deep network of a pure multi-layer perceptron (MLP) to represent the neural surface field, and based on the requirements of volume rendering, it needs to sample a large number of points on the rays emitted by each pixel on each image for volume rendering, which leads to slow optimization speed and low convergence speed. Summary of the invention

[0003] The main purpose of the present invention is to provide a method, device, storage medium and apparatus for reconstructing a three-dimensional model of an object, aiming to solve the technical problems that the three-dimensional model reconstruction algorithm of the object only uses the deep network of a pure multi-layer perception network to express the neural surface field, and based on the volume rendering requirements, a large number of points need to be sampled on the rays emitted by each pixel on each image for volume rendering, which leads to slow optimization speed and low convergence speed.

[0004] To achieve the above object, the present invention provides a method for reconstructing a three-dimensional model of an object, the method comprising the following steps:

[0005] Obtain multi-view images of the object to be reconstructed, and obtain the camera internal and external parameters corresponding to each view image through the motion recovery structure algorithm;

[0006] The rough depth image corresponding to each perspective image is obtained by a multi-perspective stereo matching algorithm;

[0007] The object to be reconstructed and the background in each perspective image are segmented by an object segmentation algorithm to obtain an object segmentation map;

[0008] According to the internal and external parameters of the camera, the rough depth image, the object segmentation map and the multi-view image, a three-dimensional model of the object to be reconstructed is reconstructed through a surface reconstruction model based on a pyramid neural body. The surface reconstruction model based on a pyramid neural body consists of a pyramid neural body and a multi-layer perception network.

[0009] Optionally, the object three-dimensional model reconstruction method further includes:

[0010] According to the camera internal and external parameters and voxel rendering requirements, a ray is emitted from the camera origin to pass through the pixel to be rendered;

[0011] Selecting a plurality of sampling points on the ray;

[0012] Calculating the color value and normal vector of the sampling point and the signed distance value between the sampling point and the object to be reconstructed;

[0013] Performing volume rendering on the pixel to be rendered based on the color values, normal vectors, and signed distance values ​​of all sampling points corresponding to the pixel to be rendered to obtain a color value of the pixel to be rendered;

[0014] The surface reconstruction model based on the pyramid neural volume is optimized based on the color value of the pixel to be rendered and the real color value of the pixel to be rendered in the multi-view image.

[0015] Optionally, selecting a plurality of sampling points on the ray includes:

[0016] Using the vertex reconstructed from the rough depth image as the sampling center and constructing the sampling width according to the confidence of the depth image to obtain the sampling range;

[0017] A preset first number of sampling points is selected within the sampling range on the ray, and a preset second number of sampling points is selected outside the sampling range on the ray, wherein the preset first number is greater than the preset second number.

[0018] Optionally, the object three-dimensional model reconstruction method further includes:

[0019] synthesizing the rough depth image into a sparse three-dimensional point cloud according to a geometric consistency constraint;

[0020] Constructing a signed distance function supervision loss corresponding to the signed distance function according to the sparse three-dimensional point cloud;

[0021] The pyramid neural network is optimized based on the signed distance function supervised loss.

[0022] Optionally, the object three-dimensional model reconstruction method further includes:

[0023] Constructing the total variational loss of the pyramid neural body according to the continuity constraint;

[0024] Construct a normal vector constraint loss based on the constraint that the normal vector modulus is a preset value;

[0025] Constructing a segmentation map loss corresponding to the signed distance function according to the object segmentation map;

[0026] generating a total supervision loss according to the signed distance function supervision loss, the volume rendering color loss, the total variational loss, the normal vector constraint loss, and the segmentation map loss;

[0027] The pyramid neural body is optimized based on the total supervision loss.

[0028] Optionally, the pyramid neural volume divides voxels by a single-scale VM volume representation;

[0029] The single-scale VM body representation method is to decompose a three-dimensional vector into multiple groups of one-dimensional vectors and two-dimensional arrays through tensor decomposition.

[0030] Optionally, the pyramidal neural body divides voxels by pyramidal VM body representation;

[0031] The pyramid VM volume representation is composed of a plurality of single-scale VM volume representations with different resolutions, and the extracted set features include global features and local features.

[0032] In addition, to achieve the above-mentioned purpose, the present invention also proposes a three-dimensional model reconstruction device for an object, which includes a memory, a processor, and a three-dimensional model reconstruction program for an object stored in the memory and executable on the processor, wherein the three-dimensional model reconstruction program for the object is configured to implement the three-dimensional model reconstruction method for the object as described above.

[0033] In addition, to achieve the above objectives, the present invention also proposes a storage medium, on which a three-dimensional model reconstruction program for an object is stored. When the three-dimensional model reconstruction program for an object is executed by a processor, the three-dimensional model reconstruction method for an object as described above is implemented.

[0034] In addition, to achieve the above-mentioned purpose, the present invention also proposes a device for reconstructing a three-dimensional model of an object, the device comprising: a camera parameter acquisition module, a depth image acquisition module, an image object segmentation module and an object model reconstruction module;

[0035] The camera parameter acquisition module is used to acquire multi-view images of the object to be reconstructed, and to acquire the camera internal and external parameters corresponding to each view image by using a motion recovery structure algorithm;

[0036] The depth image acquisition module is used to acquire a rough depth image corresponding to each perspective image through a multi-perspective stereo matching algorithm;

[0037] The image object segmentation module is used to segment the object to be reconstructed from the background in each perspective image by an object segmentation algorithm to obtain an object segmentation map;

[0038] The object model reconstruction module is used to reconstruct the three-dimensional model of the object to be reconstructed according to the internal and external parameters of the camera, the rough depth image, the object segmentation map and the multi-view image through a surface reconstruction model based on a pyramid neural body. The surface reconstruction model based on a pyramid neural body consists of a pyramid neural body and a multi-layer perception network.

[0039] The present invention discloses a method of obtaining multi-view images of an object to be reconstructed, obtaining camera internal and external parameters corresponding to each view image, obtaining a rough depth image corresponding to each view image by a multi-view stereo matching algorithm, segmenting the object to be reconstructed and the background in each view image by an object segmentation algorithm, obtaining an object segmentation map, and reconstructing a three-dimensional object model of the object to be reconstructed by a surface reconstruction model based on a pyramid neural body according to the camera internal and external parameters, the rough depth image, the object segmentation map and the multi-view images. The surface reconstruction model based on the pyramid neural body consists of a pyramid neural body and a multi-layer perception network. Since the present invention stores geometric features based on the pyramid neural body, a complex object can be described by using a very small multi-layer perception network, thereby improving the training speed and the convergence speed. In addition, the present invention uses the rough depth image to construct an initial three-dimensional model of the object, reduces the number of volume rendering ray sampling points, reduces the possibility of falling into a local optimum, thereby improving the accuracy of the reconstructed model and reducing the reconstruction time. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the structure of a device for reconstructing a three-dimensional model of an object in a hardware operating environment involved in an embodiment of the present invention;

[0041] Figure 2 It is a schematic flow chart of a first embodiment of a method for reconstructing a three-dimensional model of an object according to the present invention;

[0042] Figure 3 It is an overall schematic diagram of object three-dimensional model reconstruction according to an embodiment of the method for object three-dimensional model reconstruction of the present invention;

[0043] Figure 4 A framework diagram of a surface reconstruction model based on a pyramid neural body according to an embodiment of a method for reconstructing a three-dimensional model of an object of the present invention;

[0044] Figure 5 A schematic diagram of algorithm comparison of an embodiment of a method for reconstructing a three-dimensional model of an object according to the present invention;

[0045] Figure 6 A schematic diagram of algorithm comparison of an embodiment of a method for reconstructing a three-dimensional model of an object according to the present invention;

[0046] Figure 7 A schematic diagram of algorithm comparison of an embodiment of a method for reconstructing a three-dimensional model of an object according to the present invention;

[0047] Figure 8 It is a structural block diagram of the first embodiment of the device for reconstructing a three-dimensional model of an object according to the present invention.

[0048] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0049] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0050] Reference Figure 1 , Figure 1 It is a schematic diagram of the structure of a device for reconstructing a three-dimensional model of an object in a hardware operating environment involved in an embodiment of the present invention.

[0051] like Figure 1 As shown, the object three-dimensional model reconstruction device may include: a processor 1001, such as a central processing unit (CPU) and a graphics processing unit (GPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), and the user interface 1003 may also include a standard wired interface and a wireless interface. The wired interface of the user interface 1003 may be a USB interface in the present invention. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable memory (Non-volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0052] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the device for reconstructing a three-dimensional model of an object, and may include more or less components than those shown in the figure, or a combination of certain components, or a different arrangement of components.

[0053] like Figure 1 As shown, the memory 1005 identified as a computer storage medium may include an operating system, a network communication module, a user interface module, and an object three-dimensional model reconstruction program.

[0054] exist Figure 1In the object three-dimensional model reconstruction device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the user device; the object three-dimensional model reconstruction device calls the object three-dimensional model reconstruction program stored in the memory 1005 through the processor 1001, and executes the object three-dimensional model reconstruction method provided by the embodiment of the present invention.

[0055] Reference Figure 2 , Figure 2 1 is a flow chart of a first embodiment of a method for reconstructing a three-dimensional model of an object according to the present invention, and provides a first embodiment of a method for reconstructing a three-dimensional model of an object according to the present invention.

[0056] It should be understood that reconstructing accurate 3D model surfaces from multiple views is essential for computer vision and graphics, for example, it can be applied to various applications in virtual and augmented reality, e-commerce and robotics. The depth-based multi-view reconstruction algorithm process can include: 1) obtaining the camera internal and external parameters of each view through the Structure from Motion (SFM) algorithm (e.g., COLMAP, VisualSFM, MVE, OpenMVG); 2) estimating the depth map through corresponding point matching, for example, PMVS, OpenMVS, CMVS and other multi-view stereo matching (Multi View Stereo, MVS) algorithms use the photometric consistency between views, or the MVSNet series uses the powerful feature extraction capabilities of neural networks to more robustly estimate the depth map; 3) using the Poisson reconstruction algorithm or the Truncated Signed Distance Function (TSDF) fusion algorithm from the multi-view depth map to perform 3D surface reconstruction. However, the reconstruction quality of this process is heavily dependent on the accuracy of the corresponding point matching relationship. Even if a neural network is used to extract features, it is still very difficult to find matching relationships for low-texture areas, and it often leads to the problem of partial lack of reconstruction results. Although voxel-based 3D surface reconstruction algorithms can generate complete models, these algorithms have difficulty outputting accurate surfaces due to the limitations of voxel resolution. Compared with the above algorithms, voxel-based neural surface reconstruction algorithms can be used as an alternative to improve the reconstruction quality, and this algorithm can describe complex objects. However, due to: 1) only using a pure MLP deep network to describe the neural surface field, the training speed and convergence speed are slow; 2) in order to reconstruct an accurate surface, a large number of points must be sampled for each ray emitted by each pixel on each image, so these algorithms usually take a long time to optimize (more than ten hours to about 1 day) to reconstruct a high-quality result. In addition, this type of algorithm only uses the color difference of the re-rendered image as supervision, which is easy to fall into the local optimum, resulting in the loss of accuracy of the reconstructed 3D model.

[0057] Therefore, in order to overcome some of the above-mentioned defects, in this embodiment, geometric features are stored based on pyramid neural networks, so that complex objects can be described using a very small multi-layer perception network, thereby improving the training speed and convergence speed; and this embodiment uses a rough depth image to construct an initial three-dimensional model of the object, reduces the number of volume rendering ray sampling points, and reduces the possibility of falling into a local optimum, thereby improving the accuracy of the reconstructed model and reducing the reconstruction time.

[0058] In a first embodiment, the method for reconstructing a three-dimensional model of an object comprises the following steps:

[0059] Step S10: Acquire multi-view images of the object to be reconstructed, and obtain the camera internal and external parameters corresponding to each view image by using a structure-from-motion algorithm.

[0060] It can be understood that the executor of the method of this embodiment can be an object three-dimensional model reconstruction device with data processing, network communication and program running functions, for example, a terminal device such as a computer, or other electronic equipment that can achieve the same or similar functions. This embodiment does not impose any restrictions on this.

[0061] It should be understood that in this embodiment, the multi-perspective image of the object to be reconstructed can be obtained by photographing the object to be reconstructed from multiple angles through an image acquisition device, or by placing multiple image acquisition devices at different angles to obtain multi-perspective images of the object to be reconstructed. This embodiment does not limit this. The image acquisition device can be a mobile phone camera, a camera, or other device, and the multi-perspective image can be an RGB image of the object to be reconstructed under multiple perspectives.

[0062] Since the principle of 3D model reconstruction is what you see is what you get, the shooting requirement is that the collected images must cover all areas of the object to be reconstructed, and each area must be visible from more than two viewing angles. Generally, 10-60 images of an object are sufficient.

[0063] Considering that the acquisition speed of multi-view images of the object to be reconstructed by shooting the object to be reconstructed from multiple angles is slow, in order to overcome the above-mentioned defects, in this embodiment, the multi-view images of the object to be reconstructed can also be obtained by recording a video of the object to be reconstructed. This embodiment does not impose any restrictions on this. In practical applications, it is necessary to ensure the clarity of the recorded video to ensure that the multi-view images of the object to be reconstructed can be sampled from the recorded video.

[0064] It should be noted that the Structure from Motion (SFM) algorithm can be COLMAP, VisualSFM, MVE, OpenMVG and other algorithms, and this embodiment is not limited to this. The camera internal and external parameters can include camera internal parameters and camera external parameters, wherein the camera internal parameters are the setting parameters when the camera is shooting, such as focal length, principal point position, etc., and the camera external parameters are the parameters such as position and angle of the camera when shooting.

[0065] Step S20: obtaining a rough depth image corresponding to each perspective image through a multi-perspective stereo matching algorithm.

[0066] It should be noted that the Multi View Stereo (MVS) algorithm is a computer vision method that uses multiple images to restore three-dimensional scene information. In this embodiment, the MVS algorithm can be a traditional algorithm such as PMVS, OpenMVS, CMVS, or a multi-view depth estimation network of the MVSNet series, which is not limited in this embodiment.

[0067] Step S30: Segment the object to be reconstructed and the background in each perspective image using an object segmentation algorithm to obtain an object segmentation map.

[0068] It is understandable that in order to distinguish the object to be reconstructed from the background, in this embodiment, the object to be reconstructed and the background in each perspective image are segmented by an object segmentation algorithm to obtain an object segmentation map. The object segmentation algorithm may be GraphCut, GrabCut, DenseCut, and Segment Anything Model (SAM) and other algorithms, which are not limited in this embodiment.

[0069] Step S40: reconstructing the three-dimensional model of the object to be reconstructed through a surface reconstruction model based on a pyramid neural body according to the internal and external parameters of the camera, the rough depth image, the object segmentation map and the multi-view image, wherein the surface reconstruction model based on the pyramid neural body is composed of a pyramid neural body and a multi-layer perception network.

[0070] It should be understood that after obtaining the camera internal and external parameters, the rough depth image and the object segmentation map, the object three-dimensional model of the object to be reconstructed is reconstructed by the surface reconstruction model based on the pyramid neural body. In this embodiment, by using the pyramid neural body rendering method, when the camera internal and external parameters are given, the surface reconstruction model based on the pyramid neural body renders the image obtained under the camera internal and external parameters, and the rendered image is compared with the real image, so that the object described by the surface reconstruction model based on the pyramid neural body is close to the object to be reconstructed.

[0071] For ease of understanding, refer to Figure 3 This is for illustration, but the present invention is not limited thereto. Figure 3It is an overall schematic diagram of reconstructing a three-dimensional model of an object. In the figure, a multi-view RGB image of the object to be reconstructed is first obtained, and then the multi-view RGB image is preprocessed, wherein the preprocessing step may include: obtaining camera internal and external parameters corresponding to each view RGB image through the SFM algorithm, obtaining a rough depth image corresponding to each view image through the MVS algorithm, segmenting the object to be reconstructed and the background in each view image through the object segmentation algorithm to obtain an object segmentation map, and then inputting the camera internal and external parameters, the rough depth image, the object segmentation map and the multi-view RGB image into a surface reconstruction model based on a pyramid neural body to obtain an object three-dimensional model of the object to be reconstructed.

[0072] In the present embodiment, a method is disclosed in which a multi-view image of an object to be reconstructed is obtained, and camera internal and external parameters corresponding to each view image are obtained, a rough depth image corresponding to each view image is obtained by a multi-view stereo matching algorithm, the object to be reconstructed and the background in each view image are segmented by an object segmentation algorithm to obtain an object segmentation map, and a three-dimensional object model of the object to be reconstructed is reconstructed by a surface reconstruction model based on a pyramid neural body according to the camera internal and external parameters, the rough depth image, the object segmentation map and the multi-view image, wherein the surface reconstruction model based on the pyramid neural body is composed of a pyramid neural body and a multi-layer perception network; since the present embodiment stores geometric features based on the pyramid neural body, a complex object can be described using a very small multi-layer perception network, thereby improving the training speed and the convergence speed; and the present embodiment uses the rough depth image to construct an initial three-dimensional model of the object, reduces the number of volume rendering ray sampling points, reduces the possibility of falling into a local optimum, thereby improving the accuracy of the reconstructed model and reducing the reconstruction time.

[0073] In some embodiments, the object three-dimensional model reconstruction method further includes:

[0074] According to the camera internal and external parameters and voxel rendering requirements, a ray is emitted from the camera origin to pass through the pixel to be rendered;

[0075] Selecting a plurality of sampling points on the ray;

[0076] Calculating the color value and normal vector of the sampling point and the signed distance value between the sampling point and the object to be reconstructed;

[0077] Performing volume rendering on the pixel to be rendered based on the color values, normal vectors, and signed distance values ​​of all sampling points corresponding to the pixel to be rendered to obtain a color value of the pixel to be rendered;

[0078] The surface reconstruction model based on the pyramid neural volume is optimized based on the color value of the pixel to be rendered and the real color value of the pixel to be rendered in the multi-view image.

[0079] It should be understood that, given a spatial position (x, y, z), the pyramid neural body will output the Sign Distance Function (SDF) of the position. SDF is used to represent the distance from the position to the nearest surface of the object. It is negative inside the object and positive outside the object. During the training process, to render the color value of a pixel, according to the requirements of voxel rendering, a ray is sent connecting the pixel and the position of the optical center. Multiple points are sampled on this ray. Each point passes through the pyramid neural body to obtain the SDF, normal vector and color value of the point. Finally, all the information of these sampled points is obtained, and the rendered color value of the voxel is finally obtained through the SDF-based volume rendering strategy. The loss is constructed based on the rendered color value and the true value of the image to optimize the pyramid neural body.

[0080] In order to convert the pyramid neural body into a three-dimensional patch model, since the pyramid neural body expresses the SDF function, the space can be evenly divided into grids of a certain resolution, the SDF value of each grid location is calculated, and then the 0 isosurface is extracted by the Marching Cubes algorithm to obtain the final three-dimensional patch model. If the space is evenly divided here to save space and computing resources, octrees and hash tables can be used for division.

[0081] The SDF-based volume rendering strategy is based on the SDF-based volume rendering method of NeuS, in which the 3D surface can be defined as the zero-isosurface of the SDF function. Where S is the 3D surface and f is the SDF function. According to the voxel rendering strategy, the color value of a certain pixel is estimated It is obtained by accumulating the geometric and color attributes of N sampling points, which are obtained on the ray v with o as the center and v as the direction with a step size of t, and is defined as {p(t) = o + t i υ|0,1,...,N,t i <t i+1}. So the rendered color value can be defined as:

[0082]

[0083] In the formula, α represents the opacity value, is the cumulative transmittance. Given the SDF function f(x), NeuS proposed a field density function φ s (x) = se -sx / (1+e -sx ) 2 , where s represents a learnable hyperparameter, and as the training converges, s gradually increases. It is worth noting that φ s (x) is the Sigmoid function Φ s(x), and the Sigmoid function can be regarded as the cumulative field density distribution. After a series of derivations, α can be finally defined as:

[0084]

[0085] Furthermore, the selecting a plurality of sampling points on the ray includes:

[0086] Using the vertex reconstructed from the rough depth image as the sampling center and constructing the sampling width according to the confidence of the depth image to obtain the sampling range;

[0087] A preset first number of sampling points is selected within the sampling range on the ray, and a preset second number of sampling points is selected outside the sampling range on the ray, wherein the preset first number is greater than the preset second number.

[0088] It should be understood that in order to reconstruct an accurate surface, the relevant 3D model reconstruction algorithms have to sample a large number of points for each ray emitted by each pixel on each image, so these algorithms usually take a long time to optimize (more than ten hours to about 1 day) to reconstruct a high-quality result. Therefore, in order to overcome the above defects, this embodiment combines the rough depth image output by the MVS algorithm to construct a depth-guided sampling strategy to accelerate the process of 3D model reconstruction based on implicit neural fields.

[0089] It should be noted that, in order to ensure the stability of the optimization process, when selecting multiple sampling points on the ray, this embodiment not only selects multiple sampling points within the sampling range, but also selects a small number of sampling points outside the sampling range.

[0090] For ease of understanding, refer to Figure 4 This is for illustration, but it does not limit this solution. The depth-guided sampling strategy is shown in the Coarse Depth-based Point Sampling section in the figure. In the figure, V represents the camera position, Depth represents the rough depth image, RCB represents the multi-view image, and range of Depth represents the depth sampling range. The core idea of ​​the depth-guided sampling strategy is to focus on sampling points near the surface. Given a rough depth image, the vertex converted from the depth can be used as the center to construct a sampling range given a range, thereby narrowing the sampling range. However, the noise of the rough depth map is relatively large, especially in the contour area and low-texture area. In fact, the real surface may be outside the fixed range. Since the depth of the contour area is confused, the information of the contour area is not used for reconstruction. The contour area C can be detected by the following formula:

[0091]

[0092] Where ΔD(i) represents the number of depth differences greater than a preset threshold (20 mm) in the 7x7 area, T num Set to 20. For other areas, define two fixed ranges {[dr i , d+r i ]|i=0,1}, corresponding to the precise depth region and the rough depth region respectively. Define r0=0.04m, r1=0.1m corresponding to DTU and BlendMVS datasets. Of course, this parameter can be adjusted for different scenes to make the reconstruction result more accurate. The precise region A can be found by the following formula:

[0093] A=N gc ≥T views

[0094] Where N gc The number of views representing geometric consistency, T views Set to 5. For the depth map D j The geometric consistency G between the pixel i at the viewpoint j and t j，t (i) can be defined as:

[0095]

[0096] In the formula, rep represents D j (i) The generated point is projected to the pixel at the viewing angle t, rep i Indicates that from D t (rep) back-projects the pixels at view j. Therefore, the geometric consistency quantity map at view j is It can be obtained through the following formula:

[0097]

[0098] In the formula, represents the set of perspectives near perspective j.

[0099] In some embodiments, the pyramidal neural volume divides voxels by a single-scale VM volume representation;

[0100] The single-scale VM body representation method is to decompose a three-dimensional vector into multiple groups of one-dimensional vectors and two-dimensional arrays through tensor decomposition.

[0101] It should be understood that, assuming that a uniformly divided voxel representation is used, in order to reconstruct the model with sufficient precision, the voxels must be divided finely enough. However, this will make the model space particularly large. A neural network with a resolution of 256x256x256 requires 600Mb of space. In order to reduce the model space, the three-dimensional vector is decomposed into three groups of one-dimensional vectors + two-dimensional arrays using tensor decomposition, that is, a single-scale VM volume representation is used to divide the voxels. Voxel is the abbreviation of volume pixel, which is a unit used to represent three-dimensional space data. It is worth noting that in this embodiment, voxels can also be divided by other representations, such as octree voxel representation, uniformly divided voxel representation, etc. In a specific implementation, the single-scale VM volume representation represents the geometric feature volume. In the formula, I, J, K represent the resolution along the X, Y, and Z axes respectively, and F represents the number of feature channels. V can be decomposed along the XYZ axes into Therefore, given any point p, the geometric features can be obtained by interpolation through the following formula

[0102]

[0103] In the formula, represents the concat operation on the feature dimension. interp(p, (υ, M)) represents the interpolation from the VM tensor (v, M), which can be defined as:

[0104] interp(p,(υ,M))=interp(p,υ)interp(p,M)

[0105] Where interp(p, x) is the interpolation of tensor x at position p. Linear interpolation is used here.

[0106] Furthermore, the pyramidal neural body divides voxels by pyramidal VM body representation;

[0107] The pyramid VM volume representation is composed of a plurality of single-scale VM volume representations with different resolutions, and the extracted set features include global features and local features.

[0108] It is understandable that in order to reconstruct a smooth and accurate 3D surface, geometric features containing global and local information are indispensable. However, the geometric features obtained using a single-scale VM volume representation It can only represent local information, and its spatial receptive field depends on the size of a single voxel. Including global information will inevitably lose some detailed structural information. To solve this problem, Figure 4The PVSurF (i.e., pyramid neural volume) part shown in the figure, the pyramid volume representation method is proposed in this embodiment, so as to provide geometric features with both global and local information. Specifically, a low-resolution volume is used to provide global geometric features, and a high-resolution volume generates local detail geometric features. In this embodiment, the number of pyramid layers is 3, and a number of layers greater than 3 can be used, but it has been verified through experiments that the reconstruction effect of a pyramid less than 3 is not ideal. Therefore, the global and local geometric features are included. Can be defined as:

[0109]

[0110] In the formula, Represents the geometric features interpolated from the i-th individual. are the weights and biases in FC.

[0111] In obtaining Afterwards, this geometric feature is concat with the position code and then sent to a 2-layer MLP network to estimate the SDF value and the features for estimating the color information. Finally, the pyramid neural body estimates the color information using the normal vector, the sight direction and the features. The normal vector can be obtained by taking the first-order derivative of the SDF function.

[0112] From the above description, using a single-scale VM body can also use a very small MLP network to describe objects with complex structures. The final optimized rendering image effect is ideal, but the surface of the three-dimensional reconstruction is not smooth. On the contrary, there is a lot of noise. The first-order derivative of linear interpolation cannot be guaranteed to be completely continuous, which actually means that the continuity of the normal vector of the three-dimensional structure surface cannot be guaranteed, which will cause the reconstructed three-dimensional surface to have large noise. The discontinuity of the first-order derivative of linear interpolation exists on the interface between voxels. Therefore, in order to be able to output a continuous three-dimensional surface, two methods are provided in this embodiment: a. Use cubic spline interpolation to ensure the continuity of the first-order derivative; b. Use a pyramid structure, and the discontinuous area of ​​the coarse-grained resolution neural body can be basically ignored, so that it can ensure that the normal vector of the overall three-dimensional surface is continuous, and the fine-grained resolution neural body is used to represent the details. Because the normal vector of the detail structure (such as a right-angle structure) is actually discontinuous, and the use of cubic spline interpolation with strict continuity of the first-order derivative will lose the ability of the neural body to express details. Therefore, the pyramid neural body proposed in this embodiment is crucial in reconstructing a smooth and accurate three-dimensional surface.

[0113] In some embodiments, the object three-dimensional model reconstruction method further includes:

[0114] synthesizing the rough depth image into a sparse three-dimensional point cloud according to a geometric consistency constraint;

[0115] Constructing a signed distance function supervision loss corresponding to the signed distance function according to the sparse three-dimensional point cloud;

[0116] The pyramid neural network is optimized based on the signed distance function supervised loss.

[0117] It should be understood that the related 3D model reconstruction algorithm only uses the color difference of the re-rendered image as supervision, which is easy to fall into the local optimum, resulting in the loss of the accuracy of the reconstructed 3D model. Therefore, in order to overcome the above defects, this embodiment uses a high-precision sparse point cloud (filtered from the rough depth image) as prior knowledge to avoid some areas from falling into the local optimum, thereby improving the reconstruction accuracy.

[0118] By fitting the local plane, the normal vector map is estimated from the depth map. According to the geometric consistency filtering strategy, an accurate point cloud with normal vectors is obtained, such as Figure 4 As shown at the bottom of the figure, Depth represents the rough depth image, Pointcloud represents the sparse three-dimensional point cloud, and SDF Loss represents the signed distance function supervision loss. For each precise vertex, a point p is obtained by randomly offsetting the normal vector by t in the range of [-0.02, 0.02]. sdf =||f(p)+t|| to supervise SDF.

[0119] In some embodiments, the object three-dimensional model reconstruction method further includes:

[0120] Constructing the total variational loss of the pyramid neural body according to the continuity constraint;

[0121] Construct a normal vector constraint loss based on the constraint that the normal vector modulus is a preset value;

[0122] Constructing a segmentation map loss corresponding to the signed distance function according to the object segmentation map;

[0123] generating a total supervision loss according to the signed distance function supervision loss, the volume rendering color loss, the total variational loss, the normal vector constraint loss, and the segmentation map loss;

[0124] The pyramid neural body is optimized based on the total supervision loss.

[0125] It should be noted that constructing the normal vector constraint loss according to the constraint that the normal vector modulus is a preset value may specifically be constructing the normal vector constraint loss according to the constraint that the normal vector modulus is 1. The volume rendering color loss may be determined based on the color value of the pixel to be rendered and the actual color value of the pixel to be rendered in the multi-view image.

[0126] It can be understood that in order to avoid reconstructing a noisy point cloud, in addition to the design of the pyramid neural body structure, this embodiment also introduces a total variation loss to constrain the continuity of the neural body. The specific loss is as follows:

[0127]

[0128] In the formula, Δ 2 Represents the square of the neighborhood interpolation. N represents the total number of voxels.

[0129] In order to reconstruct a high-quality 3D model, in this embodiment, the volume rendering color loss, the normal vector constraint loss, and the segmentation map loss need to be introduced to generate a total supervision loss, which is specifically as follows:

[0130]

[0131]

[0132] L free =(1-M k )BCE(M k ,Sigmoid(-βf(p k，j ))).

[0133] L=L c +λ eik L eik +λ tυ L TV +λ sdf L sdf +λ m L m .

[0134] Where, L color and L c is the volume rendering color loss, L eik is the normal vector constraint loss, L mask is the segmentation map loss, L TV is the total variational loss, L sdf is the signed distance function supervision loss, and L is the total supervision loss.

[0135] For ease of understanding, the following is a comparative explanation. When the data set uses: DTU and BlendMVS, the quantitative comparison data with the SOTA algorithm is as follows Figure 5 As shown in the figure, ScanID represents scenes of different objects. Both w / o mask and w / mask experiments are shown. It can be seen that the overall algorithm of this embodiment is improved from 0.77 to 0.67, and the indicator used is chamfer distance.

[0136] Comparison of reconstruction time and model size Figure 6 As shown in the figure, it can be seen that compared with the high-precision reconstruction algorithm, the reconstruction time of the algorithm in this embodiment is significantly reduced. Compared with the volume representation method, not only is the accuracy significantly improved, but the model size is also relatively small.

[0137] Visualization comparison Figure 7 As shown in the figure, the algorithm of this embodiment ensures smoothness without losing detail information, while NeuS is sometimes too smooth and TensoRF has too much noise.

[0138] In addition, an embodiment of the present invention further provides a storage medium, on which a three-dimensional model reconstruction program for an object is stored. When the three-dimensional model reconstruction program for an object is executed by a processor, the three-dimensional model reconstruction method for an object as described above is implemented.

[0139] In addition, refer to Figure 8 , an embodiment of the present invention also provides a device for reconstructing a three-dimensional model of an object.

[0140] It should be understood that reconstructing accurate 3D model surfaces from multiple views is crucial for computer vision and graphics, for example, it can be applied to various applications in virtual and augmented reality, e-commerce and robotics. The traditional depth-based multi-view reconstruction algorithm process mainly includes: 1) obtaining the camera internal and external parameters of each view through the Structure from Motion (SFM) algorithm (for example: COLMAP, VisualSFM, MVE, OpenMVG); 2) estimating the depth map through corresponding point matching, for example: PMVS, OpenMVS, CMVS and other multi-view stereo matching (Multi View Stereo, MVS) algorithms use the photometric consistency between views, or the MVSNet series uses the powerful feature extraction capabilities of neural networks to estimate the depth map more robustly; 3) using the Poisson reconstruction algorithm or the Truncated Signed Distance Function (TSDF) fusion algorithm from the multi-view depth map to perform 3D surface reconstruction. However, the reconstruction quality of this process is heavily dependent on the accuracy of the matching relationship between corresponding points. Even if neural networks are used to extract features, it is still very difficult to find matching relationships for low-texture areas, and this often leads to the problem that some parts of the reconstruction results are missing. Although voxel-based 3D surface reconstruction algorithms can generate complete models, these algorithms are difficult to output accurate surfaces due to the limitations of voxel resolution. Compared with the above algorithms, voxel-based neural surface reconstruction algorithms can be used as an alternative because they have high reconstruction quality and can represent complex objects. However, due to: 1) only using a pure MLP deep network to represent the neural surface field, the training speed and convergence speed are slow; 2) in order to reconstruct an accurate surface, a large number of points must be sampled for each ray emitted by each pixel on each image, so these algorithms usually take a long optimization time (more than ten hours to about 1 day) to reconstruct a high-quality result. In addition, this type of algorithm only uses the color difference of the re-rendered image as supervision, which is easy to fall into the local optimum, resulting in impaired accuracy of the reconstructed 3D model.

[0141] Therefore, in order to overcome some of the above-mentioned defects, in this embodiment, geometric features are stored based on pyramid neural networks, so that complex objects can be described using a very small multi-layer perception network, thereby improving the training speed and convergence speed; and this embodiment uses a rough depth image to construct an initial three-dimensional model of the object, reduces the number of volume rendering ray sampling points, and reduces the possibility of falling into a local optimum, thereby improving the accuracy of the reconstructed model and reducing the reconstruction time.

[0142] In this embodiment, the object three-dimensional model reconstruction device includes: a camera parameter acquisition module 10, a depth image acquisition module 20, an image object segmentation module 30 and an object model reconstruction module 40;

[0143] The camera parameter acquisition module 10 is used to acquire multi-view images of the object to be reconstructed, and to acquire camera internal and external parameters corresponding to each view image by using a structure-from-motion algorithm.

[0144] It should be understood that in this embodiment, the multi-perspective image of the object to be reconstructed can be obtained by photographing the object to be reconstructed from multiple angles through an image acquisition device, or by placing multiple image acquisition devices at different angles to obtain multi-perspective images of the object to be reconstructed. This embodiment does not limit this. The image acquisition device can be a mobile phone camera, a camera, or other device, and the multi-perspective image can be an RGB image of the object to be reconstructed under multiple perspectives.

[0145] Since the principle of 3D model reconstruction is what you see is what you get, the shooting requirement is that the collected images must cover all areas of the object to be reconstructed, and each area must be visible from more than two viewing angles. Generally, 10-60 images of an object are sufficient.

[0146] Considering that the acquisition speed of multi-view images of the object to be reconstructed by shooting the object to be reconstructed from multiple angles is slow, in order to overcome the above-mentioned defects, in this embodiment, the multi-view images of the object to be reconstructed can also be obtained by recording a video of the object to be reconstructed. This embodiment does not impose any restrictions on this. In practical applications, it is necessary to ensure the clarity of the recorded video to ensure that the multi-view images of the object to be reconstructed can be sampled from the recorded video.

[0147] It should be noted that the Structure from Motion (SFM) algorithm can be COLMAP, VisualSFM, MVE, OpenMVG and other algorithms, and this embodiment is not limited to this. The camera internal and external parameters can include camera internal parameters and camera external parameters, wherein the camera internal parameters are the setting parameters when the camera is shooting, such as focal length, principal point position, etc., and the camera external parameters are the parameters such as position and angle of the camera when shooting.

[0148] The depth image acquisition module 20 is used to acquire a rough depth image corresponding to each view image through a multi-view stereo matching algorithm.

[0149] It should be noted that the Multi View Stereo (MVS) algorithm is a computer vision method that uses multiple images to restore three-dimensional scene information. In this embodiment, the MVS algorithm can be a traditional algorithm such as PMVS, OpenMVS, CMVS, or a multi-view depth estimation network of the MVSNet series, which is not limited in this embodiment.

[0150] The image object segmentation module 30 is used to segment the object to be reconstructed from the background in each viewing angle image by using an object segmentation algorithm to obtain an object segmentation map.

[0151] It is understandable that in order to distinguish the object to be reconstructed from the background, in this embodiment, the object to be reconstructed and the background in each perspective image are segmented by an object segmentation algorithm to obtain an object segmentation map. The object segmentation algorithm may be GraphCut, GrabCut, DenseCut, and Segment Anything Model (SAM) and other algorithms, which are not limited in this embodiment.

[0152] The object model reconstruction module 40 is used to reconstruct the three-dimensional model of the object to be reconstructed according to the internal and external parameters of the camera, the rough depth image, the object segmentation map and the multi-view image through a surface reconstruction model based on a pyramid neural body. The surface reconstruction model based on a pyramid neural body is composed of a pyramid neural body and a multi-layer perception network.

[0153] It should be understood that after obtaining the camera internal and external parameters, the rough depth image and the object segmentation map, the object three-dimensional model of the object to be reconstructed is reconstructed by the surface reconstruction model based on the pyramid neural body. In this embodiment, by using the pyramid neural body rendering method, when the camera internal and external parameters are given, the surface reconstruction model based on the pyramid neural body renders the image obtained under the camera internal and external parameters, and the rendered image is compared with the real image, so that the object described by the surface reconstruction model based on the pyramid neural body is close to the object to be reconstructed.

[0154] For ease of understanding, refer to Figure 3 This is for illustration only, but does not limit this solution. Figure 3It is an overall schematic diagram of reconstructing a three-dimensional model of an object. In the figure, a multi-view RGB image of the object to be reconstructed is first obtained, and then the multi-view RGB image is preprocessed, wherein the preprocessing step may include: obtaining camera internal and external parameters corresponding to each view RGB image through the SFM algorithm, obtaining a rough depth image corresponding to each view image through the MVS algorithm, segmenting the object to be reconstructed and the background in each view image through the object segmentation algorithm to obtain an object segmentation map, and then inputting the camera internal and external parameters, the rough depth image, the object segmentation map and the multi-view RGB image into a surface reconstruction model based on a pyramid neural body to obtain an object three-dimensional model of the object to be reconstructed.

[0155] In the present embodiment, a method is disclosed in which a multi-view image of an object to be reconstructed is obtained, and camera internal and external parameters corresponding to each view image are obtained, a rough depth image corresponding to each view image is obtained by a multi-view stereo matching algorithm, the object to be reconstructed and the background in each view image are segmented by an object segmentation algorithm to obtain an object segmentation map, and a three-dimensional object model of the object to be reconstructed is reconstructed by a surface reconstruction model based on a pyramid neural body according to the camera internal and external parameters, the rough depth image, the object segmentation map and the multi-view image, wherein the surface reconstruction model based on the pyramid neural body is composed of a pyramid neural body and a multi-layer perception network; since the present embodiment stores geometric features based on the pyramid neural body, a complex object can be described using a very small multi-layer perception network, thereby improving the training speed and the convergence speed; and the present embodiment uses the rough depth image to construct an initial three-dimensional model of the object, reduces the number of volume rendering ray sampling points, reduces the possibility of falling into a local optimum, thereby improving the accuracy of the reconstructed model and reducing the reconstruction time.

[0156] Other embodiments or specific implementations of the device for reconstructing a three-dimensional model of an object of the present invention may refer to the above-mentioned method embodiments, which will not be described in detail here.

[0157] It should be noted that in the technical solution of this specification, the operations on the data involved are in compliance with relevant regulations and do not violate public order and good morals. For example, operations on data are performed on the premise of obtaining user authorization. In this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0158] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0159] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory image (ROM) / random access memory (RAM), a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0160] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for reconstructing a three-dimensional model of an object, characterized in that: The object three-dimensional model reconstruction method comprises: Obtain multi-view images of the object to be reconstructed, and obtain the camera internal and external parameters corresponding to each view image through the motion recovery structure algorithm; The rough depth image corresponding to each perspective image is obtained by a multi-perspective stereo matching algorithm; The object to be reconstructed and the background in each perspective image are segmented by an object segmentation algorithm to obtain an object segmentation map; According to the internal and external parameters of the camera, the rough depth image, the object segmentation map and the multi-view image, a three-dimensional model of the object to be reconstructed is reconstructed through a surface reconstruction model based on a pyramid neural body. The surface reconstruction model based on a pyramid neural body consists of a pyramid neural body and a multi-layer perception network.

2. The method for reconstructing a three-dimensional model of an object according to claim 1, wherein: The object three-dimensional model reconstruction method also includes: According to the camera internal and external parameters and voxel rendering requirements, a ray is emitted from the camera origin to pass through the pixel to be rendered; Selecting a plurality of sampling points on the ray; Calculating the color value and normal vector of the sampling point and the signed distance value between the sampling point and the object to be reconstructed; Performing volume rendering on the pixel to be rendered based on the color values, normal vectors, and signed distance values ​​of all sampling points corresponding to the pixel to be rendered to obtain a color value of the pixel to be rendered; The surface reconstruction model based on the pyramid neural volume is optimized based on the color value of the pixel to be rendered and the real color value of the pixel to be rendered in the multi-view image.

3. The method for reconstructing a three-dimensional model of an object according to claim 2, wherein: The selecting a plurality of sampling points on the ray comprises: Taking the vertex reconstructed from the rough depth image as the sampling center and constructing the sampling width according to the confidence of the depth image to obtain the sampling range; A preset first number of sampling points is selected within the sampling range on the ray, and a preset second number of sampling points is selected outside the sampling range on the ray, wherein the preset first number is greater than the preset second number.

4. The method for reconstructing a three-dimensional model of an object according to claim 1, wherein: The object three-dimensional model reconstruction method also includes: synthesizing the rough depth image into a sparse three-dimensional point cloud according to a geometric consistency constraint; Constructing a signed distance function supervision loss corresponding to the signed distance function according to the sparse three-dimensional point cloud; The pyramid neural network is optimized based on the signed distance function supervised loss.

5. The method for reconstructing a three-dimensional model of an object according to claim 1, wherein: The object three-dimensional model reconstruction method also includes: Constructing the total variational loss of the pyramid neural body according to the continuity constraint; Construct a normal vector constraint loss based on the constraint that the normal vector modulus is a preset value; Constructing a segmentation map loss corresponding to the signed distance function according to the object segmentation map; generating a total supervision loss according to the signed distance function supervision loss, the volume rendering color loss, the total variational loss, the normal vector constraint loss, and the segmentation map loss; The pyramid neural body is optimized based on the total supervision loss.

6. The method for reconstructing a three-dimensional model of an object according to any one of claims 1 to 5, characterized in that: The pyramid neural body divides voxels by a single-scale VM body representation; The single-scale VM body representation method is to decompose a three-dimensional vector into multiple groups of one-dimensional vectors and two-dimensional arrays through tensor decomposition.

7. The method for reconstructing a three-dimensional model of an object according to claim 6, wherein: The pyramid neural body divides voxels by pyramid VM body representation; The pyramid VM volume representation is composed of a plurality of single-scale VM volume representations with different resolutions, and the extracted set features include global features and local features.

8. A device for reconstructing a three-dimensional model of an object, characterized in that: The object three-dimensional model reconstruction device includes: a memory, a processor, and an object three-dimensional model reconstruction program stored in the memory and executable on the processor. When the object three-dimensional model reconstruction program is executed by the processor, the object three-dimensional model reconstruction method according to any one of claims 1 to 7 is implemented.

9. A storage medium, characterized in that: The storage medium stores a three-dimensional model reconstruction program for an object, and when the three-dimensional model reconstruction program for an object is executed by a processor, the three-dimensional model reconstruction method for an object as described in any one of claims 1 to 7 is implemented.

10. A device for reconstructing a three-dimensional model of an object, characterized in that: The object three-dimensional model reconstruction device includes: a camera parameter acquisition module, a depth image acquisition module, an image object segmentation module and an object model reconstruction module; The camera parameter acquisition module is used to acquire multi-view images of the object to be reconstructed, and to acquire the camera internal and external parameters corresponding to each view image by using a motion recovery structure algorithm; The depth image acquisition module is used to acquire a rough depth image corresponding to each perspective image through a multi-perspective stereo matching algorithm; The image object segmentation module is used to segment the object to be reconstructed from the background in each perspective image by an object segmentation algorithm to obtain an object segmentation map; The object model reconstruction module is used to reconstruct the three-dimensional model of the object to be reconstructed according to the internal and external parameters of the camera, the rough depth image, the object segmentation map and the multi-view image through a surface reconstruction model based on a pyramid neural body. The surface reconstruction model based on a pyramid neural body consists of a pyramid neural body and a multi-layer perception network.

Images