Multi-view 3D reconstruction method, device, equipment and storage medium

By acquiring multi-view images in three-dimensional reconstruction technology, determining point cloud data and sampling in voxel space, the problem of redundant calculation in the prior art is solved, and the efficiency and accuracy of three-dimensional reconstruction are improved.

CN116310120BActive Publication Date: 2024-12-03NETEASE (HANGZHOU) NETWORK CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202310271277.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-03-14
Publication Date
2024-12-03
Estimated Expiration
2043-03-14

AI Technical Summary

Technical Problem

In the existing three-dimensional reconstruction technology, due to the large amount of redundancy and invalid calculations of ray sampling based on the entire bounding box, the reconstruction effect is reduced and the efficiency is reduced.

Method used

By obtaining the two-dimensional image of the target object at multiple perspectives, determining the point cloud data, and sampling the target voxel points in the preset voxel space to determine the three-dimensional surface and surface colors, and only the color network can be performed within the intersection interval. Sample to narrow the sampling range.

Benefits of technology

It improves the determination efficiency and rendering efficiency of the surface color of the target object, and improves the accuracy and detail integrity of three-dimensional reconstruction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116310120B_ABST
    Figure CN116310120B_ABST
Patent Text Reader

Abstract

The present application provides a multi-view 3D reconstruction method, device, equipment and storage medium, which relates to the field of 3D reconstruction technology. The method includes: obtaining a 2D image of a target object at multiple viewpoints; determining point cloud data at multiple viewpoints respectively according to the 2D images at multiple viewpoints; sampling point clouds in the preset voxel space in the point cloud data at multiple viewpoints based on the preset voxel space to obtain target voxel points; determining the 3D surface of the target object according to the target voxel points; determining the intersection interval between each sampling ray and the 3D surface according to the camera posture at each viewpoint; determining the surface color of the target object based on each intersection interval using a color network; and rendering the target object based on the surface color. Compared with the prior art, a large amount of redundancy and invalid calculations are avoided, and the efficiency and accuracy of 3D reconstruction are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction technology, and in particular to a multi-view three-dimensional reconstruction method, device, equipment and storage medium. Background Art

[0002] Multi-view 3D reconstruction is a technology that reconstructs the shape of a three-dimensional (3D) object by taking pictures of the object to be reconstructed from different viewpoints. In recent years, with the development of 3D games, architectural design, and virtual reality, the demand for 3D reconstruction of real-world objects has increased. Currently, 3D reconstruction based on deep learning is developing rapidly.

[0003] The prior art generally uses a neural surface reconstruction method using a signed distance function (SDF). This method obtains the initial bounding box (BBox) of the object based on the sparse point cloud calculated by the Structure From Motion (SFM) method. The observed sampling ray is designed based on the camera pose estimation result, and effective sampling is performed inside the BBox. For each target voxel point, the SDF value and the color at the target voxel point are predicted by a multilayer perceptron (MLP). Then, the SDF value is converted into a volume density value through the Logistic probability density distribution function, and the SDF representation is trained using the neural volume rendering method in combination with the predicted color of each target voxel point.

[0004] However, since the actual object to be reconstructed only occupies a part of the BBox, this reconstruction method has a lot of redundancy and invalid calculations based on the ray sampling of the entire BBox, which reduces the efficiency of optimization and leads to a decline in the final reconstruction effect. Summary of the invention

[0005] The purpose of this application is to provide a multi-view three-dimensional reconstruction method, device, equipment and storage medium to address the deficiencies in the above-mentioned prior art, so as to solve the problem that there are a large number of redundancies and invalid calculations in the prior art, which reduces the optimization efficiency and leads to a decrease in reconstruction effect.

[0006] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0007] In a first aspect, an embodiment of the present application provides a multi-view 3D reconstruction method, the method comprising:

[0008] Obtain two-dimensional images of the target object at multiple viewing angles;

[0009] Determining point cloud data at the multiple viewing angles respectively according to the two-dimensional images at the multiple viewing angles;

[0010] Based on a preset voxel space, sampling point clouds in the preset voxel space in the point cloud data under multiple viewing angles to obtain target voxel points;

[0011] Determining a three-dimensional surface of the target object according to the target voxel points;

[0012] Determining the intersection interval between each sampling ray and the three-dimensional surface according to the camera pose at each viewing angle;

[0013] Determine the surface color of the target object based on each of the intersection intervals using a color network;

[0014] The target object is rendered based on the surface color.

[0015] In a second aspect, another embodiment of the present application provides a multi-view 3D reconstruction device, the device comprising: an acquisition module, a determination module, a sampling module and a rendering module, wherein:

[0016] The acquisition module is used to acquire two-dimensional images of the target object at multiple viewing angles;

[0017] The determination module is used to respectively determine the point cloud data under the multiple viewing angles according to the two-dimensional images under the multiple viewing angles;

[0018] The sampling module is used to sample point clouds in the preset voxel space in the point cloud data under multiple viewing angles based on the preset voxel space to obtain target voxel points;

[0019] The determination module is specifically used to determine the three-dimensional surface of the target object according to the target voxel point; determine the intersection interval between each sampling ray and the three-dimensional surface according to the camera posture under each of the viewing angles; and determine the surface color of the target object based on each of the intersection intervals using a color network;

[0020] The rendering module is used to render the target object based on the surface color.

[0021] In the third aspect, another embodiment of the present application provides a multi-view three-dimensional reconstruction device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the multi-view three-dimensional reconstruction device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of any method described in the first aspect above.

[0022] In a fourth aspect, another embodiment of the present application provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any method described in the first aspect are executed.

[0023] The beneficial effect of the present application is: using the multi-view three-dimensional reconstruction method provided by the present application, two-dimensional images of the target object at multiple viewpoints are obtained, and then point cloud data at multiple viewpoints are respectively determined based on the multiple two-dimensional images, and based on the preset voxel space, the point cloud in the preset voxel space is sampled to determine the target voxel point, and the three-dimensional surface of the target object is determined based on the target voxel point. Subsequently, when determining the surface color of the target object based on the sampling ray, the intersection interval of each sampling ray and the three-dimensional surface is first determined, and then the color of each sampling ray in the focus interval is determined only based on the color network, thereby determining the surface color of the target object. This determination method can narrow the sampling range of the sampling ray, thereby improving the efficiency of determining the surface color of the target object, and improving the rendering efficiency of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A schematic diagram of a process flow of a multi-view 3D reconstruction method provided in one embodiment of the present application;

[0026] Figure 2 A schematic diagram of a flow chart of a multi-view 3D reconstruction method provided in another embodiment of the present application;

[0027] Figure 3 A schematic diagram of a flow chart of a multi-view 3D reconstruction method provided in another embodiment of the present application;

[0028] Figure 4 A schematic diagram of a flow chart of a multi-view 3D reconstruction method provided in another embodiment of the present application;

[0029] Figure 5 A schematic diagram of the structure of a multi-view 3D reconstruction device provided in one embodiment of the present application;

[0030] Figure 6 A schematic diagram of the structure of a multi-view 3D reconstruction device provided in another embodiment of the present application;

[0031] Figure 7 A schematic diagram of the structure of a multi-view 3D reconstruction device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0033] The components of the embodiments of the present application generally described and shown in the drawings herein may be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0034] In addition, the flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of order, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of this application, and can also remove one or more operations from the flowchart.

[0035] To facilitate the understanding of this application, some of the terms used in this application are explained below:

[0036] Salient object detection (SOD): aims to distinguish the most visually obvious areas. It uses traditional computer vision techniques or data-driven deep learning algorithms to mark the parts of the image that belong to the salient foreground.

[0037] Signed Distance Function (SDF) is an implicit representation of the surface of a three-dimensional object. It determines the distance from a point to the surface of the object in a finite region in space and defines the sign of the distance: the point is positive when it is inside the object, negative when it is outside, and 0 when it is on the surface. SDF can be implicitly represented with the help of a multi-layer perceptron (MLP) network.

[0038] Bounding Box (BBox), in 3D space, refers to the smallest bounding box that surrounds an object.

[0039] Structure From Motion (SFM) is a technique for estimating three-dimensional structure from a series of two-dimensional images containing visual motion information. SFM can be used to obtain camera parameters and sparse point clouds for each perspective.

[0040] Marching Cubes: Surface extraction algorithm, also known as surface rendering algorithm, is an algorithm used to extract isosurfaces in volume data. The core concept is to determine which edges of voxels intersect with isosurfaces by comparing vertices with user-specified thresholds after setting the 3D space and vertex scalar values, create triangular patches, and connect all cube faces on the isosurface boundary to obtain a surface.

[0041] The multi-view 3D reconstruction method provided in the embodiment of the present application can be executed by a terminal or a server. A terminal is any device with computing hardware that can support and execute corresponding software products. The multi-view 3D reconstruction method provided in the embodiment of the present application is explained below in combination with multiple specific application examples. Figure 1 A schematic diagram of a multi-view 3D reconstruction method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes:

[0042] S101: Acquire two-dimensional images of a target object at multiple viewing angles.

[0043] In the embodiments of the present application, in some possible embodiments, the multiple viewing angles may be, for example, 50-80 viewing angles. It should be understood that the above embodiments are only exemplary descriptions, and the two-dimensional images under the specific multiple viewing angles are only the multiple viewing angle images required for 360° surround of the target object, and may also be any integer less than 50, or any integer greater than 80. The specific number limit of the multiple viewing angles can be flexibly adjusted according to user needs, and is not limited to the above embodiments.

[0044] In some possible embodiments, a method for obtaining two-dimensional images at multiple perspectives may be, for example, obtaining multiple two-dimensional images obtained by photographing a target object in a real environment at multiple perspectives with a preset camera, where the multiple two-dimensional images correspond to multiple perspectives respectively.

[0045] S102: Determine point cloud data at multiple viewing angles respectively according to the two-dimensional images at multiple viewing angles.

[0046] In some possible embodiments, for example, multiple two-dimensional images may be estimated respectively to obtain three-dimensional point cloud data at multiple viewing angles, wherein the three-dimensional point cloud at each viewing angle may be, for example, a sparse point cloud. For example, a preset SFM network may be used to estimate the two-dimensional image at each viewing angle to obtain the three-dimensional point cloud data at each viewing angle, as well as the camera poses at multiple viewing angles.

[0047] S103: Based on the preset voxel space, sampling the point clouds in the preset voxel space in the point cloud data under multiple viewing angles to obtain target voxel points.

[0048] In one embodiment of the present application, before S103, for example, the bounding box information of the target object may be determined based on the point cloud data under multiple viewing angles, wherein the bounding box information of the target object is the information of the minimum bounding box surrounding the target object.

[0049] The method for determining the bounding box of the target object can be, for example: the camera pose at each viewing angle can be calibrated according to the three-dimensional point cloud data at each viewing angle to obtain the target camera pose at each viewing angle, and the bounding box of the target object is estimated according to the updated target camera pose, thereby obtaining the bounding box information of the target object; that is, in the embodiment of the present application, the border of the target object is estimated based on the target camera pose at multiple viewing angles after calibration to obtain the bounding box information of the target object, thereby realizing automatic estimation of the bounding box of the target object, and ensuring that the estimated bounding box of the target object is more compact.

[0050] Regarding the bounding box information of the target object, in some possible embodiments, the estimation results of the above-mentioned SFM on two-dimensional images of multiple perspectives can also be used to perform automatic object frame estimation, thereby obtaining the bounding box information of the target object.

[0051] In the embodiment of the present application, after determining the bounding box BBox of the target object, it is also necessary to normalize the coordinates of the bounding box of the target object in the three-dimensional space to [-1, 1] to obtain the normalized point cloud data, and the acquisition method is as follows:

[0052] Normalize the coordinates of the above BBox in three-dimensional space to [-1, 1], and calculate the scaling matrix as scale_mat∈R 4×4 . Among them, B∈R 3 ; i∈(x,y,z).

[0053] R 4×4 represents a 4x4 matrix, R 3 represents a three-dimensional vector space, c x 、c y 、c z Indicates the offset in the x, y, and z directions. x , B y , B z Indicates the coordinate values ​​of BBOX in the x, y, and z directions.

[0054] Since the bounding box of the target object is automatically calculated and estimated in the method provided by the present application, rather than manually estimated, the application complexity in the three-dimensional reconstruction process is reduced.

[0055] In the embodiment of the present application, the preset voxel space needs to be initialized first, and the initialization method may be, for example: according to the target accuracy of the target object, the number of voxels sampled in the preset voxel space is set.

[0056] The target accuracy indicates the degree of refinement of the three-dimensional model of the object to be obtained. Different degrees of refinement correspond to different target accuracies. The higher the degree of refinement, the higher the target accuracy. For example, medium refinement corresponds to sampling in a three-dimensional space by sampling 100 voxels in each dimension. For example, based on the preset voxel space and according to the information of the target object bounding box, the point cloud in the preset voxel space of the normalized point cloud data under multiple perspectives can be sampled to determine the target voxel points corresponding to the multiple perspectives.

[0057] Initialize the preset voxel space V∈[-1,1] in three-dimensional space 3 , and set the number of voxels sampled in each dimension to ρ, where the size of ρ is determined according to the target accuracy of the target object. The higher the target accuracy, the larger ρ is set, and the lower the target accuracy, the smaller ρ is set. The coordinates in each dimension in the three-dimensional space Define the count array of each voxel belonging to the object as O∈[-1,1] 3 .

[0058] For the i-th perspective, assume that the currently calculated transformation matrix from the world coordinate system to the current 2D image is w2c_mat i ∈R 3×4 For V∈[-1,1] 3 The coordinates of each point (i, j, k) projected to the current 2D image are The horizontal and vertical axes are represented by p=(p0,p1,p2). Then

[0059] R 3×4 Represents a 3x4 matrix, where p0, p1, and p2 represent the components of P in the x, y, and z directions. i,j,k Represents the component of the counting array O with coordinates (i, j, k) in three-dimensional space.

[0060] S104: Determine the three-dimensional surface of the target object according to the target voxel points.

[0061] In one embodiment of the present application, a method for determining the three-dimensional surface of the target object may be, for example, to determine the three-dimensional surface of the target object based on target voxel points using an isosurface extraction algorithm.

[0062] In the embodiment of the present application, the above process of determining the target voxel point is repeated for the two-dimensional images under all viewing angles to obtain the final count array O. For each position (i, j, k) in O, if If , the voxel point is considered to be inside the object, otherwise it is considered to be outside the object. The threshold γ∈(0,1). The three-dimensional surface (3D convex hull) composed of voxel points inside the object is extracted according to the Marching cubes algorithm.

[0063] S105: Determine the intersection interval between each sampling ray and the three-dimensional surface according to the camera posture at each viewing angle.

[0064] For each camera pose at each viewing angle, calculate the surface intersection of each sampling ray with the object's 3D convex hull calculated above. The intersection points of the two intersections are called near intersection and far intersection, respectively. Assume that the coordinates of the origin of the nth camera in the world coordinate system are o n , the unit vector of the ray to be sampled is (Assuming that the ray is a straight line passing through the camera origin and (i, j) in the 2D image), the distance from the center point to the camera origin is defined as Therefore, we can get the near intersection point of the ray and the BBox calculated in step 1 The far intersection point is In addition, if the ray intersects the 3D convex hull, record o n The first intersection with the 3D convex hull is o n Relative to The symmetry point is And record its first intersection with the 3D convex hull as Far intersection When the ray has no intersection with the 3D convex hull, the intersection of the ray and the BBox is used as and That is, if the ray d i,j If there is no intersection with the 3D convex hull, use the following formula instead:

[0065]

[0066] So far, we have obtained the effective near intersection point of each sampling ray. and far intersection

[0067] If there is a situation where the sampling ray is tangent to the target object, that is, the first intersection point and the second intersection point of the sampling ray coincide with each other.

[0068] S106: Determine the surface color of the target object based on each intersection interval using a color network.

[0069] Among them, the color network is a pre-trained network model, which is constructed using a multilayer perceptron (MLP) network, and the network parameters are obtained through training. The output of the network is the predicted color value of each point in the space.

[0070] That is, the target three-dimensional surface data and surface color have been obtained, that is, the three-dimensional model data of the target object has been obtained, and the target object can be directly rendered in the three-dimensional world based on the obtained three-dimensional model data.

[0071] S107: Rendering the target object based on the surface color.

[0072] After obtaining the 3D model data of the target object, the 3D model data can be stored, and according to subsequent actual business needs, the 3D model data can be opened and loaded through model rendering software to render and display the 3D model corresponding to the target object.

[0073] Since the 3D reconstruction method provided in the embodiment of the present application collects target voxel points in the 3D point cloud by estimating the bounding box of multiple 2D images under multiple perspectives to obtain the bounding box information of the target object, rather than using the bounding box information estimated manually, therefore, compared with the bounding box estimated manually, the automatic estimation method of the present application can achieve a compact estimation of the object bounding box, thereby making the sampling of the 3D point cloud more accurate, and then making the 3D model data obtained by the 3D reconstruction more accurate, thereby improving the reconstruction accuracy of the 3D model and the completeness of the model details.

[0074] The multi-view three-dimensional reconstruction method provided in the present application is adopted to obtain two-dimensional images of the target object at multiple viewpoints, and then the point cloud data and the bounding box information of the target object at multiple viewpoints are determined respectively based on the multiple two-dimensional images, and based on the preset voxel space, the point cloud in the preset voxel space is sampled to determine the target voxel point, and the three-dimensional surface of the target object is determined based on the target voxel point. Then, when determining the surface color of the target object based on the sampling ray, the intersection interval of each sampling ray and the three-dimensional surface is first determined, and then the color of each sampling ray in the focus interval is determined only based on the color network, so as to determine the surface color of the target object. This determination method can narrow the sampling range of the sampling ray, thereby improving the efficiency of determining the surface color of the target object and improving the rendering efficiency of the target object.

[0075] Optionally, based on the above embodiment, the embodiment of the present application may also provide a multi-view 3D reconstruction method. The following is an example of the implementation process of determining the surface color of the target object in the above method in combination with the accompanying drawings. Figure 2 A flowchart of a multi-view 3D reconstruction method provided by another embodiment of the present application is shown in FIG. Figure 2 As shown, S106 may include:

[0076] S111: Determine the signed distance function value, the normal vector and the target voxel distance of the target voxel point according to the signed distance function network.

[0077] In the embodiment of the present application, for the sampling ray at the nth viewing angle exist and 3D sampling is performed between the target voxel points, and the signed distance function value, normal vector and target voxel point spacing of each target voxel point are determined through the above SDF network.

[0078] S112: Determine the color value along each sampling ray according to the color network.

[0079] Similarly, when determining the color value of each sampling ray, it is also for the sampling ray under the nth viewing angle. exist and 3D sampling is performed between the two layers, and the color value along each sampling ray is determined through the color network.

[0080] S113: Obtain the target rendering color value of each sampling ray.

[0081] In an embodiment of the present application, the target rendering color value of each sampling ray may be determined by, for example, converting the weighted SDF value into a volume density value through a Logistic cumulative distribution function, and performing volume rendering according to the volume density value, color value, and sampling interval of all sampling points on each ray to obtain the final rendered color value of each ray.

[0082] Optionally, based on the above embodiment, the embodiment of the present application may further provide a multi-view 3D reconstruction method. The following is an example of the implementation process of determining the target voxel point in the above method in conjunction with the accompanying drawings. Figure 3 A flowchart of a multi-view 3D reconstruction method provided by another embodiment of the present application is shown in FIG. Figure 3 As shown, S103 may include:

[0083] S121: Determine whether each pixel point in the two-dimensional image under multiple viewing angles is a salient foreground.

[0084] The salient foreground is an object instance with a clear outline in a two-dimensional image; first, the present application can perform saliency detection based on a saliency foreground detection algorithm. Generally, an object surrounded by high-frequency details in a two-dimensional image is an object instance, that is, one or more object instances with clear outlines in a two-dimensional image under multiple perspectives are detected through saliency detection. Subsequently, it is determined whether the pixel position of each pixel is inside the object instance with a clear outline. If so, that is, the pixel falls within the area range of the object instance with a clear outline, that is, the pixel is a salient foreground pixel; otherwise, it means that the pixel is outside the object instance with a clear outline, that is, the pixel is the background and does not belong to the salient foreground.

[0085] For example, for each 2D image at each viewing angle, for each pixel in the image, if it belongs to the salient foreground, it is marked as 1, or if it does not belong to the salient foreground (belongs to the background), it is marked as 0. Assume that there are N viewing angles, and the 2D salient image at each viewing angle is I n ∈R w×h , where w and h are the image width and height respectively. If the coordinate [x, y] belongs to the salient foreground, then I n (x,y)=1, otherwise I n (x,y)=0.

[0086] R w×h The matrix representing w, x, and h. n (x,y) represents I n The pixel with coordinates (x,y) in the image.

[0087] S122: According to the determination result of whether it is a significant foreground, based on a preset voxel space, in the point cloud data under multiple perspectives, determine the target voxel point belonging to the interior of the target object.

[0088] In an embodiment of the present application, a method for determining a target voxel point belonging to the inside of a target object may be, for example: determining a first ratio based on the target voxel point and the number of perspectives of multiple perspectives; if the first ratio is greater than a preset threshold, determining that the target voxel point belongs to the inside of the target object.

[0089] Figure 4 A schematic diagram of a process flow of a multi-view 3D reconstruction method provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, a complete flowchart is used to explain the training process of the signed distance function network and the color network in the three-dimensional reconstruction method provided by this application:

[0090] S201: Input the two-dimensional image, camera pose and sparse point cloud of the target object at various viewing angles.

[0091] S202: Get BBox and normalize the sparse point cloud.

[0092] Among them, the bounding box BBox of the target object is obtained according to the sparse point cloud results obtained by SFM, and the x, y, and z coordinates of the object in three-dimensional space are normalized to [-1, 1], and the scaling matrix is ​​calculated.

[0093] S203: Calculate the salient foreground of the two-dimensional image at each viewing angle.

[0094] Calculate the salient foreground area in the two-dimensional image taken at each viewing angle. For each pixel in the two-dimensional image, if it belongs to the foreground, it is marked as 1, and if it belongs to the background, it is marked as 0.

[0095] S204: Initialize the voxel space and determine whether it belongs to the interior of the target object.

[0096] Initialize the voxel space in the unit space in the three-dimensional space, and set the voxel interval to ρ. Project each point in the above initialized voxel space into the 2D image according to the camera parameters, and determine whether each voxel point projected into the 2D image belongs to a significant foreground or background.

[0097] The proportion of each voxel point projected onto the 2D image at each viewing angle that belongs to the foreground is determined. If the proportion exceeds a threshold, the voxel point is determined to belong to the interior of the target object, otherwise it belongs to the exterior of the target object.

[0098] S205: Generate a three-dimensional surface of the target object according to the target voxel points.

[0099] The points inside the target object are determined as target voxel points, and the three-dimensional surface of the target object, that is, the 3D convex hull, is established according to the determination result of the target voxel points.

[0100] S206: Calculate the intersection interval between each sampling ray and the target object.

[0101] For the camera pose at each viewing angle, calculate the surface intersection of each sampling ray with the 3D convex hull of the object calculated above, calculate the nearest point and the farthest point, and determine the intersection interval of each sampling ray with the target object based on the nearest point and the farthest point.

[0102] S207: Randomly sample some rays and calculate three-dimensional model data.

[0103] Next, 3D sampling is performed along each observation ray between the nearest point and the farthest point of the 3D convex hull, and the SDF value, normal vector, sampling point spacing and color value along the observation ray of the sampling point are calculated through the above-mentioned Signed Distance Function (SDF) network and Color network; wherein the above-mentioned SDF value, normal vector, sampling point spacing and color value along the observation ray are collectively referred to as three-dimensional model data.

[0104] S208: Volume density rendering.

[0105] Then, the weighted SDF value is converted into a volume density value through the Logistic cumulative distribution function. Volume rendering is performed based on the volume density value, color value and sampling interval of all sampling points on each ray to obtain the final rendered color value of each ray.

[0106] S209: Calculate the loss function.

[0107] S210: Optimize network parameters.

[0108] The optimized network parameters include the optimized signed distance function network parameters and the color network network parameters.

[0109] S211: Whether the maximum number of iterations has been reached.

[0110] If not, the process returns to S207 to continue randomly sampling some rays and performing calculations.

[0111] If yes, execute S212.

[0112] S212: Save network parameters.

[0113] If the maximum number of iterations is reached, it is determined that the iterations of the signed distance function network and the color network are completed, and the iterated model is obtained and put into use.

[0114] By adopting the multi-view 3D reconstruction method provided by the present application, a method for 3D reconstruction of 2D images based on 3D surfaces is proposed to solve the problem of extrusion of 3D reconstruction and rendering of 2D images. The sampling of 3D sampling rays within the effective range around the target object can be optimized, and the color of each sampling ray within the focal interval can be determined only based on the color network, thereby determining the surface color of the target object. This determination method can narrow the sampling range of the sampling rays, thereby improving the efficiency of determining the surface color of the target object, as well as improving the rendering efficiency of the target object, and improving the efficiency and accuracy of 3D reconstruction.

[0115] The following is an explanation of the multi-view 3D reconstruction device provided by the present application in conjunction with the accompanying drawings. The multi-view 3D reconstruction device can perform the above Figure 1-Figure 4The specific implementation and beneficial effects of any multi-view 3D reconstruction method are as mentioned above and will not be described in detail below.

[0116] Figure 5 A schematic diagram of the structure of a multi-view 3D reconstruction device provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the device includes: an acquisition module 201, a determination module 202, a sampling module 203 and a rendering module 204, wherein:

[0117] An acquisition module 201 is used to acquire two-dimensional images of a target object at multiple viewing angles;

[0118] A determination module 202, configured to determine point cloud data at multiple viewing angles respectively according to the two-dimensional images at multiple viewing angles;

[0119] The sampling module 203 is used to sample the point cloud in the preset voxel space in the point cloud data under multiple viewing angles based on the preset voxel space to obtain the target voxel point;

[0120] The determination module 202 is specifically used to determine the three-dimensional surface of the target object according to the target voxel point; determine the intersection interval between each sampling ray and the three-dimensional surface according to the camera posture under each viewing angle; and determine the surface color of the target object based on each intersection interval using a color network;

[0121] The rendering module 204 is used to render the target object based on the surface color.

[0122] Optionally, the determination module 202 is specifically used to determine the signed distance function value, normal vector and target voxel point spacing of the target voxel point according to the signed distance function network; determine the color value along each sampling ray according to the color network; and obtain the target rendering color value of each sampling ray.

[0123] Optionally, the determination module 202 is specifically configured to determine the three-dimensional surface of the target object based on the target voxel points by adopting an isosurface extraction algorithm.

[0124] Optionally, based on the above embodiment, the present application embodiment may also provide a multi-view 3D reconstruction device, as shown below in conjunction with the accompanying drawings. Figure 5 The implementation process of the given device is described by way of example. Figure 6 A schematic diagram of the structure of a multi-view 3D reconstruction device provided in another embodiment of the present application is shown in FIG. Figure 6 As shown, the device further includes: a setting module 205, which is used to set the number of voxels sampled in the preset voxel space according to the target accuracy of the target object.

[0125] Optionally, the determination module 202 is specifically used to determine whether each pixel point in the two-dimensional image under multiple perspectives is a significant foreground; according to the determination result of whether it is a significant foreground, based on a preset voxel space, in the point cloud data under multiple perspectives, determine the target voxel point belonging to the inside of the target object.

[0126] Optionally, the determination module 202 is specifically configured to determine a first ratio according to the target voxel point and the number of perspectives of the multiple perspectives; if the first ratio is greater than a preset threshold, it is determined that the target voxel point belongs to the interior of the target object.

[0127] Optionally, the determination module 202 is specifically configured to determine information of a bounding box of the target object based on the point cloud data under the multiple perspectives;

[0128] The acquisition module 201 is specifically used to normalize the bounding box of the target object and obtain the normalized bounding box data;

[0129] The sampling module 203 is specifically configured to sample the point cloud in the preset voxel space in the normalized bounding box data under multiple viewing angles based on the preset voxel space to obtain the target voxel point.

[0130] The above-mentioned device is used to execute the method provided by the aforementioned embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0131] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), or one or more microprocessors, or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0132] Figure 7 This is a schematic diagram of the structure of a multi-view 3D reconstruction device provided in one embodiment of the present application. The multi-view 3D reconstruction device can be integrated into a terminal device or a chip of the terminal device.

[0133] like Figure 7 As shown, the multi-view 3D reconstruction device includes: a processor 501 , a bus 502 and a storage medium 503 .

[0134] The processor 501 is used to store programs. The processor 501 calls the program stored in the storage medium 503 to execute the above Figure 1-Figure 4 The corresponding method embodiment has similar specific implementation methods and technical effects, which will not be described here.

[0135] Optionally, the present application also provides a program product, such as a storage medium, on which a computer program is stored, including a program that, when executed by a processor, executes an embodiment corresponding to the above method.

[0136] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0137] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0138] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0139] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), disk or optical disk and other media that can store program codes.

Claims

1. A multi-view 3D reconstruction method, characterized in that: The method comprises: Obtain two-dimensional images of the target object at multiple viewing angles; Determining point cloud data at the multiple viewing angles respectively according to the two-dimensional images at the multiple viewing angles; Determining information of a bounding box of the target object according to the point cloud data under the multiple perspectives; Based on a preset voxel space, sampling point clouds in the preset voxel space in the point cloud data under multiple viewing angles to obtain target voxel points; Determining a three-dimensional surface of the target object according to the target voxel points; Determining the intersection interval between each sampling ray and the three-dimensional surface according to the camera pose at each viewing angle; Determine the surface color of the target object based on each of the intersection intervals using a color network; Rendering the target object based on the surface color; The method of sampling point clouds in the preset voxel space in the point cloud data under multiple viewing angles based on the preset voxel space to obtain target voxel points includes: Based on the preset voxel space and according to the information of the bounding box of the target object, point clouds in the preset voxel space in the point cloud data under multiple viewing angles are sampled to obtain the target voxel point.

2. The method according to claim 1, characterized in that The method of using a color network to determine the surface color of the target object based on each of the intersection intervals includes: According to the signed distance function network, the signed distance function value, the normal vector and the target voxel point spacing of the target voxel point are determined; Determine the color value along each of the sampling rays according to the color network; Get the target rendering color value of each of the sampling rays.

3. The method according to claim 1, characterized in that Determining the three-dimensional surface of the target object according to the target voxel point includes: An isosurface extraction algorithm is used to determine the three-dimensional surface of the target object based on the target voxel points.

4. The method according to claim 1, characterized in that Before sampling the point cloud in the preset voxel space in the point cloud data under multiple viewing angles based on the preset voxel space to obtain the target voxel point, the method further includes: The number of voxels sampled in the preset voxel space is set according to the target accuracy of the target object.

5. The method according to claim 1, characterized in that The method of sampling point clouds in the preset voxel space in the point cloud data under multiple viewing angles based on the preset voxel space to obtain target voxel points includes: Determining whether each pixel in the two-dimensional images under the multiple viewing angles is a salient foreground; According to the determination result of whether it is a significant foreground, based on a preset voxel space, in the point cloud data under multiple viewing angles, a target voxel point belonging to the interior of the target object is determined.

6. The method according to claim 5, characterized in that The determining of the target voxel point belonging to the interior of the target object comprises: Determining a first ratio according to the target voxel point and the number of viewing angles of the plurality of viewing angles; If the first ratio is greater than a preset threshold, it is determined that the target voxel point belongs to the interior of the target object.

7. The method according to claim 1, characterized in that Before sampling point clouds in the preset voxel space in the point cloud data under multiple perspectives based on the preset voxel space to obtain target voxel points, the method further includes: Normalizing the bounding box of the target object to obtain normalized bounding box data; The method of sampling point clouds in the preset voxel space in the point cloud data under multiple viewing angles based on the preset voxel space to obtain target voxel points includes: Based on the preset voxel space, the point cloud in the preset voxel space in the normalized bounding box data under multiple viewing angles is sampled to obtain the target voxel point.

8. A multi-view 3D reconstruction device, characterized in that: The device comprises: an acquisition module, a determination module, a sampling module and a rendering module, wherein: The acquisition module is used to acquire two-dimensional images of the target object at multiple viewing angles; The determination module is used to respectively determine the point cloud data under the multiple viewing angles according to the two-dimensional images under the multiple viewing angles; The determination module is used to determine the information of the bounding box of the target object according to the point cloud data under the multiple perspectives; The sampling module is used to sample point clouds in the preset voxel space in the point cloud data under multiple viewing angles based on the preset voxel space to obtain target voxel points; The determination module is specifically used to determine the three-dimensional surface of the target object according to the target voxel point; determine the intersection interval between each sampling ray and the three-dimensional surface according to the camera posture under each of the viewing angles; and determine the surface color of the target object based on each of the intersection intervals using a color network; The rendering module is used to render the target object based on the surface color; The sampling module is specifically used to sample the point cloud in the preset voxel space in the point cloud data under multiple perspectives based on the preset voxel space and according to the information of the bounding box of the target object to obtain the target voxel point.

9. A multi-view 3D reconstruction device, characterized in that: The device includes: a processor, a storage medium and a bus, the storage medium stores machine-readable instructions executable by the processor, and when the multi-view three-dimensional reconstruction device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to perform the method described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is executed.

Citation Information

Patent Citations

  • Three-dimensional point cloud model re-establishment method and apparatus

    CN107223268A

  • Mobile terminal three-dimensional pose estimation method based on point cloud preprocessing and ShuffleNet

    CN113362467A

  • Point cloud data generation method and device, readable storage medium and sweeping robot

    CN114782630A

  • Three-dimensional model reconstruction method and device and computer readable storage medium

    CN115294275A

  • Multi-view three-dimensional point cloud reconstruction method

    CN115731348A