Three-dimensional reconstruction method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to the technical field of three-dimensional reconstruction, in particular to a three-dimensional reconstruction method and device, electronic equipment, a storage medium and a program product, and the method comprises the steps: obtaining a plurality of two-dimensional images of a to-be-reconstructed scene and corresponding camera parameters, the two-dimensional images comprising a plurality of source images and a plurality of reference images; inputting the plurality of two-dimensional images into a convolutional neural network, and performing feature extraction on the plurality of two-dimensional images by the convolutional neural network to generate a feature map; calculating the matching cost of the feature map of each source image and the feature map of the target reference image under different parallaxes, and generating a cost body corresponding to each source image based on the matching cost; optimizing the cost body by using the stacked cyclic hourglass network, and processing the optimized cost body to obtain a disparity map; and calculating a three-dimensional coordinate of each pixel point in the target reference image based on the disparity map and the camera parameters of the target reference image so as to complete reconstruction of the scene to be reconstructed. Therefore, the problems of poor three-dimensional reconstruction effect and the like in the prior art are solved.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction technology, and in particular to a three-dimensional reconstruction method, device, electronic device, storage medium and program product. Background Art

[0002] 3D reconstruction technology refers to the technology of restoring a 2D image sequence or a single image into a 3D model through a certain algorithm. It has a wide range of applications in computer vision, robot navigation, virtual reality, medical imaging, cultural heritage protection and other fields.

[0003] In existing 3D reconstruction, if the surface texture information of the object is insufficient or repeated, the existing matching algorithm often finds it difficult to accurately find corresponding points, resulting in holes or errors in the reconstruction results. In addition, changes in lighting conditions will affect the brightness and color of the image, making it difficult for the reconstruction algorithm to work stably under different lighting conditions, resulting in poor reconstruction effects. Summary of the invention

[0004] The present application provides a three-dimensional reconstruction method, device, electronic device, storage medium and program product to solve the problems of poor three-dimensional reconstruction effect in the prior art.

[0005] The first aspect of the present application provides a three-dimensional reconstruction method, comprising the following steps: obtaining multiple two-dimensional images of a scene to be reconstructed and corresponding camera parameters, wherein the two-dimensional images include multiple source images and multiple reference images; inputting the multiple two-dimensional images into a convolutional neural network, wherein the convolutional neural network performs feature extraction on the multiple two-dimensional images to generate feature maps; selecting a target reference image from the multiple reference images, calculating the matching cost between the feature map of each source image and the feature map of the target reference image under different disparities, and generating a cost body corresponding to each source image based on the matching cost; optimizing the cost body using a stacked cyclic hourglass network, and performing convolution processing on the optimized cost body to obtain a disparity map, Among them, the front end of the stacked recurrent hourglass network uses two GRU layers as the preprocessing stage. The stacked recurrent hourglass network is composed of two repeated hourglasses stacked together, and the output of the first hourglass is used as the input of the next hourglass. In each submodule of the stacked recurrent hourglass network, there are a large number of convolutional layers and GRU layers of different sizes. After the feature reaches the lowest resolution, the decoding operation is started through two deconvolution layers of different sizes. The second hourglass integrates the feature information of the previous hourglass through jump connections; the three-dimensional coordinates of each pixel in the target reference image are calculated based on the disparity map and the camera parameters of the target reference image, wherein the reconstruction of the scene to be reconstructed is completed through disparity regression and triangulation in disparity estimation.

[0006] Preferably, before calculating the matching cost between the feature map of each source image and the feature map of the target reference image at different parallaxes, it also includes: obtaining the pixel correspondence between each source image and the target reference image; based on the pixel correspondence, the feature map of the source image and the feature map of the target reference image, calculating the matching cost between each source image and the target reference image at different parallaxes.

[0007] Preferably, the pixel correspondence relationship is: ; in, is the pixel correspondence between pixel x in source image i and pixel x in reference image i, is the fixed parameter of the camera corresponding to the feature map of the source image, and d is the depth of the source image.

[0008] Preferably, the stacked recurrent hourglass network includes a GRU module, a first hourglass module and a second hourglass module, wherein the GRU module is used to merge the features in the cost map of the cost body; the first hourglass module is used to reduce the spatial resolution of the features in the cost body and decode the features through deconvolution after the features reach the minimum resolution; the second hourglass module is used to integrate the feature information output by the first hourglass module.

[0009] Preferably, the cost body is: ; in, is the cost volume at the L layer with a depth of d, is the depth value, is the number of layers, is the feature map of the reference image, is the average value of all feature volumes for each pixel, is the number of viewing angles.

[0010] Preferably, the three-dimensional coordinates of each pixel in the target reference image are calculated based on the disparity map and the camera parameters of the target reference image, including: identifying the disparity value of each pixel in the disparity map; calculating the depth of each pixel using the triangle measurement principle and the disparity value; and calculating the three-dimensional coordinates of each pixel in the target reference image based on the depth and the camera parameters.

[0011] The second aspect of the present application provides a three-dimensional reconstruction device, including: an acquisition module, used to acquire multiple two-dimensional images of a scene to be reconstructed and corresponding camera parameters, wherein the two-dimensional images include multiple source images and multiple reference images; an input module, used to acquire multiple two-dimensional images of a scene to be reconstructed and corresponding camera parameters, wherein the two-dimensional images include multiple source images and multiple reference images; a calculation module, used to select a target reference image from the multiple reference images, calculate the matching cost between a feature map of each source image and a feature map of the target reference image under different parallaxes, and generate a cost body corresponding to each source image based on the matching cost; an optimization module, used to optimize the cost body using a stacked cyclic hourglass network, and convolute the optimized cost body The disparity map is obtained by product processing, wherein the front end of the stacked cyclic hourglass network uses two GRU layers as the preprocessing stage. The stacked cyclic hourglass network is composed of two repeated hourglasses superimposed on each other, and the output of the first hourglass is used as the input of the next hourglass. In each submodule of the stacked cyclic hourglass network, there are a large number of convolutional layers and GRU layers of different sizes. After the feature reaches the lowest resolution, the decoding operation is started through two deconvolution layers of different sizes, and the second hourglass integrates the feature information of the previous hourglass through jump connections; a reconstruction module is used to calculate the three-dimensional coordinates of each pixel in the target reference image based on the disparity map and the camera parameters of the target reference image, wherein the reconstruction of the scene to be reconstructed is completed through disparity regression and triangulation in disparity estimation.

[0012] Preferably, it also includes: a first calculation module, which is used to obtain the pixel correspondence between each source image and the target reference image before calculating the matching cost between the feature map of each source image and the feature map of the target reference image under different parallaxes; based on the pixel correspondence, the feature map of the source image and the feature map of the target reference image, calculate the matching cost between the feature map of each source image and the feature map of the target reference image under different parallaxes.

[0013] Preferably, the pixel correspondence relationship is: ; in, is the pixel correspondence between pixel x in source image i and pixel x in reference image i, is the fixed parameter of the camera corresponding to the feature map of the source image, and d is the depth of the source image.

[0014] Preferably, the stacked recurrent hourglass network includes a GRU module, a first hourglass module and a second hourglass module, wherein the GRU module is used to merge the features in the cost map of the cost body; the first hourglass module is used to reduce the spatial resolution of the features in the cost body and decode the features through deconvolution after the features reach the minimum resolution; the second hourglass module is used to integrate the feature information output by the first hourglass module.

[0015] Preferably, the cost body is: ; in, is the cost volume at the L layer with a depth of d, is the depth value, is the number of layers, is the feature map of the reference image, is the average value of all feature volumes for each pixel, is the number of viewing angles.

[0016] Preferably, the calculation module is further used to: identify the disparity value of each pixel in the disparity map; calculate the depth of each pixel using the triangle measurement principle and the disparity value; and calculate the three-dimensional coordinates of each pixel in the target reference image based on the depth and camera parameters.

[0017] The third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a three-dimensional reconstruction method as described in the above embodiment.

[0018] The fourth aspect of the present application provides a computer-readable storage medium on which a computer program or instruction is stored. The computer program or instruction is executed by a processor to implement the three-dimensional reconstruction method as described in the above embodiment.

[0019] The fifth aspect of the present application provides a computer program product, including a computer program or instructions. When the computer program or instructions are executed, the three-dimensional reconstruction method as described in the above embodiment is implemented.

[0020] Therefore, this application has at least the following beneficial effects: The embodiment of the present application can extract features from multiple two-dimensional images through a convolutional neural network, and calculate the matching cost between each source image and a target reference image under different disparities based on the feature map, thereby improving the matching accuracy and reducing the incorrect matching caused by insufficient or repeated textures, and then generating an initial cost body based on the matching cost, and optimizing the initial cost body using the target network, and further generating a disparity map through convolution processing for the optimized initial cost body, thereby enhancing the robustness of the system in handling illumination changes, blur and occlusion, and providing more reliable depth estimation, and finally calculating the three-dimensional coordinates of each pixel point according to the camera parameters based on the disparity map and the target reference image, completing the three-dimensional reconstruction of the scene to be reconstructed, and improving the quality and effect of the three-dimensional reconstruction. Additional aspects and advantages of the present application will be partially given in the following description, and partially will become apparent from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which: Figure 1 A flowchart of a three-dimensional reconstruction method provided according to an embodiment of the present application; Figure 2 A specific flow chart of a three-dimensional reconstruction method provided according to an embodiment of the present application; Figure 3 A schematic diagram of the structure of a SRH (Stacked Recurrent Hourglass) network provided according to an embodiment of the present application; Figure 4 This is an example diagram of a three-dimensional reconstruction effect provided according to an embodiment of the present application; Figure 5 This is an example diagram of a three-dimensional reconstruction device provided according to an embodiment of the present application; Figure 6 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0023] The following describes the detection method, device, vehicle and storage medium of the labeled data sample of the embodiment of the present application with reference to the accompanying drawings. Specifically, Figure 1 A schematic flow chart of a three-dimensional reconstruction method provided in an embodiment of the present application.

[0024] like Figure 1 As shown, the three-dimensional reconstruction method includes the following steps: In step S101 , a plurality of two-dimensional images of a scene to be reconstructed and corresponding camera parameters are obtained.

[0025] The two-dimensional image includes multiple source images and multiple reference images. The source image is an image taken from different viewing angles. The reference image is a graphic selected from the multiple two-dimensional images and is used as a basis for subsequent matching of other source images.

[0026] It is understandable that the embodiments of the present application can obtain multiple two-dimensional images of the scene to be reconstructed and corresponding camera parameters, so as to subsequently perform three-dimensional reconstruction of the scene to be reconstructed.

[0027] In step S102, a plurality of two-dimensional images are input into a convolutional neural network, wherein the convolutional neural network extracts features from the plurality of two-dimensional images to generate feature maps.

[0028] It is understandable that the embodiment of the present application can input multiple two-dimensional images into a convolutional neural network (CNN), and the convolutional neural network extracts features from each input image and generates a feature map. Among them, the first few layers of a pre-trained image classification network (such as VGG or ResNet) can be used as a feature extractor.

[0029] In step S103, a target reference image is selected from a plurality of reference images, a matching cost between a feature map of each source image and a feature map of the target reference image at different disparities is calculated, and a preliminary cost volume corresponding to each source image is generated based on the matching cost.

[0030] Among them, the target reference image can select the image with the best coverage, the least occlusion or the highest quality to ensure accuracy and reliability in the subsequent matching process.

[0031] It can be understood that the embodiments of the present application can select a target reference image from multiple reference images, calculate the matching cost between each source image feature map and the feature map of the target reference image under different parallaxes, and generate a preliminary cost body corresponding to each source image based on the matching cost.

[0032] It should be noted that the cost volume, also known as the cost volume, is a three-dimensional structure, in which two dimensions correspond to the spatial coordinates of the image and the other dimension corresponds to the disparity.

[0033] In an embodiment of the present application, before calculating the matching cost between the feature map of each source image and the feature map of the target reference image at different parallaxes, it also includes: obtaining the pixel correspondence between each source image and the target reference image; based on the pixel correspondence, the feature map of the source image and the feature map of the target reference image, calculating the matching cost between the feature map of each source image and the feature map of the target reference image at different parallaxes.

[0034] It can be understood that the embodiments of the present application can obtain the pixel correspondence between each source image and the target reference image, and based on the pixel correspondence, the feature map of the source image and the feature map of the target reference image, calculate the matching cost of the feature map of each source image and the feature map of the target reference image under different parallaxes. The pixel correspondence can ensure the geometric consistency between different views, and the use of feature maps instead of original pixel intensities for matching can enhance the robustness to lighting changes, texture loss, etc., and improve the matching accuracy.

[0035] In the embodiment of the present application, the pixel correspondence relationship is: ; in, is the pixel correspondence between pixel x in source image i and pixel x in reference image i, is the fixed parameter of the camera corresponding to the feature map of the source image, and d is the depth of the source image.

[0036] Specifically, the process of generating a preliminary cost body in the embodiment of the present application includes: 1. Get pixel correspondence.

[0037] The pixel correspondence between the source image and the target reference image can be described by the homography matrix Hi(d). For a given depth d, the matrix projects each pixel in the source image to the corresponding position in the target reference image. Specifically, if x is a pixel in the source image and x′ is its corresponding point in the target reference image, then x′∼Hi(d)x, where “∼” represents the projective equivalence relationship. Since the true depth is unknown, it is necessary to traverse a series of possible depth values ​​(i.e., disparity d), calculate the corresponding homology matrix for each assumed depth, and determine the pixel correspondence based on this.

[0038] 2. Calculate the matching cost based on pixel correspondence.

[0039] According to the pixel correspondence obtained above, find the corresponding position of each pixel in the source image feature map in the target reference image feature map, and extract feature vectors at these two positions for comparison. In order to increase reliability, instead of comparing individual pixels separately, a small window (such as 3×33×3 or 5×55×5) is defined around each pixel, and then the similarity measurement of feature vectors (such as absolute difference sum, squared difference sum, mutual information, etc.) is performed within this window. Traverse a series of possible disparity values ​​(i.e., depth hypotheses) and calculate the matching cost for the pixel correspondence under each hypothesis. This step generates a cost body in which each element represents the matching cost under a specific disparity hypothesis.

[0040] 3. Generate the cost body.

[0041] The matching costs under all disparity assumptions are organized into a three-dimensional structure, the cost volume, which contains information about different disparity assumptions and provides basic data for subsequent disparity estimation.

[0042] In step S104, the cost body is optimized using a stacked cyclic hourglass network, and the optimized cost body is convolved to obtain a disparity map, wherein the front end of the stacked cyclic hourglass network uses two GRU layers as a preprocessing stage, and the stacked cyclic hourglass network is composed of two repeated hourglasses superimposed, and the output of the first hourglass is used as the input of the next hourglass. In each submodule of the stacked cyclic hourglass network, there are a large number of convolutional layers and GRU layers of different sizes. After the feature reaches the lowest resolution, the decoding operation is started through two deconvolution layers of different sizes, and the second hourglass integrates the feature information of the previous hourglass through a jump connection.

[0043] It is understandable that the embodiment of the present application can use a stacked cyclic hourglass network to optimize the cost volume, and perform convolution processing on the optimized cost volume to obtain a disparity map.

[0044] In the embodiment of the present application, the cost body is: ; in, is the cost volume at the L layer with a depth of d, is the depth value, is the number of layers, is the feature map of the reference image, is the average value of all feature volumes for each pixel, is the number of viewing angles.

[0045] In an embodiment of the present application, a stacked recurrent hourglass network includes a GRU module, a first hourglass module, and a second hourglass module, wherein the GRU module is used to merge features in a cost map of a cost body; the first hourglass module is used to reduce the spatial resolution of features in the cost body, and after the features reach the minimum resolution, decode the features through deconvolution; and the second hourglass module is used to integrate feature information output by the first hourglass module.

[0046] It can be understood that the SRH network of the embodiment of the present application includes a GRU module, a first hourglass module and a second hourglass module, wherein the GRU module is mainly used to merge the features in the cost map of the cost body, and it captures the dependencies under different disparity assumptions through the gated recurrent unit, and aggregates this information. The GRU can effectively process the time series data in the cost body that changes with the disparity, help the network learn a deeper feature representation, and by integrating the features under different disparity assumptions, it can enhance useful information and suppress noise, thereby improving the quality of subsequent processing.

[0047] The first hourglass module gradually reduces the spatial resolution of features in the cost volume through a series of 3D convolutional layers, while extracting high-level abstract features. When the features reach the lowest resolution, the deconvolution (or transposed convolution) layer is used to upsample the features to restore them to the original resolution.

[0048] The second hourglass module receives the output features from the first hourglass module and further integrates and refines them to ensure the effective fusion of multi-scale information. By integrating information at multiple scales, the quality of feature representation can be further enhanced, the accuracy of the final disparity estimation can be improved, the consistency between features of different scales can be ensured, and errors caused by scale differences can be avoided.

[0049] In step S105 , the three-dimensional coordinates of each pixel in the target reference image are calculated based on the disparity map and the camera parameters of the target reference image, wherein the reconstruction of the scene to be reconstructed is completed through disparity regression and triangulation in disparity estimation.

[0050] It can be understood that the embodiments of the present application can calculate the three-dimensional coordinates of each pixel in the target reference image based on the disparity map and the camera parameters of the target reference image, and complete the reconstruction of the scene to be reconstructed through disparity regression and triangulation in disparity estimation.

[0051] In an embodiment of the present application, the three-dimensional coordinates of each pixel in the target reference image are calculated based on the disparity map and the camera parameters of the target reference image, including: identifying the disparity value of each pixel in the disparity map; calculating the depth of each pixel using the triangle measurement principle and the disparity value; and calculating the three-dimensional coordinates of each pixel in the target reference image based on the depth and the camera parameters.

[0052] Among them, the camera parameters include intrinsic parameters and extrinsic parameters. The intrinsic parameters include focal length, principal point offset, pixel ratio, etc., and the extrinsic parameters include rotation matrix, translation vector, etc.

[0053] It can be understood that the embodiments of the present application can obtain the disparity value of each pixel in the disparity map, and use the triangle measurement principle and the disparity value to calculate the depth of each pixel, and then calculate the three-dimensional coordinates of each pixel in the target reference image based on the depth and camera parameters.

[0054] Specifically, the three-dimensional coordinate technology includes: 1. Identify the disparity value of each pixel in the disparity map.

[0055] 2. Use the triangulation principle and disparity value to calculate the depth of each pixel.

[0056] Based on the principle of triangulation, using the known baseline distance B and camera focal length f , through the formula Z= Bf / d calculates the depth Z of each pixel. Taking into account the camera intrinsic parameters (such as focal length and principal point), the pixel coordinates can be converted into normalized image plane coordinates through the intrinsic parameter matrix K, so as to perform depth calculation more accurately.

[0057] 3. Calculate the three-dimensional coordinates of each pixel in the target reference image based on the depth and camera parameters.

[0058] For each pixel point (u, v) and its corresponding depth Z, first convert the pixel coordinates to normalized image plane coordinates (x′, y′) through the intrinsic matrix K. Use the depth Z to calculate the X and Y coordinates of the point in the world coordinate system: X=x′×Z = , Y=y′×Z.

[0059] Combined with the external parameter matrix [R|t], these coordinates are converted from the camera coordinate system to the world coordinate system to obtain the final three-dimensional coordinates (X w ,Y w ,Z w ).

[0060] The following describes the 3D reconstruction method of the present application through a specific embodiment. The overall process is as follows: Figure 2 As shown, including: 1. Input.

[0061] A set of reference images and multiple corresponding source images.

[0062] Camera intrinsics and extrinsics are used to map image pixels into three-dimensional space.

[0063] 2. Feature extraction.

[0064] Image feature extraction: Use a convolutional neural network (CNN) to extract features from each input image and generate a feature map. Usually the first few layers of a pre-trained image classification network (such as VGG or ResNet) are used as feature extractors.

[0065] Most previous methods use high-resolution images as network inputs and extract deep features from the input images through a two-dimensional convolutional network, but the extracted features lack a more refined representation of the image interior. For this reason, based on previous work, this application uses a feature pyramid network to divide the input image pyramid into levels, using an image pyramid formed by representing N input images. By downsampling the source image and the reference image, I use a convolutional neural network to obtain the feature representation of the lth level, i.e., the feature extraction network.

[0066] This application will represents the fixed parameters of the camera corresponding to the feature map. Let n1 be the principal axis of the reference camera, Indicates the depth of the image, It isi Feature maps and depth The homology between the reference feature maps at , the monographic transformation is represented by a 3×3 matrix: ; Each homologous Indicates the source view In and pixels in the reference view The pixel correspondence between them. This correspondence is defined as , where “∼” denotes a projective equality.

[0067] 3. Cost Volume Construction, also known as cost volume construction.

[0068] Reference image features: Select a set of reference images and extract their feature maps.

[0069] Matching cost calculation: For each source image, its feature map is matched with the reference image features through a differentiable correlation layer (such as 3D convolution or deformable convolution) to generate a matching cost volume. Each cost volume contains the matching cost under different disparities.

[0070] This application constructs a cost volume of the lowest resolution corresponding to the image at layer L, generates an initial depth map using a three-dimensional convolution module and a softmax operation, and reconstructs a feature map that is warped to a reference view given and using differentiable bilinear interpolation in a monomorphic transformation.

[0071] The cost volume of the feature at a certain depth of the definition layer in each view is: ; in, is the cost volume at the L layer with a depth of d, is the depth value, is the number of layers, is the feature map of the reference image, is the average value of all feature volumes for each pixel, is the number of viewing angles. This metric ensures that the current depth of each pixel has the minimum feature variance, which corresponds to the photometric consistency constraint.

[0072] 4.SRH structure.

[0073] The SRH network structure is as follows Figure 3As shown. The front end of the module uses two GRU layers as a preprocessing stage, which will merge the features in the cost map. The SRH module consists of two repeated hourglasses stacked together, and the output of the first hourglass is used as the input of the next hourglass. In each submodule of the SRH, there are a large number of convolutional layers and GRU layers of different sizes. After the features reach the lowest resolution, the decoding operation begins through two deconvolutional layers of different sizes. The second hourglass integrates the feature information of the previous hourglass through jump connections, making our network deeper. Compared with the traditional stacked GRU structure, the network of this application shows better performance in robustness in handling text-free areas, blur and occlusion.

[0074] 5. Cost Volume Refinement.

[0075] 3D Convolution: The cost volume is further processed using 3D convolution layers to refine the matching cost.

[0076] Multi-scale processing: may include constructing and refining cost volumes at different scales to capture structural information at different scales.

[0077] 6. Disparity Estimation: Disparity regression: The cost volume is processed through a series of convolutional layers, and finally a disparity map is output, in which the value of each pixel represents the distance from the pixel to the reference camera plane.

[0078] Experiments have shown that when this application performs 3D point depth sampling on an image, sparse sampling will work better, because the 3D point projection distance of a high-resolution image is too close, resulting in a reduced sampling range and no additional information. The depth range measured in the reference view of the scene is set to arrive This application is in arrive M parallel planes are uniformly sampled within the depth range to construct the cost volume of the reference view, and the sampling depth is .

[0079] This application applies the 3D convolution module to The cost volume pyramid constructed by the layer is used to aggregate context information and obtain the probability volume , where the rough depth map is obtained by softmax operation. Each pixel corresponding to the layer The depth estimate of is calculated as: ; in, is the probability body, d is the depth, m is the minimum depth, M is the maximum depth, and p is the probability.

[0080] Soft Argmin: In the disparity estimation stage, a soft (arg) minimization operation is usually used to calculate the most likely disparity for each pixel instead of directly selecting the minimum cost.

[0081] 7. 3D Reconstruction

[0082] Triangulation: Using the estimated disparity map and camera parameters, the 3D coordinates of each point are calculated through the principle of triangulation.

[0083] 8. Output.

[0084] 3D Point Cloud: The final output of the network is a 3D point cloud that represents the scene structure in the input image.

[0085] In summary, this application estimates the depth map from coarse to fine by constructing a cost volume pyramid structure, which achieves accurate estimation of image depth, reduces the blur on the depth map surface, and improves depth prediction accuracy. The feature extraction network of this application can significantly reduce memory requirements while improving performance. Moreover, the model of this application can adapt to most 3D reconstruction tasks and can obtain high-quality reconstruction results. Examples of the effects of 3D reconstruction of this application are as follows: Figure 4 shown.

[0086] According to the 3D reconstruction method proposed in the embodiment of the present application, features of multiple 2D images can be extracted through a convolutional neural network, and the matching cost of each source image and the target reference image under different disparities can be calculated based on the feature map, thereby improving the matching accuracy and reducing the incorrect matching caused by insufficient or repeated texture. An initial cost body is then generated based on the matching cost, and the initial cost body is optimized using the target network. A disparity map is further generated for the optimized initial cost body through convolution processing, thereby enhancing the robustness of the system in dealing with illumination changes, blur and occlusion, and providing more reliable depth estimation. Finally, the 3D coordinates of each pixel are calculated based on the camera parameters based on the disparity map and the target reference image, thereby completing the 3D reconstruction of the scene to be reconstructed and improving the quality and effect of the 3D reconstruction.

[0087] Next, the three-dimensional reconstruction device proposed according to the embodiment of the present application is described with reference to the accompanying drawings.

[0088] Figure 5 It is a block diagram of a three-dimensional reconstruction device according to an embodiment of the present application.

[0089] like Figure 5 As shown, the three-dimensional reconstruction device 10 includes: an acquisition module 100 , an input module 200 , a calculation module 300 , an optimization module 400 and a reconstruction module 500 .

[0090] Among them, the acquisition module 100 is used to obtain multiple two-dimensional images of the scene to be reconstructed and corresponding camera parameters, wherein the two-dimensional image includes multiple source images and multiple reference images; the input module 200 is used to obtain multiple two-dimensional images of the scene to be reconstructed and corresponding camera parameters, wherein the two-dimensional image includes multiple source images and multiple reference images; the calculation module 300 is used to select a target reference image from the multiple reference images, calculate the matching cost between the feature map of each source image and the feature map of the target reference image under different disparities, and generate a cost body corresponding to each source image based on the matching cost; the optimization module 400 is used to optimize the cost body using a stacked cyclic hourglass network, and convolve the optimized cost body to obtain a disparity map; the reconstruction module 500 is used to calculate the three-dimensional coordinates of each pixel in the target reference image based on the disparity map and the camera parameters of the target reference image, so as to complete the reconstruction of the scene to be reconstructed.

[0091] In the embodiment of the present application, the device 10 of the embodiment of the present application further includes: a first calculation module.

[0092] Among them, the first calculation module is used to obtain the pixel correspondence between each source image and the target reference image before calculating the matching cost between the feature map of each source image and the feature map of the target reference image under different parallaxes; based on the pixel correspondence, the feature map of the source image and the feature map of the target reference image, calculate the matching cost between the feature map of each source image and the feature map of the target reference image under different parallaxes.

[0093] In the embodiment of the present application, the pixel correspondence relationship is: ; in, is the pixel correspondence between pixel x in source image i and pixel x in reference image i, is the fixed parameter of the camera corresponding to the feature map of the source image, and d is the depth of the source image.

[0094] In an embodiment of the present application, a stacked recurrent hourglass network includes a GRU module, a first hourglass module and a second hourglass module, wherein the GRU module is used to merge features in a cost map of a cost body; the first hourglass module is used to reduce the spatial resolution of features in the cost body and decode the features through deconvolution after the features reach the minimum resolution; the second hourglass module is used to integrate feature information output by the first hourglass module.

[0095] In the embodiment of the present application, the cost body is: ; in, is the cost volume at the L layer with a depth of d, is the depth value, is the number of layers, is the feature map of the reference image, is the average value of all feature volumes for each pixel, is the number of viewing angles.

[0096] In an embodiment of the present application, the calculation module 300 is further used to: identify the disparity value of each pixel in the disparity map; calculate the depth of each pixel using the triangle measurement principle and the disparity value; and calculate the three-dimensional coordinates of each pixel in the target reference image based on the depth and camera parameters.

[0097] It should be noted that the above explanations of the three-dimensional reconstruction method embodiment are also applicable to the three-dimensional reconstruction device of this embodiment, and will not be repeated here.

[0098] According to the 3D reconstruction device proposed in the embodiment of the present application, features of multiple 2D images can be extracted through a convolutional neural network, and the matching cost of each source image and the target reference image under different disparities is calculated based on the feature map, thereby improving the matching accuracy and reducing the false matching caused by insufficient or repeated textures. An initial cost body is generated based on the matching cost, and the initial cost body is optimized using the target network. A disparity map is generated for the optimized initial cost body through convolution processing, thereby enhancing the robustness of the system in dealing with lighting changes, blur and occlusion, providing more reliable depth estimation, and finally calculating the 3D coordinates of each pixel point based on the camera parameters based on the disparity map and the target reference image, completing the 3D reconstruction of the scene to be reconstructed, and improving the quality and effect of the 3D reconstruction.

[0099] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include: A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .

[0100] When the processor 602 executes the program, the three-dimensional reconstruction method provided in the above embodiment is implemented.

[0101] Furthermore, the electronic device further comprises: The communication interface 603 is used for communication between the memory 601 and the processor 602 .

[0102] The memory 601 is used to store computer programs that can be executed on the processor 602 .

[0103] The memory 601 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0104] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the communication interface 603, the memory 601 and the processor 602 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0105] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

[0106] The processor 602 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0107] An embodiment of the present application also provides a computer-readable storage medium on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the above three-dimensional reconstruction method is implemented.

[0108] An embodiment of the present application also provides a computer program product, including a computer program or instructions, which, when executed, implements the above three-dimensional reconstruction method.

[0109] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0110] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0111] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0112] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of multiple of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0113] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0114] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A three-dimensional reconstruction method, characterized in that: include: Acquire a plurality of two-dimensional images of a scene to be reconstructed and corresponding camera parameters, wherein the two-dimensional images include a plurality of source images and a plurality of reference images; Inputting the multiple two-dimensional images into a convolutional neural network, wherein the convolutional neural network extracts features from the multiple two-dimensional images to generate feature maps; Selecting a target reference image from the multiple reference images, calculating matching costs between a feature map of each source image and a feature map of the target reference image at different disparities, and generating a cost volume corresponding to each source image based on the matching costs; The cost body is optimized by using a stacked cyclic hourglass network, and the optimized cost body is convolved to obtain a disparity map, wherein the front end of the stacked cyclic hourglass network uses two GRU layers as a preprocessing stage, the stacked cyclic hourglass network is composed of two repeated hourglasses, and the output of the first hourglass is used as the input of the next hourglass. In each submodule of the stacked cyclic hourglass network, there are a large number of convolutional layers and GRU layers of different sizes. After the feature reaches the lowest resolution, the decoding operation is started through two deconvolution layers of different sizes, and the second hourglass integrates the feature information of the previous hourglass through a jump connection; The three-dimensional coordinates of each pixel in the target reference image are calculated based on the disparity map and the camera parameters of the target reference image, wherein the reconstruction of the scene to be reconstructed is completed through disparity regression and triangulation in disparity estimation.

2. The three-dimensional reconstruction method according to claim 1, characterized in that: Before calculating the matching cost between the feature map of each source image and the feature map of the target reference image at different disparities, the method further includes: Obtaining pixel correspondence between each source image and the target reference image; Based on the pixel correspondence, the feature map of the source image and the feature map of the target reference image, a matching cost between each source image and the target reference image at different disparities is calculated.

3. The three-dimensional reconstruction method according to claim 2, characterized in that: The pixel correspondence relationship is: ； in, is the pixel correspondence between pixel x in source image i and pixel x in reference image i, is the fixed parameter of the camera corresponding to the feature map of the source image, and d is the depth of the source image.

4. The three-dimensional reconstruction method according to claim 1, characterized in that: The stacked cyclic hourglass network includes a GRU module, a first hourglass module and a second hourglass module, wherein: The GRU module is used to merge the features in the cost map of the cost volume; The first hourglass module is used to reduce the spatial resolution of the features in the cost volume and perform a decoding operation on the features through deconvolution after the features reach a minimum resolution; The second hourglass module is used to integrate the feature information output by the first hourglass module.

5. The three-dimensional reconstruction method according to claim 1, characterized in that: The cost body is: ； in, is the cost volume at the L layer with a depth of d, is the depth value, is the number of layers, is the feature map of the reference image, is the average value of all feature volumes for each pixel, is the number of viewing angles.

6. The three-dimensional reconstruction method according to claim 1, characterized in that: The calculating the three-dimensional coordinates of each pixel in the target reference image based on the disparity map and the camera parameters of the target reference image includes: Identify the disparity value of each pixel in the disparity map; Calculate the depth of each pixel using the triangulation principle and the disparity value; The three-dimensional coordinates of each pixel in the target reference image are calculated based on the depth and the camera parameters.

7. A three-dimensional reconstruction device, characterized in that: include: An acquisition module, used to acquire a plurality of two-dimensional images of a scene to be reconstructed and corresponding camera parameters, wherein the two-dimensional images include a plurality of source images and a plurality of reference images; An input module, used to obtain a plurality of two-dimensional images of a scene to be reconstructed and corresponding camera parameters, wherein the two-dimensional images include a plurality of source images and a plurality of reference images; a calculation module, configured to select a target reference image from the plurality of reference images, calculate a matching cost between a feature map of each source image and a feature map of the target reference image at different disparities, and generate a cost volume corresponding to each source image based on the matching cost; An optimization module, used for optimizing the cost body by using a stacked cyclic hourglass network, and performing convolution processing on the optimized cost body to obtain a disparity map, wherein the front end of the stacked cyclic hourglass network uses two GRU layers as a preprocessing stage, the stacked cyclic hourglass network is composed of two repeated hourglasses, and the output of the first hourglass is used as the input of the next hourglass. In each submodule of the stacked cyclic hourglass network, there are a large number of convolutional layers and GRU layers of different sizes. After the feature reaches the lowest resolution, the decoding operation is started through two deconvolution layers of different sizes, and the second hourglass integrates the feature information of the previous hourglass through a jump connection; A reconstruction module is used to calculate the three-dimensional coordinates of each pixel in the target reference image based on the disparity map and the camera parameters of the target reference image, wherein the reconstruction of the scene to be reconstructed is completed through disparity regression and triangulation in disparity estimation.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the three-dimensional reconstruction method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program or instruction is executed by a processor to implement the three-dimensional reconstruction method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed, the three-dimensional reconstruction method according to any one of claims 1 to 6 is implemented.

Images