A multi-view three-dimensional reconstruction method and system based on GRU and 3DCNN

Abstract

The application discloses a multi-view three-dimensional reconstruction method and system based on GRU and 3DCNN, and the method comprises the following steps: performing feature extraction on images of a target object under different visual angles to obtain feature maps; performing differentiable homography transformation on each feature map under different depth assumptions; dividing a source feature body and a reference feature body into multiple groups according to channels, calculating group similarity and aggregating a matching cost body; performing cost body regularization operation and calculating a depth expectation value; upsampling and normalizing a low-resolution depth map; obtaining an updated depth map according to the matching cost body and constructing a dynamic cost body; constructing a matching cost body to obtain a depth map with the same size as the original resolution; and finally completing multi-view three-dimensional reconstruction according to a corresponding loss function. The embodiment of the application can improve the high-resolution depth estimation accuracy and realize more complete three-dimensional reconstruction, and can be widely applied to the field of computer technology.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more particularly to a multi-view algorithm based on GRU and 3DCNN. Figure 3 Reconstruction methods and systems. Background Technology

[0002] 3D reconstruction is the process of generating a 3D model from a 2D image or other data source (such as sensors, LiDAR, etc.). It has wide applications in fields such as computer vision, computer graphics, and robotics. The goal of 3D reconstruction is to reconstruct the geometry, dimensions, and spatial relationships of an object or scene for analysis, visualization, or other subsequent processing. Some common 3D reconstruction techniques include: manual modeling, structured light, laser scanning, and multi-view reconstruction. Figure 3 Reconstruction, etc.

[0003] Multiview Figure 3 3D reconstruction is a technique that reconstructs the structure of a three-dimensional scene by analyzing two-dimensional images from different perspectives. In recent years, this field has received widespread attention in multiple disciplines such as computer vision, graphics, and artificial intelligence, and its applications cover a wide range of areas including virtual reality, augmented reality, autonomous driving, robot navigation, and cultural heritage preservation.

[0004] Traditional multi-view reconstruction methods typically rely on techniques such as stereo vision and motion parallax. This process generally includes several key steps: image feature extraction, feature point matching to obtain a sparse point cloud, and multi-view stereo matching to obtain a dense point cloud. However, traditional multi-view... Figure 3 3D reconstruction methods face numerous challenges when dealing with complex scenes. For example, feature point matching becomes difficult under varying lighting, viewpoint changes, or partial occlusion, leading to inaccurate reconstruction results. Traditional methods often rely on simple geometric assumptions, making them ill-suited for scenes with complex shapes and rich details. Furthermore, the computational complexity of feature extraction and matching increases significantly with the number of input images, resulting in decreased efficiency.

[0005] In recent years, the rapid development of deep learning technology has enabled multi-view... Figure 3 3D reconstruction has brought new opportunities. Deep learning, by constructing deep neural networks, can automatically extract high-level features, thereby eliminating the need for manual feature design. Multi-view based on deep learning... Figure 3 3D reconstruction demonstrates significant advantages in areas such as automatic feature learning, end-to-end training, and robustness to noise and occlusion. Deep learning models can automatically learn effective features from input images, thereby better handling complex scenes and object shapes.

[0006] In multi-view Figure 3 In the field of reconstruction, the following problems mainly exist:

[0007] 1. Error accumulation problem in cascaded architecture. Incorrect low-resolution depth map estimation in the previous stage will cause the depth assumption in the next stage to fail to cover the true depth value, thus ultimately leading to incorrect estimation results;

[0008] 2. Detail loss problem in the low-resolution stage. In the low-resolution stage, the ground truth depth value needs to be downsampled to the same low resolution for supervision. Some fine details of scene structure will be lost during the downsampling process, which will cause the neural network to fail to learn the correct details and produce incorrect depth estimates. Summary of the Invention

[0009] The main objective of this invention is to propose a multi-view method based on GRU and 3DCNN. Figure 3 The proposed 3D reconstruction method and system can improve the accuracy of high-resolution depth estimation and achieve more complete 3D reconstruction.

[0010] To achieve the above objectives, one aspect of the present invention proposes a multi-view method based on GRU and 3DCNN. Figure 3 The 3D reconstruction method includes the following steps:

[0011] Obtain a collection of images of the target object from different perspectives, and determine the reference image and source image within it;

[0012] Feature extraction is performed on the reference image and the source image to obtain the corresponding feature maps of the reference image and the source image at different resolutions;

[0013] Multiple depth hypotheses are sampled within the depth range of the scene, and differentiable homography transformations are performed on each feature map under different depth hypotheses according to camera parameters to obtain the corresponding source feature volume and reference feature volume. The source feature volume and reference feature volume are divided into multiple groups according to channels, and the group similarity measurement module is used to calculate the group similarity. The matching cost volume is constructed and aggregated based on the group similarity.

[0014] Perform cost body regularization on the matching cost body to obtain the probability body;

[0015] The expected depth value is calculated along the depth direction for the probability volume to obtain the estimated depth map corresponding to the low-resolution reference image;

[0016] The low-resolution depth map is upsampled and normalized. Then, depth assumptions are sampled within a narrowed depth sampling range around the normalized depth value to construct a dynamic matching cost body. The matching cost body is then input into a gated loop unit to obtain an updated depth map.

[0017] Centered on the updated depth value, new depth hypotheses are sampled around it, and a new dynamic cost volume is constructed. Then, the repaired depth map is obtained through multiple iterations.

[0018] Using the repaired depth maps one by one, a matching cost volume is constructed, then the cost volume is regularized and the depth expectation is calculated to obtain a depth map corresponding to the reference image with a higher resolution, until a depth map with the same size as the original resolution is obtained.

[0019] The corresponding loss function is calculated based on the depth map, ultimately completing the multi-view calculation. Figure 3 Reconstruction.

[0020] In some embodiments, the step of extracting features from the reference image and the source image to obtain corresponding feature maps of the reference image and the source image at different resolutions specifically involves: inputting the reference image and the source image into the feature extraction module, wherein the size of the input reference image and the source image is 3×H×W, and after passing through 11 convolutional layers, the final output size is [missing information]. The convolutional layers in the third, sixth, and ninth layers are downsampled to reduce their original height and width to half.

[0021] In some embodiments, sampling multiple depth hypotheses within the depth range of the scene, and performing differentiable homography transformation on each feature map under different depth hypotheses according to camera parameters to obtain corresponding source feature bodies and reference feature bodies; dividing the source feature bodies and reference feature bodies into multiple groups by channel, calculating group similarity through a group similarity measurement module, and constructing and aggregating matching cost bodies based on group similarity, includes the following steps:

[0022] Based on the principle of planar scanning algorithm, a plane perpendicular to the principal optical axis of the reference camera is used as the depth assumption plane. All feature maps are distorted into different front planes of the reference camera, and all pixel coordinates on the source image are mapped to pixel coordinates on the reference image.

[0023] The pixel coordinates of all source images are distorted to the coordinates of the reference image to obtain the distorted source feature volume. The dimension of the source feature volume is Where C represents the channel dimension and D represents the depth dimension; the expression for the coordinate warping process is: Where, K0, K i Camera intrinsics for the reference image and the source image, respectively, R i,0 Let t be the corresponding rotation matrix. i,0 Let d be the corresponding translation vector. j p represents the corresponding sampling depth value. i,j The pixel coordinates corresponding to pixel p in the source image;

[0024] The reference feature body is copied D times and expanded to obtain All feature volumes are then evenly divided into G groups according to channel dimension, with each group having C' channels;

[0025] Calculate the group similarity between the distorted source feature and the expanded reference feature. The calculation formula is: Where <·,·> represent inner product operations;

[0026] After the group similarity measurement module, N-1 matching cost bodies are obtained, which can be denoted as: Its dimensions are The variance is used to aggregate the feature volumes and calculate the final matching cost volume. The formula for calculating the matching cost volume C is as follows:

[0027] In some embodiments, performing cost body regularization on the matching cost body to obtain a probability body includes the following steps:

[0028] Regularization is performed using 3D convolution to obtain the probability volume; specifically:

[0029] A 3D-UNet network structure with four scales is used to encode and decode the original matching cost volume, and the number of channels is compressed to 1. Finally, the probability volume is normalized by applying a SoftMax operation in the depth direction. After regularization, the output probability volume has a dimension of [missing value].

[0030] The step of calculating the expected depth value along the depth direction of the probability volume to obtain the estimated depth map corresponding to the low-resolution reference image is as follows: Depth value prediction is achieved using a regression approach, and the expected depth value of the sampled depth is calculated along the depth direction. The formula for calculating the depth estimate D is: Where P(d) represents the estimated probability of each pixel at depth d, [d min ,d max [ ] represents the range of depth values, and the output depth map size is [ ].

[0031] In some embodiments, the step of upsampling and normalizing the low-resolution depth map, sampling depth assumptions within a reduced depth sampling range centered on the normalized depth value, and constructing a dynamic matching cost body; inputting the matching cost body into a gated recurrent unit to obtain an updated depth map includes the following steps:

[0032] Using bilinear interpolation Upsampling of resolution depth map The resolution is then normalized to obtain the depth map D;

[0033] Multiple depth hypotheses are sampled around this depth value to construct a dynamic matching cost body C;

[0034] The cost volume is input into the gated loop unit to obtain the updated depth map. The gated loop unit and the updated depth value can be calculated using the following formulas:

[0035] z t =σ(Conv 3×3 ([h t-1 ,x]))

[0036] r t =σ(Conv 3×3 ([h t-1 ,x]))

[0037]

[0038] Δd=Conv 3×3 (Relu(Conv 3×3 h t ))

[0039] Among them, h t and h t-1 These represent the hidden states at the current time step and the previous time step, respectively. Represents a candidate hidden state, z t and r t These represent the update gate and reset gate, respectively; σ represents the sigmoid function; ⊙ represents element-wise dot product; Δd represents the depth update value.

[0040] Based on the depth update values ​​obtained from multiple iterations, the repaired depth map is obtained, with a depth map size of [size missing]. Resolution.

[0041] In some embodiments, the calculation of a corresponding loss function based on the depth map ultimately completes multi-view processing. Figure 3 Dimensional reconstruction includes the following steps:

[0042] Calculate the loss function for the regularization part, and calculate the L1 loss by comparing the depth map obtained by the 3D CNN regularization with the ground truth depth value at the corresponding resolution;

[0043] The loss function weights of the gated recurrent unit are calculated. The difference between the depth ground truth of the current stage and the previous stage is used in each stage to obtain the details lost between each pair of stages. The loss function weights for each pixel are calculated based on the lost details to obtain the hierarchical weights.

[0044] Calculate the partial loss function of the gated recurrent unit, and use the hierarchical weights to calculate the hierarchical L1 loss by comparing the depth map repaired by the gated recurrent unit with the ground truth depth value at the corresponding resolution.

[0045] In some embodiments, the L1 loss Loss1 between the depth map obtained by 3D CNN regularization and the ground truth depth value at the corresponding resolution is calculated as follows:

[0046]

[0047] Here, 'l' represents four stages, from stage 0 to stage 3. This represents the set of valid pixels whose true depth values ​​are known at the current stage. and These represent the regularized depth estimate and the true depth value of pixel p at the current stage, respectively.

[0048] The formula for calculating the L1 loss (Loss2) of the depth map repaired by the gated recurrent unit and the corresponding resolution's true depth value is as follows:

[0049]

[0050] Where 'l' represents three stages, from stage 1 to stage 3. This represents the set of valid pixels whose true depth values ​​are known at the current stage. and W represents the gated recurrent unit depth estimate and the true depth value of pixel p at the current stage, respectively. l (p) represents the pixel weight at the current stage;

[0051] The formula for calculating the pixel weight is:

[0052] Another aspect of the present invention provides a multi-view method based on GRU and 3DCNN. Figure 3 3D reconstruction system, including:

[0053] The first module is used to acquire a set of images of the target object from different perspectives and to determine the reference image and source image within it;

[0054] The second module is used to extract features from the reference image and the source image to obtain feature maps of the reference image and the source image at different resolutions.

[0055] The third module is used to sample multiple depth hypotheses within the depth range of the scene, and perform differentiable homography transformation on each feature map under different depth hypotheses according to the camera parameters to obtain the corresponding source feature volume and reference feature volume; the source feature volume and reference feature volume are divided into multiple groups according to channels, and the group similarity measurement module is used to calculate the group similarity, and the matching cost volume is constructed and aggregated based on the group similarity.

[0056] The fourth module is used to perform cost body regularization on the matching cost body to obtain the probability body;

[0057] The fifth module is used to calculate the expected depth value of the probability volume along the depth direction to obtain the estimated depth map corresponding to the low-resolution reference image.

[0058] The sixth module is used to upsample and normalize the low-resolution depth map, sample depth assumptions within a narrowed depth sampling range around the normalized depth value, and construct a dynamic matching cost body; the matching cost body is input into the gated loop unit to obtain an updated depth map.

[0059] The seventh module is used to sample new depth hypotheses around the updated depth value, construct a new dynamic cost volume, and then obtain the repaired depth map through multiple iterations.

[0060] The eighth module is used to construct a matching cost volume by using the repaired depth map one by one, then perform cost volume regularization and calculate the depth expectation to obtain a depth map corresponding to the reference image with a higher resolution, until a depth map with the same resolution and size as the original is obtained.

[0061] The ninth module is used to calculate the corresponding loss function based on the depth map, ultimately completing the multi-view calculation. Figure 3 Reconstruction.

[0062] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0063] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0064] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0065] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a multi-view method based on GRU and 3DCNN. Figure 3A 3D reconstruction method and system are disclosed. This scheme acquires a set of images of a target object from different viewpoints and identifies a reference image and a source image. Feature extraction is performed on the reference and source images to obtain corresponding feature maps at different resolutions. Multiple depth hypotheses are sampled within the depth range of the scene, and differentiable homography transformations are applied to each feature map under different depth hypotheses based on camera parameters to obtain corresponding source and reference feature volumes. The source and reference feature volumes are divided into multiple groups by channel, and group similarity is calculated using a group similarity measurement module. A matching cost volume is constructed and aggregated based on the group similarity. Cost volume regularization is performed on the matching cost volume to obtain a probability volume. The expected depth value is calculated along the depth direction of the probability volume. The process involves: obtaining an estimated depth map corresponding to a low-resolution reference image; upsampling and normalizing the low-resolution depth map; sampling depth hypotheses within a reduced depth sampling range centered on the normalized depth value to construct a dynamic matching cost body; inputting the matching cost body into a gated recurrent unit to obtain an updated depth map; sampling new depth hypotheses within a range centered on the updated depth value to construct a new dynamic cost body, and iterating through multiple iterations to obtain a repaired depth map; using the repaired depth map one by one to construct a matching cost body, performing cost body regularization, and calculating the depth expectation to obtain a depth map corresponding to a higher-resolution reference image, until a depth map with the same size as the original resolution is obtained; calculating the corresponding loss function based on the depth map to finally complete multi-view processing. Figure 3 3D Reconstruction. Embodiments of this invention can improve the accuracy of high-resolution depth estimation and achieve more complete 3D reconstruction. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0067] Figure 2 This is a flowchart of the overall steps provided in the embodiments of the present invention;

[0068] Figure 3 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0070] It is understood that the terms “first,” “second,” etc., used in this invention may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to determination” as used herein may be interpreted as “when…” or “when…” or “in response to determination.”

[0071] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0072] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0073] Before providing a detailed description of the embodiments of the present invention, some related technologies involved in the embodiments of the present invention will be described first, as follows:

[0074] Deep learning: Deep learning is a machine learning method that mimics the way the human brain works, processing and understanding data through multi-layered neural networks. The "depth" in this term refers to the multiple layers or depth of the neural network, typically including an input layer, multiple hidden layers, and an output layer. The depth and complexity of these neural networks enable them to extract features, patterns, and correlations from large-scale and complex data, thereby achieving efficient data analysis and decision-making.

[0075] Multiview Figure 3 Multidimensional Reconstruction: Multiple Views Figure 3Multi-view stereo (MDS) is a technique for generating 3D models of objects or scenes from 2D images taken from different viewpoints. This process typically involves extracting and matching feature points from multiple images, then using geometric methods to reconstruct the 3D structural information. Figure 3 3D reconstruction is widely used in fields such as computer vision, virtual reality, augmented reality, filmmaking, geographic mapping, and robotics.

[0076] Gated Recurrent Unit (GRU): A GRU is a recurrent neural network (RNN) architecture used to process sequential data. It can be used for time series prediction, natural language processing, and other tasks that require consideration of contextual information. GRUs were designed to overcome the vanishing gradient problem in traditional RNNs with long sequences.

[0077] The multi-view based GRU and 3DCNN provided in this embodiment of the invention Figure 3 This invention relates to a 3D reconstruction method and system, which falls under the field of computer technology and can be widely applied to the deployment of automated systems such as autonomous driving and drones. The embodiments of this invention provide a multi-view reconstruction method based on GRU and 3DCNN. Figure 3 The 3D reconstruction method can be applied to terminals, servers, or software running on either terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be a multi-view implementation based on GRU and 3DCNN. Figure 3 Applications of dimensional reconstruction methods, etc., but not limited to the above forms.

[0078] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0079] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided by an embodiment of the present invention. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0080] Server 101 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0081] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0082] Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc. It can also be a vehicle-mounted terminal of the various device types described above, but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.

[0083] Exemplary based on Figure 1 The implementation environment shown in this embodiment of the invention provides a multi-view system based on GRU and 3DCNN. Figure 3The following section discusses multi-view reconstruction methods based on GRU and 3DCNN. Figure 3 The method of dimensional reconstruction is illustrated using server 101 as an example. It can be understood that this method can also be applied to terminal 102.

[0084] Reference Figure 2 , Figure 2 The multi-view application of GRU and 3DCNN based on servers provided in the embodiments of the present invention Figure 3 The flowchart illustrates the 3D reconstruction method, which can be executed by any of the aforementioned computer devices (including servers or terminals). (Refer to...) Figure 2 The method may include the following steps:

[0085] A multi-view approach based on GRU and 3DCNN Figure 3 The 3D reconstruction method includes the following steps:

[0086] Obtain a collection of images of the target object from different perspectives, and determine the reference image and source image within it;

[0087] Feature extraction is performed on the reference image and the source image to obtain the corresponding feature maps of the reference image and the source image at different resolutions;

[0088] Multiple depth hypotheses are sampled within the depth range of the scene, and differentiable homography transformations are performed on each feature map under different depth hypotheses according to camera parameters to obtain the corresponding source feature volume and reference feature volume. The source feature volume and reference feature volume are divided into multiple groups according to channels, and the group similarity measurement module is used to calculate the group similarity. The matching cost volume is constructed and aggregated based on the group similarity.

[0089] Perform cost body regularization on the matching cost body to obtain the probability body;

[0090] The expected depth value is calculated along the depth direction for the probability volume to obtain the estimated depth map corresponding to the low-resolution reference image;

[0091] The low-resolution depth map is upsampled and normalized. Then, depth assumptions are sampled within a narrowed depth sampling range around the normalized depth value to construct a dynamic matching cost body. The matching cost body is then input into a gated loop unit to obtain an updated depth map.

[0092] Centered on the updated depth value, new depth hypotheses are sampled around it, and a new dynamic cost volume is constructed. Then, the repaired depth map is obtained through multiple iterations.

[0093] Using the repaired depth maps one by one, a matching cost volume is constructed, then the cost volume is regularized and the depth expectation is calculated to obtain a depth map corresponding to the reference image with a higher resolution, until a depth map with the same size as the original resolution is obtained.

[0094] The corresponding loss function is calculated based on the depth map, ultimately completing the multi-view calculation. Figure 3 Reconstruction.

[0095] In some embodiments, the step of extracting features from the reference image and the source image to obtain corresponding feature maps of the reference image and the source image at different resolutions specifically involves: inputting the reference image and the source image into the feature extraction module, wherein the size of the input reference image and the source image is 3×H×W, and after passing through 11 convolutional layers, the final output size is [missing information]. The convolutional layers in the third, sixth, and ninth layers are downsampled to reduce their original height and width to half.

[0096] In some embodiments, sampling multiple depth hypotheses within the depth range of the scene, and performing differentiable homography transformation on each feature map under different depth hypotheses according to camera parameters to obtain corresponding source feature bodies and reference feature bodies; dividing the source feature bodies and reference feature bodies into multiple groups by channel, calculating group similarity through a group similarity measurement module, and constructing and aggregating matching cost bodies based on group similarity, includes the following steps:

[0097] Based on the principle of planar scanning algorithm, a plane perpendicular to the principal optical axis of the reference camera is used as the depth assumption plane. All feature maps are distorted into different front planes of the reference camera, and all pixel coordinates on the source image are mapped to pixel coordinates on the reference image.

[0098] The pixel coordinates of all source images are distorted to the coordinates of the reference image to obtain the distorted source feature volume. The dimension of the source feature volume is Where C represents the channel dimension and D represents the depth dimension; the expression for the coordinate warping process is: Where, K0, K i Camera intrinsics for the reference image and the source image, respectively, R i,0 Let t be the corresponding rotation matrix. i,0 Let d be the corresponding translation vector. j p represents the corresponding sampling depth value. i,j The pixel coordinates corresponding to pixel p in the source image;

[0099] The reference feature body is copied D times and expanded to obtain All feature volumes are then evenly divided into G groups according to channel dimension, with each group having C' channels;

[0100] Calculate the group similarity between the distorted source feature and the expanded reference feature. The calculation formula is: Where <·,·> represent inner product operations;

[0101] After the group similarity measurement module, N-1 matching cost bodies are obtained, which can be denoted as: Its dimensions are The variance is used to aggregate the feature volumes and calculate the final matching cost volume. The formula for calculating the matching cost volume C is as follows:

[0102] In some embodiments, performing cost body regularization on the matching cost body to obtain a probability body includes the following steps:

[0103] Regularization is performed using 3D convolution to obtain the probability volume; specifically:

[0104] A 3D-UNet network structure with four scales is used to encode and decode the original matching cost volume, and the number of channels is compressed to 1. Finally, the probability volume is normalized by applying a SoftMax operation in the depth direction. After regularization, the output probability volume has a dimension of [missing value].

[0105] The step of calculating the expected depth value along the depth direction of the probability volume to obtain the estimated depth map corresponding to the low-resolution reference image is as follows: Depth value prediction is achieved using a regression approach, and the expected depth value of the sampled depth is calculated along the depth direction. The formula for calculating the depth estimate D is: Where P(d) represents the estimated probability of each pixel at depth d, [d min ,d max [ ] represents the range of depth values, and the output depth map size is [ ].

[0106] In some embodiments, the step of upsampling and normalizing the low-resolution depth map, sampling depth assumptions within a reduced depth sampling range centered on the normalized depth value, and constructing a dynamic matching cost body; inputting the matching cost body into a gated recurrent unit to obtain an updated depth map includes the following steps:

[0107] Using bilinear interpolation Upsampling of resolution depth map The resolution is then normalized to obtain the depth map D;

[0108] Multiple depth hypotheses are sampled around this depth value to construct a dynamic matching cost body C;

[0109] The cost volume is input into the gated loop unit to obtain the updated depth map. The gated loop unit and the updated depth value can be calculated using the following formulas:

[0110] z t =σ(Conv 3×3 ([h t-1 ,x]))

[0111] r t =σ(Conv 3×3 ([h t-1 ,x]))

[0112]

[0113] Δd=Conv 3×3 (Relu(Conv 3×3 h t ))

[0114] Among them, h t and h t-1 These represent the hidden states at the current time step and the previous time step, respectively. Represents a candidate hidden state, z t and r t These represent the update gate and reset gate, respectively; σ represents the sigmoid function; ⊙ represents element-wise dot product; Δd represents the depth update value.

[0115] Based on the depth update values ​​obtained from multiple iterations, the repaired depth map is obtained, with a depth map size of [size missing]. Resolution.

[0116] In some embodiments, the calculation of a corresponding loss function based on the depth map ultimately completes multi-view processing. Figure 3 Dimensional reconstruction includes the following steps:

[0117] Calculate the loss function for the regularization part, and calculate the L1 loss by comparing the depth map obtained by the 3D CNN regularization with the ground truth depth value at the corresponding resolution;

[0118] The loss function weights of the gated recurrent unit are calculated. The difference between the depth ground truth of the current stage and the previous stage is used in each stage to obtain the details lost between each pair of stages. The loss function weights for each pixel are calculated based on the lost details to obtain the hierarchical weights.

[0119] Calculate the partial loss function of the gated recurrent unit, and use the hierarchical weights to calculate the hierarchical L1 loss by comparing the depth map repaired by the gated recurrent unit with the ground truth depth value at the corresponding resolution.

[0120] In some embodiments, the L1 loss Loss1 between the depth map obtained by 3D CNN regularization and the ground truth depth value at the corresponding resolution is calculated as follows:

[0121]

[0122] Here, 'l' represents four stages, from stage 0 to stage 3. This represents the set of valid pixels whose true depth values ​​are known at the current stage. and These represent the regularized depth estimate and the true depth value of pixel p at the current stage, respectively.

[0123] The formula for calculating the L1 loss (Loss2) of the depth map repaired by the gated recurrent unit and the corresponding resolution's true depth value is as follows:

[0124]

[0125] Where 'l' represents three stages, from stage 1 to stage 3. This represents the set of valid pixels whose true depth values ​​are known at the current stage. and W represents the gated recurrent unit depth estimate and the true depth value of pixel p at the current stage, respectively. l (p) represents the pixel weight at the current stage;

[0126] The formula for calculating the pixel weight is:

[0127] The specific implementation process of this invention will be described in detail below using a specific application scenario as an example:

[0128] The overall steps of this embodiment of the invention include:

[0129] Step 1: Obtain N images of the same object taken from different perspectives, use one of them as the reference image, and the remaining N-1 images as the source images;

[0130] Step 2: Input the reference image and the source image into the feature extraction module to obtain the corresponding feature maps of the reference image and the source image at different resolutions;

[0131] Step 3: Sample multiple depth hypotheses within the depth range of the scene, and perform differentiable homography transformation on the feature map under different depth hypotheses according to the camera parameters to obtain the feature volume;

[0132] Step 4: Divide the source feature body and the reference feature body into g groups according to channels, calculate the group similarity through the group similarity measurement module, and construct and aggregate the matching cost body based on the group similarity.

[0133] Step 5: Perform cost body regularization on the matching cost body to obtain the probability body;

[0134] Step 6: Calculate the expected depth value along the depth direction for the probability volume to obtain the estimated depth map corresponding to the low-resolution reference image;

[0135] Step 7: Upsample and normalize the low-resolution depth map, and sample depth assumptions within a reduced depth sampling range around the normalized depth value to construct a dynamic matching cost body.

[0136] Step 8: Input the cost volume into the gated loop unit to obtain the updated depth map;

[0137] Step 9: Using the updated depth value as the center, sample new depth hypotheses around it and construct a new dynamic cost body;

[0138] Step 10: Repeat steps 8 and 9, iterating multiple times to obtain the repaired depth map;

[0139] Step 11: Using the repaired depth map, construct a matching cost volume, perform cost volume regularization, and calculate the depth expectation to obtain a depth map corresponding to the higher resolution reference image.

[0140] Step 12: Repeat steps 7 to 11 until a depth map of the same size as the original resolution is obtained.

[0141] Step 13: Calculate the regularization loss function. Calculate the L1 loss by comparing the depth map obtained from the 3D CNN regularization with the ground truth depth value at the corresponding resolution.

[0142] Step 14: Calculate the loss function weights of the gated recurrent unit. In each stage, the difference between the depth ground truth of the current stage and the previous stage is used to obtain the details lost between each pair of stages. Calculate the loss function weights for each pixel based on the lost details. The more details are lost, the higher the weights are, thus obtaining hierarchical weights.

[0143] Step 15: Calculate the partial loss function of the gated recurrent unit. Using the hierarchical weights, calculate the hierarchical L1 loss by comparing the depth map repaired by the gated recurrent unit with the ground truth depth value at the corresponding resolution.

[0144] Specifically, the implementation process for each of the above steps is described in detail below:

[0145] Step 1: Acquire N images of the same object taken from different perspectives. Use one of these images as the reference image I0, and the remaining N-1 images as the source images.

[0146] Step 2: Input the above reference image and source image into the feature extraction module. The size of the input reference image and source image is 3×H×W. After 11 convolutional layers, the final output size is... The convolutional layers in layers 3, 6, and 9 are downsampled to half their original height and width. The basic structure of the entire feature extraction network is shown in Table 1. The final output reference feature map can be denoted as F0, and the source feature map can be denoted as...

[0147] Table 1 Basic Structure of Feature Extraction Network

[0148]

[0149]

[0150] Step 3, for In the resolution stage, multiple depth hypotheses are sampled within the depth range of the scene, and the feature maps are subjected to differentiable homography transformations under different depth hypotheses according to camera parameters to obtain feature volumes. The source feature volumes and reference feature volumes are divided into g groups by channel, and the group similarity measurement module calculates the group similarity. Based on the group similarity, a matching cost volume is constructed and aggregated. The specific process is as follows:

[0151] Based on the principle of planar scanning algorithm, a plane perpendicular to the principal optical axis of the reference camera is used as the depth assumption plane. All feature maps are distorted into different front planes of the reference camera, mapping all pixel coordinates on the source image to pixel coordinates on the reference image. The minimum depth is denoted as d. min The maximum depth is denoted as d. max Sampling D depths from the minimum to the maximum depth, the distortion process is called a differentiable homography transformation, and the corresponding calculation is given by equation (1):

[0152]

[0153] In equation (1), K0, K i Camera intrinsics for the reference image and the source image, respectively, R i,0 Let t be the corresponding rotation matrix. i,0 Let d be the corresponding translation vector. j p represents the corresponding sampling depth value. i,j represents the pixel coordinates corresponding to pixel p in the source image.

[0154] By twisting the pixel coordinates of all source images to the coordinates of the reference image using equation (1), the distorted source feature volume can be denoted as: Its dimensions are Where C represents the channel dimension and D represents the depth dimension. The reference feature volume is copied D times and expanded into... To maintain dimensional consistency, all feature volumes are evenly divided into G groups according to channel dimension, with each group having C' channels. and The group similarity between them is calculated by equation (2):

[0155]

[0156] In equation (2), <·,·> represents the inner product operation.

[0157] After the group similarity measurement module, N-1 matching cost bodies are obtained, which can be denoted as: Its dimensions are To accommodate any number of source image inputs, the variance is used to aggregate the feature volumes and calculate the final matching cost volume, which can be calculated by equation (3):

[0158]

[0159] In this embodiment, G=8 is set. By calculating the relevance through grouping, the relevance channels of the features can be reduced to the number of groups, reducing memory consumption. Grouping can obtain rich relevance information, making the matching cost body more accurate.

[0160] Step 4: Regularize the matching cost volume using 3DCNN to obtain the probability volume P; the specific process is as follows:

[0161] Due to non-Lambertian surfaces and occlusion, the matching cost volume obtained above contains a lot of noise. Therefore, regularization is performed using 3D convolution to obtain the probability volume. The probability volume can be used not only for pixel-by-pixel depth prediction but also to measure the confidence of the estimate. This embodiment uses a 3D-UNet network structure with four scales to encode and decode the original matching cost volume, and finally compresses the number of each channel to 1. Finally, the probability volume is normalized in the depth direction using a SoftMax operation. The output probability volume after regularization has a dimension of [missing value].

[0162] Step 5: Calculate the expected depth value along the depth direction using the above probability volume to obtain the estimated depth map corresponding to the low-resolution reference image; the specific process is as follows:

[0163] To avoid gaps and unevenness along the depth direction, a regression method is used to predict the depth value. Essentially, this method calculates the expected value of the sampled depth value along the depth direction. The depth estimate can be calculated using equation (4):

[0164]

[0165] In equation (4), P(d) represents the estimated probability of each pixel at depth d, [d min ,d max [ ] represents the range of depth values, and the output depth map size is [ ].

[0166] Step 6: After upsampling and normalizing the above depth map, sample depth hypotheses within a reduced depth sampling range centered on the normalized depth value to construct a dynamic matching cost body. Input the cost body into the gated recurrent unit to obtain an updated depth map; the specific process is as follows:

[0167] Using bilinear interpolation Upsampling of resolution depth map The resolution is normalized to obtain the depth map D. Multiple depth hypotheses are sampled around this depth value to construct a dynamic matching cost body C. The cost body is input into a gated recurrent unit to obtain an updated depth map. The gated recurrent unit and the updated depth value can be calculated by equations (5) to (9):

[0168] z t =σ(Conv 3×3 ([h t-1 Formula (5)

[0169] r t =σ(Conv 3×3 ([h t-1 Formula (6)

[0170]

[0171] Δd=Conv 3×3 (Relu(Conv 3×3 h t Equation (9)

[0172] In equations (5) to (8), h t and h t-1 These represent the hidden states at the current time step and the previous time step, respectively. Represents a candidate hidden state, z t and r t Let represent the update gate and reset gate, respectively; σ represents the sigmoid function; and ⊙ represents element-wise multiplication. The hidden state output by the gated recurrent unit in equation (9) can be used to obtain the depth update value Δd through two convolutional layers. Iterating the above process multiple times yields the repaired depth map, the size of which is... The resolution remains the same as the current stage.

[0173] Step 7: Using the repaired depth map, construct a matching cost volume, perform cost volume regularization, and calculate the depth expectation to obtain a depth map corresponding to the higher resolution reference image.

[0174] Step 8: Repeat steps 6 and 7 to obtain the following results. The goal is to estimate the depth map of the reference image by using both the original resolution and the depth map of the original resolution.

[0175] Step 9: Calculate the loss function. The loss function consists of two parts: the multiple repaired depth maps generated by the gated recurrent unit iterations and the depth map generated by the regularization part. The loss function needs to be calculated for the depth maps at intermediate scales in each stage. The loss function can be calculated by equation (10):

[0176] Loss = Loss1 + Loss2 (Equation 10)

[0177] Where Loss1 represents the L1 loss between the depth map generated by the regularization part and the ground truth depth:

[0178]

[0179] In equation (11), l represents four stages, from stage 0 to stage 3. This represents the set of valid pixels whose true depth values ​​are known at the current stage. and These represent the regularized depth estimate and the true depth value of pixel p at the current stage, respectively.

[0180] Loss2 represents the hierarchical L1 loss between the depth map generated by the gated recurrent unit and the ground truth depth value:

[0181]

[0182] In equation (12), l represents three stages, from stage 1 to stage 3. This represents the set of valid pixels whose true depth values ​​are known at the current stage. and W represents the gated recurrent unit depth estimate and the true depth value of pixel p at the current stage, respectively. l (p) represents the pixel weight at the current stage. W l (p) can be calculated using equation (13):

[0183]

[0184] In equation (13), the true depth value D is... gt downsampling to The resolution, upsampled back to the original resolution, can be denoted as: Similarly, we can obtain and When calculating the loss function at the intermediate size stage, the downsampled ground truth depth value is used. This ground truth depth value loses some details during the downsampling process, and the degree of detail loss varies depending on the downsampling ratio. Therefore, the gated recurrent unit used in the next stage of this embodiment can recover the lost details from a higher resolution. Using the percentage of depth difference as the weight of the pixel loss function can highlight the proportion of details, making the network more focused on learning to recover detailed structures.

[0185] Step 10: Using the obtained final depth value and the camera parameters corresponding to the image, project the image back into 3D space to obtain a partial point cloud. By projecting all images back into 3D space in sequence, a unified dense 3D point cloud model can be obtained.

[0186] In summary, the present invention has the following characteristics:

[0187] 1. The multi-view implementation of this invention Figure 3 In 3D reconstruction technology, gated recurrent units are used to recover the detailed information lost during low-resolution reconstruction, so that the depth estimation of each level can be performed within a more accurate depth range, thereby making the final reconstruction result more accurate and complete.

[0188] 2. The multi-view functionality implemented in this invention Figure 3 In 3D reconstruction technology, by constructing a hierarchical loss function, the detailed information that needs to be recovered at each level can be accurately perceived, thereby enhancing the ability of the gated recurrent unit to recover and fit the loss details.

[0189] 3. Cascaded multi-view stereo geometry framework with embedded gated recurrent units: This invention uses gated recurrent units to recover the detailed information lost in low-resolution depth estimation, adjusts the depth value of each level to the correct depth range, and then uses the 3D convolution of the cascaded architecture for accurate depth estimation.

[0190] 4. Normalized gated recurrent unit: The normalized depth sampling used in this invention can be generalized to scenes with arbitrary depth intervals. Even if the depth interval of the application scene is different from that used in training, accurate results can still be obtained.

[0191] 5. Hierarchical Loss Function: The hierarchical loss function used in this invention aims to strengthen the loss function of lost details and improve the ability of neural networks to recover lost detailed structures.

[0192] Compared with the prior art, the present invention has the following advantages:

[0193] 1. High accuracy: Avoids the problem of accumulated errors caused by misestimation;

[0194] 2. High integrity: It restores the detailed structure lost due to depth ground truth downsampling;

[0195] 3. High versatility: The framework structure of this invention has strong versatility, and the feature extraction and regularization parts can be replaced with any feature extractor and regularization algorithm;

[0196] 4. High generalization ability: The normalized depth sampling method adopted in this invention can be generalized to scenes with arbitrary depth intervals. Even if the depth interval of the application scene is different from that used in training, accurate results can still be obtained.

[0197] Another aspect of the present invention provides a multi-view method based on GRU and 3DCNN. Figure 3 3D reconstruction system, including:

[0198] The first module is used to acquire a set of images of the target object from different perspectives and to determine the reference image and source image within it;

[0199] The second module is used to extract features from the reference image and the source image to obtain feature maps of the reference image and the source image at different resolutions.

[0200] The third module is used to sample multiple depth hypotheses within the depth range of the scene, and perform differentiable homography transformation on each feature map under different depth hypotheses according to the camera parameters to obtain the corresponding source feature volume and reference feature volume; the source feature volume and reference feature volume are divided into multiple groups according to channels, and the group similarity measurement module is used to calculate the group similarity, and the matching cost volume is constructed and aggregated based on the group similarity.

[0201] The fourth module is used to perform cost body regularization on the matching cost body to obtain the probability body;

[0202] The fifth module is used to calculate the expected depth value of the probability volume along the depth direction to obtain the estimated depth map corresponding to the low-resolution reference image.

[0203] The sixth module is used to upsample and normalize the low-resolution depth map, sample depth assumptions within a narrowed depth sampling range around the normalized depth value, and construct a dynamic matching cost body; the matching cost body is input into the gated loop unit to obtain an updated depth map.

[0204] The seventh module is used to sample new depth hypotheses around the updated depth value, construct a new dynamic cost volume, and then obtain the repaired depth map through multiple iterations.

[0205] The eighth module is used to construct a matching cost volume by using the repaired depth map one by one, then perform cost volume regularization and calculate the depth expectation to obtain a depth map corresponding to the reference image with a higher resolution, until a depth map with the same resolution and size as the original is obtained.

[0206] The ninth module is used to calculate the corresponding loss function based on the depth map, ultimately completing the multi-view calculation. Figure 3 Reconstruction.

[0207] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0208] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned multi-view based on GRU and 3DCNN. Figure 3 3D reconstruction method. The electronic device can be any smart terminal, including tablets, in-vehicle computers, etc.

[0209] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0210] Please see Figure 3 , Figure 3 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0211] The processor 301 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0212] The memory 302 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 302 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 302 and is called and executed by the processor 301 to execute the multi-view based GRU and 3DCNN embodiments of the present invention. Figure 3 3D reconstruction method;

[0213] Input / output interface 303 is used to implement information input and output;

[0214] The communication interface 304 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0215] Bus 305 transmits information between various components of the device (e.g., processor 301, memory 302, input / output interface 303, and communication interface 304);

[0216] The processor 301, memory 302, input / output interface 303, and communication interface 304 are connected to each other within the device via bus 305.

[0217] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned multi-view based on GRU and 3DCNN. Figure 3 Dimensional reconstruction method.

[0218] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0219] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0220] It should be noted that in various specific embodiments of the present invention, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of the present invention require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data for the normal operation of the embodiments of the present invention acquired.

[0221] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0222] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0223] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0224] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0225] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0226] It should be understood that in this invention, "at least one (item)" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0227] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0228] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0229] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0230] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0231] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.

Claims

1. A multi-view 3D reconstruction method based on GRU and 3DCNN, characterized in that, Includes the following steps: Obtain a collection of images of the target object from different perspectives, and determine the reference image and source image within it; Feature extraction is performed on the reference image and the source image to obtain the corresponding feature maps of the reference image and the source image at different resolutions; Multiple depth hypotheses are sampled within the depth range of the scene, and differentiable homography transformations are performed on each feature map under different depth hypotheses according to camera parameters to obtain the corresponding source feature volume and reference feature volume. The source feature volume and reference feature volume are divided into multiple groups according to channels, and the group similarity measurement module is used to calculate the group similarity. The matching cost volume is constructed and aggregated based on the group similarity. Perform cost body regularization on the matching cost body to obtain the probability body; The expected depth value is calculated along the depth direction for the probability volume to obtain the estimated depth map corresponding to the low-resolution reference image; The low-resolution depth map is upsampled and normalized. Then, depth assumptions are sampled within a narrowed depth sampling range around the normalized depth value to construct a dynamic matching cost body. The matching cost body is then input into a gated loop unit to obtain an updated depth map. Centered on the updated depth value, new depth hypotheses are sampled around it, and a new dynamic cost volume is constructed. Then, the repaired depth map is obtained through multiple iterations. Using the repaired depth maps one by one, a matching cost volume is constructed, then the cost volume is regularized and the depth expectation is calculated to obtain a depth map corresponding to the reference image with a higher resolution, until a depth map with the same size as the original resolution is obtained. The corresponding loss function is calculated based on the depth map to finally complete the multi-view 3D reconstruction. The process of calculating the corresponding loss function based on the depth map to ultimately complete the multi-view 3D reconstruction includes the following steps: Calculate the loss function for the regularization part, and calculate the L1 loss by comparing the depth map obtained by the 3D CNN regularization with the ground truth depth value at the corresponding resolution; The loss function weights of the gated recurrent unit are calculated. The difference between the depth ground truth of the current stage and the previous stage is used in each stage to obtain the details lost between each pair of stages. The loss function weights for each pixel are calculated based on the lost details to obtain the hierarchical weights. Calculate the partial loss function of the gated recurrent unit, and use the hierarchical weights to calculate the hierarchical L1 loss by comparing the depth map repaired by the gated recurrent unit with the ground truth depth value at the corresponding resolution.

2. The multi-view 3D reconstruction method based on GRU and 3DCNN according to claim 1, characterized in that, The step of extracting features from the reference image and the source image to obtain corresponding feature maps of the reference image and the source image at different resolutions specifically involves: inputting the reference image and the source image into the feature extraction module, wherein the size of the input reference image and source image is [missing information]. After 11 convolutional layers, the final output size is The convolutional layers in the third, sixth, and ninth layers are downsampled to half their original height and width.

3. The multi-view 3D reconstruction method based on GRU and 3DCNN according to claim 1, characterized in that, The process involves sampling multiple depth hypotheses within the scene's depth range, and performing differentiable homography transformations on each feature map under different depth hypotheses based on camera parameters to obtain corresponding source and reference feature volumes. The source and reference feature volumes are then divided into multiple groups by channel, and group similarity is calculated using a group similarity measurement module. Based on the group similarity, a matching cost volume is constructed and aggregated. This process includes the following steps: Based on the principle of planar scanning algorithm, a plane perpendicular to the principal optical axis of the reference camera is used as the depth assumption plane. All feature maps are distorted into different front planes of the reference camera, and all pixel coordinates on the source image are mapped to pixel coordinates on the reference image. The pixel coordinates of all source images are distorted to the coordinates of the reference image to obtain the distorted source feature volume. The dimension of the source feature body is Where C represents the channel dimension and D represents the depth dimension; the expression for the coordinate warping process is: ,in, , Camera intrinsics for the reference image and the source image, respectively. Here is the corresponding rotation matrix. For the corresponding translation vector, For the corresponding sampling depth value, The pixel coordinates corresponding to pixel p in the source image; The reference feature body is copied D times and expanded to obtain And divide all feature bodies into G groups evenly according to the channel dimension, with each group having C' channels; Calculate the group similarity between the distorted source feature and the expanded reference feature. The calculation formula is: ,in, It is an inner product operation; After the group similarity measurement module, N-1 matching cost bodies are obtained, which can be denoted as: Its dimensions are The variance is used to aggregate the feature volumes and calculate the final matching cost volume. The calculation formula is: .

4. The multi-view 3D reconstruction method based on GRU and 3DCNN according to claim 1, characterized in that, The step of performing cost body regularization on the matching cost body to obtain the probability body includes the following steps: Regularization is performed using 3D convolution to obtain the probability volume; specifically: A 3D-UNet network structure with four scales is used to encode and decode the original matching cost volume, and the number of channels is compressed to 1. Finally, the probability volume is normalized by applying a SoftMax operation in the depth direction. After regularization, the output probability volume has a dimension of [missing value]. ; The process of calculating the expected depth value along the depth direction of the probability volume to obtain the estimated depth map corresponding to the low-resolution reference image specifically involves: using a regression approach to predict the depth value, calculating the expected depth value of the sampled depth value along the depth direction, and obtaining the estimated depth value. The calculation formula is: , where P( d ) indicates depth d The estimated probability of each pixel at [location], , [ ] represents the range of depth values, and the output depth map size is [ ]. .

5. The multi-view 3D reconstruction method based on GRU and 3DCNN according to claim 1, characterized in that, The process of upsampling and normalizing the low-resolution depth map, sampling depth hypotheses within a reduced depth sampling range centered on the normalized depth value, and constructing a dynamic matching cost body; inputting the matching cost body into a gated recurrent unit to obtain an updated depth map includes the following steps: Using bilinear interpolation Upsampling of resolution depth map The resolution is then normalized to obtain the depth map D; Multiple depth hypotheses are sampled around this depth value to construct a dynamic matching cost body C; The cost volume is input into the gated loop unit to obtain the updated depth map. The gated loop unit and the updated depth value can be calculated using the following formulas: in, and These represent the hidden states at the current time step and the previous time step, respectively. This represents a candidate hidden state. and These represent updating the door and resetting the door, respectively. This represents the sigmoid function. Indicates element-wise product; Represents the depth update value; Based on the depth update values ​​obtained from multiple iterations, the repaired depth map is obtained, with a depth map size of [size missing]. Resolution.

6. The multi-view 3D reconstruction method based on GRU and 3DCNN according to claim 1, characterized in that, L1 loss between the depth map obtained by 3D CNN regularization and the ground truth depth value at the corresponding resolution The calculation formula is: in, The four phases represent phases 0 through 3. This represents the set of valid pixels whose true depth values ​​are known at the current stage. and These represent the regularized depth estimate and the true depth value of pixel p at the current stage, respectively. L1 loss of depth map repaired by gated recurrent unit and corresponding resolution depth ground truth The calculation formula is: in, The three phases are represented by phases 1 to 3. This represents the set of valid pixels whose true depth values ​​are known at the current stage. and These represent the gated recurrent unit depth estimate and the true depth value of pixel p at the current stage, respectively. This represents the pixel weight at the current stage; The formula for calculating the pixel weight is: .

7. A multi-view 3D reconstruction system based on GRU and 3DCNN, characterized in that, include: The first module is used to acquire a set of images of the target object from different perspectives and to determine the reference image and source image within it; The second module is used to extract features from the reference image and the source image to obtain feature maps of the reference image and the source image at different resolutions. The third module is used to sample multiple depth hypotheses within the depth range of the scene, and perform differentiable homography transformation on each feature map under different depth hypotheses according to the camera parameters to obtain the corresponding source feature volume and reference feature volume; the source feature volume and reference feature volume are divided into multiple groups according to channels, and the group similarity measurement module is used to calculate the group similarity, and the matching cost volume is constructed and aggregated based on the group similarity. The fourth module is used to perform cost body regularization on the matching cost body to obtain the probability body; The fifth module is used to calculate the expected depth value of the probability volume along the depth direction to obtain the estimated depth map corresponding to the low-resolution reference image. The sixth module is used to upsample and normalize the low-resolution depth map, sample depth assumptions within a narrowed depth sampling range around the normalized depth value, and construct a dynamic matching cost body; the matching cost body is input into the gated loop unit to obtain an updated depth map. The seventh module is used to sample new depth hypotheses around the updated depth value, construct a new dynamic cost volume, and then obtain the repaired depth map through multiple iterations. The eighth module is used to construct a matching cost volume by using the repaired depth map one by one, then perform cost volume regularization and calculate the depth expectation to obtain a depth map corresponding to the reference image with a higher resolution, until a depth map with the same resolution and size as the original is obtained. The ninth module is used to calculate the corresponding loss function based on the depth map, and finally complete the multi-view 3D reconstruction. The ninth module is specifically used for: Calculate the loss function for the regularization part, and calculate the L1 loss by comparing the depth map obtained by the 3D CNN regularization with the ground truth depth value at the corresponding resolution; The loss function weights of the gated recurrent unit are calculated. The difference between the depth ground truth of the current stage and the previous stage is used in each stage to obtain the details lost between each pair of stages. The loss function weights for each pixel are calculated based on the lost details to obtain the hierarchical weights. Calculate the partial loss function of the gated recurrent unit, and use the hierarchical weights to calculate the hierarchical L1 loss by comparing the depth map repaired by the gated recurrent unit with the ground truth depth value at the corresponding resolution.

8. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 6.

Images