Multi-view three-dimensional reconstruction method and system based on GRU and 3DCNN

Abstract

The invention discloses a multi-view three-dimensional reconstruction method and system based on GRU and 3DCNN. The method comprises the following steps: carrying out feature extraction on images of a target object at different view angles to obtain a feature map; performing microhomography on each feature map at different depth assumptions; dividing the source feature body and the reference feature body into a plurality of groups according to channels, calculating group similarity in groups, and aggregating and matching a cost body; carrying out cost body regularization operation and calculating a depth expected value; carrying out up-sampling on the low-resolution depth map and then carrying out normalization; obtaining an updated depth map according to the matched cost body, and constructing a dynamic cost body; constructing a matching cost body to obtain a depth map with the same resolution as the original resolution; and finally completing multi-view three-dimensional reconstruction according to the corresponding loss function. According to the embodiment of the invention, high-resolution depth estimation accuracy can be improved, more complete three-dimensional reconstruction can be realized, and the method can be widely applied to the technical field of computers.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a multi-view based on GRU and 3DCNN. Figure 3 Maintenance and reconstruction method and system. Background Art

[0002] 3D reconstruction is the process of generating a 3D model from a 2D image or other data source (such as sensors, lidar, etc.). It has a wide range of applications in computer vision, computer graphics, and robotics. The goal of 3D reconstruction is to reconstruct the geometry, size, and spatial relationships of an object or scene for analysis, visualization, or other subsequent processing. Some common 3D reconstruction techniques are: manual modeling, structured light, laser scanning, and multi-view Figure 3 Reconstruction, etc.

[0003] Multi-view Figure 3 3D reconstruction is a technique for restoring the structure of a 3D scene by analyzing 2D images from different viewpoints. In recent years, this field has received extensive attention in multiple disciplines such as computer vision, graphics, and artificial intelligence, with applications in virtual reality, augmented reality, autonomous driving, robot navigation, and cultural heritage protection.

[0004] Traditional multi-view reconstruction methods usually rely on technologies such as stereo vision and motion parallax. This process generally includes several key steps: image feature extraction, feature point matching to obtain sparse point cloud, and multi-view stereo matching to obtain dense point cloud. Figure 3 Dimensional reconstruction methods face many challenges when dealing with complex scenes. For example, in the case of changing illumination, changing perspective, or partial occlusion, it becomes difficult to match feature points, resulting in inaccurate reconstruction results. Traditional methods usually rely on simple geometric assumptions and have difficulty dealing with scenes with complex shapes and rich details. In addition, as the number of input images increases, the computational complexity of feature extraction and matching also increases significantly, resulting in decreased efficiency.

[0005] In recent years, the rapid development of deep learning technology has led to the Figure 3 dimensional reconstruction brings new opportunities. Deep learning can automatically extract high-level features by building deep neural networks, thus eliminating the need for manual feature design. Figure 3 Dimensional reconstruction shows significant advantages in automatic feature learning, end-to-end training, and robustness to noise and occlusion. Deep learning models can automatically learn effective features from input images to better handle complex scenes and object shapes.

[0006] In Multiview Figure 3 In the field of maintenance and reconstruction, the following problems exist:

[0007] 1. The error accumulation problem of the cascade architecture. The low-resolution depth map estimated in the previous stage is wrong, which will result in the depth hypothesis in the next stage not covering the true depth value, which will eventually lead to wrong estimation results;

[0008] 2. Detail loss problem in the low-resolution stage. In the low-resolution stage, the true depth value needs to be downsampled to the same low resolution for supervision. Some fine scene structure details will be lost in the downsampling process, resulting in the inability of the neural network to learn the correct details and produce incorrect depth estimation. Summary of the invention

[0009] The main purpose of the embodiment of the present invention is to propose a multi-view based on GRU and 3DCNN Figure 3 The three-dimensional reconstruction method and system can improve the accuracy of high-resolution depth estimation and achieve more complete three-dimensional reconstruction.

[0010] To achieve the above objectives, an embodiment of the present invention proposes a multi-view based on GRU and 3DCNN. Figure 3 The reconstruction method comprises the following steps:

[0011] Obtain a set of images of the target object at different viewing angles, and determine the reference image and the source image therein;

[0012] Performing feature extraction on the reference image and the source image to obtain corresponding feature maps of the reference image and the source image at different resolutions;

[0013] Sample multiple depth hypotheses within the depth range of the scene, and perform differentiable homography transformation on each feature map at different depth hypotheses according to the camera parameters to obtain the corresponding source feature volume and reference feature volume; divide the source feature volume and the reference feature volume into multiple groups according to the channel, calculate the group similarity by group similarity measurement module, and construct and aggregate the matching cost volume based on the group similarity;

[0014] Performing a cost body regularization operation on the matching cost body to obtain a probability body;

[0015] Calculating an expected depth value for the probability volume along a depth direction to obtain an estimated depth map corresponding to a low-resolution reference image;

[0016] The low-resolution depth map is upsampled and normalized, and a depth hypothesis is sampled within a reduced depth sampling range around the normalized depth value to construct a dynamic matching cost volume; the matching cost volume is input into a gated recurrent unit to obtain an updated depth map;

[0017] Taking the updated depth value as the center, new depth hypotheses are sampled around it and a new dynamic cost volume is constructed, and then the repaired depth map is obtained through multiple iterations;

[0018] Use the repaired depth maps one by one to construct the matching cost volume, then perform cost volume regularization and calculate the depth expectation to obtain the depth map corresponding to the reference image with a higher resolution, until a depth map with the same resolution as the original is obtained;

[0019] According to the depth map, the corresponding loss function is calculated and the multi-view is finally completed. Figure 3 Reconstruction.

[0020] In some embodiments, the feature extraction of the reference image and the source image is performed to obtain the corresponding feature maps of the reference image and the source image at different resolutions, specifically: the reference image and the source image are input into the feature extraction module, the size of the input reference image and the source image is 3×H×W, after 11 convolution layers, the final output size is The convolutional layers in the third, sixth and ninth layers are downsampled to half of their original height and width.

[0021] In some embodiments, the method of sampling multiple depth hypotheses within the depth range of the scene, and performing differentiable homography transformation on each feature map at different depth hypotheses according to camera parameters to obtain corresponding source feature volumes and reference feature volumes; dividing the source feature volumes and the reference feature volumes into multiple groups according to channels, calculating group similarities by grouping through a group similarity measurement module, and constructing and aggregating matching cost volumes based on group similarities, includes the following steps:

[0022] According to the principle of plane scanning algorithm, the plane perpendicular to the main optical axis of the reference camera is used as the depth hypothesis plane, all feature maps are warped into different front planes of the reference camera, and all pixel coordinates on the source image are mapped to the pixel coordinates on the reference image;

[0023] Warp the pixel coordinates on all source images to the coordinates on the reference image to obtain the distorted source feature volume The dimensions of the source feature volume are Where C represents the channel dimension and D represents the depth dimension; the expression of the coordinate distortion process is: Among them, K 0 , K i are the camera intrinsic parameters of the reference image and the source image, R i,0 is the corresponding rotation matrix, t i,0 is the corresponding translation vector, d j is the corresponding sampling depth value, p i,j is the pixel coordinate corresponding to pixel p in the source image;

[0024] Expand the reference feature volume by copying D copies to obtain And all feature bodies are evenly divided into G groups according to the channel dimension, and the number of channels in each group is C';

[0025] Calculate the group similarity between the distorted source feature body and the expanded reference feature body, group similarity The calculation formula is: Among them, <·,·> is the inner product operation;

[0026] After the group similarity measurement module, N-1 matching cost bodies are obtained, which can be recorded as Its dimension is The feature volume is aggregated and calculated using the variance to finally obtain the matching cost volume. The calculation formula for the matching cost volume C is:

[0027] In some embodiments, performing a cost body regularization operation on the matching cost body to obtain a probability body comprises the following steps:

[0028] Regularization is performed through 3D convolution to obtain a probability body; specifically:

[0029] The network structure of 3D-UNet with four scales is used to encode and decode the original matching cost volume, and the number of channels is compressed to 1. Finally, the probability value is normalized using the SoftMax operation in the depth direction. The output probability volume dimension after regularization is

[0030] The depth expectation value of the probability body is calculated along the depth direction to obtain an estimated depth map corresponding to the low-resolution reference image. Specifically, the depth value is predicted by regression, and the expectation of the sampled depth value is calculated along the depth direction. The depth estimation value D is calculated as follows: Where P(d) represents the estimated probability of each pixel at depth d, [d min ,d max ] is the range of depth values, and the output depth map size is

[0031] In some embodiments, the step of upsampling the low-resolution depth map and normalizing it, sampling depth hypotheses within a reduced depth sampling range around the normalized depth value as the center, and constructing a dynamic matching cost volume; and inputting the matching cost volume into a gated recurrent unit to obtain an updated depth map comprises the following steps:

[0032] By bilinear interpolation Resolution depth map upsampled to Resolution and normalization to obtain the depth map D;

[0033] Sample multiple depth hypotheses around the depth value to construct a dynamic matching cost volume C;

[0034] The cost volume is input into the gated recurrent unit to obtain the updated depth map. The gated recurrent unit and the depth update value can be calculated by the following formulas:

[0035] z t =σ(Conv 3×3 ([h t-1 ,x]))

[0036] r t =σ(Conv 3×3 ([h t-1 ,x]))

[0037]

[0038] Δd=Conv 3×3 (Relu(Conv 3×3 h t ))

[0039] Among them, h t and h t-1 Represent the hidden states of the current time step and the previous time step respectively, represents the candidate hidden state, z t and r t They represent the update gate and reset gate respectively, σ represents the sigmoid function, ⊙ represents the element-wise multiplication; Δd represents the depth update value;

[0040] According to the depth update value obtained by multiple iterations, the repaired depth map is obtained, and the depth map size is Resolution.

[0041] In some embodiments, the corresponding loss function is calculated according to the depth map to finally complete the multi-view Figure 3 The reconstruction process includes the following steps:

[0042] Calculate the regularization partial loss function, and calculate the L1 loss between the depth map obtained by 3D CNN regularization and the depth truth value of the corresponding resolution;

[0043] Calculate the weights of some loss functions of the gated recurrent unit. In each stage, the depth truth value of the current stage is used as the difference between the previous stage to obtain the details lost between each stage. The pixel-by-pixel loss function weights are calculated based on the lost details to obtain the hierarchical weights.

[0044] The partial loss function of the gated recurrent unit is calculated, and the hierarchical L1 loss is calculated by comparing the depth map inpainted by the gated recurrent unit with the true depth value of the corresponding resolution using the hierarchical weights.

[0045] In some embodiments, the L1 loss Loss between the depth map obtained by 3D CNN regularization and the depth truth value of the corresponding resolution 1 The calculation formula is:

[0046]

[0047] Among them, l represents 4 stages from stage 0 to stage 3. Represents the valid pixel set with known true depth values ​​at the current stage. and Respectively represent the regularized depth estimation value and the true depth value of the pixel point p at the current stage;

[0048] L1 loss between the depth map restored by the gated recurrent unit and the depth truth value of the corresponding resolution 2 The calculation formula is:

[0049]

[0050] Among them, l represents 3 stages from stage 1 to stage 3. Represents the valid pixel set with known true depth values ​​at the current stage. and They represent the estimated depth value and true depth value of the gated cycle unit of the pixel p at the current stage, respectively, and W l (p) represents the pixel weight at the current stage;

[0051] The calculation formula of the pixel weight is:

[0052] Another aspect of the embodiment of the present invention further provides a multi-view based on GRU and 3DCNN Figure 3 Maintenance and reconstruction system, including:

[0053] The first module is used to obtain a set of images of the target object at different viewing angles and determine a reference image and a source image therein;

[0054] The second module is used to extract features from the reference image and the source image to obtain corresponding feature maps of the reference image and the source image at different resolutions;

[0055] The third module is used to sample multiple depth hypotheses within the depth range of the scene, and perform differentiable homography transformation on each feature map at different depth hypotheses according to the camera parameters to obtain the corresponding source feature volume and reference feature volume; the source feature volume and the reference feature volume are divided into multiple groups according to the channels, and the group similarity is calculated by the group similarity measurement module, and the matching cost volume is constructed and aggregated based on the group similarity;

[0056] The fourth module is used to perform a cost body regularization operation on the matching cost body to obtain a probability body;

[0057] A fifth module is used to calculate the expected depth value of the probability volume along the depth direction to obtain an estimated depth map corresponding to the low-resolution reference image;

[0058] The sixth module is used to normalize the low-resolution depth map after upsampling, sample the depth hypothesis within the reduced depth sampling range around the normalized depth value, and construct a dynamic matching cost body; input the matching cost body into the gated recurrent unit to obtain an updated depth map;

[0059] The seventh module is used to sample new depth hypotheses around the updated depth value and construct a new dynamic cost volume, and then obtain the repaired depth map through multiple iterations;

[0060] The eighth module is used to use the repaired depth maps one by one to construct a matching cost volume, then perform a cost volume regularization operation and calculate the depth expectation to obtain a depth map corresponding to a higher resolution reference image, until a depth map with the same resolution as the original is obtained;

[0061] The ninth module is used to calculate the corresponding loss function according to the depth map, and finally complete the multi-view Figure 3 Reconstruction.

[0062] To achieve the above objective, another aspect of an embodiment of the present invention provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0063] To achieve the above objective, another aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method is implemented.

[0064] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0065] The embodiments of the present invention at least include the following beneficial effects: The present invention provides a multi-view based on GRU and 3DCNN Figure 3A method and system for dimensional reconstruction, the scheme obtains a set of images of a target object at different viewing angles, and determines a reference image and a source image therein; performs feature extraction on the reference image and the source image to obtain corresponding feature maps of the reference image and the source image at different resolutions; samples multiple depth hypotheses within the depth range of the scene, and performs differentiable homography transformation on each feature map at different depth hypotheses according to camera parameters to obtain corresponding source feature volumes and reference feature volumes; divides the source feature volumes and the reference feature volumes into multiple groups according to channels, calculates group similarities by grouping through a group similarity measurement module, constructs and aggregates matching cost volumes based on group similarities; performs cost volume regularization operation on the matching cost volume to obtain a probability volume; calculates depth expectation values ​​along the depth direction of the probability volume to obtain To the estimated depth map corresponding to the low-resolution reference image; the low-resolution depth map is upsampled and normalized, and the normalized depth value is centered, and the depth hypothesis is sampled within the reduced depth sampling range around it to construct a dynamic matching cost volume; the matching cost volume is input into the gated recurrent unit to obtain an updated depth map; the updated depth value is centered, and new depth hypotheses are sampled around it, and a new dynamic cost volume is constructed, and then the repaired depth map is obtained through multiple iterations; the repaired depth maps are used one by one to construct the matching cost volume, and then the cost volume regularization operation is performed and the depth expectation is calculated to obtain the depth map corresponding to the higher-resolution reference image, until a depth map with the same resolution as the original is obtained; the corresponding loss function is calculated according to the depth map, and finally the multi-view Figure 3 The embodiments of the present invention can improve the accuracy of high-resolution depth estimation and achieve more complete three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0067] Figure 2 is a flow chart of the overall steps provided by an embodiment of the present invention;

[0068] Figure 3 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0070] It is understood that the terms "first", "second", etc. used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0071] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0072] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0073] Before describing the embodiments of the present invention in detail, some related technologies involved in the embodiments of the present invention are first described as follows:

[0074] Deep learning: Deep learning is a machine learning method that mimics the way the human brain works, processing and understanding data through multi-layer neural networks. The "deep" in this term refers to the fact that the neural network has multiple levels or depths, usually including an input layer, multiple hidden layers (also called hidden layers), and an output layer. The depth and complexity of these neural networks enable them to extract features, patterns, and associations from large-scale and complex data, thereby achieving efficient data analysis and decision-making.

[0075] Multi-view Figure 3 Dimensional reconstruction: multi-view Figure 3Multi-View Stereo is a technique for generating a 3D model of an object or scene using 2D images from different viewpoints. This process usually involves extracting and matching feature points from multiple images, and then using geometric methods to restore 3D structure information. Figure 3 Dimensional reconstruction is widely used in computer vision, virtual reality, augmented reality, filmmaking, geographic mapping, and robotics.

[0076] Gated Recurrent Unit: Gated Recurrent Unit (GRU) is a recurrent neural network (RNN) architecture for processing sequence data. It can be used for time series prediction, natural language processing, and other tasks that require contextual information. GRU is designed to overcome the gradient vanishing problem of traditional RNN in long sequences.

[0077] Multi-view based on GRU and 3DCNN provided by the embodiment of the present invention Figure 3 The invention relates to a multi-dimensional reconstruction method and system, and relates to the field of computer technology, and can be widely used in the deployment of automated systems such as autonomous driving and unmanned aerial vehicles. Figure 3 The reconstruction method can be applied to a terminal or a server, or it can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, but is not limited thereto; the server side can be configured as an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be a multi-view based on GRU and 3DCNN. Figure 3 The application of dimensional reconstruction method, etc., but is not limited to the above forms.

[0078] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0079] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.

[0080] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0081] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0082] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 102 may also be a vehicle-mounted terminal of various device types described above, but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the present invention.

[0083] Based on the example Figure 1 The implementation environment shown in the figure, the embodiment of the present invention provides a multi-view based on GRU and 3DCNN Figure 3dimensional reconstruction method, the following is a multi-view based on GRU and 3DCNN Figure 3 The dimension reconstruction method is applied in the server 101 as an example for explanation. It can be understood that the method can also be applied in the terminal 102 .

[0084] Reference Figure 2 , Figure 2 A multi-view algorithm based on GRU and 3DCNN applied to a server provided in an embodiment of the present invention Figure 3 The flowchart of the method for reconstruction is shown in FIG. 1 , and the execution subject of the method can be any of the aforementioned computer devices (including a server or a terminal). Figure 2 , the method may include the following steps:

[0085] A multi-view based on GRU and 3DCNN Figure 3 The reconstruction method comprises the following steps:

[0086] Obtain a set of images of the target object at different viewing angles, and determine the reference image and the source image therein;

[0087] Performing feature extraction on the reference image and the source image to obtain corresponding feature maps of the reference image and the source image at different resolutions;

[0088] Sample multiple depth hypotheses within the depth range of the scene, and perform differentiable homography transformation on each feature map at different depth hypotheses according to the camera parameters to obtain the corresponding source feature volume and reference feature volume; divide the source feature volume and the reference feature volume into multiple groups according to the channel, calculate the group similarity by group similarity measurement module, and construct and aggregate the matching cost volume based on the group similarity;

[0089] Performing a cost body regularization operation on the matching cost body to obtain a probability body;

[0090] Calculating an expected depth value for the probability volume along a depth direction to obtain an estimated depth map corresponding to a low-resolution reference image;

[0091] The low-resolution depth map is upsampled and normalized, and a depth hypothesis is sampled within a reduced depth sampling range around the normalized depth value to construct a dynamic matching cost volume; the matching cost volume is input into a gated recurrent unit to obtain an updated depth map;

[0092] Taking the updated depth value as the center, new depth hypotheses are sampled around it and a new dynamic cost volume is constructed, and then the repaired depth map is obtained through multiple iterations;

[0093] Use the repaired depth maps one by one to construct the matching cost volume, then perform cost volume regularization and calculate the depth expectation to obtain the depth map corresponding to the reference image with a higher resolution, until a depth map with the same resolution as the original is obtained;

[0094] According to the depth map, the corresponding loss function is calculated and the multi-view is finally completed. Figure 3 Reconstruction.

[0095] In some embodiments, the feature extraction of the reference image and the source image is performed to obtain the corresponding feature maps of the reference image and the source image at different resolutions, specifically: the reference image and the source image are input into the feature extraction module, the size of the input reference image and the source image is 3×H×W, after 11 convolution layers, the final output size is The convolutional layers in the third, sixth and ninth layers are downsampled to half of their original height and width.

[0096] In some embodiments, the method of sampling multiple depth hypotheses within the depth range of the scene, and performing differentiable homography transformation on each feature map at different depth hypotheses according to camera parameters to obtain corresponding source feature volumes and reference feature volumes; dividing the source feature volumes and the reference feature volumes into multiple groups according to channels, calculating group similarities by grouping through a group similarity measurement module, and constructing and aggregating matching cost volumes based on group similarities, includes the following steps:

[0097] According to the principle of plane scanning algorithm, the plane perpendicular to the main optical axis of the reference camera is used as the depth hypothesis plane, all feature maps are warped into different front planes of the reference camera, and all pixel coordinates on the source image are mapped to the pixel coordinates on the reference image;

[0098] Warp the pixel coordinates on all source images to the coordinates on the reference image to obtain the distorted source feature volume The dimensions of the source feature volume are Where C represents the channel dimension and D represents the depth dimension; the expression of the coordinate distortion process is: Among them, K 0 , K i are the camera intrinsic parameters of the reference image and the source image, R i,0 is the corresponding rotation matrix, t i,0 is the corresponding translation vector, d j is the corresponding sampling depth value, p i,j is the pixel coordinate corresponding to pixel p in the source image;

[0099] Expand the reference feature volume by copying D copies to obtain And all feature bodies are evenly divided into G groups according to the channel dimension, and the number of channels in each group is C';

[0100] Calculate the group similarity between the distorted source feature body and the expanded reference feature body, group similarity The calculation formula is: Among them, <·,·> is the inner product operation;

[0101] After the group similarity measurement module, N-1 matching cost bodies are obtained, which can be recorded as Its dimension is The feature volume is aggregated and calculated using the variance to finally obtain the matching cost volume. The calculation formula for the matching cost volume C is:

[0102] In some embodiments, performing a cost body regularization operation on the matching cost body to obtain a probability body comprises the following steps:

[0103] Regularization is performed through 3D convolution to obtain a probability body; specifically:

[0104] The network structure of 3D-UNet with four scales is used to encode and decode the original matching cost volume, and the number of channels is compressed to 1. Finally, the probability value is normalized using the SoftMax operation in the depth direction. The output probability volume dimension after regularization is

[0105] The depth expectation value of the probability body is calculated along the depth direction to obtain an estimated depth map corresponding to the low-resolution reference image. Specifically, the depth value is predicted by regression, and the expectation of the sampled depth value is calculated along the depth direction. The depth estimation value D is calculated as follows: Where P(d) represents the estimated probability of each pixel at depth d, [d min ,d max ] is the range of depth values, and the output depth map size is

[0106] In some embodiments, the step of upsampling the low-resolution depth map and normalizing it, sampling depth hypotheses within a reduced depth sampling range around the normalized depth value as the center, and constructing a dynamic matching cost volume; and inputting the matching cost volume into a gated recurrent unit to obtain an updated depth map comprises the following steps:

[0107] By bilinear interpolation Resolution depth map upsampled to Resolution and normalization to obtain the depth map D;

[0108] Sample multiple depth hypotheses around the depth value to construct a dynamic matching cost volume C;

[0109] The cost volume is input into the gated recurrent unit to obtain the updated depth map. The gated recurrent unit and the depth update value can be calculated by the following formulas:

[0110] z t =σ(Conv 3×3 ([h t-1 ,x]))

[0111] r t =σ(Conv 3×3 ([h t-1 ,x]))

[0112]

[0113] Δd=Conv 3×3 (Relu(Conv 3×3 h t ))

[0114] Among them, h t and h t-1 Represent the hidden states of the current time step and the previous time step respectively, represents the candidate hidden state, z t and r t They represent the update gate and reset gate respectively, σ represents the sigmoid function, ⊙ represents the element-wise multiplication; Δd represents the depth update value;

[0115] According to the depth update value obtained by multiple iterations, the repaired depth map is obtained, and the depth map size is Resolution.

[0116] In some embodiments, the corresponding loss function is calculated according to the depth map to finally complete the multi-view Figure 3 The reconstruction process includes the following steps:

[0117] Calculate the regularization partial loss function, and calculate the L1 loss between the depth map obtained by 3D CNN regularization and the depth truth value of the corresponding resolution;

[0118] Calculate the weights of some loss functions of the gated recurrent unit. In each stage, the depth truth value of the current stage is used as the difference between the previous stage to obtain the details lost between each stage. The pixel-by-pixel loss function weights are calculated based on the lost details to obtain the hierarchical weights.

[0119] The partial loss function of the gated recurrent unit is calculated, and the hierarchical L1 loss is calculated by comparing the depth map inpainted by the gated recurrent unit with the true depth value of the corresponding resolution using the hierarchical weights.

[0120] In some embodiments, the L1 loss Loss between the depth map obtained by 3D CNN regularization and the depth truth value of the corresponding resolution1 The calculation formula is:

[0121]

[0122] Among them, l represents 4 stages from stage 0 to stage 3. Represents the valid pixel set with known true depth values ​​at the current stage. and Respectively represent the regularized depth estimation value and the true depth value of the pixel point p at the current stage;

[0123] L1 loss between the depth map restored by the gated recurrent unit and the depth truth value of the corresponding resolution 2 The calculation formula is:

[0124]

[0125] Among them, l represents 3 stages from stage 1 to stage 3. Represents the valid pixel set with known true depth values ​​at the current stage. and They represent the estimated depth value and true depth value of the gated cycle unit of the pixel p at the current stage, respectively, and W l (p) represents the pixel weight at the current stage;

[0126] The calculation formula of the pixel weight is:

[0127] The following describes the specific implementation process of the embodiment of the present invention in detail by taking a specific application scenario as an example:

[0128] The overall steps of the embodiment of the present invention include:

[0129] Step 1: Obtain N images of the same object taken from different perspectives, use one of them as a reference image, and the remaining N-1 images as source images;

[0130] Step 2: Input the reference image and the source image into the feature extraction module to obtain the corresponding feature maps of the reference image and the source image at different resolutions;

[0131] Step 3: Sample multiple depth hypotheses within the depth range of the scene, and perform differentiable homography transformation on the feature map at different depth hypotheses according to the camera parameters to obtain a feature volume;

[0132] Step 4: Divide the source feature body and the reference feature body into g groups according to the channels, calculate the group similarity by grouping through the group similarity measurement module, and construct and aggregate the matching cost body based on the group similarity;

[0133] Step 5: Perform a cost body regularization operation on the matching cost body to obtain a probability body;

[0134] Step 6: Calculate the expected depth value of the probability volume along the depth direction to obtain an estimated depth map corresponding to the low-resolution reference image;

[0135] Step 7: normalize the low-resolution depth map after upsampling, sample the depth hypothesis within the reduced depth sampling range around the normalized depth value, and construct a dynamic matching cost volume;

[0136] Step 8: Input the cost volume into the gated recurrent unit to obtain an updated depth map;

[0137] Step 9: Taking the updated depth value as the center, sampling new depth hypotheses around it and constructing a new dynamic cost volume;

[0138] Step 10: Repeat steps 8 and 9 for multiple iterations to obtain the repaired depth map.

[0139] Step 11: Use the repaired depth map to construct a matching cost volume, perform a cost volume regularization operation and calculate the depth expectation to obtain a depth map corresponding to a reference image with a higher resolution;

[0140] Step 12: Repeat steps 7 to 11 until a depth map with the same resolution as the original is obtained.

[0141] Step 13: Calculate the regularization partial loss function, and calculate the L1 loss between the depth map obtained by 3D CNN regularization and the depth truth value of the corresponding resolution;

[0142] Step 14: Calculate the weight of the partial loss function of the gated recurrent unit. In each stage, the depth truth value of the current stage and the previous stage is used as the difference to obtain the details lost between each stage. The pixel-by-pixel loss function weight is calculated based on the lost details. The more details are lost, the higher the weight is, and the hierarchical weight is obtained.

[0143] Step 15: Calculate the partial loss function of the gated recurrent unit, and use the hierarchical weights to calculate the hierarchical L1 loss between the depth map restored by the gated recurrent unit and the true depth value of the corresponding resolution.

[0144] Specifically, the specific implementation process of each of the above steps is described in detail as follows:

[0145] Step 1: Get N images of the same object taken from different perspectives, and use one of them as the reference image I 0 , and the remaining N-1 images are used as source images

[0146] Step 2: Input the reference image and source image into the feature extraction module. The size of the input reference image and source image is 3×H×W. After 11 convolutional layers, the final output size is The convolution layers in the third, sixth and ninth layers are downsampled to half of their original height and width. The basic structure of the entire feature extraction network can be seen in Table 1. The final output reference feature map can be recorded as F 0 , the source feature map can be recorded as

[0147] Table 1 Basic structure of feature extraction network

[0148]

[0149]

[0150] Step 3: For In the resolution stage, multiple depth hypotheses are sampled within the depth range of the scene, and the above feature maps are transformed into different depth hypotheses according to the camera parameters to obtain feature volumes; the source feature volumes and reference feature volumes are divided into g groups by channel, and the group similarity is calculated through the group similarity measurement module, and the matching cost volume is constructed and aggregated based on the group similarity. The specific process is as follows:

[0151] According to the principle of the plane scanning algorithm, the plane perpendicular to the main optical axis of the reference camera is used as the depth hypothesis plane. All feature maps are distorted to different front planes of the reference camera, and all pixel coordinates on the source image are mapped to the pixel coordinates on the reference image. The minimum depth is recorded as d min , the maximum depth is denoted as d max , sampling D depth hypotheses from the minimum depth to the maximum depth, the distortion process is called a differentiable homography transformation, and the corresponding calculation is given by formula (1):

[0152]

[0153] In formula (1), K 0 , K i are the camera intrinsic parameters of the reference image and the source image, R i,0 is the corresponding rotation matrix, t i,0 is the corresponding translation vector, d j is the corresponding sampling depth value, p i,j is the pixel coordinate corresponding to pixel p in the source image.

[0154] By using formula (1), the pixel coordinates of all source images are distorted to the coordinates of the reference image, and the distorted source feature volume can be recorded as Its dimension is Where C represents the channel dimension and D represents the depth dimension. To keep the dimensions consistent, all feature bodies are evenly divided into G groups according to the channel dimension, and the number of channels in each group is C'. and The group similarity between them is calculated by formula (2):

[0155]

[0156] The <·,·> in formula (2) is an inner product operation.

[0157] After the group similarity measurement module, N-1 matching cost bodies are obtained, which can be recorded as Its dimension is In order to adapt to any number of source image inputs, the feature volume is aggregated and calculated using the variance, and the final matching cost volume can be calculated by formula (3):

[0158]

[0159] In this embodiment, G=8 is set. By calculating the correlation by grouping, the correlation channels of the features can be reduced to the number of groups, reducing memory consumption. Grouping can obtain rich correlation information, making the matching cost body more accurate.

[0160] Step 4: Regularize the matching cost volume through 3DCNN to obtain the probability volume P. The specific process is as follows:

[0161] Due to non-Lambertian surfaces and occlusions, the matching cost volume obtained above contains a lot of noise, so it is regularized through 3D convolution to obtain a probability volume. The probability volume can not only be used for pixel-by-pixel depth prediction, but also can be used to measure the confidence of the estimate. This embodiment uses a 3D-UNet network structure containing four scales to encode and decode the original matching cost volume, and finally compresses the number of channels to 1, and finally uses the SoftMax operation to normalize the probability value of the probability volume in the depth direction. The dimension of the probability volume output after regularization is

[0162] Step 5: Calculate the expected depth value of the above probability volume along the depth direction to obtain the estimated depth map corresponding to the low-resolution reference image; the specific process is:

[0163] In order to avoid gaps and unevenness along the depth direction, a regression method is used to predict the depth value. Its essence is to calculate the expectation of the sampled depth value along the depth direction. The depth estimation value can be calculated by formula (4):

[0164]

[0165] In formula (4), P(d) represents the estimated probability of each pixel at depth d, [d min ,d max ] is the range of depth values, and the output depth map size is

[0166] Step 6: normalize the above depth map after upsampling, sample the depth hypothesis in the reduced depth sampling range around the normalized depth value, construct a dynamic matching cost body, and input the cost body into the gated recurrent unit to obtain an updated depth map; the specific process is:

[0167] By bilinear interpolation Resolution depth map upsampled to The resolution is normalized to obtain a depth map D. Multiple depth hypotheses are sampled around the depth value to construct a dynamic matching cost volume C. The cost volume is input into the gated recurrent unit to obtain an updated depth map. The gated recurrent unit and the depth update value can be calculated by equations (5) to (9):

[0168] z t =σ(Conv 3×3 ([h t-1 ,x])) Formula (5)

[0169] r t =σ(Conv 3×3 ([h t-1 ,x])) Formula (6)

[0170]

[0171] Δd=Conv 3×3 (Relu(Conv 3×3 h t )) Formula (9)

[0172] In formula (5) to formula (8), h t and h t-1 Represent the hidden states of the current time step and the previous time step respectively, represents the candidate hidden state, z t and r t Denote the update gate and reset gate respectively, σ denotes the sigmoid function, and ⊙ denotes the element-wise multiplication. The hidden state output by the gated recurrent unit in formula (9) can be obtained by passing through two convolutional layers to obtain the depth update value Δd. The above process is iterated multiple times to obtain the repaired depth map. The depth map size is Resolution, consistent with the current stage.

[0173] Step 7: Use the repaired depth map to construct a matching cost volume, perform a cost volume regularization operation, and calculate the depth expectation to obtain a depth map corresponding to a reference image with a higher resolution.

[0174] Step 8: Repeat steps 6 and 7 to get The depth map of the original resolution and the original resolution is finally used to estimate the depth map of the reference image.

[0175] Step 9: Calculate the loss function. The loss function consists of two parts: the multiple repaired depth maps generated by multiple iterations of the gated recurrent unit and the depth map generated by the regularization part. The loss function needs to be calculated for the depth maps of the intermediate scales at each stage. The loss function can be calculated by formula (10):

[0176] Loss=Loss 1 +Loss 2 Formula (10)

[0177] Among them, Loss 1 L represents the difference between the depth map generated by the regularization part and the true depth value 1 loss:

[0178]

[0179] In formula (11), l represents four stages from stage 0 to stage 3. Represents the valid pixel set with known true depth values ​​at the current stage. and They represent the regularized depth estimation value and the true depth value of the pixel point p at the current stage respectively.

[0180] Loss 2 Represents the hierarchical relationship between the depth map generated by the gated recurrent unit and the depth truth value L 1 loss:

[0181]

[0182] In formula (12), l represents three stages from stage 1 to stage 3. Represents the valid pixel set with known true depth values ​​at the current stage. and They represent the estimated depth value and true depth value of the gated cycle unit of the pixel p at the current stage, respectively, and W l (p) represents the pixel weight at the current stage. l (p) can be calculated by formula (13):

[0183]

[0184] In formula (13), the depth truth value D gt Downsample to Resolution, and then upsampled back to the original resolution, can be recorded as The same can be said and When calculating the loss function at the intermediate size stage, the downsampled depth truth value is used. The depth truth value loses some details in the downsampling process, and the degree of lost details varies depending on the downsampling ratio. Therefore, the gated recurrent unit used in the next stage of the embodiment of the present invention can restore the lost details from a higher resolution. Using the depth difference percentage as the loss function weight of the pixel can highlight the proportion of the detail part, so that the network pays more attention to learning to restore the detail structure.

[0185] Step 10: Use the final depth value and the camera parameters corresponding to the image to project the image back into the three-dimensional space to obtain a partial point cloud. All images are projected back into the three-dimensional space in turn to fuse them into a unified dense three-dimensional point cloud model.

[0186] In summary, the present invention has the following characteristics:

[0187] 1. Multi-viewing realized by the present invention Figure 3 In the 3D reconstruction technology, the gated recurrent unit is used to restore the detail information lost in low-resolution reconstruction, so that the depth estimation of each level can be carried out in a more accurate depth range, making the final reconstruction result more accurate and complete;

[0188] 2. Multi-viewing realized by the present invention Figure 3 In the multi-dimensional reconstruction technology, by constructing a hierarchical loss function, the detail information that needs to be restored at each level can be accurately perceived, and the gated recurrent unit's ability to recover and fit the lost details can be enhanced.

[0189] 3. Cascaded multi-view stereo geometry framework embedded with gated recurrent units: The present invention uses gated recurrent units to recover the detail information lost by low-resolution depth estimation, adjusts the depth value of each level to the correct depth range, and then uses the cascaded architecture's three-dimensional convolution for accurate depth estimation;

[0190] 4. Normalized gated recurrent unit: The normalized depth sampling used in the present invention can be generalized to scenes with arbitrary depth intervals, and accurate results can be obtained even if the depth interval of the application scene is different from that used in training;

[0191] 5. Hierarchical loss function: The hierarchical loss function adopted in the present invention aims to strengthen the loss function of the lost detail part and improve the ability of the neural network to restore the lost detail structure.

[0192] Compared with the prior art, the present invention has the following advantages:

[0193] 1. High accuracy: avoids the problem of error accumulation caused by misestimation;

[0194] 2. High integrity: The detailed structure lost due to downsampling of the depth truth is restored;

[0195] 3. High versatility: The framework structure of the present invention has strong versatility, and the feature extraction and regularization parts can be replaced by any feature extractor and regularization algorithm;

[0196] 4. High generalization: The normalized depth sampling method adopted by the present invention can be generalized to scenes with arbitrary depth intervals. Even if the depth interval of the application scene is different from that used in training, accurate results can be obtained.

[0197] Another aspect of the embodiment of the present invention further provides a multi-view based on GRU and 3DCNN Figure 3 Maintenance and reconstruction system, including:

[0198] The first module is used to obtain a set of images of the target object at different viewing angles and determine a reference image and a source image therein;

[0199] The second module is used to extract features from the reference image and the source image to obtain corresponding feature maps of the reference image and the source image at different resolutions;

[0200] The third module is used to sample multiple depth hypotheses within the depth range of the scene, and perform differentiable homography transformation on each feature map at different depth hypotheses according to the camera parameters to obtain the corresponding source feature volume and reference feature volume; the source feature volume and the reference feature volume are divided into multiple groups according to the channels, and the group similarity is calculated by the group similarity measurement module, and the matching cost volume is constructed and aggregated based on the group similarity;

[0201] The fourth module is used to perform a cost body regularization operation on the matching cost body to obtain a probability body;

[0202] A fifth module is used to calculate the expected depth value of the probability volume along the depth direction to obtain an estimated depth map corresponding to the low-resolution reference image;

[0203] The sixth module is used to normalize the low-resolution depth map after upsampling, sample the depth hypothesis within the reduced depth sampling range around the normalized depth value, and construct a dynamic matching cost body; input the matching cost body into the gated recurrent unit to obtain an updated depth map;

[0204] The seventh module is used to sample new depth hypotheses around the updated depth value and construct a new dynamic cost volume, and then obtain the repaired depth map through multiple iterations;

[0205] The eighth module is used to use the repaired depth maps one by one to construct a matching cost volume, then perform a cost volume regularization operation and calculate the depth expectation to obtain a depth map corresponding to a higher resolution reference image, until a depth map with the same resolution as the original is obtained;

[0206] The ninth module is used to calculate the corresponding loss function according to the depth map, and finally complete the multi-view Figure 3 Reconstruction.

[0207] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0208] The embodiment of the present invention further provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the multi-view based on GRU and 3DCNN when executing the computer program Figure 3 The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0209] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0210] See also Figure 3 , Figure 3 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0211] The processor 301 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention;

[0212] The memory 302 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 302 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 302 and is called by the processor 301 to execute the multi-view based on GRU and 3DCNN of the embodiment of the present invention. Figure 3 Dimensional reconstruction method;

[0213] Input / output interface 303, used to implement information input and output;

[0214] The communication interface 304 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0215] A bus 305 that transmits information between the various components of the device (e.g., the processor 301, the memory 302, the input / output interface 303, and the communication interface 304);

[0216] The processor 301 , the memory 302 , the input / output interface 303 and the communication interface 304 are connected to each other in communication within the device via the bus 305 .

[0217] The embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the multi-view based on GRU and 3DCNN. Figure 3 Dimensional reconstruction method.

[0218] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0219] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0220] It should be noted that in various specific embodiments of the present invention, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present invention needs to obtain the user's sensitive personal information, it will obtain the user's separate permission or consent through a pop-up window or jump to a confirmation page, and after clearly obtaining the user's separate permission or consent, it will obtain the necessary user-related data for the normal operation of the embodiment of the present invention.

[0221] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0222] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0223] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0224] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0225] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0226] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0227] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0228] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0229] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0230] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0231] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. A multi-view 3D reconstruction method based on GRU and 3DCNN, characterized in that: The following steps are involved: Obtain a set of images of the target object at different viewing angles, and determine the reference image and the source image therein; Performing feature extraction on the reference image and the source image to obtain corresponding feature maps of the reference image and the source image at different resolutions; Sample multiple depth hypotheses within the depth range of the scene, and perform differentiable homography transformation on each feature map at different depth hypotheses according to the camera parameters to obtain the corresponding source feature volume and reference feature volume; divide the source feature volume and the reference feature volume into multiple groups according to the channel, calculate the group similarity by group similarity measurement module, and construct and aggregate the matching cost volume based on the group similarity; Performing a cost body regularization operation on the matching cost body to obtain a probability body; Calculating an expected depth value for the probability volume along a depth direction to obtain an estimated depth map corresponding to a low-resolution reference image; The low-resolution depth map is upsampled and normalized, and a depth hypothesis is sampled within a reduced depth sampling range around the normalized depth value to construct a dynamic matching cost volume; the matching cost volume is input into a gated recurrent unit to obtain an updated depth map; Taking the updated depth value as the center, new depth hypotheses are sampled around it and a new dynamic cost volume is constructed, and then the repaired depth map is obtained through multiple iterations; Use the repaired depth maps one by one to construct the matching cost volume, then perform cost volume regularization and calculate the depth expectation to obtain the depth map corresponding to the reference image with a higher resolution, until a depth map with the same resolution as the original is obtained; The corresponding loss function is calculated according to the depth map, and finally the multi-view 3D reconstruction is completed.

2. The multi-view 3D reconstruction method based on GRU and 3DCNN according to claim 1, characterized in that: The feature extraction of the reference image and the source image is performed to obtain the corresponding feature maps of the reference image and the source image at different resolutions, specifically: the reference image and the source image are input into the feature extraction module, the size of the input reference image and the source image is 3×H×W, after 11 convolution layers, the final output size is The convolutional layers in the third, sixth and ninth layers are downsampled to half of their original height and width.

3. The multi-view 3D reconstruction method based on GRU and 3DCNN according to claim 1, characterized in that: The method comprises the following steps: sampling multiple depth hypotheses within the depth range of the scene, and performing differentiable homography transformation on each feature map at different depth hypotheses according to camera parameters to obtain corresponding source feature volumes and reference feature volumes; dividing the source feature volumes and the reference feature volumes into multiple groups according to channels, calculating group similarities by grouping through a group similarity measurement module, and constructing and aggregating matching cost volumes based on group similarities. According to the principle of plane scanning algorithm, the plane perpendicular to the main optical axis of the reference camera is used as the depth hypothesis plane, all feature maps are warped into different front planes of the reference camera, and all pixel coordinates on the source image are mapped to the pixel coordinates on the reference image; Warp the pixel coordinates on all source images to the coordinates on the reference image to obtain the distorted source feature volume The dimensions of the source feature volume are Where C represents the channel dimension and D represents the depth dimension; the expression of the coordinate distortion process is: Among them, K0, K i are the camera intrinsic parameters of the reference image and the source image, R i,0 is the corresponding rotation matrix, t i,0 is the corresponding translation vector, d j is the corresponding sampling depth value, p i,j is the pixel coordinate corresponding to pixel p in the source image; Expand the reference feature volume by copying D copies to obtain And all feature bodies are evenly divided into G groups according to the channel dimension, and the number of channels in each group is C'; Calculate the group similarity between the distorted source feature body and the expanded reference feature body, group similarity The calculation formula is: g=0,1,…,G-1,i=1,2,…,N-1,where <·,·> is the inner product operation; After the group similarity measurement module, N-1 matching cost bodies are obtained, which can be recorded as Its dimension is The feature volume is aggregated and calculated using the variance to finally obtain the matching cost volume. The calculation formula for the matching cost volume C is:

4. The multi-view 3D reconstruction method based on GRU and 3DCNN according to claim 1, characterized in that: The step of performing a cost body regularization operation on the matching cost body to obtain a probability body comprises the following steps: Regularization is performed through 3D convolution to obtain a probability body; specifically: The network structure of 3D-UNet with four scales is used to encode and decode the original matching cost volume, and the number of channels is compressed to 1. Finally, the probability value is normalized using the SoftMax operation in the depth direction. The output probability volume dimension after regularization is The depth expectation value of the probability body is calculated along the depth direction to obtain an estimated depth map corresponding to the low-resolution reference image. Specifically, the depth value is predicted by regression, and the expectation of the sampled depth value is calculated along the depth direction. The depth estimation value D is calculated as follows: Where P(d) represents the estimated probability of each pixel at depth d, [d min ,d max ] is the range of depth values, and the output depth map size is 5. The multi-view 3D reconstruction method based on GRU and 3DCNN according to claim 1, characterized in that: The low-resolution depth map is upsampled and normalized, and the normalized depth value is used as the center, and the depth hypothesis is sampled within the reduced depth sampling range around it to construct a dynamic matching cost body; the matching cost body is input into the gated recurrent unit to obtain an updated depth map, including the following steps: By bilinear interpolation Resolution depth map upsampled to Resolution and normalization to obtain the depth map D; Sample multiple depth hypotheses around the depth value to construct a dynamic matching cost volume C; The cost volume is input into the gated recurrent unit to obtain the updated depth map. The gated recurrent unit and the depth update value can be calculated by the following formulas: z t =σ(Conv 3×3 ([h t-1 ,x])) r t =σ(Conv 3×3 ([h t-1 ,x])) Δd=Conv 3×3 (Release(Conv 3×3 h t )) Among them, h t and h t-1 Represent the hidden states of the current time step and the previous time step respectively, represents the candidate hidden state, z t and r t They represent the update gate and reset gate respectively, σ represents the sigmoid function, ⊙ represents the element-wise multiplication; Δd represents the depth update value; According to the depth update value obtained by multiple iterations, the repaired depth map is obtained, and the depth map size is Resolution.

6. The multi-view 3D reconstruction method based on GRU and 3DCNN according to claim 1, characterized in that: The step of calculating a corresponding loss function according to the depth map and finally completing multi-view 3D reconstruction comprises the following steps: Calculate the regularization partial loss function, and calculate the L1 loss between the depth map obtained by 3D CNN regularization and the depth truth value of the corresponding resolution; Calculate the weights of some loss functions of the gated recurrent unit. In each stage, the depth truth value of the current stage is used as the difference between the previous stage to obtain the details lost between each stage. The pixel-by-pixel loss function weights are calculated based on the lost details to obtain the hierarchical weights. The partial loss function of the gated recurrent unit is calculated, and the hierarchical L1 loss is calculated by comparing the depth map inpainted by the gated recurrent unit with the true depth value of the corresponding resolution using the hierarchical weights.

7. The multi-view 3D reconstruction method based on GRU and 3DCNN according to claim 6, characterized in that: The calculation formula of L1 loss Loss1 between the depth map obtained by 3D CNN regularization and the depth truth value of the corresponding resolution is: Among them, l represents 4 stages from stage 0 to stage 3. Represents the valid pixel set with known true depth values ​​at the current stage. and Respectively represent the regularized depth estimation value and the true depth value of the pixel point p at the current stage; The calculation formula of L1 loss Loss2 between the depth map restored by the gated cycle unit and the depth truth value of the corresponding resolution is: Among them, l represents 3 stages from stage 1 to stage 3. Represents the valid pixel set with known true depth values ​​at the current stage. and They represent the estimated depth value and true depth value of the gated cycle unit of the pixel p at the current stage, respectively, and W l (p) represents the pixel weight at the current stage; The calculation formula of the pixel weight is:

8. A multi-view 3D reconstruction system based on GRU and 3DCNN, characterized in that: include: The first module is used to obtain a set of images of the target object at different viewing angles and determine a reference image and a source image therein; The second module is used to extract features from the reference image and the source image to obtain corresponding feature maps of the reference image and the source image at different resolutions; The third module is used to sample multiple depth hypotheses within the depth range of the scene, and perform differentiable homography transformation on each feature map at different depth hypotheses according to the camera parameters to obtain the corresponding source feature volume and reference feature volume; the source feature volume and the reference feature volume are divided into multiple groups according to the channels, and the group similarity is calculated by the group similarity measurement module, and the matching cost volume is constructed and aggregated based on the group similarity; The fourth module is used to perform a cost body regularization operation on the matching cost body to obtain a probability body; A fifth module is used to calculate the expected depth value of the probability volume along the depth direction to obtain an estimated depth map corresponding to the low-resolution reference image; The sixth module is used to normalize the low-resolution depth map after upsampling, sample the depth hypothesis within the reduced depth sampling range around the normalized depth value, and construct a dynamic matching cost body; input the matching cost body into the gated recurrent unit to obtain an updated depth map; The seventh module is used to sample new depth hypotheses around the updated depth value and construct a new dynamic cost volume, and then obtain the repaired depth map through multiple iterations; The eighth module is used to use the repaired depth maps one by one to construct a matching cost volume, then perform a cost volume regularization operation and calculate the depth expectation to obtain a depth map corresponding to a higher resolution reference image, until a depth map with the same resolution as the original is obtained; The ninth module is used to calculate the corresponding loss function according to the depth map, and finally complete the multi-view 3D reconstruction.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 7.

Images