Learning-based lightweight adaptive multi-stage dense reconstruction method and apparatus

Abstract

The application discloses a learning-based lightweight adaptive multi-stage dense reconstruction method and device, which comprises the following steps: obtaining multiple original images; generating a feature map of multiple scales of each original image through a preset feature pyramid based on a depth separable convolution; determining a feature body of multiple scales of multiple source images corresponding to each reference image; using an intra-view and inter-view double-flow adaptive aggregation mode to aggregate the cost volume with weights; at the top scale, generating a cost volume of the reference image, and using a hierarchical recursive convolution network to generate a depth map; at other scales, generating a residual cost volume of the reference image by referring to the depth map of the last scale, using the hierarchical recursive convolution network to generate a residual depth map, and adding the residual depth map and the up-sampling of the depth map of the last scale to obtain the depth map at the current scale; and fusing the depth maps generated at the multiple bottom scales to obtain a dense point cloud. The method can solve the problem of weak texture extraction.

Description

Technical Field

[0001] This application relates to the field of remote sensing mapping and geographic information technology, and in particular to a learning-based lightweight adaptive multi-stage dense reconstruction method and apparatus. Background Technology

[0002] Dense reconstruction, also known as multi-view stereo geometry, aims to calculate the 3D points corresponding to each pixel in an image, given the camera pose, to obtain a dense 3D point cloud of the scene's object surface. Dense reconstruction algorithms seek to extract a 3D dense point cloud model of a real scene from multiple images. Traditional methods use manually calculated similarity metrics and photometric consistency to estimate the depth map and obtain the 3D dense point cloud.

[0003] Traditional methods can achieve good results in ideal Lambertian scenes and scenes without weak texture regions. However, they are difficult to extract surface features in weak texture regions and struggle to take into account multi-scale information. Manually calculated similarity metrics become unreliable, leading to incomplete dense image reconstruction results. Summary of the Invention

[0004] In this application embodiment, a learning-based lightweight adaptive multi-stage dense reconstruction method and apparatus are provided, which effectively overcomes the problems of the previous deep learning-based MVS algorithm having difficulty in extracting surface features in weak texture regions and having difficulty in taking into account multi-scale information, and the unreliability of manually calculated similarity metrics, resulting in incomplete dense reconstruction results of images, and realizes dense reconstruction of images.

[0005] In a first aspect, embodiments of this application provide a learning-based lightweight adaptive multi-stage dense reconstruction method, which includes: acquiring multiple original images; wherein the original images include reference images and source images; generating feature maps of multiple scales for each of the original images through a preset feature pyramid based on depthwise separable convolution; determining feature volumes of multiple scales for multiple source images corresponding to each reference image based on the feature maps of the original images at multiple scales; performing weighted aggregation of cost volumes using a dual-process adaptive aggregation method of intra-view and inter-view; generating the cost volume of the reference image at the top-level scale and generating a depth map using a hierarchical recurrent convolutional network; generating the residual cost volume of the reference image at other scales by referring to the depth map of the previous scale, generating a residual depth map using a hierarchical recurrent convolutional network, and upsampling and adding the residual depth map to the depth map of the previous scale to obtain a depth map at the current scale; and fusing the depth maps generated at multiple bottom-level scales to obtain a dense point cloud.

[0006] In conjunction with the first aspect, in one possible implementation, the intra-view adaptive aggregation method is to determine the feature volumes of multiple source images at multiple scales corresponding to each reference image based on the feature maps of multiple scales of the original image; the inter-view adaptive aggregation method is to construct a cost body based on weights using an inter-view weight aggregation module.

[0007] In conjunction with the first aspect, in one possible implementation, the hierarchical recurrent convolutional network includes multiple parallel recurrent modules in the vertical direction, each of which is used to transmit the recurrent convolution result of the previous parallel recurrent module to the next parallel recurrent module; the hierarchical recurrent convolutional network has a planar U-Net structure in the horizontal direction.

[0008] In conjunction with the first aspect, in one possible implementation, the fusion of multiple depth maps generated at the underlying scale includes: fusing multiple depth maps generated at the underlying scale through a dynamic consistency check.

[0009] In conjunction with the first aspect, in one possible implementation, the method further includes: using a multi-scale aggregation module to refine the feature map.

[0010] In conjunction with the first aspect, in one possible implementation, the multi-scale aggregation module includes: deformable convolution and bilinear interpolation units.

[0011] In conjunction with the first aspect, in one possible implementation, the method further includes: training the hierarchical recurrent convolutional network and analyzing the hierarchical recurrent convolutional network using a total loss function; the total loss function is defined as follows: Among them, L k Let λ represent the loss function at the k-th scale. k This indicates the corresponding loss weight.

[0012] Secondly, embodiments of this application provide a low-memory-occupancy, multi-stage dense reconstruction optimization apparatus. The apparatus includes: an acquisition module for acquiring multiple original images, wherein the original images include reference images and source images; a feature map generation module for generating feature maps at multiple scales for each original image using a preset feature pyramid based on depthwise separable convolution; a feature volume determination module for determining feature volumes at multiple scales for multiple source images corresponding to each reference image based on the feature maps at multiple scales of the original images; a cost volume aggregation module for weighted aggregation of cost volumes using a dual-flow adaptive aggregation method (intra-view and inter-view); a depth map generation module for generating the cost volume of the reference image at the top-level scale and generating a depth map using a hierarchical recurrent convolutional network; at other scales, generating the remaining cost volume of the reference image by referring to the depth map of the previous scale, generating a residual depth map using a hierarchical recurrent convolutional network, and upsampling and adding the residual depth map to the depth map of the previous scale to obtain a depth map at the current scale; and a fusion module for fusing multiple depth maps generated at the bottom-level scales to obtain a dense point cloud.

[0013] In conjunction with the second aspect, in one possible implementation, the fusion module is specifically used to: fuse multiple depth maps generated at the underlying scale through dynamic consistency checks.

[0014] In conjunction with the second aspect, in one possible implementation, the apparatus further includes a multi-scale aggregation module for refining the feature map using the multi-scale aggregation module.

[0015] In conjunction with the second aspect, in one possible implementation, the multi-scale aggregation module includes: deformable convolution and bilinear interpolation units.

[0016] In conjunction with the second aspect, in one possible implementation, the apparatus further includes: a loss analysis module, used to train the hierarchical recurrent convolutional network and analyze the hierarchical recurrent convolutional network using a total loss function; the total loss function is defined as follows: Among them, L k Let λ represent the loss function at the k-th scale. k This indicates the corresponding loss weight.

[0017] Thirdly, embodiments of this application provide a learning-based lightweight adaptive multi-stage dense reconstruction server, including a memory and a processor; the memory is used to store computer-executable instructions; the processor is used to execute the computer-executable instructions to implement the method described in the first aspect or any possible implementation of the first aspect.

[0018] Fourthly, embodiments of this application provide a computer-readable storage medium storing executable instructions, which, when executed by a computer, enable the implementation of the method described in accordance with the first aspect or any possible implementation thereof.

[0019] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0020] This application provides a learning-based lightweight adaptive multi-stage dense reconstruction method. In implementation, this method generates feature maps at multiple scales for each original image by pre-setting a feature pyramid based on depthwise separable convolution. At each scale of the feature pyramid, deformable convolutions and bilinear interpolation units of different scales are used to refine the feature maps. A dual-process adaptive aggregation method, both intra-view and inter-view, is used to aggregate the cost volume with weights. A staged strategy is then used to infer the depth map from coarse to fine on the cost volume regularization. This solves the problems of difficulty in extracting surface features in weakly textured regions and the difficulty in simultaneously considering multi-scale information, thus achieving dense image reconstruction. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments of the present invention or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart of learning-based lightweight adaptive multi-stage dense reconstruction provided for embodiments of this application;

[0023] Figure 2 This is a schematic diagram of the structure of a depthwise separable convolutional feature pyramid provided in an embodiment of this application;

[0024] Figure 3 A schematic diagram of the convolutional structure for the calculation process of the inter-view weight aggregation module provided in an embodiment of this application;

[0025] Figure 4 A schematic diagram of the structure of a hierarchical recurrent convolutional network provided in an embodiment of this application;

[0026] Figure 5 This is a schematic diagram of the structure of the U-LSTMCONV module provided in the embodiments of this application;

[0027] Figure 6 This is a schematic diagram of the structure of the aggregation module provided in an embodiment of this application;

[0028] Figure 7 A schematic diagram of a learning-based lightweight adaptive multi-stage dense reconstruction device provided in an embodiment of this application;

[0029] Figure 8 A schematic diagram of a learning-based lightweight adaptive multi-stage dense reconstruction server provided in an embodiment of this application;

[0030] Figure 9 This is a schematic diagram illustrating the overall steps provided in the embodiments of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0032] First, a brief introduction to the relevant technologies or concepts involved in the embodiments of this application will be given.

[0033] Deep learning (DL) is a new research direction in the field of machine learning (ML). It was introduced into machine learning to bring it closer to its original goal—artificial intelligence (AI). Deep learning learns the inherent patterns and hierarchical representations of sample data. The information gained during this learning process greatly aids in the interpretation of data such as text, images, and sound. Its ultimate goal is to enable machines to possess analytical and learning capabilities like humans, capable of recognizing data such as text, images, and sound. Deep learning has achieved significant results in search technology, data mining, machine learning, machine translation, natural language processing, multimedia learning, speech recognition, recommendation and personalization technologies, and other related fields. Deep learning enables machines to mimic human activities such as sight, hearing, and thought, solving many complex pattern recognition problems and leading to significant advancements in AI-related technologies.

[0034] Homography is defined as a projection mapping from one plane to another. For example, the mapping of a point on a two-dimensional plane to the image of a camera is an example of planar homography.

[0035] A feature pyramid network is a fundamental component in a recognition system used to detect objects at different scales. It can improve model accuracy by extracting and fusing feature information from multiple scales.

[0036] Depthwise separable convolution is an algorithm in convolutional neural networks that improves upon standard convolution computation. By separating the correlation between spatial dimension and channel (depth) dimension, it reduces the number of parameters required for convolution computation and has been shown in some studies to improve the efficiency of convolution kernel parameter utilization.

[0037] The prototype of depthwise separable convolution can be considered to originate from the Inception module in convolutional neural networks. Its convolution computation is divided into two parts: first, spatial convolution is performed on each channel (depth), and the outputs are concatenated; then, pointwise convolution is performed using unit convolution kernels to obtain feature maps. In terms of applications, depthwise separable convolution is used to build micro-neural networks and to optimize the structure of large-scale convolutional neural networks. Deep learning algorithms using depthwise separable convolution include Xception and MobileNet.

[0038] This application provides a learning-based lightweight adaptive multi-stage dense reconstruction method. It generates feature maps at multiple scales for each original image by pre-setting a feature pyramid based on depthwise separable convolution. At each scale of the feature pyramid, deformable convolutions and bilinear interpolation units of different scales are used to refine the feature maps. A dual-process adaptive aggregation method, both intra-view and inter-view, is used to aggregate the cost volume with weights. A staged strategy is applied to the cost volume regularization to infer the depth map from coarse to fine. This method solves the problems of difficulty in extracting surface features in weakly textured regions and the difficulty in taking into account multi-scale information, thus achieving dense image reconstruction.

[0039] Figure 1 This application provides a learning-based lightweight adaptive multi-stage dense reconstruction method, such as... Figure 1 and Figure 9 As shown, the method includes steps S101 to S107.

[0040] S101: Acquire multiple original images. The original images include a reference image and a source image.

[0041] S102: Generate feature maps of multiple scales for each original image by using a preset feature pyramid based on depthwise separable convolution.

[0042] The preset feature pyramid based on depthwise separable convolution in this embodiment can be considered as an encoding and decoding structure. The original image can be processed through the preset feature pyramid to obtain feature maps at multiple scales. While the feature map at the top level contains high-level semantic features, it lacks low-level details; while the feature maps at the bottom level contain feature details, they lack sufficient semantic information. Therefore, the features extracted from multiple scales by the preset feature pyramid based on depthwise separable convolution are more comprehensive and can accurately describe the image's features.

[0043] For example, Figure 2 This is a schematic diagram of the feature pyramid structure based on depthwise separable convolution provided in an embodiment of this application. The preset number of layers is three, and each original image is processed... Figure 2 The preset depthwise separable convolution-based feature pyramid shown can generate feature maps at three scales. Of course, the number of layers in the preset depthwise separable convolution-based feature pyramid in this embodiment can also be set to two, four, or other numbers; this application is not limited to three layers. Furthermore, the size of the preset depthwise separable convolution-based feature pyramid provided in this embodiment is preset manually. For example, the feature scale of each layer of the preset depthwise separable convolution-based feature pyramid is twice that of the layer above it. Figure 2 The preset top layer of the feature pyramid based on depthwise separable convolution shown has a scale of W / 4×H / 4×32, the middle layer has a scale of W / 2×H / 2×16, and the bottom layer has a scale of W×H×8; where W and H are the original image scales. Of course, in this embodiment, the feature size of each layer of the preset feature pyramid based on depthwise separable convolution can also be other multiples of the previous layer, such as three times, four times, etc., and this application is not limited to two times.

[0044] Specifically, Figure 2 The left side of the preset feature pyramid based on depthwise separable convolution uses depthwise separable convolution. Depthwise separable convolution is an algorithm in convolutional neural networks that improves upon standard convolution calculations. By splitting the correlation between spatial and channel (depth) dimensions, it reduces the number of parameters required for convolution calculations, making feature extraction more lightweight—that is, extracting fewer network parameters—and thus improving the efficiency of convolution kernel parameter utilization. The right side shows the feature map obtained by upsampling the preset feature pyramid based on depthwise separable convolution. Figure 2 The arrows in the diagram represent addition operations. The left side represents shallow convolution, while the right side represents deeper convolution. Adding the shallow convolution on the left side to the deeper convolution on the right side through addition operations enables the preset feature pyramid based on depth-separable convolution to extract more comprehensive and accurate image features from multiple scales.

[0045] Specifically, depthwise separable convolution is performed in two steps: depthwise convolution (DW) and pointwise convolution (PW). In depthwise convolution, one kernel is responsible for one channel, and each channel is convolved by only one kernel. This process produces a feature map with the same number of channels as the input layer, but it cannot expand the number of channels in the feature map and does not effectively utilize the feature information of different channels at the same spatial location. Therefore, pointwise convolution is needed to combine these feature maps to generate new feature maps. The operation of pointwise convolution is very similar to that of regular convolution. Its kernel size is 1×1×M, where M is the number of channels in the previous layer. The convolution operation weights and combines the feature maps from the previous step in the depth direction to generate new feature maps. In this process, the number of feature maps output is equal to the number of convolution kernels. Deformable convolution requires approximately 1 / 3 of the parameters of a single operation as ordinary convolution, thus significantly reducing the parameters required in the operation and making the process lightweight.

[0046] S103: Based on the feature maps at multiple scales of the original image, determine the feature volumes at multiple scales of the multiple source images corresponding to each reference image. Specifically, each reference image at each scale corresponds to the feature volumes of the multiple source images.

[0047] S104: The cost volume is aggregated with weights using a dual-process adaptive aggregation method that combines intra-view and inter-view approaches. Specifically, the intra-view adaptive aggregation method determines the feature volumes at multiple scales of the source images corresponding to each reference image based on the feature maps at multiple scales of the original image; the inter-view adaptive aggregation method constructs the cost volume from the multiple feature volumes using an inter-view weighted aggregation module.

[0048] Specifically, the adaptive aggregation method within the first flow view refers to obtaining the features of a reference image and N-1 source images through feature extraction. Then, based on homography transformation, each source image is projected onto each depth level to form a feature volume. Finally, interpolation is used to ensure that each projection has the same size. Theoretically, each reference image will have N-1 corresponding feature volumes.

[0049] The general approach is to fuse the feature volumes after construction into a cost volume based on variance, treating each view fairly and giving it the same weight. However, this method is not ideal because different shooting angles result in different lighting conditions, such as occlusion and non-Lambertian surfaces (Lambertian surfaces are those where the surface brightness of an object is the same from all viewing directions under the same lighting distribution), which increases the difficulty of depth estimation. Therefore, the second process of adaptive aggregation between views adopted in this embodiment specifically refers to designing an inter-view weight aggregation module for the network. This involves first calculating a weight coefficient, then weighting and aggregating the feature volumes to generate a feature map, and finally calculating the cost map to form the cost volume. The weight calculation logic is as follows: when the correlation between some images and other images is low, the weights are reduced during calculation; conversely, when the correlation between some images and other images is high, the weights are increased during calculation.

[0050] The formula for the view weight aggregation module is as follows: in, Let ω() represent the Hadamard product, and ω() represent the weight map adaptively generated based on the feature volume. In this embodiment, using 1+ω() for calculation better prevents excessive image smoothing. A schematic diagram of the convolution structure in the inter-view weight aggregation module calculation process provided in this embodiment is shown below. Figure 3 As shown, v1 represents the feature map of the feature volume at a certain depth, c represents the cost map of the cost volume at a certain depth, 301 represents a 3x3 convolution, 302 represents a 1x1 convolution, and 303 represents a Hadamard product. Specifically, the feature map of the feature volume at a certain depth is processed by convolutions with 4, 4, 4, and 1 channels respectively to generate a weighted map. Then, a Hadamard product is performed with the original feature map, followed by averaging to calculate the cost map. The cost map is then used to construct the cost volume.

[0051] S105: At the top-level scale, generate the cost volume of the reference image and use a hierarchical recurrent convolutional network to generate a depth map.

[0052] S106: At other scales, the residual cost volume of the reference image is generated by referring to the depth map of the previous scale. A hierarchical recurrent convolutional network is used to generate a residual depth map, and the residual depth map is upsampled and added to the depth map of the previous scale to obtain the depth map at the current scale. The hierarchical recurrent convolutional network includes multiple parallel recurrent modules in the vertical direction, each of which transmits the recurrent convolution result of the previous parallel recurrent module to the next parallel recurrent module. The hierarchical recurrent convolutional network has a planar U-Net structure in the horizontal direction.

[0053] S107: Fusing multiple depth maps generated at the lower-level scale to obtain a dense point cloud. The depth maps are obtained by fusing multiple lower-level scales through dynamic consistency checks.

[0054] Figure 4 This is a schematic diagram of the structure of a hierarchical recurrent convolutional network provided in an embodiment of this application, as shown below. Figure 4 As shown, the hierarchical recurrent convolutional network includes multiple parallel recurrent modules in the vertical direction. Each parallel recurrent module is used to pass the recurrent convolution result of the previous parallel recurrent module to the next parallel recurrent module. The number of parallel recurrent modules is preset manually. This embodiment of the application... Figure 4 The hierarchical recurrent convolutional network module in this application has five parallel recurrent modules in the vertical direction. However, the number of parallel recurrent modules in this embodiment can also be four, six, etc., and is not limited to five. Figure 4 The stack-like module shown can absorb contextual information at multiple scales and can efficiently process cost volumes.

[0055] like Figure 5 As shown, the hierarchical recurrent convolutional network has a planar U-Net structure in the horizontal direction. The planar U-Net structure is the horizontal structure of the U-LSTMCONV module. The planar U-Net structure includes LSTMConvCell unit 501, Deconv unit 502, and MaxPooling unit 503, with the positions of each unit as shown below. Figure 5 As shown, LSTMConvCell possesses both the temporal characteristics of LSTM and the ability to characterize local features like CNNs (Convolutional Neural Networks).

[0056] The input to a hierarchical recurrent convolutional network is the original image and the calibrated camera pose. The original image is used for feature extraction, and the camera pose information is used for homography transformation. Figure 2 Taking the preset feature pyramid based on depthwise separable convolution and a hierarchical recurrent convolutional network provided in this application embodiment as examples, the steps are described in detail below.

[0057] Using the depth map obtained at the top-level scale as a reference, depth map estimation is performed at the intermediate-level scale. The intermediate-level feature map is used to sample within the remaining depth range, referencing the first stage to construct the residual cost body, with a scale of W / 2×H / 2×16×32. This residual cost body is then passed through a hierarchical recurrent convolutional network to generate the residual depth map, ultimately producing the intermediate-level depth map. Specifically, the formula for constructing the residual cost body through homography transformation is as follows:

[0058]

[0059] in, This represents the depth predicted by layer k at pixel m. This represents the residual depth of the m-th pixel in the (k+1)-th layer that needs to be estimated. K, R, and t represent the camera's intrinsic parameters, I is the identity matrix, and n is the principal optical axis of the reference image.

[0060] The final depth map is constructed by using the depth map estimated at the intermediate layer scale in the same way to form the residual cost volume, with a scale of W×H×8×8. The constructed residual cost volume is then passed through a hierarchical recurrent convolutional network to generate a residual depth map, ultimately producing the depth map at the lowest layer scale. In fact, the number of feature pyramid layers (network stages) can be infinite; iteration can stop once the required depth accuracy is achieved.

[0061] For example, Figure 2 The preset feature pyramid shown is based on depthwise separable convolution and has three layers, therefore it is similar to... Figure 2 The preset layered recurrent convolutional network shown is based on a feature pyramid with depthwise separable convolutions and consists of three stages. Figure 6 The diagram illustrates the structure of the aggregation module provided in this embodiment, including deformable convolution and bilinear interpolation units. In each stage, three deformable convolutions of different scales are used, and multi-scale information is extracted through bilinear interpolation and concatenation. The feature map obtained in the first stage has a scale of W / 4×H / 4×32, the feature map obtained in the second stage has a scale of W / 2×H / 2×16, and the feature map obtained in the third stage has a scale of W×H×8.

[0062] Specifically, deformable convolution is defined as follows: Where f(p) represents the feature value of pixel p, w k and p k This represents the kernel parameters and fixed offset defined in a normal convolution operation, Δp. k and Δm k These are the learned offsets and weights of deformable convolutions. The advantage of deformable convolutions is that they support arbitrary shapes and perform better.

[0063] Furthermore, after obtaining the lowest-level depth map that meets the accuracy requirements, step S107 specifically involves fusing multiple depth maps generated at the lowest-level scale through dynamic consistency checks.

[0064] Currently, in depth map fusion based on geometric constraints, the consistency of depth estimation measurements across multiple views mostly relies on pre-fixed parameters, such as pixel reprojection error and depth reprojection error. Using fixed parameters fails to filter out a sufficient number of mismatched pixels in different scenarios, making it unreliable for various situations. The method provided in this application employs a dynamic consistency check to fuse multiple depth maps generated at the underlying scale. Based on the algorithm's dynamic constraints and the consistency of adjacent views, it can obtain a more accurate and complete dense point cloud.

[0065] The dynamic consistency test method for matching different views is defined as follows: Where ε p ε represents the pixel reprojection error. d Let λ represent the depth reprojection error, and λ be the weight of the two reprojection errors. The matching consistency of all views is fused to obtain the global dynamic multi-view geometric consistency, defined as: Finally, outliers are filtered out using τ. For example, λ = 200 and τ = 1.8 are set.

[0066] The method provided in this application embodiment further includes: training a hierarchical recurrent convolutional network and analyzing the hierarchical recurrent convolutional network using a total loss function.

[0067] The hierarchical recurrent convolutional network in this application embodiment is multi-stage. The cost body of multiple stages can generate two intermediate depth maps and a final depth map. Before obtaining the depth map, the hierarchical recurrent convolutional network needs to be used to regularize the cost body to obtain the probability body.

[0068] When calculating the total loss function, all stages need to be considered. The total loss function is defined as follows:

[0069]

[0070] Among them, L k Let λ represent the loss function at the k-th scale. k This represents the corresponding loss weight. In the embodiments of this application, it is used... Figure 2 When the preset is based on a feature pyramid with depthwise separable convolutions, the hierarchical recurrent convolutional network sets N to 3. Generally, the higher the resolution of the generated depth map, the larger the weights are set.

[0071] The loss function for each scale is calculated as follows:

[0072]

[0073] Where, x validLet G(i,x) represent the valid set of pixels, G(i,x) represent the one-hot encoding of the true depth map at the i-th depth of pixel x, and P(i,x) represent the pixels in the probability volume.

[0074] Based on the results of multiple experiments, it can be determined that the method provided in this application significantly reduces GPU memory consumption, with a runtime memory usage of 27% of MVSNet. At the same time, it uses a dual-flow adaptive aggregation method of intra-view and inter-view to aggregate the cost volume, which is more advantageous in weak texture processing.

[0075] This application also provides a learning-based lightweight adaptive multi-stage dense reconstruction device 700, such as... Figure 7 As shown, the device includes: an acquisition module 701, a feature map generation module 702, a feature volume determination module 703, a cost volume aggregation module 704, a depth map generation module 705, and a fusion module 706.

[0076] The acquisition module 701 is used to acquire multiple original images; wherein the original images include a reference image and a source image.

[0077] The feature map generation module 702 is used to generate feature maps of multiple scales for each original image through a preset feature pyramid based on depthwise separable convolution.

[0078] The feature volume determination module 703 is used to determine the feature volumes at multiple scales of multiple source images corresponding to each reference image based on the feature maps at multiple scales of the original image;

[0079] The cost body aggregation module 704 is used to perform weighted aggregation of cost bodies using a dual-flow adaptive aggregation method that is both within and between views.

[0080] The depth map generation module 705 is used to generate the cost volume of the reference image at the top-level scale and generate a depth map using a hierarchical recurrent convolutional network; at other scales, it generates the residual cost volume of the reference image with reference to the depth map of the previous scale, generates a residual depth map using a hierarchical recurrent convolutional network, and adds the residual depth map to the depth map of the previous scale by upsampling, thus obtaining the depth map at the current scale.

[0081] The fusion module 706 is used to fuse multiple depth maps generated at the lower-level scale to obtain a dense point cloud. Specifically, it is used to fuse multiple depth maps generated at the lower-level scale through dynamic consistency checks.

[0082] The learning-based lightweight adaptive multi-stage dense reconstruction device 700 also includes a multi-scale aggregation module, comprising deformable convolution and bilinear interpolation units.

[0083] The learning-based lightweight adaptive multi-stage dense reconstruction device 700 also includes: a loss analysis module for training the hierarchical recurrent convolutional network and analyzing the hierarchical recurrent convolutional network using a total loss function; the total loss function is defined as follows: Among them, L k Let λ represent the loss function at the k-th scale. k This indicates the corresponding loss weight.

[0084] The apparatus or module described in the above embodiments can be implemented by a computer chip or physical entity, or by a product with a certain function. For ease of description, the above apparatus is described by dividing it into various modules according to their functions. In implementing this application, the functions of each module can be implemented in one or more software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0085] The methods, apparatus, or modules described in this application can be implemented in a computer-readable program code manner. The controller can be implemented in any suitable manner, such as a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of a memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code manner, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included within it for implementing various functions can also be considered as structures within the hardware component. Alternatively, the device used to implement various functions can be viewed as either a software module that implements the method or a structure within a hardware component.

[0086] Some modules in the apparatus described in this application can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0087] This application also provides a depth map server for 3D reconstruction, such as... Figure 8 As shown, it includes a memory 801 and a processor 802; the memory 801 is used to store computer-executable instructions; the processor 802 is used to execute computer-executable instructions to implement the low video memory usage multi-stage dense reconstruction method provided in the embodiments of this application.

[0088] This application also provides a computer-readable storage medium storing executable instructions. When a computer executes the executable instructions, it can implement the low-memory-occupancy, multi-stage dense reconstruction method provided in this application.

[0089] The aforementioned storage media include, but are not limited to, Random Access Memory (RAM), Read-Only Memory (ROM), Cache, Hard Disk Drive (HDD), or Memory Card. The memory can be used to store computer program instructions.

[0090] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, or it can be embodied in the process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0091] The various embodiments described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. All or part of this application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, mobile communication terminals, multiprocessor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0092] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of this application.

Claims

1. A learning-based lightweight adaptive multi-stage dense reconstruction method, characterized in that, include: Acquire multiple original images; wherein the original images include a reference image and a source image; Multiple feature maps at various scales are generated for each original image by pre-setting a feature pyramid based on depthwise separable convolution; Based on the feature maps at multiple scales of the original image, determine the feature volumes at multiple scales of multiple source images corresponding to each reference image; We use a dual-flow adaptive aggregation method, both within and between views, to perform weighted aggregation of the cost body. At the top-level scale, the cost volume of the reference image is generated, and a depth map is generated using a hierarchical recurrent convolutional network. At other scales, the residual cost volume of the reference image is generated by referring to the depth map of the previous scale, a hierarchical recurrent convolutional network is used to generate a residual depth map, and the residual depth map is upsampled and added to the depth map of the previous scale to obtain the depth map at the current scale. By fusing multiple depth maps generated at the underlying scale, a dense point cloud is obtained.

2. The method according to claim 1, characterized in that, The in-view adaptive aggregation method is to determine the feature volumes of multiple source images at multiple scales corresponding to each reference image based on the feature maps of multiple scales of the original image. The adaptive aggregation method between views involves using an inter-view weight aggregation module to construct a cost body from multiple feature bodies based on weights.

3. The method according to claim 1, characterized in that, The hierarchical recursive convolutional network includes multiple parallel recursive modules in the vertical direction. Each parallel recursive module is used to transmit the recursive convolution result of the previous parallel recursive module to the next parallel recursive module. The hierarchical recurrent convolutional network has a planar U-Net structure in the horizontal direction.

4. The method according to claim 1, characterized in that, The fusion of multiple depth maps generated at the lower-level scale includes: Multiple depth maps generated at the underlying scale are fused using dynamic consistency checks.

5. The method according to claim 1, characterized in that, Also includes: The feature map is refined using a multi-scale aggregation module.

6. The method according to claim 5, characterized in that, The multi-scale aggregation module includes deformable convolution and bilinear interpolation units.

7. The method according to claim 1, characterized in that, Also includes: The hierarchical recurrent convolutional network is trained, and the total loss function is used to analyze the hierarchical recurrent convolutional network. The total loss function is defined as follows: Among them, L k Let λ represent the loss function at the k-th scale. k This indicates the corresponding loss weight.

8. A learning-based lightweight adaptive multi-stage dense reconstruction optimization device, characterized in that, include: An acquisition module is used to acquire multiple original images; wherein the original images include a reference image and a source image; The feature map generation module is used to generate feature maps of multiple scales for each of the original images by means of a preset feature pyramid based on depthwise separable convolution. The feature volume determination module is used to determine the feature volumes at multiple scales of multiple source images corresponding to each reference image based on the feature maps at multiple scales of the original image; The cost body aggregation module is used to perform weighted aggregation of cost bodies using a dual-flow adaptive aggregation method that combines in-view and inter-view processes. The depth map generation module is used to generate the cost volume of the reference image at the top-level scale and generate a depth map using a hierarchical recurrent convolutional network; at other scales, it generates the residual cost volume of the reference image with reference to the depth map of the previous scale, generates a residual depth map using a hierarchical recurrent convolutional network, and adds the residual depth map to the depth map of the previous scale by upsampling, thereby obtaining the depth map at the current scale. The fusion module is used to fuse multiple depth maps generated at the lower level to obtain a dense point cloud.

9. A learning-based lightweight adaptive multi-stage dense reconstruction server, characterized in that, Including memory and processor; The memory is used to store computer-executable instructions; The processor is configured to execute the computer-executable instructions to implement the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores executable instructions, which, when executed by a computer, enable the implementation of the method as described in any one of claims 1-7.

Images