A method and device for recursive dense reconstruction based on attention enhanced matching

Abstract

The application discloses a method and device for recursive dense reconstruction based on attention-enhanced matching, which comprises the following steps: obtaining multiple original images; introducing an ASPP network at the lowest scale of a feature pyramid based on dilated convolution, outputting the final multi-scale aggregated features at the lowest scale, and making the resolution of the final multi-scale aggregated features the same as that of the original input of the ASPP network; up-sampling from the top layer scale to obtain the total aggregated features; calculating the weight information of the total aggregated features according to the global average pooling result of the total aggregated features, calculating the weighted cost surface, and obtaining the cost volume; constructing a cascade regularization network to regularize the cost volume and obtain the probability volume; normalizing the probability value of the probability volume by using a normalized exponential function, and regressing the depth map; and performing joint training of the probability volume loss and the depth value loss of the depth map, so as to solve the problems that it is difficult to extract surface features in weak texture areas, multi-scale information is difficult to be considered, the reconstruction effect on a reflective surface is poor, and the dense reconstruction result is incomplete.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a recursive dense reconstruction method and apparatus based on attention-enhanced matching. Background Technology

[0002] Dense reconstruction aims to generate high-precision 3D scene models from image or video data captured from multiple perspectives. This technology is significant in many application areas, including virtual reality, augmented reality, robot navigation, and 3D modeling. To achieve dense reconstruction, researchers and engineers have developed various algorithms and methods. One common approach combines multi-view geometry and optical flow estimation techniques. Multi-view geometry involves deriving the relative poses and projection relationships between images, while optical flow estimation estimates camera motion and scene depth information by analyzing pixel displacements in the image. Another common dense reconstruction method utilizes structured light or texture projection for 3D reconstruction. Structured light technology uses a light projector and camera to capture textures or light spots in the scene, thereby obtaining the 3D shape of objects. Texture projection methods achieve 3D reconstruction by projecting textures or special encodings onto the object surface.

[0003] Traditional methods can achieve good results in ideal Lambertian scenes and scenes without weak texture regions. However, they have difficulty extracting surface features in weak texture regions and cannot take into account multi-scale information. Furthermore, they have poor reconstruction results on reflective surfaces, resulting in incomplete dense reconstruction results of the image. Summary of the Invention

[0004] In this embodiment, a multi-stage dense reconstruction method and apparatus based on uncertainty is provided, which solves the problem that traditional methods can achieve good results in ideal Lambertian scenes and scenes without weak texture regions. However, it is difficult to extract surface features in weak texture regions and it is difficult to take into account multi-scale information. Furthermore, the reconstruction effect of reflective surfaces is also poor, thus leading to incomplete dense reconstruction results of the image.

[0005] In a first aspect, embodiments of this application provide a recursive dense reconstruction method based on attention-enhanced matching. The method includes: acquiring multiple original images; wherein the original images include a reference image and a source image; introducing an ASPP network at the lowest scale of a preset feature pyramid based on dilated convolution, outputting the final multi-scale aggregated features at the lowest scale, and ensuring that the resolution is the same as the original input of the ASPP network; upsampling from the top scale of the feature pyramid to obtain total aggregated features; calculating the weight information of the total aggregated features based on the global average pooling result of the total aggregated features, calculating the weighted cost surface and obtaining the cost volume; constructing a cascaded regularization network to regularize the cost volume to obtain a probability volume, normalizing the probability values ​​of the probability volume using a normalized exponential function, and regressing the depth map; and jointly training the probability volume loss and the depth value loss of the depth map.

[0006] In conjunction with the first aspect, in one possible implementation, prior to acquiring multiple original images, the method further includes: inputting a deep learning dataset and constructing the feature pyramid based on dilated convolution.

[0007] In conjunction with the first aspect, in one possible implementation, the output of the final multi-scale aggregated features at the lowest scale includes: extracting the original multi-scale features at the lowest scale using average pooling and multiple dilated convolutions with different dilation parameters in the ASPP network; performing feature concatenation on the extracted original multi-scale features to output the final multi-scale aggregated features at the lowest scale.

[0008] In conjunction with the first aspect, in one possible implementation, calculating the weight information of the total aggregated features based on the global average pooling result of the total aggregated features includes: according to the formula Calculate the weight information; where N is the number of input feature maps, f i This is the result of global average pooling on a single feature map. f i The average value of , where a1, a2, b1, and b2 are constant terms.

[0009] In conjunction with the first aspect, in one possible implementation, calculating the weighted cost surface and obtaining the cost volume includes: performing a differentiable homography transformation on the total aggregate features to obtain multiple initial cost surfaces; according to the formula... Calculate the weighted final cost surfaces separately; where V j The initial cost surface is used; the cost volume is obtained by superimposing and summing pixels at the same position along the depth direction of multiple final cost surfaces.

[0010] In conjunction with the first aspect, in one possible implementation, the joint training of the probabilistic volume loss and the depth value loss of the depth map includes: according to the formula Calculate the true probability volume; where, α and β are constants, P j (x) is the probability body, d j (x) represents the depth value of the depth map, x represents the number of pixels in the depth map, and d'(x) represents the ground truth value; according to the formula Calculate the probabilistic volume loss, and then perform joint training of the probabilistic volume loss and the depth value loss of the depth map.

[0011] In conjunction with the first aspect, one possible implementation also includes using point convolution to reduce the number of channels resulting from performing convolution.

[0012] Secondly, embodiments of this application provide a recursive dense reconstruction device based on attention-enhanced matching. The device includes: an acquisition module for acquiring multiple original images, wherein the original images include reference images and source images; an introduction module for introducing an ASPP network at the lowest scale of a preset feature pyramid based on dilated convolution, outputting the final multi-scale aggregated features at the lowest scale, and ensuring that the resolution is the same as the original input of the ASPP network; a sampling module for upsampling from the top scale of the feature pyramid to obtain total aggregated features; a calculation module for calculating the weight information of the total aggregated features based on the global average pooling result of the total aggregated features, calculating the weighted cost surface, and obtaining the cost volume; a normalization module for constructing a cascaded regularization network to regularize the cost volume to obtain a probability volume, using a normalization exponential function to normalize the probability values ​​of the probability volume, and regressing the depth map; and a training module for jointly training the probability volume loss and the depth value loss of the depth map.

[0013] In conjunction with the second aspect, in one possible implementation, before acquiring multiple original images, the method further includes: inputting a deep learning dataset and constructing the feature pyramid based on dilated convolution.

[0014] In conjunction with the second aspect, in one possible implementation, the output of the final multi-scale aggregated features at the lowest scale includes: extracting the original multi-scale features at the lowest scale using average pooling and multiple dilated convolutions with different dilation parameters in the ASPP network; and performing feature concatenation on the extracted original multi-scale features to output the final multi-scale aggregated features at the lowest scale.

[0015] In conjunction with the second aspect, in one possible implementation, calculating the weight information of the total aggregated features based on the global average pooling result of the total aggregated features includes: according to the formula... Calculate the weight information; where N is the number of input feature maps, f i This is the result of global average pooling on a single feature map. f i The average value of , where a1, a2, b1, and b2 are constant terms.

[0016] In conjunction with the second aspect, in one possible implementation, calculating the weighted cost surface and obtaining the cost volume includes: performing a differentiable homography transformation on the total aggregate features to obtain multiple initial cost surfaces; according to the formula... Calculate the weighted final cost surfaces separately; where V j The initial cost surface is used; the cost volume is obtained by superimposing and summing pixels at the same position along the depth direction of multiple final cost surfaces.

[0017] In conjunction with the second aspect, in one possible implementation, the joint training of the probabilistic volume loss and the depth value loss of the depth map includes: according to the formula Calculate the true probability volume; where, α and β are constants, P j (x) is the probability body, d j (x) represents the depth value of the depth map, x represents the number of pixels in the depth map, and d'(x) represents the ground truth value; according to the formula Calculate the probabilistic volume loss, and then perform joint training of the probabilistic volume loss and the depth value loss of the depth map.

[0018] In conjunction with the second aspect, one possible implementation also includes using pointwise convolution to reduce the number of channels resulting from performing convolution.

[0019] Thirdly, embodiments of this application provide a recursive dense reconstruction server based on attention-enhanced matching, including a memory and a processor; the memory is used to store computer-executable instructions; the processor is used to execute the computer-executable instructions to implement the method described in the first aspect or any possible implementation of the first aspect.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium storing executable instructions, which, when executed by a computer, enable the method described in the first aspect or any possible implementation thereof.

[0021] One or more technical solutions provided in the embodiments of this application have at least the following technical effects:

[0022] This application provides a recursive dense reconstruction method based on attention-enhanced matching. In implementation, this method introduces an ASPP network at the lowest scale of the feature pyramid to simultaneously consider local detail information and global contextual information, thus avoiding the loss of image features. The calculated weights are used to construct a weighted cost volume that ignores unimportant features. A cascaded regularization network is then constructed to regularize the cost volume, resulting in a probability volume that selects more useful features while ignoring unimportant ones. This helps address the vanishing gradient problem and requires very few parameters. Furthermore, this network possesses strong contextual semantic information capture capabilities and excellent detail recovery capabilities, significantly improving the cost volume regularization ability. Finally, joint training of the probability volume loss and the depth value loss of the depth map improves the accuracy of the estimated probability volume. This addresses the problem that traditional methods, while achieving good results in ideal Lambertian scenes and scenes without weak texture regions, struggle with extracting surface features in weak texture regions and failing to consider multi-scale information, resulting in poor reconstruction of reflective surfaces and incomplete dense image reconstruction results. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments of the present invention or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart of the attention-enhanced matching-based recursive dense reconstruction method provided in the embodiments of this application;

[0025] Figure 2 A flowchart illustrating the specific steps for implementing the final multi-scale aggregated feature at the lowest output scale, as provided in the embodiments of this application;

[0026] Figure 3 This application provides a flowchart illustrating the specific steps involved in calculating the weighted cost surface and obtaining the cost volume in an embodiment of the present application.

[0027] Figure 4 A flowchart illustrating the specific steps involved in jointly training the probabilistic volume loss and the depth value loss of the depth map for the implementation provided in this application embodiment;

[0028] Figure 5 A schematic diagram of a pre-defined feature pyramid based on dilated convolution and an ASPP network provided for embodiments of this application;

[0029] Figure 6 A schematic diagram of a cascaded regularized network provided in an embodiment of this application;

[0030] Figure 7 A schematic diagram of recursive dense reconstruction based on attention-enhanced matching provided for an embodiment of this application;

[0031] Figure 8 The recursive dense reconstruction apparatus based on attention-enhanced matching provided in the embodiments of this application;

[0032] Figure 9 This application provides a recursive dense reconstruction server based on attention-enhanced matching. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0034] First, a brief introduction to the relevant technologies or concepts involved in the embodiments of this application will be given.

[0035] Deep learning (DL) is a new research direction in the field of machine learning (ML). It was introduced into machine learning to bring it closer to its original goal—artificial intelligence (AI). Deep learning learns the inherent patterns and hierarchical representations of sample data. The information gained during this learning process greatly aids in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to possess analytical and learning capabilities like humans, capable of recognizing data such as text, images, and sound. Deep learning has achieved significant results in search technology, data mining, machine learning, machine translation, natural language processing, multimedia learning, speech recognition, recommendation and personalization technologies, and other related fields. Deep learning enables machines to mimic human activities such as sight, hearing, and thinking, solving many complex pattern recognition problems and leading to significant advancements in artificial intelligence-related technologies.

[0036] A feature pyramid network is a fundamental component in a recognition system used to detect objects at different scales. It can improve model accuracy by extracting and fusing feature information from multiple scales.

[0037] This application provides a recursive dense reconstruction method based on attention-enhanced matching, such as... Figure 1 As shown, the method includes steps S101 to S106.

[0038] S101: Acquire multiple raw images. The raw images include a reference image and a source image.

[0039] Before acquiring multiple original images, the process includes: inputting a deep learning dataset and constructing a feature pyramid based on dilated convolutions. Specifically, the input deep learning dataset can be a DTU or BlendedMVS dataset. Of course, other datasets are also possible, and this application is not limited to the aforementioned datasets. The dataset is divided into a training set and a test set; the training set is used to train the model, and the test set is used to test the model's performance.

[0040] S102: An ASPP network is introduced at the lowest scale of the pre-defined dilated convolution-based feature pyramid, outputting the final multi-scale aggregated features at the lowest scale, with the same resolution as the original input to the ASPP network. Specifically, the pre-defined dilated convolution-based feature pyramid can be considered as a whole network. Since frequent downsampling of the feature pyramid to extract features can lead to image feature loss, introducing the ASPP network at the lowest scale of the feature pyramid can simultaneously consider local detail information and global context information, thereby avoiding the loss of image features. Furthermore, outputting the final multi-scale aggregated features with the same resolution as the original input to the ASPP network helps preserve the spatial information of the input image and makes feature alignment and fusion operations easier in subsequent processing.

[0041] Figure 2 The flowchart of the specific steps for implementing the final multi-scale aggregated feature with the lowest output scale in S102 provided in the embodiments of this application includes steps S201 to S202.

[0042] S201: Extract the lowest-scale multi-scale original features using average pooling operations and multiple dilated convolutions with different dilation parameters in the ASPP network.

[0043] Figure 5 The schematic diagram of the pre-defined feature pyramid based on dilated convolution and the ASPP network provided for the embodiments of this application is as follows: Figure 5As shown, this application constructs a four-layer feature pyramid and introduces an ASPP network at the lowest scale. The ASPP network uses dilated convolutions with dilation parameters of 1, 3, 5, and 7 to extract multi-scale original features at the lowest scale. The advantage of choosing dilated convolutions is that they can increase the size of the receptive field without increasing network parameters and computational cost. Dilated convolutions with smaller dilation parameters, such as dilated convolutions with dilation parameter of 1, provide a local receptive field, while dilated convolutions with larger dilation parameters, such as dilated convolutions with dilation parameter of 7, provide a wider receptive field. By using different dilation parameters, feature information at different scales in the image can be captured. Of course, dilated convolutions with dilation parameters of other values ​​can also be selected. This application is not limited to the values ​​of the dilation parameters mentioned above.

[0044] S202: Perform feature concatenation operation on the extracted multi-scale original features to output the final multi-scale aggregated features at the lowest scale.

[0045] Specifically, the extracted multi-scale original features can be concatenated according to the channel dimension to output the final multi-scale aggregated features at the lowest scale. Using average pooling can reduce the risk of overfitting. Since the output channel number is typically 32 or 64, pointwise convolution can generally be used to reduce the number of channels to 16. Pointwise convolution is a 1×1 convolution. The benefits of reducing the number of channels are reduced computation and saved GPU memory. Of course, other methods can be used to reduce the number of channels generated by convolution, and this application is not limited to the methods described above.

[0046] S103: Upsample from the top scale of the feature pyramid to obtain the total aggregated features.

[0047] S104: Calculate the weight information of the total aggregated features based on the global average pooling result of the total aggregated features, calculate the weighted cost surface and obtain the cost body.

[0048] The weight information of the total aggregated features is calculated based on the global average pooling result of the total aggregated features, including: according to the formula Calculate the weight information; where N is the number of input feature maps, f i This is the result of global average pooling on a single feature map. f i The average value is given, where a1, a2, b1, and b2 are constant terms. Specifically, the constant terms can be determined using methods such as Bayesian optimization; of course, other methods can also be used, and this application is not limited to the methods mentioned above. Furthermore, the calculation of the above weight information adopts an attention mechanism.

[0049] Figure 3The flowchart of the specific steps for calculating the weighted cost surface and obtaining the cost body in S104 provided in the embodiments of this application includes steps S301 to S303.

[0050] S301: Perform a differentiable homography transformation on the total aggregation features to obtain multiple initial cost surfaces.

[0051] S302: According to the formula Calculate the weighted final cost surfaces separately. Where V j Let D be the initial cost surfaces. Specifically, assume the number of depth samples is D. Since the number of depth samples is the same as the number of initial cost surfaces, there are D initial cost surfaces, denoted as: The value of each initial cost surface, i.e., V j The value is determined by the matching cost between each source image and the reference image. The values ​​of multiple initial cost surfaces, i.e., V... j The numerical value is multiplied by the weight information obtained from the attention mechanism to obtain multiple weighted final cost surfaces. The attention mechanism dynamically learns weights based on the importance of different input features. By learning the weights, the attention mechanism model can focus on features that contribute more to the task, thereby improving the performance of the attention mechanism model.

[0052] S303: The cost volume is obtained by superimposing and summing the pixels at the same position along the depth direction of multiple final cost surfaces. Specifically, the obtained cost volume contains the superposition information of multiple cost surfaces in the depth direction.

[0053] S105: Construct a cascaded regularized network to regularize the cost volume and obtain the probability volume. Use a normalized exponential function to normalize the probability values ​​of the probability volume and then regress the depth map.

[0054] Figure 6 A schematic diagram of the cascaded regularized network provided in the embodiments of this application is shown below. Figure 6As shown, the Cascade GRUU-Net (CG-UNet) constructed in this application consists of two encoders and decoders. Each encoder and decoder is composed of a U-Net network modified with GRU. Specifically, GRU modules are added to the skip connections of U-Net to enhance feature extraction. This network combines the selective memory characteristics of GRU modules, enabling it to select more useful feature information and ignore unimportant feature information. This helps to solve the vanishing gradient problem and requires very few parameters. Furthermore, this network has strong contextual semantic information capture capabilities and excellent detail recovery capabilities, thus greatly improving the cost volume regularization ability. In short, this network is based on U-Net, replacing ordinary convolutions with convolution + LeakyReLU and replacing skip connections with GRU.

[0055] S106: Perform joint training of probabilistic volume loss and depth value loss of depth map.

[0056] Figure 4 The flowchart of the specific steps for joint training of probabilistic volume loss and depth value loss of depth map in S106 provided in the embodiments of this application includes steps S401 to S402.

[0057] S401: According to the formula Calculate the true probability volume. Wherein, α and β are constants, P j (x) is the probability body, d j Let d'(x) be the depth value of the depth map, x be the number of pixels in the depth map, and d'(x) be the ground truth value. Specifically, the ground truth value can be obtained from known depth maps in the dataset, and the constant term can be determined by methods such as Bayesian optimization. Of course, it can also be determined by other methods, and this application is not limited to the above methods.

[0058] S402: According to the formula The probabilistic volume loss is calculated, and then the probabilistic volume loss and the depth value loss of the depth map are jointly trained. Specifically, the formula is to calculate the probabilistic volume loss using cross-entropy. Joint training with the probabilistic volume loss and the depth value loss of the depth map can enhance the robustness and generalization of the overall network.

[0059] Figure 7 A schematic diagram of recursive dense reconstruction based on attention-enhanced matching provided in this application embodiment is shown below. Figure 7 As shown in the figure, DPFN refers to the pre-defined feature pyramid based on dilated convolution. Loss1 in the figure represents training with probabilistic volume loss, and Loss2 represents training with depth value loss of the depth map.

[0060] This application also provides a recursive dense reconstruction apparatus 800 based on attention-enhanced matching, such as... Figure 8 As shown, the device includes: an acquisition module 801, an introduction module 802, a sampling module 803, a calculation module 804, a normalization module 805, and a training module 806.

[0061] The acquisition module 801 is used to acquire multiple original images; wherein the original images include a reference image and a source image.

[0062] The introduction module 802 is used to introduce the ASPP network at the lowest scale of the preset dilated convolution-based feature pyramid, output the final multi-scale aggregated features at the lowest scale, and make them the same as the resolution of the original input of the ASPP network.

[0063] The sampling module 803 is used to upsample from the top scale of the feature pyramid to obtain the total aggregated features.

[0064] The calculation module 804 is used to calculate the weight information of the total aggregated features based on the global average pooling result of the total aggregated features, calculate the weighted cost surface and obtain the cost body.

[0065] The normalization module 805 is used to construct a cascaded regularized network, regularize the cost volume to obtain the probability volume, normalize the probability values ​​of the probability volume using a normalization exponential function, and regress the depth map.

[0066] Training module 806 is used for joint training of probabilistic volume loss and depth value loss of depth map.

[0067] Before acquiring multiple original images, the attention-enhanced matching-based recursive dense reconstruction device 800 also includes an input module for inputting a deep learning dataset and constructing a feature pyramid based on dilated convolution.

[0068] The module 802 is specifically used to extract the lowest-scale multi-scale original features using average pooling and multiple dilated convolutions with different dilation parameters in the ASPP network; and to perform feature concatenation on the extracted multi-scale original features to output the lowest-scale final multi-scale aggregated features.

[0069] Calculation module 804 is specifically used to calculate according to the formula Calculate the weight information; where N is the number of input feature maps, f i This is the result of global average pooling on a single feature map. f i The average value of , where a1, a2, b1, and b2 are constant terms.

[0070] The calculation module 804 is also specifically used to perform a differentiable homography transformation on the total aggregation features to obtain multiple initial cost surfaces; according to the formula Calculate the weighted final cost surfaces separately; where V j The initial cost surface is used; the cost volume is obtained by superimposing and summing the pixels at the same position along the depth direction of multiple final cost surfaces.

[0071] Training module 806 is specifically used to train according to the formula Calculate the true probability volume; where, α and β are constants, P j (x) is the probability body, d j (x) represents the depth value of the depth map, x represents the number of pixels in the depth map, and d'(x) represents the ground truth value; according to the formula Calculate the probabilistic volume loss, and then perform joint training using the probabilistic volume loss and the depth value loss of the depth map.

[0072] The attention-enhanced matching-based recursive dense reconstruction apparatus 800 also includes a channel reduction module for using point convolution to reduce the number of channels resulting from performing convolution.

[0073] The apparatus or module described in the above embodiments can be implemented by a computer chip or physical entity, or by a product with a certain function. For ease of description, the above apparatus is described by dividing it into various modules according to their functions. In implementing this application, the functions of each module can be implemented in one or more software and / or hardware. Of course, a module that implements a certain function can also be implemented by combining multiple sub-modules or sub-units.

[0074] The methods, apparatus, or modules described in this application can be implemented in a computer-readable program code manner. The controller can be implemented in any suitable manner, such as a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of a memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code manner, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included within it for implementing various functions can also be considered as structures within the hardware component. Alternatively, the device used to implement various functions can be viewed as either a software module that implements the method or a structure within a hardware component.

[0075] Some modules in the apparatus described in this application can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0076] This application also provides a recursive dense reconstruction server based on attention-enhanced matching, such as... Figure 9 As shown, it includes a memory 901 and a processor 902; the memory 901 is used to store computer-executable instructions; the processor 902 is used to execute the computer-executable instructions to implement the attention-enhanced matching-based recursive dense reconstruction method provided in the embodiments of this application.

[0077] This application also provides a computer-readable storage medium storing executable instructions. When a computer executes the executable instructions, it can implement the attention-enhanced matching-based recursive dense reconstruction method provided in this application.

[0078] The aforementioned storage media include, but are not limited to, Random Access Memory (RAM), Read-Only Memory (ROM), Cache, Hard Disk Drive (HDD), or Memory Card. The memory can be used to store computer program instructions.

[0079] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, or it can be embodied in the process of data migration. The computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0080] The various embodiments described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. All or part of this application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, mobile communication terminals, multiprocessor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0081] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of this application.

Claims

1. A recursive dense reconstruction method based on attention-enhanced matching, characterized in that, include: Acquire multiple original images; wherein the original images include a reference image and a source image; An ASPP network is introduced at the lowest scale of a pre-defined feature pyramid based on dilated convolution, and the final multi-scale aggregated features at the lowest scale are output, with the same resolution as the original input of the ASPP network. The total aggregated features are obtained by upsampling starting from the top scale of the feature pyramid; The weight information of the total aggregated features is calculated based on the global average pooling result of the total aggregated features, and the weighted cost surface is calculated to obtain the cost volume. A cascaded regularization network is constructed to regularize the cost volume to obtain a probability volume. The probability values ​​of the probability volume are normalized using a normalized exponential function, and then the depth map is regressed. Perform joint training using probabilistic volume loss and depth value loss of the depth map; The step of calculating the weight information of the total aggregated features based on the global average pooling result of the total aggregated features includes: according to the formula Calculate the weight information; where N is the number of input feature maps. This is the result of global average pooling on a single feature map. for The average value, For constant terms; The calculation of the weighted cost surface and obtaining the cost volume includes: performing a differentiable homography transformation on the total aggregate features to obtain multiple initial cost surfaces; according to the formula Calculate the weighted final cost surfaces separately; among them, The initial cost surface is used; the cost volume is obtained by superimposing and summing pixels at the same position along the depth direction of multiple final cost surfaces.

2. The method according to claim 1, characterized in that, Before acquiring multiple original images, the process also includes: inputting a deep learning dataset and constructing the feature pyramid based on dilated convolution.

3. The method according to claim 1, characterized in that, The output of the final multi-scale aggregated feature at the lowest scale includes: The lowest-scale multi-scale original features are extracted using average pooling and multiple dilated convolutions with different dilation parameters in the ASPP network. The extracted multi-scale original features are subjected to feature concatenation operation to output the final multi-scale aggregated features at the lowest scale.

4. The method according to claim 1, characterized in that, Also includes: Use point convolution to reduce the number of channels generated by performing convolution.

5. A recursive dense reconstruction device based on attention-enhanced matching, characterized in that, include: An acquisition module is used to acquire multiple original images; wherein the original images include a reference image and a source image; An introduction module is used to introduce the ASPP network at the lowest scale of a preset feature pyramid based on dilated convolution, output the final multi-scale aggregated features at the lowest scale, and make them have the same resolution as the original input of the ASPP network. A sampling module is used to upsample from the top scale of the feature pyramid to obtain the total aggregated features; The calculation module is used to calculate the weight information of the total aggregated features based on the global average pooling result of the total aggregated features, calculate the weighted cost surface and obtain the cost body; The normalization module is used to construct a cascaded regularization network to regularize the cost body to obtain the probability body, normalize the probability value of the probability body using a normalization exponential function, and regress the depth map. The training module is used for joint training of the probabilistic volume loss and the depth value loss of the depth map; The step of calculating the weight information of the total aggregated features based on the global average pooling result of the total aggregated features includes: according to the formula Calculate the weight information; where N is the number of input feature maps. This is the result of global average pooling on a single feature map. for The average value, For constant terms; The calculation of the weighted cost surface and obtaining the cost volume includes: performing a differentiable homography transformation on the total aggregate features to obtain multiple initial cost surfaces; according to the formula Calculate the weighted final cost surfaces separately; among them, The initial cost surface is used; the cost volume is obtained by superimposing and summing pixels at the same position along the depth direction of multiple final cost surfaces.

6. A recursive dense reconstruction server based on attention-enhanced matching, characterized in that, Including memory and processor; The memory is used to store computer-executable instructions; The processor is configured to execute the computer-executable instructions to implement the method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores executable instructions, which, when executed by a computer, enable the implementation of the method as described in any one of claims 1-4.

Images