Large-scale scene multi-view reconstruction method based on global depth regularization

Abstract

The embodiment of the invention provides a large-scale scene multi-view reconstruction method based on global depth regularization, and the method comprises the steps: inputting a reference image and a plurality of source images in a multi-view view into a cascade feature extraction network, so as to obtain a multi-scale feature map and a multi-scale weight matrix of the reference image and each source image; and inputting the multi-scale feature maps and the multi-scale weight matrixes of the reference image and each source image into a global dynamic propagation network to obtain a first probability body. And inputting the first probability body into a depth perception regularization network to obtain a second probability body of the reference image. And determining a depth map of the reference image according to the second probability body, and performing multi-view reconstruction according to the depth map of the reference image. The embodiment of the invention can improve the reconstruction precision.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image processing, in particular, to a large-scale scene multi-view reconstruction method based on global depth regularization. BACKGROUND

[0002] With the development of technology, the multi-view reconstruction method has been widely used in three-dimensional reconstruction, city modeling, remote sensing mapping and other fields, especially in high-precision and large-scale scene reconstruction, which shows important value. The multi-view reconstruction method obtains multi-view views from multiple perspectives and restores complex three-dimensional structures according to the multi-view views to make up for the lack of single-view images in spatial information expression, and provides key technical support for intelligent manufacturing, virtual reality and other applications.

[0003] At present, the learning-based multi-view reconstruction method gradually replaces the traditional algorithm, which relies on deep neural networks to extract the potential geometric information between multi-view views, thereby improving the matching accuracy and reconstruction quality. The existing learning-based multi-view reconstruction method mostly realizes multi-view reconstruction through feature extraction (single-scale feature extraction), feature matching and local regularization of multi-view views.

[0004] However, in the existing learning-based multi-view reconstruction method, only the extracted single-scale features are used as the data basis for subsequent matching and regularization, which has the problem of single data, reducing the reconstruction accuracy. In addition, in large-scale scene reconstruction, local regularization is adopted, which is difficult to adapt to multi-scale geometric changes, resulting in unstable depth estimation and reducing the reconstruction accuracy. SUMMARY

[0005] Embodiments of the present application provide a large-scale scene multi-view reconstruction method based on global depth regularization, which can improve the reconstruction accuracy.

[0006] In a first aspect, the embodiments of the present application provide a large-scale scene multi-view reconstruction method based on global depth regularization, comprising: obtaining multi-view images, wherein the multi-view images comprise a reference image and a plurality of source images; inputting the reference image and the plurality of source images into a cascaded feature extraction network of a multi-view reconstruction model respectively, to obtain multi-scale feature maps and multi-scale weight matrices of the reference image, and multi-scale feature maps and multi-scale weight matrices of each of the source images; wherein the multi-scale feature maps of the reference image correspond one-to-one to the multi-scale weight matrices of the reference image, and the multi-scale feature maps of the source images correspond one-to-one to the multi-scale weight matrices of the source images; the weight matrix of one scale is used to indicate the importance of the semantics of each feature point in the corresponding scale feature map; the cascaded feature extraction network is used to perform multi-scale cascaded feature extraction on the reference image and the plurality of source images based on residual connection and spatial attention mechanism respectively; inputting the multi-scale feature maps and multi-scale weight matrices of the reference image, and the multi-scale feature maps and multi-scale weight matrices of each of the source images into a global dynamic propagation network of the multi-view reconstruction model, to obtain a first probability volume; wherein the first probability volume is used to indicate the probability of each pixel in the reference image being at different depths respectively; the global dynamic propagation network is used to determine, for each of the source images, a first cost volume of the source image according to the multi-scale feature maps and multi-scale weight matrices of the source image, and the multi-scale feature maps and multi-scale weight matrices of the reference image, and fuse the first cost volumes of the plurality of source images to obtain the first probability volume; wherein the first cost volume of the source image is used to indicate the matching cost of the source image and the reference image; inputting the first probability volume into a depth-aware regularization network of the multi-view reconstruction model, to obtain a second probability volume; wherein the depth-aware regularization network is used to regularize the first probability volume by a first convolution kernel to obtain a third probability volume, and regularize a fusion of the third probability volume and the first probability volume by a second convolution kernel to obtain a fourth probability volume, and regularize a fusion of the fourth probability volume and the first probability volume by a third convolution kernel to obtain the second probability volume; wherein the first convolution kernel is larger than the second convolution kernel, and the second convolution kernel is larger than the third convolution kernel; determining a depth map of the reference image according to the second probability volume, and performing multi-view reconstruction according to the depth map of the reference image.

[0007] Optionally, the global dynamic flow network comprises: a deformation network configured to deform, in a depth direction, a multi-scale feature map of the reference image and a multi-scale feature map of each of the source images respectively to obtain a multi-scale feature volume of the reference image and a multi-scale feature volume of each of the source images; and deform, in the depth direction, a multi-scale weight matrix of the reference image and a multi-scale weight matrix of each of the source images respectively to obtain a multi-scale weight volume of the reference image and a multi-scale weight volume of each of the source images; a cost volume determination network configured to determine, for each of the source images, a same-scale second cost volume of the source image according to the same-scale feature volume of the source image and the reference image to obtain a multi-scale second cost volume of the source image; wherein the second cost volume of one scale of the source image is used to indicate a matching cost of the same-scale feature volume of the source image and the reference image; a correction network configured to correct, for each of the source images, the same-scale second cost volume of the source image according to the same-scale weight volume of the source image and the reference image to obtain a multi-scale third cost volume of the source image; a first fusion network configured to fuse, for each of the source images, the multi-scale third cost volume of the source image through a cross-scale attention mechanism to obtain a first cost volume of the source image; and a second fusion network configured to fuse the first cost volumes of the plurality of source images to obtain the first probability volume.

[0008] Optionally, the depth-aware regularization network comprises: a first regularization network configured to regularize the first probability volume through the first convolution kernel and in combination with epipolar attention and depth gradient-aware normalization to obtain a third probability volume; a second regularization network configured to regularize a fusion volume of the third probability volume and the first probability volume according to the second convolution kernel and in combination with depth gradient-aware normalization to obtain a fourth probability volume; and a third regularization network configured to regularize a fusion volume of the fourth probability volume and the first probability volume according to the third convolution kernel and in combination with depth gradient-aware normalization to obtain the second probability volume.

[0009] Optionally, the determining the depth map of the reference image according to the second probability volume comprises: obtaining, from the second probability volume, a probability that each pixel in the reference image is at a different depth; determining, for each pixel, a depth with the largest probability as a depth of the pixel; and constructing a depth map of the reference image according to the depth of each pixel in the reference image.

[0010] Optionally, before the obtaining the multi-view image, the method further comprises: obtaining a multi-view reconstruction data set, wherein the multi-view reconstruction data set comprises a plurality of sample data, each of the sample data comprises a multi-view sample image, a second sample probability volume, and a third sample probability volume; dividing the multi-view reconstruction data set into a training data set and a testing data set; training the multi-view reconstruction model in an initial state by using a stochastic gradient descent method or an adaptive moment estimation, and by minimizing a loss function and the training data set; and adjusting parameters of the trained multi-view reconstruction model by using the testing data set, to obtain the multi-view reconstruction model.

[0011] Optionally, the training the multi-view reconstruction model in an initial state by using a stochastic gradient descent method or an adaptive moment estimation, and by minimizing a loss function and the training data set comprises: enhancing the training data set; and training the multi-view reconstruction model in an initial state by using a stochastic gradient descent method or an adaptive moment estimation, and by minimizing a loss function and the enhanced training data set.

[0012] In a second aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor; the memory is configured to store a computer program; the processor is configured to implement the multi-view reconstruction method based on global depth regularization for large-scale scenes according to any one of the first aspect when executing the computer program.

[0013] In a third aspect, an embodiment of the present application provides a computer readable storage medium, the storage medium stores a computer program, and when the computer program is executed by a processor, the multi-view reconstruction method based on global depth regularization for large-scale scenes according to any one of the first aspect is implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a chip system, comprising a processor, the processor is coupled with a memory, the processor executes a computer program stored in the memory, and the multi-view reconstruction method based on global depth regularization for large-scale scenes according to any one of the first aspect is implemented.

[0015] The multi-view reconstruction method based on global depth regularization for large-scale scenes according to the embodiments of the present application has the following advantages:

[0016] By inputting the reference image and the plurality of source images into the cascade feature extraction network of the multi-view reconstruction model respectively, not only the multi-scale feature maps of the reference image and each source image can be obtained, but also the multi-scale weight matrices of the reference image and each source image (i.e. the importance of the semantics of each feature point in the feature maps of each scale of the reference image and the source image) can be obtained. That is, by feature extraction, not only the feature maps are obtained, but also the semantics of each feature point in the feature maps are obtained. Compared with the existing single-scale feature extraction, rich data are provided for subsequent reconstruction, thereby improving the reconstruction accuracy.

[0017] In addition, the multi-scale cascade feature extraction is performed on the reference image and the plurality of source images based on the residual connection and the spatial attention mechanism respectively. Since the residual connection can alleviate the feature degradation of the deep network in the multi-scale cascade feature extraction, the effective features of the shallow layer can be transmitted to the deep layer, the integrity of the features in the cascade process is ensured, the spatial attention mechanism can focus on the key spatial area and enhance the feature response of the important position, and the features obtained through the multi-scale cascade feature extraction are more targeted. In this way, the multi-scale feature maps of the reference image and each source image and the multi-scale weight matrices of the reference image and each source image can have the detailed information of different levels and the enhanced features of the key areas, and the feature transmission is stable and robust. In this way, rich-scale features and weight bases can be provided for subsequent reconstruction, which helps to capture geometric details at different levels, thereby improving the accuracy and integrity of the reconstruction.

[0018] In addition, for each source image, the first cost volume of the source image can be determined according to the multi-scale feature maps and the multi-scale weight matrices of the source image, the multi-scale feature maps and the multi-scale weight matrices of the reference image. The first cost volume of the source image is used to indicate the matching cost of the source image and the reference image. That is, the construction of the first cost volume of the source image takes into account the multi-scale feature maps and the multi-scale weight matrices of the source image and the reference image, enhances the robustness of the first cost volume at different scales, realizes the alignment of the semantics and features at different scales, and improves the accuracy of the first cost volume. In this way, by fusing the first cost volumes of the plurality of source images, the fusion of the first cost volumes of the source images at different angles is realized, that is, the interaction of the features and semantics across scales and angles is realized, the consistency modeling capability of the multi-angle view geometry structure is improved, the accuracy and stability of the first probability volume obtained by fusion are improved, that is, the accuracy and stability of the depth estimation are improved, thereby improving the accuracy and stability of the subsequent reconstruction.

[0019] And, since in the process of regularization, the first probability body is sequentially regularized by the large-to-small convolution kernel, and in the process of non-first regularization, the probability body obtained after the previous regularization is also substituted, that is, the global structure consistency is first strengthened and the global noise is suppressed by the large convolution kernel, and then the local details are refined by the small convolution kernel, and the previous results are combined to make the regularization not only guarantee the global rationality of the probability body, but also retain the key local features, so that the final obtained second probability body can simultaneously consider global features and local details, not only providing data support for subsequent view reconstruction of different scales, but also improving the accuracy and stability of subsequent reconstruction. And since the regularization is based on multiple convolution kernels of different sizes, compared with local regularization, the accuracy and stability of depth estimation can be further improved, and the reconstruction accuracy of large-scale scenes can be improved.

[0020] In this way, the depth map of the reference image is determined according to the second probability body, and the multi-view reconstruction is performed according to the depth map of the reference image, which can improve the accuracy and stability of the multi-view reconstruction, and also consider multi-view reconstruction of different scales. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 A flowchart of a large-scale scene multi-view reconstruction method based on global depth regularization provided by an embodiment of the present application is shown in the figure.

[0022] Figure 2 A structure diagram of a multi-view reconstruction model provided by an embodiment of the present application is shown in the figure.

[0023] Figure 3 A reconstruction effect diagram provided by an embodiment of the present application is shown in the figure.

[0024] Figure 4 A structure diagram of an electronic device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0025] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms, and should not be interpreted as being limited to the embodiments described herein, on the contrary, these embodiments are provided to make the present application more thorough and complete. It should be understood that the drawings and embodiments of the present application are only for illustrative purposes, and are not intended to limit the scope of protection of the present application.

[0026] It should be understood that each step described in the method embodiments of the present application can be executed in different order and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present application is not limited in this respect.

[0027] The term "include," and variations thereof, as used in this document, is open-ended and means "comprising" or "including" but not limited to; the term "based on," as used in this document, is the conventional attachment sense of "based on," that is, "on the basis of," or "whereupon" and not the causal sense of "based on" that is, "on the result of; the terms "a or "an," as used in this document, mean "one or more;" the term "another," as used in this document, means "at least one," i.e. one or more; the term "another" as used in this document when used in the context as "another embodiment" means "at least one additional embodiment." The term "some" as used in this document means "at least some" but not "all." The term "optional" as used in this document means "optional but not required." The related definitions further describe specific use cases for these terms in the description. It is to be noted that the terms "first," "second," and the like as used in this document are meant as identifiers, not as an indication of relative importance of the entities so designated. The terms "coupled" and "connected" as used in this document mean to be directly or indirectly connected or coupled.

[0028] It is to be noted that the terms "one," "multiple," and the like as used in this document are illustrative and not limiting, and those skilled in the art will understand that "one" or "a" should be interpreted as "one or more" unless the context clearly indicates otherwise.

[0029] The names of the messages or information exchanged between the devices in the embodiments of the present application are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0030] The embodiment of the present application provides a large-scale scene multi-view reconstruction method based on global depth regularization. The method can be executed by an electronic device, including but not limited to a notebook computer, a desktop computer, a tablet computer and other electronic devices with data processing capability. As shown in the figure, the method can include the following steps: Figure 1

[0031] S101. Obtain a multi-view view, wherein the multi-view view includes a reference image and a plurality of source images.

[0032] Illustratively, the multi-view view is a plurality of views obtained by photographing the first region based on different perspectives. The reference image can be any one of the multi-view view, and the plurality of source images are all views in the multi-view view except the reference image. The first region includes but is not limited to a school, a community, a factory, a park, etc.

[0033] Next, since the following text relates to a multi-view reconstruction model, the multi-view reconstruction model will be described before the following steps are described. As shown in the figure, the multi-view reconstruction model includes a cascaded feature extraction network 210, a global dynamic propagation network 220, and a depth perception regularization network 230. Since the operation process of each network will be described below, it will not be described here. Figure 2

[0034] ​​S102. input the reference image and the plurality of source images into a cascaded feature extraction network 210 of the multi-view reconstruction model respectively to obtain multi-scale feature maps and multi-scale weight matrices of the reference image and multi-scale feature maps and multi-scale weight matrices of each source image.

[0035] The cascaded feature extraction network 210 is configured to perform multi-scale cascaded feature extraction on the reference image and the plurality of source images based on residual connection and spatial attention mechanism respectively. Specifically, a convolution structure with shared weights is used to extract feature maps and weight matrices of different scales of the reference image and each source image respectively.

[0036] The multi-scale feature maps of the reference image correspond to the multi-scale weight matrices of the reference image one by one, and the multi-scale feature maps of the source image correspond to the multi-scale weight matrices of the source image one by one. The weight matrix of one scale is used to indicate the importance of the semantics of each feature point in the feature map of the corresponding scale (for example, the weight of the object edge and the texture-rich area is high, and the weight of the background is low). The higher the importance of the semantics of the feature point, the greater the influence of the feature point. The lower the importance of the semantics of the feature point, the less important the feature point, and the influence of the feature point is inhibited.

[0037] The residual connection can alleviate the feature degradation of the deep network in the multi-scale cascaded feature extraction, so that the effective features of the shallow layer can be transmitted to the deep layer, and the integrity of the features in the cascaded process is ensured. The spatial attention mechanism can focus on the key spatial area and enhance the feature response of the important position, so that the features extracted by the multi-scale cascaded feature extraction have more pertinence. In this way, by using the residual connection and the spatial attention mechanism in the multi-scale cascaded feature extraction, the multi-scale feature maps of the reference image and each source image and the multi-scale weight matrices of the reference image and each source image have both different levels of detail information and enhanced features of key areas, and the feature transmission is stable and has strong robustness.

[0038] S103. input the multi-scale feature maps and the multi-scale weight matrices of the reference image and the multi-scale feature maps and the multi-scale weight matrices of each source image into a global dynamic propagation network 220 of the multi-view reconstruction model to obtain a first probability volume.

[0039] The first probability volume is used to indicate the probability of each pixel in the reference image being at different depths.

[0040] The global dynamic propagation network 220 is configured to determine, for each source image, a first cost volume of the source image according to the multi-scale feature maps and the multi-scale weight matrices of the source image and the multi-scale feature maps and the multi-scale weight matrices of the reference image, and fuse the first cost volumes of the plurality of source images to obtain the first probability volume.

[0041] The first cost volume of the source image is used to indicate a matching cost of the source image and the reference image. Specifically, the matching cost of the source image and the reference image is used to indicate a difference of each corresponding pixel in the source image and the reference image at different depths. The smaller the difference of a corresponding pixel in the source image and the reference image at a depth is, the lower the matching cost of the corresponding pixel in the source image and the reference image at the depth is. The greater the difference of a corresponding pixel in the source image and the reference image at a depth is, the higher the matching cost of the corresponding pixel in the source image and the reference image at the depth is.

[0042] S104. Input the first probability volume to the depth-aware regularization network 230 of the multi-view reconstruction model to obtain a second probability volume.

[0043] The depth-aware regularization network 230 is configured to regularize the first probability volume by a first convolution kernel to obtain a third probability volume, regularize a fusion of the third probability volume and the first probability volume by a second convolution kernel to obtain a fourth probability volume, and regularize a fusion of the fourth probability volume and the first probability volume by a third convolution kernel to obtain the second probability volume. The first convolution kernel is larger than the second convolution kernel, and the second convolution kernel is larger than the third convolution kernel.

[0044] The fusion of the two probability volumes can be understood as fusing the probabilities of the same pixel in the two probability volumes at the same depth, so that the fused fusion volume also indicates the probabilities of each pixel in the reference image at different depths. It should be noted that for the fusion volume, the probability of a pixel at a depth is the weighted sum or average of the probabilities of the same pixel in the two probability volumes at the same depth.

[0045] S105. Determine a depth map of the reference image according to the second probability volume, and perform multi-view reconstruction according to the depth map of the reference image.

[0046] For example, determining the depth map of the reference image according to the second probability volume can include the following steps:

[0047] First, the probabilities of each pixel in the reference image at different depths are obtained from the second probability volume. Then, for each pixel, the depth with the maximum probability is determined as the depth of the pixel. Finally, according to the depth of each pixel in the reference image, a depth map of the reference image is constructed.

[0048] The process of performing multi-view reconstruction according to the depth map of the reference image may, for example, be: first generating an initial three-dimensional point cloud according to the depth map of the reference image, then estimating camera poses through feature matching between other views and the reference image, combining multi-view geometry constraints to expand the point cloud and optimize, to complete multi-view reconstruction.

[0049] It should be noted that the above process for multi-view reconstruction is only exemplary and does not limit the specific implementation process of multi-view reconstruction.

[0050] As can be seen from the above, by inputting the reference image and the plurality of source images into the cascaded feature extraction network of the multi-view reconstruction model respectively, not only the multi-scale feature maps of the reference image and each source image are obtained, but also the multi-scale weight matrices of the reference image and each source image are obtained (i.e. the importance of the semantics of each feature point in the feature maps of each scale of the reference image and the source image), that is, through feature extraction, not only the feature maps are obtained, but also the semantics of each feature point in the feature maps is obtained, compared with the existing single-scale feature extraction, rich data is provided for subsequent reconstruction, thereby improving the reconstruction accuracy.

[0051] Moreover, the multi-scale cascaded feature extraction is performed on the reference image and the plurality of source images based on the residual connection and the spatial attention mechanism respectively, since the residual connection can alleviate the feature degradation of the deep network in the multi-scale cascaded feature extraction, let the effective features of the shallow layer be transmitted to the deep layer, guarantee the integrity of the features in the cascaded process, and the spatial attention mechanism can focus on the key spatial area and enhance the feature response of the important position, so that the features obtained through the multi-scale cascaded feature extraction are more targeted. In this way, the multi-scale feature maps of the reference image and each source image and the multi-scale weight matrices of the reference image and each source image can have different levels of detail information and enhanced features of key areas, and the feature transmission is stable and robust. In this way, rich-scale features and weight bases can be provided for subsequent reconstruction, which helps to capture different levels of geometric details, so as to improve the accuracy and integrity of the reconstruction.

[0052] Moreover, for each source image, the first cost volume of the source image can be determined according to the multi-scale feature map and the multi-scale weight matrix of the source image, the multi-scale feature map and the multi-scale weight matrix of the reference image. The first cost volume of the source image is used to indicate the matching cost of the source image and the reference image. That is, the construction of the first cost volume of the source image takes into account the multi-scale feature map and the multi-scale weight matrix of the source image and the reference image, enhances the robustness of the first cost volume at different scales, realizes the alignment of the semantics and features at different scales, and improves the accuracy of the first cost volume. In this way, by fusing the first cost volumes of the plurality of source images, the fusion of the first cost volumes of the source images at different views is realized, that is, the interaction of the features and semantics across scales and views is realized, the consistency modeling capability of the multi-view view geometry structure is improved, the accuracy and stability of the first probability volume obtained by fusion are improved, that is, the accuracy and stability of the depth estimation are improved, thereby improving the accuracy and stability of the subsequent reconstruction.

[0053] And, in the process of regularization, the first probability body is sequentially regularized by the convolution kernel from large to small, and in the process of non-first regularization, the probability body obtained after the previous regularization is also substituted, that is, the global structure consistency is first strengthened and the global noise is inhibited by using a large convolution kernel, and then the local details are refined by using a small convolution kernel, and the previous results are combined to ensure the global rationality of the probability body and retain key local features, so that the second probability body obtained can simultaneously consider global features and local details, not only providing data support for subsequent view reconstruction of different scales, but also improving the accuracy and stability of subsequent reconstruction. And, since the regularization is based on multiple convolution kernels of different sizes, compared with local regularization, the accuracy and stability of depth estimation can be further improved, and the reconstruction accuracy of large-scale scenes can be improved.

[0054] In this way, the depth map of the reference image is determined according to the second probability body, and the multi-view reconstruction is performed according to the depth map of the reference image, which can improve the accuracy and stability of the multi-view reconstruction, and also consider multi-view reconstruction of different scales.

[0055] It should be noted that the large-scale scene can be understood as a large-scale scene.

[0056] In some possible implementation manners, the global dynamic relay network 220 can include: a deformation network 221, a cost volume determination network 222, a correction network 223, a first fusion network 224, and a second fusion network 225.

[0057] The deformation network 221 is configured to deform, in the depth direction, the multi-scale feature maps of the reference image and the multi-scale feature maps of each source image respectively, to obtain the multi-scale feature bodies of the reference image and the multi-scale feature bodies of each source image. And, in the depth direction, deform the multi-scale weight matrices of the reference image and the multi-scale weight matrices of each source image respectively, to obtain the multi-scale weight bodies of the reference image and the multi-scale weight bodies of each source image.

[0058] It can be understood that the multi-scale feature body of the reference image and the multi-scale feature Figure 1 correspond, that is, the feature body of one scale of the reference image is obtained by deforming the feature map of the scale of the reference image in the depth direction, and the deformation can be understood as transforming the feature points in the feature map from (H, W) to (H, W, D), where the value range of D is pre-set.

[0059] It should be noted that the obtaining manners of the multi-scale feature bodies and the multi-scale weight bodies of the source image, and the multi-scale weight bodies of the reference image can be referred to the above description, which will not be described here.

[0060] The cost volume determination network 222 is configured to determine, for each source image, a second cost volume of a same scale of the source image according to the feature volume of the same scale of the source image and the reference image, to obtain a multi-scale third cost volume of the source image.

[0061] The second cost volume of one scale of the source image is used to indicate a matching cost of the feature volume of the same scale of the source image and the reference image. Specifically, the matching cost of the feature volume of the same scale of the source image and the reference image can be used to indicate a difference of each pair of corresponding feature points in the feature volume of the same scale of the source image and the reference image at different depths. The smaller the difference of each pair of corresponding feature points in the feature volume of the same scale of the source image and the reference image at a depth is, the lower the matching cost of the pair of corresponding feature points in the feature volume of the same scale of the source image and the reference image at the depth is. The greater the difference of each pair of corresponding feature points in the feature volume of the same scale of the source image and the reference image at a depth is, the higher the matching cost of the pair of corresponding feature points in the feature volume of the same scale of the source image and the reference image at the depth is.

[0062] Specifically, the feature points in the feature volume of the same scale of the source image can be re-projected into the feature volume of the same scale of the reference image by using the disparity assumption, and then the matching cost of each pair of corresponding feature points at different depths can be determined by calculating the similarity (for example, by calculating the similarity by inner product or difference) of each pair of corresponding feature points at different depths, to obtain the second cost volume of the same scale of the source image. It should be noted that the higher the similarity is, the smaller the difference is, and the lower the similarity is, the greater the difference is.

[0063] The correction network 223 is configured to correct, for each source image, the second cost volume of the same scale of the source image according to the weight volume of the same scale of the source image and the reference image, to obtain a multi-scale third cost volume of the source image.

[0064] Specifically, the matching cost of each pair of corresponding feature points at different depths in the second cost volume of the same scale of the source image can be corrected according to the weight of each pair of corresponding feature points at different depths in the weight volume of the same scale of the source image and the reference image, to obtain the multi-scale third cost volume of the source image.

[0065] The first fusion network 224 is configured to fuse, for each source image, the multi-scale third cost volume of the source image by using a cross-scale attention mechanism, to obtain a first cost volume of the source image.

[0066] For example, the features in the third cost volume of the source image in multiple scales are dynamically aligned through the cross-scale attention mechanism, and the information in the third cost volume of the source image in a small scale is propagated to the third cost volume of the source image in a large scale through the skip connection, so that the propagation ability of the global constraint is enhanced, and the detail information is reserved. For example, the third cost volumes of the source image in multiple scales are sorted in the order of scales from small to large, the propagation weights of the third cost volumes of the source image ranked in the first two positions are generated through the cross-scale attention mechanism, the third cost volumes of the source image ranked in the first two positions are weighted and fused through the propagation weights, the propagation weights of the fused cost volume and the third cost volume of the source image ranked in the third position are generated through the cross-scale attention mechanism, the fused cost volume and the third cost volume of the source image ranked in the third position are weighted and fused based on the propagation weights, and the foregoing process is repeated until the fusion of the third cost volumes of the source image in multiple scales is completed, so as to obtain the first cost volume of the source image.

[0067] It should be noted that the third cost volume indicates the feature points, and the first cost volume indicates the pixels, so in the foregoing process, the matching cost of the feature points in different depths after fusion needs to be mapped to the pixels to determine the matching cost of the mapped pixels in different depths.

[0068] The second fusion network 225 is configured to fuse the first cost volumes of the plurality of source images to obtain a first probability volume.

[0069] For example, the first cost volumes of the plurality of source images are first aligned in the spatial and depth dimensions to eliminate the perspective difference, and then the matching cost of the corresponding pixels and depths is combined through a strategy such as weighted average, minimum value or weighted based on matching confidence, to obtain a fused cost volume, and the fused cost volume is converted into a first probability volume, that is, the matching cost of the pixels in different depths is converted into the probability of the pixels in different depths. Specifically, the lower the matching cost, the higher the probability, and the higher the matching cost, the lower the probability.

[0070] In a possible implementation, the depth-aware regularization network 230 can include a first regularization network 231, a second regularization network 232 and a third regularization network 233, where:

[0071] The first regularization network 231 is configured to regularize the first probability volume through a first convolution kernel and in combination with the epipolar attention and the depth gradient-aware normalization to obtain a third probability volume.

[0072] The second regularization network 232 is configured to regularize the fusion of the third probability volume and the first probability volume according to a second convolution kernel and in combination with the depth gradient-aware normalization to obtain a fourth probability volume.

[0073] The third regularization network 233 is configured to regularize the fusion of the fourth probability volume and the first probability volume according to a third convolution kernel and in combination with deep gradient-aware normalization, to obtain a second probability volume.

[0074] Based on the epipolar attention and in combination with the regularization process of the larger convolution kernel (i.e., the first convolution kernel), the full receptive field can be covered to enhance the modeling capability of the global spatial structure. Based on the regularization process of the smaller convolution kernel, the detailed features near the depth edges can be preserved. The deep gradient-aware normalization is introduced to adaptively adjust the activation value according to the depth distribution statistical characteristics of the current pixel, and to improve the prediction accuracy at the edge region (such as occlusion and depth mutation). In this way, the accuracy of the second probability volume is improved, and the multi-scale adaptability is improved.

[0075] In a possible implementation, before obtaining the multi-view views, the following steps can be further included:

[0076] First, a multi-view reconstruction data set is obtained; wherein the multi-view reconstruction data set includes a plurality of sample data, each sample data including a multi-view sample view, a second sample probability volume to a fourth sample probability volume, and the multi-view sample view including a reference sample image and a plurality of source sample images.

[0077] It should be noted that the second sample probability volume to the fourth sample probability volume can refer to the related description of the second probability volume to the fourth probability volume above, which will not be repeated here.

[0078] Then, the multi-view reconstruction data set is divided into a training data set and a test data set.

[0079] For example, eighty percent can be taken as the training data set, and twenty percent can be taken as the test training set.

[0080] Next, the random gradient descent method or the adaptive moment estimation is used to train the multi-view reconstruction model in the initial state by minimizing the loss function and the training data set.

[0081] Finally, the parameters of the trained multi-view reconstruction model are adjusted through the test data set to obtain the multi-view reconstruction model.

[0082] For example, the parameters of the trained multi-view reconstruction model can be adjusted according to the test results of the test data set. For example, the learning rate, momentum and other parameters are adjusted to realize the continuous optimization of the trained multi-view reconstruction model, and the parameter configuration with the best effect on the test data set is obtained as the final multi-view reconstruction model.

[0083] In a possible implementation, the training of the multi-view reconstruction model in the initial state by minimizing the loss function and the training data set can include:

[0084] First, the training data set is augmented.

[0085] For example, the multi-view sample views in the training data set can be augmented by flipping and rotating, and the ground truth in the training data set can be augmented by direct loading.

[0086] Then, the multi-view reconstruction model in the initial state is trained by minimizing the loss function and the augmented training data set using the stochastic gradient descent method or the adaptive moment estimation.

[0087] Obviously, by augmenting the data, the diversity of the data can be improved, and the accuracy of the trained model can be higher.

[0088] For example, Table 1 shows the comparison results of different reconstruction methods. The reconstruction methods include EG-MVSNet, AggrMVS, RingMo-Aerial, and the reconstruction method provided in the application. It should be noted that by comparing the values of the three performance parameters MAE, ACC-06 and Inter-3, the performance of various algorithms in the reconstruction task can be intuitively evaluated. Therefore, the reconstruction performance of different reconstruction methods is mainly compared based on the three performance parameters MAE, ACC-06 and Inter-3.

[0089] Method MAE ACC-06 Inter-3 EG-MVSNet 0.097 98.0 98.2 AggrMVS 0.154 98.1 98.1 RingMo-Aerial 0.154 97.4 98.3 The method presented in the present application 0.091 98.5 98.6

[0090] Generally, the lower the value of MAE and the higher the values of ACC-06 and Inter-3, the better the algorithm performs in the task.

[0091] Thus, from Table 1, it can be seen that the method provided in the application is superior to other algorithms in the above three indicators.

[0092] It should be noted that the performance simulation and comparison process described above is simulated on a central processing unit of i7-6800K 3.40GHz CPU, NVIDIA GeForce GTX 3090 GPU, Ubuntu operating system, using Python software and PyTorch deep learning framework. The WHU-MVS data set is selected for algorithm verification. The WHU-MVS data set is collected under real conditions at a long distance and consists of 1776 images, each with a resolution of 684x512 pixels.

[0093] For example, Figure 3The reconstruction effect schematic diagram provided by the embodiment of the present application can be known from Figure 3 In the overall result display, the overall appearance is displayed, the global feature is reserved, in the local zoom display, the local details can be clearly displayed, the detail feature is reserved, therefore, the embodiment of the present application can meet the multi-scale reconstruction requirement.

[0094] As shown in Figure 4 The electronic device 400 provided by the embodiment of the present application can include a processor 410 and a memory 420; the memory 420 is used for storing a computer program; the processor 410 is used for realizing the large-scale scene multi-view reconstruction method based on global depth regularization as described above when the computer program is executed.

[0095] The computer readable storage medium provided by the embodiment of the present application has a computer program stored thereon, when the computer program is executed by the processor, the large-scale scene multi-view reconstruction method based on global depth regularization as described above is realized.

[0096] The chip system provided by the embodiment of the present application includes a processor, the processor is coupled with a memory, the processor executes the computer program stored in the memory, and realizes the large-scale scene multi-view reconstruction method based on global depth regularization as described above.

[0097] Now, the electronic device 400 which can be a server or a client of the present application will be described, which is an example of a hardware device that can be applied to various aspects of the present application. The electronic device 400 is intended to represent various forms of digital electronic computing devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device 400 can also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections, and their functions, as described herein, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed herein.

[0098] The electronic device 400 includes a computing unit, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) or a computer program loaded into a random access memory (RAM) from a storage unit. In the RAM, various programs and data required for device operation can also be stored. The computing unit, the ROM, and the RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0099] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware, and the program can be stored in a computer readable storage medium. When the program is executed, the program can include the processes of the above-mentioned embodiment methods. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM), a random access memory (RAM), or the like. In this application, the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present application. In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically independently, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0100] Although the present application is disclosed as above, the protection scope of the present application is not limited to this. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application, and these changes and modifications will fall within the protection scope of the present application.

Claims

1. A method for large-scale scene multi-view reconstruction based on global depth regularization, characterized in that, The method comprises: obtaining multi-view images, wherein the multi-view images comprise a reference image and a plurality of source images; inputting the reference image and the plurality of source images into a cascaded feature extraction network of a multi-view reconstruction model respectively to obtain multi-scale feature maps and multi-scale weight matrices of the reference image, and multi-scale feature maps and multi-scale weight matrices of each of the source images; wherein the multi-scale feature maps of the reference image correspond to the multi-scale weight matrices of the reference image one by one, and the multi-scale feature maps of the source images correspond to the multi-scale weight matrices of the source images one by one; the weight matrix of one scale is used to indicate the importance of the semantics of each feature point in the corresponding scale feature map; the cascaded feature extraction network is used for multi-scale cascaded feature extraction of the reference image and the plurality of source images based on residual connection and spatial attention mechanism respectively; inputting the multi-scale feature maps and multi-scale weight matrices of the reference image, and the multi-scale feature maps and multi-scale weight matrices of each of the source images into a global dynamic propagation network of the multi-view reconstruction model to obtain a first probability volume; wherein the first probability volume is used to indicate the probability of each pixel in the reference image being at different depths respectively; the global dynamic propagation network is used to determine, for each of the source images, a first cost volume of the source image according to the multi-scale feature maps and multi-scale weight matrices of the source image, the multi-scale feature maps and multi-scale weight matrices of the reference image, and fuse the first cost volumes of the plurality of source images to obtain the first probability volume; wherein the first cost volume of the source image is used to indicate the matching cost of the source image and the reference image; inputting the first probability volume into a depth-aware regularization network of the multi-view reconstruction model to obtain a second probability volume; wherein the depth-aware regularization network is used to regularize the first probability volume by a first convolution kernel to obtain a third probability volume, regularize a fusion of the third probability volume and the first probability volume by a second convolution kernel to obtain a fourth probability volume, and regularize a fusion of the fourth probability volume and the first probability volume by a third convolution kernel to obtain the second probability volume; wherein the first convolution kernel is larger than the second convolution kernel, and the second convolution kernel is larger than the third convolution kernel; determining a depth map of the reference image according to the second probability volume, and performing multi-view reconstruction according to the depth map of the reference image.

2. The method of claim 1, wherein, The global dynamic propagation network comprises: a deformation network configured to deform, in a depth direction, a multi-scale feature map of the reference image and a multi-scale feature map of each of the source images respectively to obtain a multi-scale feature volume of the reference image and a multi-scale feature volume of each of the source images, and deform, in the depth direction, a multi-scale weight matrix of the reference image and a multi-scale weight matrix of each of the source images respectively to obtain a multi-scale weight volume of the reference image and a multi-scale weight volume of each of the source images; a cost volume determination network configured to determine, for each of the source images, a second cost volume of a same scale of the source image according to the same scale feature volume of the source image and the reference image to obtain a multi-scale second cost volume of the source image; wherein the second cost volume of one scale of the source image is used to indicate a matching cost of the same scale feature volume of the source image and the reference image; a correction network configured to correct, for each of the source images, the second cost volume of the same scale of the source image according to the same scale weight volume of the source image and the reference image to obtain a multi-scale third cost volume of the source image; a first fusion network configured to fuse, for each of the source images, the multi-scale third cost volume of the source image through a cross-scale attention mechanism to obtain a first cost volume of the source image; a second fusion network configured to fuse the first cost volumes of the plurality of source images to obtain the first probability volume.

3. The method of claim 1, wherein, The depth-aware regularization network comprises: a first regularization network configured to regularize the first probability volume through the first convolution kernel and in combination with a epipolar attention and a depth gradient-aware normalization to obtain a third probability volume; a second regularization network configured to regularize a fusion volume of the third probability volume and the first probability volume according to the second convolution kernel and in combination with the depth gradient-aware normalization to obtain a fourth probability volume; a third regularization network configured to regularize a fusion volume of the fourth probability volume and the first probability volume according to the third convolution kernel and in combination with the depth gradient-aware normalization to obtain a second probability volume.

4. The method of claim 1, wherein, The determination of the depth map of the reference image according to the second probability volume comprises: obtaining, from the second probability volume, a probability that each pixel in the reference image is at a different depth respectively; determining, for each pixel, a depth with the largest probability as the depth of the pixel; constructing a depth map of the reference image according to the depth of each pixel of the reference image.

5. The method of claim 1, wherein, The method further comprises, before the obtaining of the multi-view view: obtaining a multi-view reconstruction data set; wherein the multi-view reconstruction data set comprises a plurality of sample data, and each of the sample data comprises a multi-view sample view, a second sample probability volume to a fourth sample probability volume, and the multi-view sample view comprises a reference sample image and a plurality of source sample images; dividing the multi-view reconstruction data set into a training data set and a test data set; training the multi-view reconstruction model in an initial state by using a stochastic gradient descent method or an adaptive moment estimation, and by minimizing a loss function and the training data set; and outputting the multi-view reconstruction model trained. Adjusting parameters of the trained multi-view reconstruction model by the test dataset to obtain the multi-view reconstruction model.

6. The method of claim 4, wherein, The training of the multi-view reconstruction model in the initial state by minimizing the loss function and the training dataset using the stochastic gradient descent method or the adaptive moment estimation comprises: Augmenting the training dataset; The training of the multi-view reconstruction model in the initial state by minimizing the loss function and the augmented training dataset using the stochastic gradient descent method or the adaptive moment estimation.

7. An electronic device, comprising: comprising a memory and a processor; The memory is configured to store a computer program. The processor is configured to implement the method of any one of claims 1-6 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The storage medium has a computer program stored thereon, and the computer program, when executed by a processor, implements the method of any one of claims 1-6.

9. A chip system, characterized by The processor is coupled with the memory, and the processor executes a computer program stored in the memory to implement the method of any one of claims 1-6.

Images