An Adaptive Matching Cost Construction Method and System Based on Multi-View Image Features

Abstract

This invention discloses an adaptive matching cost construction method and system based on multi-view image features, belonging to the field of stereo matching and 3D reconstruction of multi-view images. The method includes: performing pairwise matching of multi-view image features and concatenating two matched views along the feature dimension to obtain image feature combinations of the two matched views; extracting similarity features between views from the image feature combinations; combining the similarity features between each pair of matched views into a similarity feature sequence, and calculating the normalized weights corresponding to different pairs of matched views through a soft-max operation; and performing a normalized weighted summation of multiple initial cost volumes of matched views using their corresponding normalized weights to generate the multi-view matching cost. This invention comprehensively considers the differences between different views, enhancing the robustness of the constructed matching cost, especially in weakly textured regions where matching is difficult, resulting in a significant improvement in predicted depth and scene reconstruction quality.

Description

Technical Field

[0001] This invention belongs to the field of stereo matching and 3D reconstruction of multi-view images, and more specifically, relates to an adaptive matching cost construction method and system based on multi-view image features. Background Technology

[0002] Multi-view stereo matching technology is used for image-based 3D scene reconstruction. Specifically, it reconstructs the corresponding 3D structure of a scene from a sequence of images from multiple viewpoints of the same scene. During camera imaging, real-world 3D points are mapped to 2D points in the image, resulting in the loss of depth information. Therefore, reconstructing the 3D structure from a 2D image requires solving for the corresponding depth information. The key to solving multi-view depth information is the matching relationship of the same 3D point in the scene across multiple imaging views. If the correct matching relationship can be obtained, the correct depth can be calculated using differences in imaging positions and camera parameters.

[0003] Currently, deep learning-based multi-view stereo matching algorithms, such as MVSNet, have achieved good reconstruction results and have a significant speed advantage. The key to deep learning-based methods lies in constructing a multi-view matching cost, and then regularizing the matching cost using a 3D convolutional neural network to predict the correct depth. Current methods construct a cost volume between every two views using homography transformation, and then aggregate multiple cost volumes between two views using variance metric to construct the matching cost volume. This method is relatively simple and flexible, applicable to any number of views as input. Its premise is that the matching cost between all views is equal, thus eliminating the differences in matching costs between different views through variance calculation. However, due to differences in camera position and imaging angle between different views, the imaging of the same 3D point differs across views. Ignoring these differences in the matching cost construction results in insufficient robustness, leading to inaccurate depth maps inferred from this matching cost, especially in weakly textured (i.e., large areas of repetitive texture) regions, where large-scale errors occur, affecting the quality of the final reconstructed 3D scene structure. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides an adaptive matching cost construction method and system based on multi-view image features. Its purpose is to solve the technical problem that the constructed matching cost is not robust enough due to ignoring the differences between views, resulting in inaccurate predicted depth and affecting the quality of the final reconstructed 3D scene structure.

[0005] To achieve the above objectives, according to one aspect of the present invention, an adaptive matching cost construction method based on multi-view image features is provided, comprising:

[0006] S1. Perform pairwise matching of multi-view image features and stitch the two matched views together in the feature dimension to obtain the image feature combination of the two matched views;

[0007] S2. Obtain similarity features between views from image feature combinations;

[0008] S3. Combine the similarity features between each pair of matched views into a similarity feature sequence, and calculate the normalized weights corresponding to different pairs of matched views through a soft-max operation;

[0009] S4. The initial cost bodies of multiple matching views are normalized and weighted by their corresponding normalized weights to generate the multi-view matching cost.

[0010] Furthermore, convolutional neural networks are used to obtain similarity features between views from image feature combinations or to calculate the SSIM structural similarity between two views.

[0011] Furthermore, the convolutional neural network has a symmetrical U-shaped structure, including seven convolutional layers, each followed by a batch normalization layer; the first and second convolutional layers downsample the features to 1 / 4 and 1 / 16 respectively, and the fourth and fifth convolutional layers upsample the features to 1 / 4 and the original size respectively, and perform skip connection operations on the downsampled features to perform cross-layer feature fusion.

[0012] Furthermore, the cost of multi-view matching is:

[0013]

[0014] W i V represents the aggregation weight corresponding to the source viewpoint. i is the initial matching cost corresponding to the source viewpoint, and n represents the number of source viewpoints.

[0015] According to another aspect of the present invention, an adaptive matching cost construction system based on multi-view image features is provided, comprising:

[0016] The matching and stitching module is used to match the features of multi-view images one by one, and stitch the two matched views in the feature dimension to obtain the image feature combination of the two matched views;

[0017] The similarity feature acquisition module is used to obtain similarity features between views from the combination of image features;

[0018] The normalized weight calculation module is used to combine the similarity features between each pair of matched views into a similarity feature sequence, and calculate the normalized weights corresponding to different pairs of matched views through a soft-max operation.

[0019] The weighted summation module is used to perform normalized weighted summation on the initial cost bodies of multiple matching views using their corresponding normalized weights to generate multi-view matching costs.

[0020] Furthermore, the similarity feature acquisition module uses a convolutional neural network to obtain similarity features between views from the combination of image features or to calculate the SSIM structural similarity between two views.

[0021] Furthermore, the convolutional neural network has a symmetrical U-shaped structure, including seven convolutional layers, each followed by a batch normalization layer; the first and second convolutional layers downsample the features to 1 / 4 and 1 / 16 respectively, and the fourth and fifth convolutional layers upsample the features to 1 / 4 and the original size respectively, and perform skip connection operations on the downsampled features to perform cross-layer feature fusion.

[0022] Furthermore, the cost of multi-view matching is:

[0023]

[0024] W i V represents the aggregation weight corresponding to the source viewpoint. i is the initial matching cost corresponding to the source viewpoint, and n represents the number of source viewpoints.

[0025] Overall, the above-described technical solutions conceived by this invention can achieve the following beneficial effects compared with the prior art.

[0026] The matching cost construction method of this invention comprehensively considers the differences between different views. For more similar views, the aggregation weight of adaptive prediction is larger, thus enhancing the matching cost. Conversely, for views with lower similarity, the weight of adaptive prediction is smaller, thus suppressing the matching cost. Therefore, the robustness of the final constructed matching cost will be enhanced, especially in weakly textured regions where matching is difficult. Based on this, the predicted depth and scene reconstruction quality will be significantly improved. Attached Figure Description

[0027] Figure 1 This is an optional implementation flowchart of the method of the present invention.

[0028] Figure 2 This is a visual comparison of the depth estimation results of the present invention and existing methods.

[0029] Figure 3This is a visual comparison of the 3D scene reconstruction results of the present invention and existing methods. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0031] This invention provides an adaptive matching cost construction method based on multi-view image features, comprising: obtaining effective multi-view image features through a feature extraction network, then constructing an initial matching cost body by homography transformation of the multi-view image features; simultaneously matching the multi-view image features pairwise and concatenating the features of the two matched views in the feature dimension to obtain a combination of image features of the two matched views; and then mining the similarity between views from the combination of image features.

[0032] To extract similarities between views from combinations of image features, as a preferred embodiment, such as Figure 1 As shown, this invention designs a typical multi-layer convolutional neural network structure. Its input is a concatenation of multiple pairs of source and reference view image features. The network consists of seven (transposed) convolutional layers, each with a 3×3 kernel. A batch normalization layer and a rectified linear unit are added after each convolutional layer as activation functions. The entire convolutional neural network is designed with a symmetrical U-shaped structure. In the first and second layers, the convolutional stride is set to 2, downsampling the features to 1 / 4 and 1 / 16 of their original size, respectively. In the fourth and fifth layers, the features are upsampled to 1 / 4 and their original size, respectively. Skip connections are then used to perform cross-layer feature fusion on the downsampled features.

[0033] The present invention can also calculate the SSIM (structural similarity) between two views to obtain the similarity features between the views.

[0034] Multiple similarity features form a feature sequence, and then a soft-max operation is used to calculate and obtain the aggregate weight based on the similarity features for each source view, as shown in formula (1.1), where f i W is the similarity feature corresponding to the i-th source viewpoint. i The aggregate weight is calculated based on the similarity features of the i-th source viewpoint and through a soft-max operation.

[0035]

[0036] Then, the aggregation weight W corresponding to the source viewpoint is used. i The initial matching cost V corresponding to the source viewpoint i The final matching cost (Cost Volume) is constructed by performing weighted summation and normalization, as shown in Equation (1.2).

[0037]

[0038] Compared with existing methods for matching cost aggregation and construction based on variance aggregation, the method of this invention constructs a more robust matching cost that can estimate more accurate depth in weakly textured regions where matching is difficult, thereby improving the quality of depth estimation and the quality of 3D scene reconstruction. Figure 2 As shown, the boxes mark weakly textured regions that are difficult to match. The first column shows the depth map estimated using the matching cost constructed with existing methods and the corresponding confidence map. The second column shows the depth map estimated using the robust matching cost constructed using the method of this invention and the corresponding confidence map. It can be seen that the more robust matching cost constructed using this invention can significantly improve the depth estimation quality of the weakly textured regions marked with boxes that are difficult to match. Correspondingly, the depth confidence map also reflects the improvement in depth estimation quality. In addition, as... Figure 3 As shown in the figure, the boundary region integrity of the reconstruction results obtained by using the robust matching cost construction method proposed in this paper for multi-view stereo matching reconstruction is significantly improved, as shown in the area marked by the black solid line box.

[0039] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for constructing adaptive matching cost based on multi-view image features, characterized in that, Comprising: S1. performing pairwise matching on multi-view image features, and performing stitching on two matched views in feature dimension to obtain an image feature combination of the two matched views; S2. obtaining similarity features between views from the image feature combination; S3. combining the similarity features between each pair of matched views into a similarity feature sequence, and calculating normalized weights corresponding to different two matched views through a soft-max operation; S4. performing normalized weighted summation on multiple matched view initial cost volumes using the corresponding normalized weights to generate a multi-view matching cost; wherein the similarity features between views are obtained from the image feature combination or the SSIM structural similarity between two views is calculated using a convolutional neural network; the convolutional neural network is a symmetrical U-shaped structure, comprising seven convolutional layers, and a batch normalization layer is connected after each convolutional layer; the first convolutional layer and the second convolutional layer down-sample the features to 1 / 4 and 1 / 16, the fourth convolutional layer and the fifth convolutional layer up-sample the features to 1 / 4 and the original size, and a jump connection operation is performed on the down-sampled features to perform cross-level feature fusion; the multi-view matching cost is: a aggregated weight corresponding to a source view point, an initial matching cost corresponding to a source view point, n represents the number of source view points.

2. A multi-view image feature based adaptive matching cost construction system, characterized by, Comprising: a matching and stitching module configured to perform pairwise matching on multi-view image features, and perform stitching on two matched views in feature dimension to obtain an image feature combination of the two matched views; a similarity feature obtaining module configured to obtain similarity features between views from the image feature combination; a normalized weight calculating module configured to combine the similarity features between each pair of matched views into a similarity feature sequence, and calculate normalized weights corresponding to different two matched views through a soft-max operation; a weighted summation module configured to perform normalized weighted summation on multiple matched view initial cost volumes using the corresponding normalized weights to generate a multi-view matching cost; wherein the similarity feature obtaining module obtains the similarity features between views from the image feature combination or calculates the SSIM structural similarity between two views using a convolutional neural network; the convolutional neural network is a symmetrical U-shaped structure, comprising seven convolutional layers, and a batch normalization layer is connected after each convolutional layer; the first convolutional layer and the second convolutional layer down-sample the features to 1 / 4 and 1 / 16, the fourth convolutional layer and the fifth convolutional layer up-sample the features to 1 / 4 and the original size, and a jump connection operation is performed on the down-sampled features to perform cross-level feature fusion; the multi-view matching cost is: a aggregated weight corresponding to a source view point, an initial matching cost corresponding to a source view point, n represents the number of source view points.

3. An electronic device, comprising: Comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is executed by the processor to implement the steps of the method of claim 1.

Images