Multi-view stereo matching method based on optical flow consistency

Abstract

This disclosure provides a multi-view stereo matching method based on optical flow consistency, belonging to the field of image processing technology. Specifically, it includes: performing downsampling using an image pyramid algorithm; calculating the depth map and normal vector map of each downsampled image; selecting the best neighboring image; calculating the optical flow motion vector of the downsampled image based on the dense optical flow algorithm and the best neighboring image; constructing optical flow consistency; jointly estimating the optical flow consistency and photometric consistency; recalculating the depth map and normal vector map of each downsampled image according to a cost function; comparing the obtained depth map and normal vector map using a detail restorer, and selecting the depth value and normal vector for each pixel using a selection function to obtain the final depth map and normal vector map for the next layer; finally, fusing the depth map and normal vector map into a three-dimensional point cloud model. This disclosure improves the accuracy and adaptability of image matching.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a multi-view stereo matching method based on optical flow consistency. Background Technology

[0002] Currently, multi-view stereo matching algorithms have been an important research topic in computer vision for many years. 3D reconstruction technology is now widely used in various fields such as autonomous driving, robotics, AR, and industrial manufacturing. Multi-view stereo matching is a commonly used passive reconstruction method in 3D reconstruction. It involves taking a set of multi-angle images of an object or scene, given the camera's internal parameters and pose, calculating the depth of each pixel through pose relationships to obtain a set of corresponding depth maps, and finally fusing these depth maps to reconstruct a 3D point cloud model of the object or scene.

[0003] Therefore, the core of multi-view stereo matching technology is to calculate the depth value of each pixel. A mainstream and efficient multi-view depth estimation method is the PatchMatch algorithm proposed by Barnes et al. The core principle of the PatchMatch algorithm is to randomly assign a depth value to each pixel, construct a matching window for that pixel, and search in the neighborhood or neighboring frames of the current frame through this matching window. Based on photometric consistency, a suitable depth value is obtained and propagated. Photometric consistency can accurately match the corresponding pixels in scenes with rich textures, calculating the correct depth value and normal vector. However, in areas with large planes and low texture (e.g., walls, tabletops, floors), the matching strategy based on photometric consistency often suffers from matching ambiguity due to the high similarity between pixels within the matching window and neighboring pixels, leading to local optima. Incorrect depth values ​​and normal vectors significantly reduce the completeness and accuracy of the reconstructed 3D model.

[0004] It is evident that there is an urgent need for a multi-view stereo matching method based on optical flow consistency that is both accurate and adaptable. Summary of the Invention

[0005] In view of this, the present disclosure provides a multi-view stereo matching method based on optical flow consistency, which at least partially solves the problems of poor accuracy and adaptability in the prior art.

[0006] This disclosure provides a multi-view stereo matching method based on optical flow consistency, including:

[0007] Step 1: Use the image pyramid algorithm to perform downsampling operations on the original image a preset number of times and at a preset scale to obtain multiple downsampled images;

[0008] Step 2: Calculate the depth map and normal map for each downsampled image, starting from the top-level downsampled image;

[0009] Step 3: Using stereo pair selection under the OpenMVS framework, select a set of neighboring images for each downsampled image. Score each neighboring image based on the camera center angle, resolution similarity, and common viewing area of ​​the two images. Select the neighboring image with the highest score as the best neighboring image of the corresponding downsampled image.

[0010] Step 4: Convert the downsampled image and its corresponding best neighbor image into grayscale images, and calculate the optical flow motion vector of the downsampled image based on the dense optical flow algorithm and the best neighbor image.

[0011] Step 5: Compare the similarity between the optical flow motion vector and the geometric motion vector obtained by using the projection relationship, and perform normalization processing to construct optical flow consistency.

[0012] Step 6: Jointly estimate the optical flow consistency and the photometric consistency of the PatchMatch algorithm to form a cost function;

[0013] Step 7: Recalculate the depth map and normal map of each downsampled image according to the cost function. After obtaining the depth map and normal map of the current layer, upsample them to the next layer.

[0014] Step 8: Between two adjacent image scales, the depth map and normal vector map obtained in Step 2 and Step 7 are compared using the detail restorer. After selecting the depth value and normal vector for each pixel using the selection function, the final depth map and normal vector map of the next layer are obtained. Steps 2 to 8 are repeated until the depth map and normal vector map of the original image are calculated. Finally, the depth map and normal vector map are fused into a three-dimensional point cloud model.

[0015] According to a specific implementation of this disclosure, the formula for calculating the optical flow motion vector is as follows:

[0016]

[0017] Where φ(Δp) is the weighting function of the pixel neighborhood, and A(p+Δp) and ΔB(p+Δp) are the polynomial expansion coefficients of the image. This is the optical flow motion vector.

[0018] According to a specific implementation of this disclosure, the formula for calculating optical flow consistency is as follows:

[0019]

[0020] Where × represents the cross product of vectors, and |·| represents the magnitude of the vector. The optical flow motion vector, This is the geometric motion vector.

[0021] According to a specific implementation of this disclosure, the expression for the cost function is as follows:

[0022] Cost = (1-ω)Cost photo +ωCost flow

[0023] Cost photo Cost is the cost term for photometric consistency matching. flow ω is the optical flow consistency matching cost term, and ω is a constant.

[0024] According to a specific implementation of this disclosure, the calculation formula for the selection function is as follows:

[0025]

[0026] Among them, C down (q) represents the matching cost of the upsampled pixel, C up (q) represents the pixel matching cost calculated at the current scale, and γ is a selection threshold.

[0027] The multi-view stereo matching scheme based on optical flow consistency in this embodiment includes: Step 1, using the image pyramid algorithm to perform downsampling operations on the original image a preset number of times and at a preset scale to obtain multiple downsampled images; Step 2, calculating the depth map and normal vector map of each downsampled image starting from the top-level downsampled image; Step 3, using stereo pair selection under the OpenMVS framework, selecting a group of neighboring images for each downsampled image, scoring each neighboring image based on the camera center angle, resolution similarity, and shared viewing area of ​​the two images, and selecting the neighboring image with the highest score as the best neighboring image of the corresponding downsampled image; Step 4, converting the downsampled image and its corresponding best neighboring image into grayscale images, and calculating the optical flow motion vector of the downsampled image based on the dense optical flow algorithm and the best neighboring image; Step 5, converting the optical flow... Step 6: The similarity of the motion vectors is compared with the geometric motion vectors obtained by projection relationship, and normalization is performed to construct the optical flow consistency. Step 7: The optical flow consistency and the photometric consistency of the PatchMatch algorithm are jointly estimated to form a cost function. Step 8: The depth map and normal vector map of each downsampled image are recalculated according to the cost function. After obtaining the depth map and normal vector map of the current layer, they are upsampled to the next layer. Step 9: Between two adjacent image scales, the depth map and normal vector map obtained in Step 2 and Step 7 are compared using the detail restorer. After selecting the depth value and normal vector for each pixel using the selection function, the final depth map and normal vector map of the next layer are obtained. Steps 2 to 8 are repeated until the depth map and normal vector map of the original image are calculated. Finally, the depth map and normal vector map are fused into a three-dimensional point cloud model.

[0028] The beneficial effects of this disclosure are as follows: The scheme of this disclosure incorporates an image pyramid algorithm within the OpenMVS framework to solve the mismatch problem in the PatchMatch algorithm based on photometric consistency when reconstructing high-resolution images. Furthermore, it calculates the depth map and normal vector map starting from the smallest scale image, and then performs upsampling. The upsampled depth map and normal vector map can serve as reference values ​​for the next layer of calculation, which greatly increases the accuracy of depth and normal vector estimation in weakly textured regions. Based on optical flow, it proposes optical flow consistency and adds a detail restorer within adjacent scales to perform motion estimation for each pixel in the image, constructing a new matching cost term to compensate for the shortcomings of photometric consistency. The detail restorer can retain the correct estimation of weakly textured regions at the coarse scale and inherit the accurate calculation of details at the fine scale. The use of the detail restorer strengthens the connection between adjacent scales, ensuring geometric invariance across multiple scales, and to some extent solves the problems of the image pyramid algorithm. Finally, the improved algorithm can correctly estimate the depth of the input high-resolution image, thereby reconstructing a more complete and accurate 3D point cloud, improving the accuracy and adaptability of image matching. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 A flowchart illustrating a multi-view stereo matching method based on optical flow consistency provided in this disclosure embodiment;

[0031] Figure 2 A schematic diagram illustrating the specific implementation process of a multi-view stereo matching method based on optical flow consistency provided in this disclosure embodiment;

[0032] Figure 3 This is a schematic diagram of an actual matching window provided in an embodiment of this disclosure. Detailed Implementation

[0033] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0034] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0035] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0036] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0037] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0038] This disclosure provides a multi-view stereo matching method based on optical flow consistency, which can be applied to image reconstruction in computer vision application scenarios.

[0039] See Figure 1 This is a flowchart illustrating a multi-view stereo matching method based on optical flow consistency provided in an embodiment of this disclosure. Figure 1 and Figure 2 As shown, the method mainly includes the following steps:

[0040] Step 1: Use the image pyramid algorithm to perform downsampling operations on the original image a preset number of times and at a preset scale to obtain multiple downsampled images;

[0041] In practical implementation, the image pyramid algorithm can be used to downsample the high-resolution image three times consecutively, with each downsampled image size being half the size of the previous layer, resulting in four layers of images with progressively decreasing scales. Whether the photometric consistency-based PatchMatch algorithm can calculate the correct depth value and normal vector for each pixel depends on the discriminative power of the matching window. In weakly textured regions, pixels within the matching window are very similar to their neighbors, resulting in low distinguishability and leading to fuzzy matching. Incorrect depth values ​​and normal vectors obtained from fuzzy matching are filtered out during the fusion process, causing the point cloud reconstructed by the photometric consistency-based PatchMatch algorithm to be very sparse in weakly textured regions. This design incorporates the image pyramid algorithm within the OpenMVS framework, effectively avoiding the fuzzy matching problem. The image pyramid algorithm downsamples the original image three times consecutively, ensuring that the final image size is 1 / 8 of the original. Figure 3 As shown, with the same matching window size, the texture information within the matching window is richer in the downsampled image. During spatial matching, the enhanced texture information guides the matching window to find the corresponding pixels, thereby obtaining the correct depth value and normal vector.

[0042] Step 2: Calculate the depth map and normal map for each downsampled image, starting from the top-level downsampled image;

[0043] In practice, the depth map and normal map can be calculated starting from the top-level image (the image scale is 1 / 8 of the original image).

[0044] Step 3: Using stereo pair selection under the OpenMVS framework, select a set of neighboring images for each downsampled image. Score each neighboring image based on the camera center angle, resolution similarity, and common viewing area of ​​the two images. Select the neighboring image with the highest score as the best neighboring image of the corresponding downsampled image.

[0045] In practice, stereo pair selection under the OpenMVS framework can be used to select a set of neighboring images for each current downsampled image. Each neighboring image is scored based on the camera center angle between the two images, resolution similarity, and shared viewing area. The neighboring image with the highest score is selected as the best neighboring image for the current downsampled image.

[0046] Step 4: Convert the downsampled image and its corresponding best neighbor image into grayscale images, and calculate the optical flow motion vector of the downsampled image based on the dense optical flow algorithm and the best neighbor image.

[0047] Furthermore, the formula for calculating the optical flow motion vector is as follows:

[0048]

[0049] Where φ(Δp) is the weighting function of the pixel neighborhood, and A(p+Δp) and ΔB(p+Δp) are the polynomial expansion coefficients of the image. This is the optical flow motion vector.

[0050] Furthermore, the formula for calculating the optical flow consistency is as follows:

[0051]

[0052] Where × represents the cross product of vectors, and |·| represents the magnitude of the vector. The optical flow motion vector, This is the geometric motion vector.

[0053] In practical implementation, the current image and its best neighboring image can be converted into grayscale images. The dense optical flow algorithm is then used to calculate the optical flow motion vector map of the current image based on the best neighboring image. Optical flow is a method for estimating object motion. The dense optical flow algorithm is based on the motion estimation algorithm of all pixels in two consecutive frames, realizing the dense optical flow field through the displacement vectors of all pixels in the two frames. This design selects the best neighboring frame I from the candidate frames of the stereo pair.best Pair with the current frame to calculate the motion vectors of all pixels. Converting the current frame and the best neighboring frame into grayscale images, the image can be viewed as a function I(p) of a two-dimensional signal, with the pixel coordinates p(x,y) as the dependent variable. Expanding the function I(p) binomially, we can approximate it as:

[0054] I(p)≈r1+r2x+r3y+r4x 2 +r5y 2 +r6xy=pAp T +B T p T +C

[0055] Assuming the displacement of a certain pixel Then this pixel is in the best neighboring frame I best The location is:

[0056]

[0057] If we let:

[0058]

[0059]

[0060] We can obtain:

[0061]

[0062]

[0063] To determine the optical flow motion vector We construct an objective function:

[0064]

[0065] The neighborhood information of this pixel can be used to reduce the impact of noise on the results, and the motion vector can be solved by weighted optimization of the objective function:

[0066]

[0067] Project the pixel p onto the best neighbor frame I best Find the corresponding pixel q. Therefore, we can obtain the geometric motion vector of pixel p projected using the homography matrix.

[0068]

[0069] Based on the above, we obtain the optical flow motion vector calculated by the optical flow method. Geometric motion vectors calculated in agreement with photometric purity Ideally, the two motion vectors should be consistent, which is also known as optical flow consistency.

[0070] Step 5: Compare the similarity between the optical flow motion vector and the geometric motion vector obtained by using the projection relationship, and perform normalization processing to construct optical flow consistency.

[0071] In practical implementation, optical flow consistency can be added to photometric consistency, constructing a new cost function for the PatchMatch algorithm. To unify the value range of the new cost function with the traditional range of [0,2], we normalize the optical flow consistency. Furthermore, to more closely integrate photometric and optical flow consistency, we designed a parameter for harmonization.

[0072] Step 6: Jointly estimate the optical flow consistency and the photometric consistency of the PatchMatch algorithm to form a cost function;

[0073] Based on the above embodiments, the expression for the cost function is as follows:

[0074] Cost = (1-ω)Cost photo +ωCost flow

[0075] Cost photo Cost is the cost term for photometric consistency matching. flow ω is the optical flow consistency matching cost term, and ω is a constant.

[0076] In practical implementation, considering the reduction in image scale and the unchanged matching window size, the matching window contains more information. In two high-resolution images (especially in weakly textured regions), finding the corresponding position of each pixel using only visual information is extremely difficult, which is why the photometric consistency-based PatchMatch algorithm is prone to matching ambiguity. The image pyramid algorithm makes the matching window representing the pixel more distinguishable in weakly textured regions, solving the matching ambiguity problem caused by photometric consistency. Simultaneously, after obtaining relatively reliable depth information in weakly textured regions at a coarser scale, this design upsamples the depth map and uses multi-scale geometric invariance to guide the propagation of depth value assumptions to finer scales. Although the pixel depth is well recovered in weakly textured regions, these depth values ​​are calculated at a coarser scale, inevitably leading to the loss of scene details (e.g., blurred boundaries between different planes, missing fine structures, etc.). This design constructs a detail restorer based on optical flow consistency in adjacent scales. Based on photometric consistency, optical flow consistency is added, constructing a matching cost function that combines photometric and optical flow to recover these details. The added optical flow consistency term increases the OpenMVS system's sensitivity to detail. The matching cost function is as follows (the traditional photometric consistency-based PatchMatch algorithm only has Cost). photo One item):

[0077] Cost = (1-ω)Cost photo +ωCost flow

[0078] Cost photo Cost is the cost term for photometric consistency matching. flow Let ω be the cost term for optical flow consistency matching, where ω is a constant. (Optical flow consistency matching cost term - Cost) flow Defined as:

[0079]

[0080] Where × represents the cross product of vectors, and |·| represents the magnitude of the vector. The optical flow motion vector, This is the geometric motion vector.

[0081] Step 7: Recalculate the depth map and normal map of each downsampled image according to the cost function. After obtaining the depth map and normal map of the current layer, upsample them to the next layer.

[0082] In practice, depth maps and normal maps can be calculated for each image based on the new cost function. After obtaining the depth map and normal map of the current layer, we upsample them to the next layer (twice the scale of the current layer image). Based on the aforementioned cost function, we calculate the depth map and normal map for the image of the next layer.

[0083] Step 8: Between two adjacent image scales, the depth map and normal vector map obtained in Step 2 and Step 7 are compared using the detail restorer. After selecting the depth value and normal vector for each pixel using the selection function, the final depth map and normal vector map of the next layer are obtained. Steps 2 to 8 are repeated until the depth map and normal vector map of the original image are calculated. Finally, the depth map and normal vector map are fused into a three-dimensional point cloud model.

[0084] Furthermore, the calculation formula for the selection function is as follows:

[0085]

[0086] Among them, C down (q) represents the matching cost of the upsampled pixel, C up (q) represents the pixel matching cost calculated at the current scale, and γ is a selection threshold.

[0087] In practice, the newly constructed cost function can be used to calculate depth information and cost C at a finer scale. up (q), and then the cost C is calculated only under photometric consistency from the depth map sampled at a coarse scale. down (q), then compare the cost of the two and take the better depth information:

[0088]

[0089] Here, γ is a small constant value that increases robustness in distinguishing incorrect estimates. The detail restorer strengthens the connection between the two scales. It can recover sacrificed detail information while preserving depth information in weakly textured regions. The result of the detail restorer serves as initialization information for the next scale, providing a good reference when calculating the depth map and normal map at that scale. Then, between two adjacent image scales, we use the detail restorer to compare the newly calculated depth map with the normal map and the upsampled one, selecting the better depth value and normal vector for each pixel. After selection, the final depth map and normal map for the next layer are obtained. The above steps are repeated until the depth map and normal map of the original image are calculated. Finally, the depth map and normal map are fused together to form a 3D point cloud model.

[0090] This embodiment provides a multi-view stereo matching method based on optical flow consistency. By incorporating an image pyramid algorithm within the OpenMVS framework, it solves the mismatch problem encountered by the photometric consistency-based PatchMatch algorithm when reconstructing high-resolution images. Weak-texture regions are common and widespread in scenes. Due to the extremely high pixel similarity and severe homogenization within these regions, reconstructing a complete 3D point cloud presents numerous challenges. The image pyramid algorithm continuously downsamples the original image, minimizing the area of ​​weak-texture regions and increasing the information within the matching window, thus mitigating mismatches. Furthermore, we calculate the depth map and normal vector map starting from the smallest image, then perform upsampling. The upsampled depth map and normal vector map can serve as reference values ​​for the next layer of calculation, significantly increasing the accuracy of depth and normal vector estimation in weak-texture regions.

[0091] After introducing the image pyramid algorithm, the continuous downsampling process makes the photometric consistency insufficient for recognizing scene details, resulting in a significant loss of detail in the reconstructed 3D point cloud. To enhance the sensitivity of the PatchMatch algorithm for detail recognition, we propose optical flow consistency based on optical flow and add a detail restorer within adjacent scales. Experiments have verified that the optical flow method is highly accurate in estimating the motion of detailed objects. Based on the idea of ​​optical flow, we perform motion estimation for each pixel in the image, constructing a new matching cost term to compensate for the shortcomings of photometric consistency. The detail restorer can retain the correct estimation of weakly textured regions at the coarse scale and inherit the accurate calculation of details at the fine scale. The inclusion of the detail restorer strengthens the connection between adjacent scales, ensures geometric invariance across multiple scales, and solves the problems of the image pyramid algorithm to some extent. Finally, the improved algorithm can accurately estimate the depth of the input high-resolution image, thereby reconstructing a more complete and accurate 3D point cloud.

[0092] The units described in the embodiments of this disclosure can be implemented in software or in hardware.

[0093] It should be understood that the various parts of this disclosure can be implemented in hardware, software, firmware, or a combination thereof.

[0094] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A multi-view stereo matching method based on optical flow consistency, characterized in that, include: Step 1: Use the image pyramid algorithm to perform downsampling operations on the original image a preset number of times and at a preset scale to obtain multiple downsampled images; Step 2: Calculate the depth map and normal map for each downsampled image, starting from the top-level downsampled image; Step 3: Using stereo pair selection under the OpenMVS framework, select a set of neighboring images for each downsampled image. Score each neighboring image based on the camera center angle, resolution similarity, and common viewing area of ​​the two images. Select the neighboring image with the highest score as the best neighboring image of the corresponding downsampled image. Step 4: Convert the downsampled image and its corresponding best neighbor image into grayscale images, and calculate the optical flow motion vector of the downsampled image based on the dense optical flow algorithm and the best neighbor image. Step 5: Compare the similarity between the optical flow motion vector and the geometric motion vector obtained using the projection relationship, and perform normalization processing to construct the optical flow consistency. The calculation formula for the optical flow consistency is as follows: ； in, Represents the cross product of vectors. The modulus representing the orientation quantity The optical flow motion vector, It is a geometric motion vector; Step 6: Jointly estimate the optical flow consistency and the photometric consistency of the PatchMatch algorithm to form a cost function, wherein the expression of the cost function is as follows: ； in, For photometric consistency matching cost term, For optical flow consistency matching cost term, It is a constant; Step 7: Recalculate the depth map and normal map of each downsampled image according to the cost function. After obtaining the depth map and normal map of the current layer, upsample them to the next layer. Step 8: Between two adjacent image scales, the depth map and normal vector map obtained in Steps 2 and 7 are compared using a detail restorer. A selection function is used to select a depth value and normal vector for each pixel, resulting in the final depth map and normal vector map for the next layer. Steps 2 to 8 are repeated until the depth map and normal vector map of the original image are calculated. Finally, the depth map and normal vector map are fused into a 3D point cloud model. The calculation formula for the selection function is as follows: ； in, The matching cost of the upsampled pixels, This represents the pixel matching cost calculated at the current scale. This is a selection threshold.

2. The method according to claim 1, characterized in that... The formula for calculating the optical flow motion vector is as follows: ； in, A weighting function for the neighborhood of pixels. and where are the polynomial expansion coefficients of the image.

Images