A method and system for selecting a first reference image for MVS dense reconstruction

Abstract

The application discloses a method and system for selecting a first reference image of MVS dense reconstruction, and belongs to the field of three-dimensional reconstruction, which comprises the following steps: inputting source image data; calculating the incidence angle of the source image; in each loop, calculating the incidence angle of other source images relative to the current source image, and obtaining the number of other source images within a certain threshold relative to the incidence angle of each source image; normalizing the number of other source images corresponding to each source image to obtain a first parameter for measuring the index; performing feature detection on all source images; performing feature matching and recording the total sum of feature point matching pairs of each source image; normalizing the total sum of feature point matching pairs of each source image to obtain a second parameter for measuring the index; calculating the product of the first parameter and the second parameter of each source image, and selecting the source image with the largest product result as the first reference image. The application can obtain a first reference image that can reconstruct a larger number of three-dimensional points, thereby obtaining a more accurate reconstruction in MVS and improving the efficiency of reconstruction.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional reconstruction, and more particularly to a method and system for selecting the first reference image for dense MVS reconstruction. Background Technology

[0002] With the emergence of the demand for 3D reconstruction, the traditional multi-view stereo vision (MVS) method has arisen. The MVS method aims to recover the 3D structure of a scene from images from multiple viewpoints. Its main goal is to recover the geometric structure and depth information of the scene by analyzing and processing image data from multiple viewpoints. The background of this method can be traced back to the research fields of computer vision and computer graphics. The development of the traditional MVS method began with the exploration of image processing, feature extraction, and matching techniques. With the advancement of digital image processing technology, it is possible to extract and match feature points in images more accurately, thus achieving image alignment capabilities. Dense reconstruction using MVS further expands the point cloud based on sparse reconstruction (SFM), resulting in better reconstruction effects.

[0003] The first reference image is used for the initial reconstruction in MVS and is the foundation of the entire reconstruction. The effect of the initial reconstruction has a significant impact on the final reconstruction effect. In this case, MVS has many defects. For example: (1) the initial reconstruction fails because there are very few matching points between the first reference image and the associated images; (2) the accuracy of the first depth map may be poor because there are very few associated images of the first reference image, and more time needs to be spent to fill in the points in the subsequent reconstruction, which reduces efficiency; (3) the first reference image is not the best reference image, which causes higher accuracy points to be discarded in the subsequent reconstruction. Summary of the Invention

[0004] The purpose of this invention is to overcome the problems in the selection of the first reference image in existing multi-view stereo vision methods, and to provide a method and system for selecting the first reference image for MVS reconstruction.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] Firstly, a method for selecting the first reference image for dense MVS reconstruction is provided, comprising the following steps:

[0007] S1, Input source image data;

[0008] S2. Iterate through and calculate the incident angle of the source images. In each loop, calculate the incident angle of other source images relative to the current source image to obtain the number of other source images whose incident angle relative to each source image is within a certain threshold.

[0009] S3. Normalize the number of other source images corresponding to each source image to obtain the first parameter for measuring the metric.

[0010] S4. Perform feature detection on all source images;

[0011] S5. Perform feature matching and record the sum of feature point matching pairs for each source image;

[0012] S6. Normalize the sum of feature point matching pairs for each source image to obtain the second parameter for measuring the metric.

[0013] S7. Calculate the product of the first and second parameters of each source image, and select the source image with the largest product as the first reference image.

[0014] Preferably, the angle of incidence of other source images relative to the current source image is calculated using the following formula:

[0015]

[0016] Where m represents other source images, l represents the pixels of the current source image, p represents the 3D point obtained from the current source image in incremental SFM, c represents the projection center of other source images, pc represents the ray direction from the point to the camera, and n represents the normal direction.

[0017] Preferably, the threshold value of the incident angle is 45°.

[0018] Preferably, the first parameter is calculated using the following formula:

[0019]

[0020] Among them, o max o is the maximum number of other source images min This represents the minimum number of other source images.

[0021] Preferably, the SIFT feature extraction algorithm is used to perform feature detection on all source images to obtain the feature descriptor corresponding to each source image.

[0022] Preferably, step S5 specifically includes:

[0023] The Euclidean distance between feature descriptors is used to match features in the image to obtain information about matching pairs.

[0024] Preferably, the second parameter is calculated using the following formula:

[0025]

[0026] Where, m max m is the maximum value of the sum of feature point matching pairs. minThis is the minimum value of the sum of the feature point matching pairs.

[0027] Secondly, a system for selecting the first reference image for dense MVS reconstruction is provided, comprising:

[0028] The image input module is configured to input source image data;

[0029] The incident angle calculation module is configured to iterate through and calculate the incident angle of the source images. In each loop, it calculates the incident angle of other source images relative to the current source image, and obtains the number of other source images whose incident angle relative to each source image is within a certain threshold.

[0030] The first parameter calculation module is configured to normalize the number of other source images corresponding to each source image to obtain the first parameter for measuring the metric.

[0031] The feature extraction and matching module is configured to perform feature detection and feature matching on all source images, and record the sum of feature point matching pairs for each source image;

[0032] The first parameter calculation module is configured to normalize the sum of feature point matching pairs for each source image to obtain the second parameter for measuring the metric.

[0033] The reference image selection module is configured to calculate the product of the first and second parameters of each source image, and select the source image with the largest product as the first reference image.

[0034] It should be further noted that the technical features corresponding to the above options can be combined or substituted to form new technical solutions if there is no conflict.

[0035] Compared with the prior art, the beneficial effects of the present invention are:

[0036] This invention, by comprehensively comparing the number of incident angles at a certain threshold and the total number of matching feature point pairs for a given image from other source images, can determine which image has more relevant images that can be used for reconstruction. This allows for the selection of an image with the most suitable perspective for reconstruction, resulting in more complete and accurate point cloud depth information and obtaining a first reference image with a larger number of 3D points. Consequently, higher accuracy reconstruction can be achieved in MVS, and reconstruction efficiency can be improved. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating a method for selecting the first reference image for MVS reconstruction, as shown in an embodiment of the present invention. Detailed Implementation

[0038] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] In the description of this invention, it should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0040] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0041] Before describing the embodiments of the present invention, the English terms involved are explained:

[0042] MVS: Multiple View Stereo, is a group of techniques that use two or more images to match the main cues in a stereoscopic manner.

[0043] Sparse reconstruction (SFM) is a common method for 3D reconstruction that uses camera movement to determine the spatial and geometric relationships of a target.

[0044] SIFT, or Scale-invariant feature transform, is a descriptor used in image processing. This descriptor is scale-invariant and can detect key points in an image; it is a local feature descriptor.

[0045] Reference Figure 1 In one exemplary embodiment, a method for selecting the first reference image for dense MVS reconstruction is provided, comprising the following steps:

[0046] S1. Start multi-view dense 3D reconstruction, input source image data;

[0047] S2. Iterate through and calculate the incident angle of the source images. In each loop, calculate the incident angle of other source images relative to the current source image to obtain the number of other source images whose incident angle relative to each source image is within a certain threshold.

[0048] S3. Normalize the number of other source images corresponding to each source image to obtain the first parameter for measuring the metric.

[0049] S4. Perform feature detection on all source images;

[0050] S5. Perform feature matching and record the sum of feature point matching pairs for each source image;

[0051] S6. Normalize the sum of feature point matching pairs for each source image to obtain the second parameter for measuring the metric.

[0052] S7. Calculate the product of the first and second parameters of each source image, and select the source image with the largest product as the first reference image.

[0053] Specifically, in step S2, the angle of incidence of other source images relative to the current source image is calculated using the following formula:

[0054]

[0055] Where m represents other source images, l represents the pixels of the current source image, p represents the 3D point obtained from the current source image in incremental SFM, c represents the projection center of other source images, pc represents the ray direction from the point to the camera, and n represents the normal direction.

[0056] Furthermore, the threshold for the incident angle is 45°. The incident angles relative to other source images are calculated to obtain the number of source images o with incident angles between 0 and 45°. This step requires obtaining the point cloud information of the current image obtained during sparse reconstruction. This information is then used to calculate the incident angle with the source images according to the formula. Since a smaller angle results in a larger overlap between the two images, reconstruction results are generally better when the angle is less than 50°. Considering the limited number of captured images, further reducing the threshold might lead to roughly the same number o for each source image; therefore, 45° is chosen.

[0057] Further, in step S3, the first parameter is calculated using the following formula:

[0058]

[0059] Among them, o max o is the maximum number of other source images min This represents the minimum number of other source images.

[0060] Further, in step S4, the SIFT feature extraction algorithm is used to perform feature detection on all source images to obtain the feature descriptor corresponding to each source image. Specifically, scale-space extremum detection is first performed. For each detected local extremum point, a surface is fitted in scale space using a second-order Gaussian function. The stability of a keypoint is determined by comparing its intensity, scale, and surface curvature. An orientation is assigned to each keypoint, and a local image patch centered on the keypoint is constructed in the scale space surrounding it. The local image patch is divided into small sub-regions. For each sub-region, the gradient magnitude and direction of its internal pixels are calculated. The gradient direction histogram is used to statistically analyze the gradients within the sub-regions, generating a feature vector. The feature vectors of all sub-regions are concatenated to form the final feature descriptor.

[0061] Further, step S5 specifically includes:

[0062] The Euclidean distance between feature descriptors is used to match features in the image, obtaining matching pair information. Specifically, for each pair of feature descriptors, the Euclidean distance between them is calculated. Nearest neighbor matching is used to compare each feature descriptor with feature descriptors in other images, and the closest feature is selected as the match, ultimately obtaining the matching pair information. The formula for calculating the Euclidean distance is as follows:

[0063]

[0064] In the formula x i and x j Let x represent a pair of feature descriptors, where x is a point on the image and k represents the k-th feature value.

[0065] Further, in step S6, the second parameter is calculated using the following formula:

[0066]

[0067] Where, m max m is the maximum value of the sum of feature point matching pairs. min This is the minimum value of the sum of the feature point matching pairs.

[0068] Finally, the product of a and k is calculated for each image, and the image with the largest result is selected as the first reference image. By comparing the number of incident angles between 0 and 45° in other source images for a given image with the total number of matched feature points, it can be determined which image has more relevant images that can be used for reconstruction, and which image can be reconstructed to obtain more 3D points, thereby achieving higher accuracy and improving reconstruction efficiency in MVS.

[0069] In another exemplary embodiment, the present invention provides a system for selecting the first reference image for dense MVS reconstruction, comprising:

[0070] The image input module is configured to input source image data;

[0071] The incident angle calculation module is configured to iterate through and calculate the incident angle of the source images. In each loop, it calculates the incident angle of other source images relative to the current source image, and obtains the number of other source images whose incident angle relative to each source image is within a certain threshold.

[0072] The first parameter calculation module is configured to normalize the number of other source images corresponding to each source image to obtain the first parameter for measuring the metric.

[0073] The feature extraction and matching module is configured to perform feature detection and feature matching on all source images, and record the sum of feature point matching pairs for each source image;

[0074] The first parameter calculation module is configured to normalize the sum of feature point matching pairs for each source image to obtain the second parameter for measuring the metric.

[0075] The reference image selection module is configured to calculate the product of the first and second parameters of each source image, and select the source image with the largest product as the first reference image.

[0076] The above detailed embodiments are a description of the present invention. It should not be considered that the specific embodiments of the present invention are limited to these descriptions. For those skilled in the art, several simple deductions and substitutions can be made without departing from the concept of the present invention, and all of these should be considered to fall within the protection scope of the present invention.

Claims

1. A method for selecting the first reference image for dense MVS reconstruction, characterized in that, Includes the following steps: S1, Input source image data; S2. Iterate through and calculate the incident angle of the source images. In each loop, calculate the incident angle of other source images relative to the current source image to obtain the number of other source images whose incident angle relative to each source image is within a certain threshold. S3. Normalize the number of other source images corresponding to each source image to obtain the first parameter for measuring the metric. S4. Perform feature detection on all source images; S5. Perform feature matching and record the sum of feature point matching pairs for each source image; S6. Normalize the sum of feature point matching pairs for each source image to obtain the second parameter for measuring the metric. S7. Calculate the product of the first and second parameters of each source image, and select the source image with the largest product as the first reference image.

2. The method for selecting the first reference image for dense MVS reconstruction according to claim 1, characterized in that, The angle of incidence of other source images relative to the current source image is calculated using the following formula: Where m represents other source images, l represents the pixels of the current source image, p represents the 3D point obtained from the current source image in incremental SFM, c represents the projection center of other source images, pc represents the ray direction from the point to the camera, and n represents the normal direction.

3. The method for selecting the first reference image for dense MVS reconstruction according to claim 1, characterized in that, The threshold for the incident angle is 45°.

4. The method for selecting the first reference image for dense MVS reconstruction according to claim 1, characterized in that, The first parameter is calculated using the following formula: Among them, o max o is the maximum number of other source images min This represents the minimum number of other source images.

5. The method for selecting the first reference image for dense MVS reconstruction according to claim 1, characterized in that, The SIFT feature extraction algorithm is used to detect features in all source images, and feature descriptors corresponding to each source image are obtained.

6. The method for selecting the first reference image for dense MVS reconstruction according to claim 5, characterized in that, Step S5 specifically includes: The Euclidean distance between feature descriptors is used to match features in the image to obtain information about matching pairs.

7. The method for selecting the first reference image for dense MVS reconstruction according to claim 1, characterized in that, The second parameter is calculated using the following formula: Where, m max m is the maximum value of the sum of feature point matching pairs. min This is the minimum value of the sum of the feature point matching pairs.

8. A system for selecting the first reference image for dense MVS reconstruction, characterized in that, include: The image input module is configured to input source image data; The incident angle calculation module is configured to iterate through and calculate the incident angle of the source images. In each loop, it calculates the incident angle of other source images relative to the current source image, and obtains the number of other source images whose incident angle relative to each source image is within a certain threshold. The first parameter calculation module is configured to normalize the number of other source images corresponding to each source image to obtain the first parameter for measuring the metric. The feature extraction and matching module is configured to perform feature detection and feature matching on all source images, and record the sum of feature point matching pairs for each source image; The first parameter calculation module is configured to normalize the sum of feature point matching pairs for each source image to obtain the second parameter for measuring the metric. The reference image selection module is configured to calculate the product of the first and second parameters of each source image, and select the source image with the largest product result as the first reference image.

Images