Point cloud extraction method, system and equipment based on epipolar ray image pyramid and medium

Abstract

The invention provides a point cloud extraction method, system and device based on an epipolar ray image pyramid and a medium, and the method comprises the steps: carrying out the feature point extraction of a collected three-dimensional epipolar ray image through an initial SIFT feature matching algorithm, and obtaining the layer number of the pyramid epipolar ray image through the parallax calculation of all extracted feature points; calculating the optimal parallax of each layer of image layer by layer from the top layer image to the middle layer image according to the bottom layer parallax of the bottom layer feature points in the bottom layer image in the pyramid epipolar line image, and iteratively calculating the optimal parallax of the bottom layer feature points according to the optimal parallax of each layer of image; and matching the acquired current three-dimensional epipolar ray image through the optimal parallax of the bottom-layer feature points to obtain homonymous feature points, and generating dense point clouds according to the homonymous feature points. According to the method, the overall information of the image is considered, the method is a matching method with global constraints, and high-reliability dense point clouds can be obtained.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, relates to point cloud extraction technology, and specifically relates to a point cloud extraction method, system, equipment and medium based on an epipolar image pyramid. Background Art

[0002] In recent years, image matching of multi-degree overlapping image data has become a research focus. For multi-baseline stereo matching technology, a common method is to use the grayscale information of the local range of the image point. However, since the overall information of the image is not considered and there is a lack of global constraints, the final matching results usually have certain gross errors.

[0003] At the same time, semi-global image matching (SGM) is a matching method with global constraints that can obtain highly reliable matching results by constructing a disparity cost function and multi-path aggregation to find the global optimal solution and obtain dense image points with the same name. However, this method is usually only used for matching between two images. Therefore, how to achieve dense point cloud extraction of multi-baseline and multi-view images is the key to complete high-precision DSM / DEM extraction.

[0004] Compared with the overall solution problem of the relaxation method in digital photogrammetry, the semi-global matching (SGM) method can effectively avoid the problem of non-convergence in the process of multiple iterative processing. However, in the case of inaccurate initial values ​​of image matching and large parallax changes, the SGM image matching method requires a large amount of memory space and has a slow calculation speed. This phenomenon is particularly obvious for large-scale remote sensing images. Summary of the invention

[0005] In order to solve the technical problems that when obtaining dense point clouds through existing image matching methods, there is a lack of global constraints due to not considering the overall information of the image, which leads to gross errors in the matching results, and the SGM image matching method occupies a large memory space and has a slow calculation speed, the present invention discloses a point cloud extraction method based on an epipolar image pyramid, and the dense point cloud extraction method includes:

[0006] S1, extracting feature points from the collected stereo epipolar image by using an initial SIFT feature matching algorithm, and calculating the number of layers of a pyramid epipolar image by using the parallax of all the extracted feature points, wherein the pyramid epipolar image includes a bottom layer image, a middle layer image, and a top layer image;

[0007] S2, calculating the optimal disparity of the top feature points in the top image according to the bottom disparity of the bottom feature points in the bottom image, calculating the optimal disparity of each layer of image from the top image to the middle image according to the optimal disparity of the top feature points, and iteratively calculating the optimal disparity of the bottom feature points according to the optimal disparity of each layer of image, wherein the optimal disparity is the matching correlation coefficient of all feature points in each layer of image and the corresponding disparity when it is minimum;

[0008] S3. Match the collected current stereo epipolar line image through the optimal parallax of the underlying feature points to obtain feature points with the same name, and generate a dense point cloud based on the feature points with the same name.

[0009] Furthermore, in the specific implementation of the above step S1, the feature points of the collected stereo epipolar image are extracted, and the number of layers of the pyramid epipolar image is obtained by calculating the parallax of all the extracted feature points, including:

[0010] S11, performing feature point matching on the collected stereo epipolar image by using an initial SIFT feature matching algorithm to obtain the disparity of each feature point, and calculating the disparity mean and variance of all the feature points;

[0011] S12, downsampling the stereo epipolar image at different resolutions to obtain a pyramid epipolar image;

[0012] S13. Calculate the number of layers of the pyramid epipolar image by using a three-pixel averaging method according to the disparity mean and the variance.

[0013] Furthermore, in the above step S13, the three-pixel averaging method is used to calculate the number of layers of the pyramid epipolar image according to the disparity mean and variance, including:

[0014] By formula The number of layers of the pyramid epipolar image is calculated, where λ is the number of layers and θ is the given threshold. To indicate rounding down, represents the mean disparity, which is the mean of the disparity of all feature points, and σ is the variance.

[0015] Furthermore, in the specific implementation of step S2, the optimal disparity of the top feature points in the top image is calculated based on the bottom disparity of the bottom feature points in the bottom image, and the optimal disparity of each layer of image is calculated layer by layer from the top image to the middle image based on the optimal disparity of the top feature points, including:

[0016] S21, calculating the disparity search range of the top feature point corresponding to the bottom feature point in the top image according to the disparity mean, the variance and the bottom feature point of the bottom image;

[0017] S22, calculating the optimal disparity of the middle-layer feature points corresponding to the top-layer feature points in the middle-layer image according to the disparity search range of the top-layer feature points.

[0018] Furthermore, when the above step S21 is specifically implemented, the disparity search range of the top feature point corresponding to the bottom feature point in the top image is calculated based on the disparity mean, the variance and the bottom feature point of the bottom image, including:

[0019] S211, performing feature point matching on the bottom image and the top image respectively by using a SIFT feature matching algorithm to obtain top feature points in the top image corresponding to the bottom feature points;

[0020] S212, calculating an initial disparity search range of the top-level feature point according to the disparity mean, the variance and the number of layers;

[0021] S213, calculating first matching correlation coefficients of the top-level feature point and its multiple neighborhood epipolar line image points within the initial disparity search range, and summing all the first matching correlation coefficients to obtain a first coefficient sum;

[0022] S214 . Selecting the first coefficient and the smallest top-level disparity of the top-level feature point within the initial disparity search range as the optimal disparity.

[0023] Furthermore, when the above step S22 is specifically implemented, the optimal disparity of the middle-layer feature point corresponding to the top-layer feature point in the middle-layer image is calculated based on the disparity search range of the top-layer feature point, including:

[0024] S221, determining a disparity search range of middle-layer feature points in the middle-layer image by using the optimal disparity of the top-layer feature points;

[0025] S222, calculating the second matching correlation coefficients of the middle-layer feature point and a plurality of neighboring epipolar line image points within the disparity search range of the middle-layer feature point, and summing all the second matching correlation coefficients to obtain a second coefficient sum;

[0026] S223 . Selecting, within the disparity search range of the middle-layer feature point, the second coefficient and the smallest middle-layer disparity of the middle-layer feature point as the optimal disparity.

[0027] Preferably, in the above step S213 and step S222, the neighborhood epipolar line image points include 8 epipolar line image points uniformly distributed circumferentially with the target feature point (ie, the middle-layer feature point / top-layer feature point) as the center.

[0028] Furthermore, in the specific implementation of step S3, generating a dense point cloud based on the feature points with the same name includes:

[0029] According to the feature points with the same name and camera parameters, the three-dimensional coordinates of the feature points with the same name are calculated, and the three-dimensional coordinates are converted into point cloud data to obtain a dense point cloud.

[0030] The embodiment of the present invention further provides a point cloud extraction system based on an epipolar image pyramid, comprising a pyramid epipolar image layer number calculation module, an optimal disparity calculation module and a dense point cloud generation module.

[0031] The pyramid epipolar image layer number calculation module is used to extract feature points from the collected stereo epipolar image by using the initial SIFT feature matching algorithm, and calculate the number of layers of the pyramid epipolar image by using the parallax of all the extracted feature points. The pyramid epipolar image includes a bottom layer image, a middle layer image and a top layer image.

[0032] The optimal disparity calculation module is used to calculate the optimal disparity of the top feature points in the top image according to the bottom disparity of the bottom feature points in the bottom image, calculate the optimal disparity of each layer of image from the top image to the middle image layer by layer according to the optimal disparity of the top feature points, and iteratively calculate the optimal disparity of the bottom feature points according to the optimal disparity of each layer of image;

[0033] The dense point cloud generation module is used to match the optimal parallax of the underlying feature points with the collected current stereo epipolar line image to obtain feature points with the same name, and generate a dense point cloud based on the feature points with the same name.

[0034] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned point cloud extraction methods based on the kernel line image pyramid when executing the computer program, so as to solve the technical problems of obtaining dense point clouds through existing image matching methods, such as lack of global constraints due to failure to consider the overall information of the image, which leads to gross errors in matching results, and the SGM image matching method, which occupies a large memory space and has a slow calculation speed.

[0035] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program for executing any of the above-mentioned point cloud extraction methods based on the core line image pyramid, so as to solve the lack of global constraints caused by not considering the overall information of the image when obtaining dense point clouds through existing image matching methods, which in turn leads to gross errors in matching results, and the technical problems of large memory space occupation and slow calculation speed of the SGM image matching method.

[0036] Compared with the prior art, the beneficial effects that can be achieved by at least one of the above-mentioned technical solutions adopted in the embodiments of this specification include at least the following: the point cloud extraction method based on the core line image pyramid of the present invention, first, uses the initial SIFT feature matching algorithm to extract feature points from the collected stereo core line image, and calculates the number of layers of the pyramid core line image; secondly, through the disparity of the bottom layer of the pyramid core line image, the optimal disparity of each layer is calculated layer by layer from the top layer to the middle layer, and then the optimal disparity of the feature points of the bottom layer image is iteratively calculated; finally, the current stereo core line image is matched with the optimal disparity of the feature points of the bottom layer image, and a dense point cloud is generated by the feature points of the same name obtained by matching.

[0037] The method of the present invention is a multi-resolution epipolar image disparity search method, which can generate dense generation by finding a global optimal solution to obtain multiple image points with the same name. The method is a matching method with global constraints, which can obtain highly reliable matching results and dense point clouds. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 It is a flow chart of a point cloud extraction method based on epipolar image pyramid disclosed in an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of the distribution of neighborhood epipolar line image points disclosed in an embodiment of the present invention;

[0041] Figure 3 It is an architecture diagram of a point cloud extraction system based on epipolar image pyramid disclosed in an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of a computer device disclosed in an embodiment of the present invention;

[0043] Figure 5 A comparison diagram of dense point clouds generated by the method of the present invention and the traditional epipolar image dense matching results in a scenario;

[0044] Figure 6 The figure is a comparison diagram of the dense point cloud generated by the method of the present invention and the traditional epipolar image dense matching result in another scenario.

[0045] Among them, 301 is a pyramid epipolar line image layer calculation module; 302 is an optimal disparity calculation module; 303 is a dense point cloud generation module; 401 is a memory; 402 is a processor. DETAILED DESCRIPTION

[0046] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0047] The following describes the implementation methods of the present application through specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and the features of the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0048] The present invention discloses a point cloud extraction method based on an epipolar image pyramid. The method uses the disparity of feature points obtained by a stereo image matching method as an initial value, and then performs a multi-resolution epipolar image disparity search on this basis. The disparity is calculated from the top layer of the pyramid epipolar image and gradually propagated to the bottom layer, and finally the global optimal disparity, that is, the optimal disparity of the bottom layer, is obtained. Figure 1 As shown, the dense point cloud extraction method includes:

[0049] S1, extracting feature points from the collected stereo epipolar image by using an initial SIFT feature matching algorithm, and calculating the number of layers of a pyramid epipolar image by using the parallax of all the extracted feature points, wherein the pyramid epipolar image includes a bottom layer image, a middle layer image, and a top layer image;

[0050] S2, calculating the optimal disparity of the top feature points in the top image according to the bottom disparity of the bottom feature points in the bottom image, calculating the optimal disparity of each layer of image from the top image to the middle image according to the optimal disparity of the top feature points, and iteratively calculating the optimal disparity of the bottom feature points according to the optimal disparity of each layer of image, wherein the optimal disparity is the matching correlation coefficient of all feature points in each layer of image and the corresponding disparity when it is minimum;

[0051] S3. Match the collected current stereo epipolar line image through the optimal parallax of the underlying feature points to obtain feature points with the same name, and generate a dense point cloud based on the feature points with the same name.

[0052] Furthermore, in the specific implementation of the above step S1, the feature points of the collected stereo epipolar image are extracted, and the number of layers of the pyramid epipolar image is obtained by calculating the parallax of all the extracted feature points, including:

[0053] S11, performing feature point matching on the collected stereo epipolar image by using an initial SIFT feature matching algorithm to obtain the disparity of each feature point, and calculating the disparity mean and variance of all the feature points;

[0054] S12, downsampling the stereo epipolar image at different resolutions to obtain a pyramid epipolar image;

[0055] S13. Calculate the number of layers of the pyramid epipolar image by using a three-pixel averaging method according to the disparity mean and the variance.

[0056] Among them, the SIFT (Scale-Invariant Feature Transform) feature matching algorithm is an algorithm for detecting and describing local features in digital images, which mainly includes scale space extreme value detection, key point location, direction assignment, feature description and feature matching. For example, first, the image is Gaussian blurred at different scales by the SIFT algorithm, and then the DoG (Difference of Gaussians) operator is used to detect the extreme points in the image, which usually correspond to the key features in the image. Secondly, after the scale space extreme points are detected, the SIFT algorithm will accurately locate these extreme points to obtain more accurate key point position and scale information. Then, for each key point, the SIFT algorithm will assign one or more main directions according to the local image gradient direction, so that the key point has rotation invariance. Thirdly, for each key point, the SIFT algorithm will extract the local feature descriptor of its surrounding area, and usually use the histogram statistics method based on the gradient direction to describe the local image features. Finally, the SIFT algorithm uses feature descriptors to perform feature matching, usually using Euclidean distance or other similarity metrics to measure the similarity between two features, thereby finding the best feature matching pair.

[0057] Furthermore, in the above step S13, the three-pixel averaging method is used to calculate the number of layers of the pyramid epipolar image according to the disparity mean and variance, including:

[0058] By formula The number of layers of the pyramid epipolar image is calculated, where λ is the number of layers, θ is a given threshold, generally θ≤15, and θ=10 is selected in the present invention. To indicate rounding down, represents the mean disparity, which is the mean of the disparity of all feature points, and σ is the variance.

[0059] Among them, the three-pixel average method is a technique used in image processing and analysis to reduce noise in an image or smooth an image. It can group adjacent pixels together to calculate their average value and assign the average value to the pixel at their center. More specifically, the three-pixel average method refers to considering a pixel together with its surrounding 8 adjacent pixels to form a 3x3 pixel window, then calculating the average value of these 9 pixels and assigning the average value to the central pixel. Its advantage is that it can effectively reduce noise and is relatively simple and easy to implement.

[0060] Furthermore, in the specific implementation of step S2, the optimal disparity of the top feature points in the top image is calculated based on the bottom disparity of the bottom feature points in the bottom image, and the optimal disparity of each layer of image is calculated layer by layer from the top image to the middle image based on the optimal disparity of the top feature points, including:

[0061] S21, calculating the disparity search range of the top feature point corresponding to the bottom feature point in the top image according to the disparity mean, the variance and the bottom feature point of the bottom image;

[0062] S22, calculating the optimal disparity of the middle-layer feature points corresponding to the top-layer feature points in the middle-layer image according to the disparity search range of the top-layer feature points.

[0063] The optimal disparity refers to the disparity value corresponding to the minimum matching correlation coefficient of all feature points in the image.

[0064] Furthermore, generally speaking, when calculating the disparity search range of the top-level pyramid image, since the bottom-level pyramid image has a higher resolution and rich detail information, it can provide a more accurate disparity estimation, while the top-level pyramid image has a lower resolution and a relatively large search range, the disparity of the bottom-level pyramid image is usually used for guidance.

[0065] The disparity result in the bottom pyramid image can be mapped to the top pyramid image, and then the disparity search range of the top pyramid image can be determined according to the mapping relationship. Specifically, when the above step S21 is implemented, the disparity search range of the top feature point in the top image corresponding to the bottom feature point is calculated based on the disparity mean, the variance and the bottom feature point of the bottom image, including:

[0066] S211 , performing feature point matching on the bottom image and the top image respectively by using a SIFT feature matching algorithm to obtain top feature points in the top image corresponding to the bottom feature points.

[0067] S212: Calculate an initial disparity search range of the top-level feature points according to the disparity mean, the variance, and the number of layers.

[0068] S213 . Calculate first matching correlation coefficients of the top-level feature point and its multiple neighborhood epipolar line image points respectively within the initial disparity search range, and sum all the first matching correlation coefficients to obtain a first coefficient sum.

[0069] S214 . Selecting the first coefficient and the smallest top-level disparity of the top-level feature point within the initial disparity search range as the optimal disparity.

[0070] When the above steps S211 to S214 are specifically implemented, first, the parallax search range κ(λ) can be obtained by the formula Calculated; secondly, in order to improve the robustness of matching, in the disparity search range δ λ ∈κ(λ), the matching correlation coefficients of the paths (ρ1~ρ8) between the top feature points and the neighboring epipolar line image points in the eight directions around them are calculated respectively. The matching correlation coefficients are calculated by the following formula:

[0071] Among them, c(p,δ λ ) is the initial target feature point p (i.e., the top feature point p) and the initial disparity value δ λ The correlation coefficient between ρ (p,i) is the neighborhood kernel image point p on the path ρ ρ and the matching correlation coefficient with disparity i, φ1 and φ2 are hyperparameters with smooth constraints, satisfying φ1<φ2. Again, the multipath correlation coefficients of the target feature point p in the top image are summed to obtain the total first coefficient and T(p,δ λ ), can be calculated by the following formula:

[0072] That is, the matching coefficients of the eight neighboring epipolar image points of ρ1 to ρ8 are summed up. Finally, within δ(λ)∈κ(λ), the formula The optimal disparity δ′ of the target feature point p can be obtained λ , the optimal disparity acquisition method for other target feature points is the same as that for the target feature point p.

[0073] Furthermore, when the above step S22 is specifically implemented, the optimal disparity of the middle-layer feature point corresponding to the top-layer feature point in the middle-layer image is calculated based on the disparity search range of the top-layer feature point, including:

[0074] S221, determining a disparity search range of middle-layer feature points in the middle-layer image by using the optimal disparity of the top-layer feature points;

[0075] S222, calculating the second matching correlation coefficients of the middle-layer feature point and a plurality of neighboring epipolar line image points within the disparity search range of the middle-layer feature point, and summing all the second matching correlation coefficients to obtain a second coefficient sum;

[0076] S223 . Selecting, within the disparity search range of the middle-layer feature point, the second coefficient and the smallest middle-layer disparity of the middle-layer feature point as the optimal disparity.

[0077] In the specific implementation of the above steps S221 to S223, the middle-layer image is located between the bottom-layer image and the top-layer image, which can have one layer or multiple layers. For each layer in the middle-layer image, the optimal disparity is searched layer by layer from top to bottom. Specifically, the optimal disparity δ′ is obtained for the top feature point p of the top pyramid. λ After that, the next layer of images is transferred and the optimal disparity is searched. The disparity search range of the next layer of images is determined by the formula κ(λ-1)=(δ′ λ -6,δ′ λ +6) is determined, where κ(λ-1) is the parallax search range of the next layer (λ-1 layer), δ′ λ The optimal disparity of the previous layer is calculated by the same method as steps S213 and S214 to obtain the optimal disparity δ′ of the feature point in this layer corresponding to the top feature point p. λ-1 .

[0078] Preferably, in the above steps S213 and S222, see Figure 2 As shown, the neighborhood epipolar line image points include 8 epipolar line image points uniformly distributed circumferentially with the target feature point (ie, the middle-layer feature point / top-layer feature point) as the center.

[0079] Furthermore, when the above step S2 is implemented, the optimal disparity of the bottom feature point is iteratively calculated according to the optimal disparity of each layer of the image. The optimal disparity of each layer of the pyramid image can be iteratively calculated in a loop through steps S21 and S22 until the optimal disparity of the bottom image is obtained, that is, the optimal disparity δ′0 of the target feature point p. The optimal disparity δ′0 of the target feature point p can be calculated by the formula Calculated.

[0080] Furthermore, when the above step S3 is specifically implemented, generating a dense point cloud based on the feature points with the same name includes: calculating the three-dimensional coordinates of the feature points with the same name based on the feature points with the same name and camera parameters, and converting the three-dimensional coordinates into point cloud data to obtain a dense point cloud.

[0081] Based on the same inventive concept, a point cloud extraction system based on a nuclear line image pyramid is also provided in an embodiment of the present invention, as described in the following embodiment. Since the principle of solving the problem by the point cloud extraction system based on the nuclear line image pyramid is similar to that of the point cloud extraction method based on the nuclear line image pyramid in the above embodiment, the implementation of the dense point cloud extraction system can refer to the implementation of the dense point cloud extraction method described in the above embodiment, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements predetermined functions. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0082] Figure 3 is a structural block diagram of a point cloud extraction system based on an epipolar image pyramid according to an embodiment of the present invention. Figure 3 As shown, the dense point cloud extraction system includes a pyramid epipolar line image layer number calculation module 301, an optimal disparity calculation module 302 and a dense point cloud generation module 303, and the structure is described below.

[0083] The pyramid epipolar image layer number calculation module 301 is used to extract feature points from the collected stereo epipolar image by using the initial SIFT feature matching algorithm, and calculate the number of layers of the pyramid epipolar image by using the parallax of all the extracted feature points. The pyramid epipolar image includes a bottom layer image, a middle layer image and a top layer image.

[0084] The optimal disparity calculation module 302 is used to calculate the optimal disparity of the top feature points in the top image according to the bottom disparity of the bottom feature points in the bottom image, calculate the optimal disparity of each layer of image from the top image to the middle image layer by layer according to the optimal disparity of the top feature points, and iteratively calculate the optimal disparity of the bottom feature points according to the optimal disparity of each layer of image.

[0085] The dense point cloud generation module 303 is used to match the optimal parallax of the underlying feature points with the collected current stereo epipolar line image to obtain feature points with the same name, and generate a dense point cloud based on the feature points with the same name.

[0086] The principle of the point cloud extraction method based on the epipolar image pyramid of the present invention is: first, the feature points of the collected stereo epipolar image are extracted by using the initial SIFT feature matching algorithm, and the number of layers of the pyramid epipolar image is calculated; second, the optimal disparity of each layer is calculated layer by layer from the top layer to the middle layer through the disparity of the bottom layer of the pyramid epipolar image, and then the optimal disparity of the feature points of the bottom layer image is iteratively calculated; finally, the current stereo epipolar image is matched with the optimal disparity of the feature points of the bottom layer image, and a dense point cloud is generated by the feature points of the same name obtained by matching.

[0087] The method of the present invention is a multi-resolution epipolar image disparity search method. By finding the global optimal solution, multiple image points with the same name can be obtained and then dense generation can be generated. This method is a matching method with global constraints, which can obtain highly reliable matching results and dense point clouds. In two scenarios, the dense point cloud generated by the matching method of the present invention is compared with the dense point cloud generated by the dense matching of traditional epipolar images. The raster images generated by sampling with the same parameters are compared. See Figure 5 and Figure 6 As shown, it can be seen that the point cloud data extracted by the method of the present invention is more complete, more precise and less noisy.

[0088] The method of the present invention is a multi-resolution epipolar image disparity search method, which can generate dense generation by finding a global optimal solution to obtain multiple image points with the same name. The method is a matching method with global constraints, which can obtain highly reliable matching results and dense point clouds.

[0089] In this embodiment, a computer device is provided, such as Figure 4 As shown, it includes a memory 401, a processor 402, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any of the above-mentioned point cloud extraction methods based on epipolar image pyramid is implemented.

[0090] Specifically, the computer device may be a computer terminal, a server or a similar computing device.

[0091] In this embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program for executing any of the above-mentioned point cloud extraction methods based on epipolar image pyramid.

[0092] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0093] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0094] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A point cloud extraction method based on epipolar image pyramid, characterized in that: include: The feature points of the collected stereo epipolar image are extracted by using the initial SIFT feature matching algorithm, and the number of layers of the pyramid epipolar image is obtained by using the parallax calculation of all the extracted feature points. The pyramid epipolar image includes a bottom layer image, a middle layer image and a top layer image; Calculating the optimal disparity of the top feature points in the top image according to the bottom disparity of the bottom feature points in the bottom image, calculating the optimal disparity of each layer of image from the top image to the middle image according to the optimal disparity of the top feature points, and iteratively calculating the optimal disparity of the bottom feature points according to the optimal disparity of each layer of image, wherein the optimal disparity is the matching correlation coefficient of all feature points in each layer of image and the corresponding disparity when it is minimum; The currently collected stereo epipolar image is matched through the optimal parallax of the underlying feature points to obtain feature points with the same name, and a dense point cloud is generated based on the feature points with the same name.

2. The point cloud extraction method based on epipolar image pyramid according to claim 1, characterized in that: The feature point extraction is performed on the collected stereo epipolar line image, and the number of layers of the pyramid epipolar line image is obtained by calculating the parallax of all the extracted feature points, including: The collected stereo epipolar images are matched with feature points by using the initial SIFT feature matching algorithm to obtain the disparity of each feature point, and the mean and variance of the disparity of all the feature points are calculated; Downsampling the stereo epipolar image at different resolutions to obtain a pyramid epipolar image; The number of layers of the pyramid epipolar image is calculated using a three-pixel averaging method according to the disparity mean and the variance.

3. The point cloud extraction method based on epipolar image pyramid according to claim 2, characterized in that: The method of calculating the number of layers of the pyramid epipolar image by using a three-pixel averaging method based on the disparity mean and variance includes: By formula The number of layers of the pyramid epipolar image is calculated, where λ is the number of layers and θ is the given threshold. To indicate rounding down, represents the mean disparity, which is the mean of the disparity of all feature points, and σ is the variance.

4. The point cloud extraction method based on epipolar image pyramid according to claim 1, characterized in that: The step of calculating the optimal disparity of the top-layer feature points in the top-layer image based on the bottom-layer disparity of the bottom-layer feature points in the bottom-layer image, and calculating the optimal disparity of each layer of image from the top-layer image to the middle-layer image layer by layer based on the optimal disparity of the top-layer feature points, comprises: Calculate the disparity search range of the top feature point in the top image corresponding to the bottom feature point according to the disparity mean, the variance and the bottom feature point of the bottom image; The optimal disparity of the middle-layer feature points corresponding to the top-layer feature points in the middle-layer image is calculated according to the disparity search range of the top-layer feature points.

5. The point cloud extraction method based on epipolar image pyramid according to claim 4, characterized in that: The calculating, based on the disparity mean, the variance and the bottom feature points of the bottom image, the disparity search range of the top feature points corresponding to the bottom feature points in the top image comprises: The bottom image and the top image are matched with each other using a SIFT feature matching algorithm to obtain top feature points in the top image corresponding to the bottom feature points; Calculating an initial disparity search range of the top-level feature point according to the disparity mean, the variance and the number of layers; In the initial disparity search range, first matching correlation coefficients of the top-level feature point and its multiple neighboring epipolar line image points are calculated respectively, and a first coefficient sum is obtained by summing all the first matching correlation coefficients; In the initial disparity search range, the first coefficient and the smallest top-level disparity of the top-level feature point are selected as the optimal disparity.

6. The point cloud extraction method based on epipolar image pyramid according to claim 4, characterized in that: The step of calculating the optimal disparity of the middle-layer feature points corresponding to the top-layer feature points in the middle-layer image according to the disparity search range of the top-layer feature points includes: Determining a disparity search range of middle-layer feature points in the middle-layer image by using the optimal disparity of the top-layer feature points; In the parallax search range of the middle-layer feature point, respectively calculating the second matching correlation coefficients of the middle-layer feature point and a plurality of neighboring epipolar line image points, and summing all the second matching correlation coefficients to obtain a second coefficient sum; In the disparity search range of the middle-layer feature points, the second coefficient and the smallest middle-layer disparity of the middle-layer feature points are selected as the optimal disparity.

7. The point cloud extraction method based on epipolar image pyramid according to claim 1, characterized in that: The generating of a dense point cloud according to the feature points with the same name comprises: According to the feature points with the same name and camera parameters, the three-dimensional coordinates of the feature points with the same name are calculated, and the three-dimensional coordinates are converted into point cloud data to obtain a dense point cloud.

8. A point cloud extraction system based on epipolar image pyramid, characterized in that: include: A pyramid epipolar image layer number calculation module, which is used to extract feature points from the collected stereo epipolar image by using an initial SIFT feature matching algorithm, and calculate the number of layers of the pyramid epipolar image by using the parallax of all the extracted feature points, wherein the pyramid epipolar image includes a bottom layer image, a middle layer image and a top layer image; An optimal disparity calculation module, the optimal disparity calculation module is used to calculate the optimal disparity of the top feature points in the top image according to the bottom disparity of the bottom feature points in the bottom image, calculate the optimal disparity of each layer of image from the top image to the middle image layer by layer according to the optimal disparity of the top feature points, and iteratively calculate the optimal disparity of the bottom feature points according to the optimal disparity of each layer of image; A dense point cloud generation module is used to match the optimal parallax of the underlying feature points with the collected current stereo epipolar line image to obtain feature points with the same name, and generate a dense point cloud based on the feature points with the same name.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the point cloud extraction method based on the epipolar image pyramid described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the point cloud extraction method based on epipolar image pyramid according to any one of claims 1 to 7.

Images