A method and system for fast and dense reconstruction of three-dimensional point clouds in large scenes

Abstract

The present invention provides a method and system for fast dense reconstruction of three-dimensional point clouds in large scenes. The method includes the following steps: obtaining continuous images of the scene to be measured taken by a camera, and using a redundant image screening method to extract key frames from the continuous images to obtain a key frame image sequence, performing sparse reconstruction on the key frame image sequence, and obtaining the camera pose, 2D key points, and 3D sparse points of each image to select the viewing angle of the tracking image, estimating the parameter information of each image using a method based on photometric features and planar features based on the sparse point cloud generated by the point cloud sparse reconstruction and the results of the viewing angle selection, then using a fast fusion method to densely reconstruct the point cloud, and finally performing scale recovery and coordinate transformation on the dense point cloud. The present invention improves the speed and quality of dense reconstruction in large scene environments, increases the detailed features of the surface of small target objects, and provides effective technical support for accurate perception of large scene environments.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision and three-dimensional image processing, and in particular to a method and system for fast and dense reconstruction of three-dimensional point clouds in large scenes. Background Art

[0002] In recent years, deep learning technology has demonstrated remarkable success in various 2D image processing tasks, achieving accuracy far exceeding that of traditional methods across a wide range of datasets. With the growing demand for 3D vision and 3D structural data, researchers have begun applying deep learning to 3D vision. As a classic computer vision problem, 3D reconstruction, combined with neural networks to achieve higher-precision reconstructions, has become a widely studied topic.

[0003] The ultimate goal of 3D reconstruction is to restore the 3D structure of a target scene. Image sequence-based 3D reconstruction is one of the primary methods for obtaining this 3D structure. 3D reconstruction technology uses a 2D image sequence as input and, through a complex series of processing steps, reconstructs the 3D information of the scene depicted by the 2D image. This method can be considered the inverse of photography. It is relatively inexpensive, requiring only a standard camera, and is easy to operate and carry. Feature point matching and multi-view stereo reconstruction based on image sequences are fundamental research areas and key areas of focus in computer vision.

[0004] However, existing 3D reconstruction methods have the following problems under high-altitude perspectives and large scenes: the reconstructed point cloud loses scale and cannot reflect the true size of the object; the 3D reconstruction time is long; there are missing point clouds in weak-texture areas in the reconstructed image, and the description of detailed features on the object surface is poor; and the fidelity of weak-texture areas in the reconstructed terrain is low. Summary of the Invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method and system for fast and dense reconstruction of three-dimensional point clouds in large scenes, which effectively solves the problems of long point cloud reconstruction time and missing point clouds in weak texture areas in large scenes.

[0006] In a first aspect, the present invention provides a method for fast dense reconstruction of three-dimensional point clouds in a large scene, the method comprising the following steps:

[0007] S1, obtaining continuous images of a scene to be tested taken by a camera, and extracting key frames from the continuous images using a redundant image screening method to obtain a key frame image sequence;

[0008] S2. Extract and match feature points of the key frame image sequence to obtain matching feature points, and use the SFM algorithm to perform point cloud sparse reconstruction on the matching feature points;

[0009] S3. Selecting a viewing angle for the tracking image of each image in the key frame image sequence according to the result of the point cloud sparse reconstruction to obtain a resource image of each image;

[0010] S4. Calculating the depth information, normal information, and matching cost of pixels in each image based on the sparse point cloud generated by the point cloud sparse reconstruction and the result of the viewpoint selection using a method based on photometric features and planar features, and performing fast dense reconstruction of the sparse point cloud using a fast fusion method;

[0011] S5. Perform scale recovery and coordinate conversion on the dense point cloud generated by dense reconstruction, and output the fast dense reconstruction result of the three-dimensional point cloud.

[0012] Preferably, in step S1, extracting key frames from the continuous images using a redundant image screening method to obtain a key frame image sequence includes:

[0013] S11, selecting a frame image from the continuous image, extracting adjacent frames as frames to be evaluated, and calculating the similarity Cs between two adjacent frames using image block color features;

[0014] S12, if the similarity Cs is less than 0.43, the adjacent frames are stored in a key frame group; otherwise, the global texture similarities D1, D2, and D3 between the two adjacent frames are calculated using LBP_max texture feature global statistical classification;

[0015] S13, if the global texture similarity D1>4.61, D2>10, D3>14.8, then the adjacent frames are stored in the key frame group, otherwise the local texture similarities P1, P2 and P3 between the two adjacent frames are calculated using the LBP_max texture feature local statistical classification;

[0016] S14, if the local texture similarity P1>72, P2>69, P3>58, then save the adjacent frames in the key frame group; otherwise, classify the adjacent frames as redundant images;

[0017] S15 , extracting images in sequence from the images in the key frame group using a certain margin method, controlling the number of key frame images to be within 40 frames, and obtaining a key frame image sequence.

[0018] Preferably, the step S2 specifically includes:

[0019] S21, using a deep neural network SuperPoint to extract evenly distributed feature points and descriptors for each image in the key frame image sequence, and then using a feature point matching network SuperGlue to match the feature points and descriptors to obtain a feature matching relationship between the images;

[0020] S22. Based on the feature matching relationship between the images, select a group of images with the highest matching degree to estimate the initial pose and transformation matrix of the camera, reconstruct a first sparse point cloud based on the principle of triangulation, and perform bundle adjustment optimization on the sparse point cloud;

[0021] S23. Repeat step S22 for the remaining image sequences to perform point cloud sparse reconstruction to obtain the camera pose, 2D key points, and 3D sparse points of each image.

[0022] Preferably, the step S3 specifically includes:

[0023] The score between each image and the corresponding tracking image is calculated according to the camera pose, 2D key points and 3D sparse points of each image, and the top N tracking images with the highest scores are selected as the resource images of each image.

[0024] Preferably, the step S4 specifically includes:

[0025] S41. Generate an initial map based on the photometric consistency features of each image and the corresponding resource image, and calculate the initial depth, initial normal, and initial matching cost of each pixel in each image based on the camera intrinsic parameters;

[0026] S42, selecting the pixel point with the minimum local matching cost according to the initial matching cost to perform triangulation, calculating the features of the plane where the three coordinate points corresponding to each triangle are located, and assigning the plane features to the pixel points that meet the depth condition;

[0027] S43, generating a new map based on the photometric features and planar features of each pixel, and updating the initial depth, initial normal, and initial matching cost of each pixel in each image based on the camera intrinsic parameters to generate depth information, normal information, and matching cost for dense reconstruction;

[0028] S44, reprojecting the depth information of each image using GPU acceleration according to the photometric characteristics and spatial geometric characteristics;

[0029] S45, comparing the matching cost of each image pixel point according to the reprojection calculation result, and marking the pixel points that meet the matching cost parameter threshold and the spatial point distance threshold;

[0030] S46. Convert the marked pixel points into spatial coordinate points using a transformation matrix according to the camera intrinsic parameters and the camera position, and then convert the generated spatial coordinate points into the initial camera coordinate system according to the position of each image to obtain a dense reconstruction result.

[0031] Preferably, performing scale restoration on the dense point cloud generated by dense reconstruction in step S5 specifically includes:

[0032] S51, performing random downsampling on the dense point cloud generated by the dense reconstruction, and selecting 1 / 5 of the points as sampling retention points;

[0033] S52: Use a plane fitting algorithm to extract the plane where the ground is located for the sampled retention points, and calculate the distance P between the coordinate origin and the plane where the ground is located. z ;

[0034] S53, determine a calibration object, and give the actual length Tl, width Tw, height Th of the calibration object and the actual distance H between the camera and the ground;

[0035] S54. Calculate the scale parameters kx, ky, and kz of the point cloud in the X, Y, and Z directions based on the length Pl, width Pw, and height Ph of the calibration object in the point cloud, where:

[0036] kx=Tl÷Pl

[0037] ky=Tw÷Pw

[0038] kz=Th÷Ph

[0039] S55. Use the formula to restore the coordinates of each point in the dense point cloud in sequence. The formula is as follows:

[0040]

[0041] Among them, x, y, and z represent the horizontal coordinate, vertical coordinate, and vertical coordinate of the image point in the camera coordinate system, respectively.

[0042] Preferably, the coordinate conversion in step S5 includes:

[0043] The camera position is transformed according to the camera parameters of the first frame image in the key frame sequence image, and then the origin of the coordinate system is transformed by subtracting the camera coordinates from the coordinates of the densely reconstructed point cloud. Finally, the point cloud in the camera coordinate system is converted to the device coordinate system for point cloud display.

[0044] In a second aspect, the present invention provides a system for fast and dense reconstruction of three-dimensional point clouds in a large scene, the system comprising:

[0045] An image acquisition and key frame extraction unit is used to acquire continuous images of the scene to be measured taken by a camera, and to extract key frames from the continuous images using a redundant image screening method to obtain a key frame image sequence;

[0046] A point cloud sparse reconstruction unit is used to extract and match feature points of the key frame image sequence to obtain matching feature points, and use the SFM algorithm to perform point cloud sparse reconstruction on the matching feature points;

[0047] a perspective selection unit, configured to select a perspective of the tracking image of each image in the key frame image sequence according to the result of the point cloud sparse reconstruction, so as to obtain a resource image of each image;

[0048] a point cloud dense reconstruction unit, configured to calculate the depth information, normal information, and matching cost of pixels in each image based on the sparse point cloud generated by the point cloud sparse reconstruction and the result of the view selection using a method based on photometric features and planar features, and to perform fast dense reconstruction of the sparse point cloud using a fast fusion method;

[0049] The scale recovery and coordinate transformation unit is used to perform scale recovery and coordinate transformation on the dense point cloud generated by dense reconstruction, and output the fast dense reconstruction result of the three-dimensional point cloud.

[0050] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, the steps of the method for fast and dense reconstruction of three-dimensional point clouds in large scenes as described in the first aspect of the present invention are implemented.

[0051] In a fourth aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for fast and dense reconstruction of three-dimensional point clouds in large scenes as described in the first aspect of the present invention.

[0052] The present invention provides a method for fast dense reconstruction of three-dimensional point clouds in large scenes. The method calculates the viewing angle score of the image through the result of sparse reconstruction, and selects the image with higher score for dense reconstruction. It solves the problems of long point cloud reconstruction time and missing point clouds in weak texture areas in large scenes, improves the speed of dense reconstruction in large scene environments, and increases the detailed features of the surface of small target objects; at the same time, it improves the quality of dense reconstruction, and obtains a high-precision, high-density, high-quality environment model, which provides effective technical support for accurate perception of large scene environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 This is a flow chart of a method for fast and dense reconstruction of three-dimensional point clouds in a large scene provided by an embodiment of the present invention;

[0055] Figure 2 This is a flow chart of a redundant image screening method according to an embodiment of the present invention;

[0056] Figure 3 is a schematic diagram of the result of point cloud sparse reconstruction in an embodiment of the present invention;

[0057] Figure 4 is a schematic diagram of the result of dense reconstruction of point clouds in an embodiment of the present invention;

[0058] Figure 5 This is a structural diagram of a system for fast and dense reconstruction of 3D point clouds in large scenes provided by an embodiment of the present invention;

[0059] Figure 6 It is a structural diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] To make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be further clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. It should be noted that the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0061] The terms "first" and "second" in the embodiments of the present application are only used for descriptive purposes and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present application, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a system, product or device comprising a series of components or units is not limited to the listed components or units, but may optionally also include components or units that are not listed, or may optionally also include other components or units that are inherent to these products or devices. In the description of the present application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0062] Existing 3D reconstruction methods have the following problems under high-altitude perspectives and large scenes: the reconstructed point cloud loses scale and cannot reflect the true size of the object; the 3D reconstruction time is long; point clouds are missing in weak-texture areas in the reconstructed image, and the description of detailed features on the object surface is poor; and the fidelity of weak-texture areas in the reconstructed terrain is low.

[0063] The embodiments of the present invention provide a method for fast and dense reconstruction of three-dimensional point clouds in large scenes, which effectively solves the problems of long point cloud reconstruction time and missing point clouds in weak texture areas in large scenes.

[0064] In this embodiment of the present invention, continuous images of the scene to be measured taken by a drone equipped with a monocular camera are selected as the data source. The resolution of the camera is 2048x1536. The continuous images collected by the monocular camera are used as input for 3D reconstruction to generate a dense point cloud. Figure 1 This is a flow chart of a method for fast dense reconstruction of 3D point clouds in a large scene provided by an embodiment of the present invention. Figure 1 As shown, the method specifically includes:

[0065] Step S1, obtaining continuous images of a scene to be tested taken by a camera, and extracting key frames from the continuous images using a redundant image screening method to obtain a key frame image sequence;

[0066] First, continuous images of the scene to be tested are obtained from the camera, and then a redundant image screening method is used to select clear, complete and valid key frame image sequences. Figure 2 FIG. 1 is a flow chart of a redundant image screening method according to an embodiment of the present invention. Figure 2 As shown, the method specifically includes:

[0067] Select a frame from the continuous image, extract adjacent frames as the frames to be evaluated, and classify them based on image block color feature matching. Convert the image RGB channels to HSI channels, extract only the S channel image to obtain a 1517-dimensional feature vector, and then obtain the normalized vector S, where:

[0068]

[0069] In the above formula, i∈[1,2,...,1517], u and σ are the mean and variance of all components of vector C respectively, C i Represents the i-th component of the color feature vector, S i Represents the i-th component of the standard normalized color feature vector;

[0070] The similarity of color feature vectors between two adjacent frames is measured by cosine angle, and its formula is as follows:

[0071]

[0072] The more similar the two frames are, the smaller the θ value is and the larger the Cs value is.

[0073] If the similarity Cs is less than 0.43, the adjacent frames are saved in the key frame group. Otherwise, the global texture similarities D1, D2 and D3 between the two adjacent frames are calculated by global statistical classification of LBP_max texture features, specifically including:

[0074] A three-layer pyramid is constructed to extract H channel images from different levels to analyze their textures, and the global texture feature vectors [Y1, Y2, Y3] and [Y1', Y2', Y3'] of two adjacent frames are obtained. The weighted Manhattan distance is then used to calculate the global texture similarity D1, D2 and D3 between the two adjacent frames.

[0075] If the global texture similarity D1>4.61, D2>10, D3>14.8, the adjacent frames are saved in the key frame group. Otherwise, the local texture similarity P1, P2 and P3 between two adjacent frames are calculated using the local statistical classification of the LBP_max texture feature, which includes:

[0076] Compare the texture features of the same position of the global texture feature vector [Y1, Y2, Y3] and [Y1', Y2', Y3']. If the texture features are the same, the statistical value is increased by one. Then, the local texture similarity P1, P2 and P3 of the three-layer pyramid are calculated based on the statistical results of the local features of each layer.

[0077] If the local texture similarity P1>72, P2>69, P3>58, the adjacent frames are saved in the key frame group, otherwise the adjacent frames are classified as redundant images.

[0078] The redundant image screening method adopted in the embodiment of the present invention designs a cascade classification process. The sample is judged as a key frame in the previous level and will not be subjected to the next level of feature extraction and classification, thereby speeding up the identification and classification of key frames and redundant frames.

[0079] For the key frames obtained, images are extracted sequentially with a certain interval margin to control the number of images within 40 frames.

[0080] Step S2: extracting and matching feature points of the key frame image sequence to obtain matching feature points, and using the SFM algorithm to perform point cloud sparse reconstruction on the matching feature points;

[0081] First, the deep neural network SuperPoint is used to extract evenly distributed feature points and descriptors from each image, and then the feature point matching network SuperGlue is used to match the detected feature points to obtain the feature matching relationship between images.

[0082] SuperPoint, a deep neural network, is a self-supervised deep learning feature point extraction algorithm with excellent image understanding and feature point description capabilities. Its computing speed on GPUs far exceeds that of traditional feature extraction algorithms. SuperGlue, a feature point matching network based on attention-based GNNs adapted for SuperPoint, uses the attention mechanism to simulate the human image matching process.

[0083] Then, based on the feature matching relationship between the images, a group of images with the highest matching degree is selected to estimate the initial pose and transformation matrix of the camera, and the first sparse point cloud is reconstructed based on the principle of triangulation, and the sparse point cloud is bundled and adjusted and optimized. Figure 3 Schematic diagram of the result of point cloud sparse reconstruction in an embodiment of the present invention.

[0084] Finally, sparse point cloud reconstruction and bundle adjustment optimization are performed on each of the remaining images, and the camera pose, 2D key points and 3D sparse points of each image are saved.

[0085] Step S3: selecting a viewing angle for the tracking image of each image in the key frame image sequence according to the result of the point cloud sparse reconstruction to obtain a resource image of each image;

[0086] First, the score between each image and the tracking image is calculated based on the camera pose, 2D key points, and 3D sparse points of each image. The specific calculation process is as follows:

[0087] Extract a 3D point p from the 3D sparse points in sequence and obtain all tracking images of the point p;

[0088] Then traverse all tracking images and calculate the total score between the current image k and the remaining tracking images j:

[0089]

[0090] In the above formula, p is the common three-dimensional coordinate point of image k and image j, θ kj is the angle between the common trajectory point of views k and j and the two ends of the baseline, and its calculation formula is as follows:

[0091] θ kj (p)=(180 / π)arccos((c k -p)·(c j -p))

[0092] In the above formula, c k is the position point of image k, c j is the location point of image j.

[0093] At the same time, a piecewise Gaussian function g(θ) is used to calculate the score between the current image k and the remaining tracking images j, and its expression is:

[0094]

[0095] In the above formula, θ0, σ1, σ2 are all constant terms, where θ0 = 5°, σ1 = 1, σ2 = 10.

[0096] Finally, after all 3D sparse points are processed, a score table of all view images is obtained, and the top N images with the highest scores are selected as resource images for the current image k according to the score sorting.

[0097] S4. Calculating the depth information, normal information, and matching cost of pixels in each image based on the sparse point cloud generated by the point cloud sparse reconstruction and the result of the viewpoint selection using a method based on photometric features and planar features, and performing fast dense reconstruction of the sparse point cloud using a fast fusion method;

[0098] The depth information, normal information and matching cost of each pixel in each image are calculated using a method based on photometric features and planar features as follows:

[0099] First, an initial map is generated based on the photometric features of each image k and the corresponding resource image j, and the initial depth D(r,c), initial normal N(r,c) and initial matching cost V(r,c) of each pixel in each image are calculated using the camera intrinsic parameters.

[0100] Then, based on the initial matching cost, the pixel point with the minimum local matching cost is selected for triangulation. Subsequently, the features of the plane where the three coordinate points corresponding to each triangle are located are calculated based on the results of the triangulation calculation, and the plane features are assigned to the pixel points in the triangle plane that meet the depth conditions.

[0101] Finally, a new map is generated based on the photometric and planar features of each pixel, and the initial depth, initial normal, and initial matching cost of each pixel in each image are updated according to the camera intrinsic parameters to generate depth information, normal information, and matching cost for dense reconstruction.

[0102] The fast fusion method is used to quickly and densely reconstruct sparse point clouds as follows:

[0103] First, the depth information of each image is reprojected based on the photometric features and spatial geometric features.

[0104] Then, by comparing the matching costs of the pixels in each image, the spatial point p1 corresponding to the current pixel that meets the matching cost threshold is calculated; at the same time, the pixel position and the corresponding depth value of the spatial point in the resource image are calculated, and then the spatial point p2 of the pixel in the resource image is calculated based on the depth value. The pixels whose distance between the two spatial points is less than the threshold are marked from the perspective, and the unmarked pixels in each image are determined to be non-abnormal points.

[0105] Then use the transformation matrix according to the camera intrinsic parameters and camera position:

[0106]

[0107] Convert the marked pixel points into spatial coordinate points. By transforming the transformation matrix, the world coordinate point (x, y, z) of the image point (u, v) can be obtained as follows:

[0108]

[0109] In the above formula, d c is the depth value of pixel (u, v), K is the camera intrinsic parameter, R is the rotation matrix, and T is the position of the current image camera.

[0110] Finally, the generated spatial coordinate points are converted to the initial camera coordinate system according to the position of each image to obtain the dense reconstruction result. Figure 4 Schematic diagram of the result of dense reconstruction of point cloud in an embodiment of the present invention.

[0111] S5. Perform scale recovery and coordinate conversion on the dense point cloud generated by dense reconstruction, and output the fast dense reconstruction result of the three-dimensional point cloud.

[0112] First, the generated dense point cloud is randomly downsampled, and 1 / 5 of the points are selected as sampling retention points to reduce the time of the scale recovery stage;

[0113] Next, use the plane fitting algorithm to perform plane fitting on the retained point cloud to obtain the normal of the plane where the ground is located. Calculate the distance between the coordinate origin and the ground plane

[0114] Then determine a calibration object, and give the real length Tl, width Tw, height Th of the calibration object and the real distance H between the camera and the ground, and calculate the scale parameters kx, ky, kz of the point cloud in the X, Y, and Z directions according to the length Pl, width Pw, and height Ph of the calibration object in the point cloud, where:

[0115]

[0116] Finally, the coordinates of each point in the dense point cloud are restored in sequence using the formula as follows:

[0117]

[0118] Among them, x, y, and z represent the horizontal coordinate, vertical coordinate, and vertical coordinate of the image point in the camera coordinate system, respectively.

[0119] Finally, the coordinate transformation method is used to transform the dense point cloud into the coordinate system of the drone aerial photography environment, accurately measuring the distance information between each object on the ground and the drone, including:

[0120] First, the camera position is transformed according to the camera parameters of the first frame in the sequence of images. Then, the origin of the coordinate system is transformed by subtracting the camera coordinates from the coordinates of the densely reconstructed point cloud. Finally, the point cloud in the camera coordinate system is converted to the device coordinate system for point cloud display according to the control parameters of the drone movement.

[0121] Based on the method for fast dense reconstruction of three-dimensional point clouds in a large scene in the above embodiment, a second aspect of the embodiment of the present invention provides a system for fast dense reconstruction of three-dimensional point clouds in a large scene. Figure 5 This is a structural diagram of a system for fast and dense reconstruction of 3D point clouds in large scenes provided by an embodiment of the present invention. Figure 5 As shown, the system includes:

[0122] The image acquisition and key frame extraction unit 510 is used to acquire continuous images of the scene to be measured taken by the camera, and extract key frames from the continuous images using a redundant image screening method to obtain a key frame image sequence;

[0123] The point cloud sparse reconstruction unit 520 is used to extract and match feature points of the key frame image sequence to obtain matching feature points, and perform point cloud sparse reconstruction on the matching feature points using the SFM algorithm;

[0124] A perspective selection unit 530 is configured to select a perspective for the tracking image of each image in the key frame image sequence according to the result of the point cloud sparse reconstruction to obtain a resource image of each image;

[0125] a point cloud dense reconstruction unit 540 for calculating depth information, normal information, and matching cost of pixels in each image based on the sparse point cloud generated by the point cloud sparse reconstruction and the result of the view selection using a method based on photometric features and planar features, and performing fast dense reconstruction of the sparse point cloud using a fast fusion method;

[0126] The scale recovery and coordinate conversion unit 550 is used to perform scale recovery and coordinate conversion on the dense point cloud generated by dense reconstruction, and output a fast dense reconstruction result of the three-dimensional point cloud.

[0127] Based on the same concept, an embodiment of the present invention further provides an electronic device, Figure 6 FIG. 1 is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Figure 6As shown, the device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the steps of the method for fast dense reconstruction of a three-dimensional point cloud in a large scene as described in the above embodiments, for example, including:

[0128] Step S1, obtaining continuous images of a scene to be tested taken by a camera, and extracting key frames from the continuous images using a redundant image screening method to obtain a key frame image sequence;

[0129] Step S2: extracting and matching feature points of the key frame image sequence to obtain matching feature points, and using the SFM algorithm to perform point cloud sparse reconstruction on the matching feature points;

[0130] Step S3: selecting a viewing angle for the tracking image of each image in the key frame image sequence according to the result of the point cloud sparse reconstruction to obtain a resource image of each image;

[0131] Step S4: Calculate the depth information, normal information, and matching cost of each pixel in each image using a method based on photometric features and planar features based on the sparse point cloud generated by the point cloud sparse reconstruction and the result of the viewpoint selection, and perform fast dense reconstruction of the sparse point cloud using a fast fusion method;

[0132] Step S5: Perform scale recovery and coordinate conversion on the dense point cloud generated by dense reconstruction, and output a fast dense reconstruction result of the three-dimensional point cloud.

[0133] The processor 610 may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0134] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0135] The memory 630 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0136] Based on the same concept, an embodiment of the present invention further provides a non-transitory computer-readable storage medium storing a computer program. The computer program includes at least one code segment, which can be executed by a main control device to control the main control device to implement the steps of the method for fast dense reconstruction of a three-dimensional point cloud in a large scene as described in the above embodiments, for example, including:

[0137] Step S1, obtaining continuous images of a scene to be tested taken by a camera, and extracting key frames from the continuous images using a redundant image screening method to obtain a key frame image sequence;

[0138] Step S2: extracting and matching feature points of the key frame image sequence to obtain matching feature points, and using the SFM algorithm to perform point cloud sparse reconstruction on the matching feature points;

[0139] Step S3: selecting a viewing angle for the tracking image of each image in the key frame image sequence according to the result of the point cloud sparse reconstruction to obtain a resource image of each image;

[0140] Step S4: Calculate the depth information, normal information, and matching cost of each pixel in each image based on the sparse point cloud generated by the point cloud sparse reconstruction and the result of the viewpoint selection using a method based on photometric features and planar features, and perform fast dense reconstruction of the sparse point cloud using a fast fusion method;

[0141] Step S5: Perform scale recovery and coordinate conversion on the dense point cloud generated by dense reconstruction, and output a fast dense reconstruction result of the three-dimensional point cloud.

[0142] Based on the same technical concept, an embodiment of the present application also provides a computer program, which, when executed by a main control device, is used to implement the above method embodiment.

[0143] The program may be stored in whole or in part on a storage medium packaged with the processor, or may be stored in whole or in part on a memory not packaged with the processor.

[0144] Based on the same technical concept, the embodiment of the present application further provides a processor, which is used to implement the above method embodiment. The above processor can be a chip.

[0145] In summary, the present invention provides a method for fast dense reconstruction of three-dimensional point clouds in large scenes, which extracts key image sequences from continuous images for sparse reconstruction, obtains the camera pose, 2D key points, and 3D sparse points of each image to select the viewing angle of the tracking image, estimates the parameter information of each image using a method based on photometric features and planar features, and then uses a fast fusion method to densely reconstruct the point cloud. Finally, the dense point cloud is scaled and transformed. The method provided by the present invention calculates the viewing angle score of the image through the result of sparse reconstruction, selects images with higher scores for dense reconstruction, solves the problems of long point cloud reconstruction time and missing point clouds in weak texture areas in large scenes, improves the speed of dense reconstruction in large scene environments, and increases the detailed features of the surface of small target objects; at the same time, it improves the quality of dense reconstruction, and obtains a high-precision, high-density, high-quality environment model, which provides effective technical support for accurate perception of large scene environments and has important theoretical significance and engineering application value.

[0146] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for fast dense reconstruction of 3D point clouds in large scenes, characterized by: The method comprises the following steps: S1, obtaining continuous images of a scene to be tested taken by a camera, and extracting key frames from the continuous images using a redundant image screening method to obtain a key frame image sequence; S2. Extract and match feature points of the key frame image sequence to obtain matching feature points, and use the SFM algorithm to perform point cloud sparse reconstruction on the matching feature points; S3. Selecting a viewing angle for the tracking image of each image in the key frame image sequence according to the result of the point cloud sparse reconstruction to obtain a resource image of each image; S4. Calculating the depth information, normal information, and matching cost of pixels in each image based on the sparse point cloud generated by the point cloud sparse reconstruction and the result of the viewpoint selection using a method based on photometric features and planar features, and performing fast dense reconstruction of the sparse point cloud using a fast fusion method; S5. Perform scale recovery and coordinate conversion on the dense point cloud generated by dense reconstruction, and output the fast dense reconstruction result of the three-dimensional point cloud.

2. The method for fast dense reconstruction of 3D point clouds in large scenes according to claim 1, characterized in that: In step S1, a redundant image screening method is used to extract key frames from the continuous images to obtain a key frame image sequence, which includes: S11, selecting a frame image from the continuous image, extracting adjacent frames as frames to be evaluated, and calculating the similarity Cs between two adjacent frames using image block color features; S12, if the similarity Cs is less than 0.43, the adjacent frames are stored in a key frame group; otherwise, the global texture similarities D1, D2, and D3 between the two adjacent frames are calculated using LBP_max texture feature global statistical classification; S13, if the global texture similarity D1>4.61, D2>10, D3>14.8, then the adjacent frames are stored in the key frame group, otherwise the local texture similarities P1, P2 and P3 between the two adjacent frames are calculated using the LBP_max texture feature local statistical classification; S14, if the local texture similarity P1>72, P2>69, P3>58, then save the adjacent frames in the key frame group; otherwise, classify the adjacent frames as redundant images; S15 , extracting images in sequence from the images in the key frame group using a certain margin method, controlling the number of key frame images to be within 40 frames, and obtaining a key frame image sequence.

3. The method for fast dense reconstruction of 3D point clouds in large scenes according to claim 1, characterized in that: The step S2 specifically includes: S21, using a deep neural network SuperPoint to extract evenly distributed feature points and descriptors for each image in the key frame image sequence, and then using a feature point matching network SuperGlue to match the feature points and descriptors to obtain a feature matching relationship between the images; S22. Based on the feature matching relationship between the images, select a group of images with the highest matching degree to estimate the initial pose and transformation matrix of the camera, reconstruct a first sparse point cloud based on the principle of triangulation, and perform bundle adjustment optimization on the sparse point cloud; S23. Repeat step S22 for the remaining image sequences to perform point cloud sparse reconstruction to obtain the camera pose, 2D key points, and 3D sparse points of each image.

4. The method for fast dense reconstruction of 3D point clouds in large scenes according to claim 3, characterized in that: The step S3 specifically includes: The score between each image and the corresponding tracking image is calculated based on the camera pose, 2D key points and 3D sparse points of each image, and the top N tracking images with the highest scores are selected as the resource images of each image. The specific calculation process of the score is: Extract a 3D point p from the 3D sparse points in sequence and obtain all tracking images of the point p; Then traverse all tracking images and calculate the total score between the current image k and the remaining tracking images j: In the above formula, p is the common three-dimensional coordinate point of image k and image j, θ kj is the angle between the common trajectory point of views k and j and the two ends of the baseline, and its calculation formula is as follows: θ kj (p)S(180 / π)arcs((c k -p)·(c j -p)) In the above formula, c k is the position point of image k, c j is the position point of image j; At the same time, a piecewise Gaussian function g(θ) is used to calculate the score between the current image k and the remaining tracking images j, and its expression is: In the above formula, θ0, σ1, and σ2 are all constant terms.

5. The method for fast dense reconstruction of 3D point clouds in large scenes according to claim 1, characterized in that: The step S4 specifically includes: S41. Generate an initial map based on the photometric consistency features of each image and the corresponding resource image, and calculate the initial depth, initial normal, and initial matching cost of each pixel in each image based on the camera intrinsic parameters; S42, selecting the pixel point with the minimum local matching cost according to the initial matching cost to perform triangulation, calculating the features of the plane where the three coordinate points corresponding to each triangle are located, and assigning the plane features to the pixel points that meet the depth condition; S43, generating a new map based on the photometric features and planar features of each pixel, and updating the initial depth, initial normal, and initial matching cost of each pixel in each image based on the camera intrinsic parameters to generate depth information, normal information, and matching cost for dense reconstruction; S44, reprojecting the depth information of each image using GPU acceleration according to the photometric characteristics and spatial geometric characteristics; S45, comparing the matching cost of each image pixel point according to the reprojection calculation result, and marking the pixel points that meet the matching cost parameter threshold and the spatial point distance threshold; S46. Convert the marked pixel points into spatial coordinate points using a transformation matrix according to the camera intrinsic parameters and the camera position, and then convert the generated spatial coordinate points into the initial camera coordinate system according to the position of each image to obtain a dense reconstruction result.

6. The method for fast dense reconstruction of 3D point clouds in large scenes according to claim 1, characterized in that: The scale restoration of the dense point cloud generated by dense reconstruction in step S5 specifically includes: S51, performing random downsampling on the dense point cloud generated by the dense reconstruction, and selecting 1 / 5 of the points as sampling retention points; S52: Use a plane fitting algorithm to extract the plane where the ground is located for the sampled retention points, and calculate the distance P between the coordinate origin and the plane where the ground is located. z ; S53, determine a calibration object, and give the actual length Tl, width Tw, height Th of the calibration object and the actual distance H between the camera and the ground; S54. Calculate the scale parameters kx, ky, and kz of the point cloud in the X, Y, and Z directions based on the length Pl, width Pw, and height Ph of the calibration object in the point cloud, where: kx=Tl÷Pl ky=Tw÷Pw kz=Th÷Ph S55. Use the formula to restore the coordinates of each point in the dense point cloud in sequence. The formula is as follows: Among them, x, y, and z represent the horizontal coordinate, vertical coordinate, and vertical coordinate of the image point in the camera coordinate system, respectively.

7. The method for fast dense reconstruction of 3D point clouds in large scenes according to claim 1, characterized in that: The coordinate conversion in step S5 includes: The camera position is transformed according to the camera parameters of the first frame image in the key frame sequence image, and then the origin of the coordinate system is transformed by subtracting the camera coordinates from the coordinates of the densely reconstructed point cloud. Finally, the point cloud in the camera coordinate system is converted to the device coordinate system for point cloud display.

8. A fast dense reconstruction system for 3D point clouds in large scenes, characterized by: The system comprises: An image acquisition and key frame extraction unit is used to acquire continuous images of the scene to be measured taken by a camera, and to extract key frames from the continuous images using a redundant image screening method to obtain a key frame image sequence; A point cloud sparse reconstruction unit is used to extract and match feature points of the key frame image sequence to obtain matching feature points, and use the SFM algorithm to perform point cloud sparse reconstruction on the matching feature points; a perspective selection unit, configured to select a perspective of the tracking image of each image in the key frame image sequence according to the result of the point cloud sparse reconstruction, so as to obtain a resource image of each image; a point cloud dense reconstruction unit, configured to calculate the depth information, normal information, and matching cost of pixels in each image based on the sparse point cloud generated by the point cloud sparse reconstruction and the result of the view selection using a method based on photometric features and planar features, and to perform fast dense reconstruction of the sparse point cloud using a fast fusion method; The scale recovery and coordinate transformation unit is used to perform scale recovery and coordinate transformation on the dense point cloud generated by dense reconstruction, and output the fast dense reconstruction result of the three-dimensional point cloud.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method for fast and dense reconstruction of three-dimensional point clouds in a large scene as described in any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for fast dense reconstruction of three-dimensional point clouds in a large scene are implemented as claimed in any one of claims 1 to 7.

Images