Dense point cloud data generation method and device, equipment and storage medium

Abstract

The invention discloses a dense point cloud data generation method, apparatus and device, and a storage medium, and aims at performing edge extraction on a ground image during dense reconstruction of an outdoor scene, generating a first image containing edge information, and then identifying a target of a specified category in the first image, and assigning a pixel corresponding to the target of the specified category as a specified color in the original ground image to obtain a second image. In a subsequent process, after edge extraction and pixel assignment, interference of a specified category on a dense reconstruction algorithm can be reduced to a great extent, in addition, pixels of a target of the specified category are filled with specified colors, and after a fusion point cloud is obtained by using a second image, the fusion point cloud is obtained through a color filtering mode. According to the method, interference points can be removed on the premise that other correct dense points are hardly influenced, so that cleaner outdoor scene dense point cloud is obtained, and dense point cloud data with more complete structure and plane can be obtained.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, apparatus, device and storage medium for generating dense point cloud data. Background Art

[0002] Image dense matching technology plays an important role in many fields, especially in 3D reconstruction, robot navigation, augmented reality and autonomous driving. With the development of technology, the demand for high-precision 3D models continues to grow. As a key technology for obtaining disparity maps, image dense matching is crucial for 3D reconstruction. In the dense reconstruction task of outdoor scenes, some special surfaces in outdoor scenes, such as the sky, lack sufficient texture information, making it difficult to extract and match feature points, affecting the accuracy of matching. In particular, the pixels of the sky account for a large proportion in the image, and the long distance and weak texture characteristics of the sky make it an important interference source for dense reconstruction of outdoor scenes, which easily leads to a large number of divergent outliers in the generated dense point cloud. However, when generating dense point clouds, the existing technology does not take into account the influence of these special surfaces, resulting in a large amount of interference data in the generated dense point cloud data. Summary of the Invention

[0003] The purpose of the embodiments of the present invention is to provide a method, apparatus, device and storage medium for generating dense point cloud data, which can effectively remove interference data in dense point clouds and obtain cleaner dense point cloud data of outdoor scenes.

[0004] To achieve the above objectives, an embodiment of the present invention provides a method for generating dense point cloud data, comprising:

[0005] Perform edge extraction on the ground image to generate a first image containing edge information;

[0006] Identifying the categories of all objects in the first image, and assigning a specified color to pixels corresponding to objects belonging to a specified category in the ground image, to obtain a second image;

[0007] Performing sparse reconstruction on the second image to obtain sparse point cloud data of the pose and scene of the second image;

[0008] Using two dense reconstruction methods to process the sparse point cloud data of the scene, the second image and its pose, respectively, to obtain two types of dense point cloud data;

[0009] Fuse the two dense point cloud data to obtain fused point cloud data;

[0010] Color filtering is performed on the fused point cloud data to obtain target point cloud data.

[0011] As an improvement to the above solution, identifying the categories of all objects in the first image includes:

[0012] Perform semantic segmentation on the first image to identify categories of all objects in the first image.

[0013] As an improvement to the above solution, the sparse reconstruction of the second image to obtain the sparse point cloud data of the pose and scene of the second image includes:

[0014] performing feature point extraction on the second image;

[0015] matching the feature points of the second images based on the similarity between the image position information of the second images and the extracted feature point descriptors to obtain a matching relationship;

[0016] Generate a scene graph based on the matching relationship;

[0017] According to the scene graph, the second image is positioned and posed to obtain the pose of the second image, and sparse point cloud data of the scene is generated.

[0018] As an improvement to the above solution, the two dense reconstruction methods are used to process the pose of the second image and the sparse point cloud data of the scene, respectively, to obtain two types of dense point cloud data, including:

[0019] Processing the pose of the second image and the sparse point cloud data of the scene using a dense reconstruction method based on a traditional MVS algorithm to obtain first dense point cloud data;

[0020] The pose of the second image and the sparse point cloud data of the scene are processed using a dense reconstruction method based on the deep learning MVS algorithm to obtain second dense point cloud data.

[0021] As an improvement to the above solution, after obtaining the two types of dense point cloud data, the method further includes:

[0022] Performing denoising on the first dense point cloud data to obtain denoised point cloud data;

[0023] Plane feature calculation is performed on the second dense point cloud data to obtain surface point cloud data.

[0024] As an improvement to the above solution, the two dense point cloud data are fused to obtain fused point cloud data, including:

[0025] For each reference point in the surface point cloud data, calculating a first distance from all reference points to a target point in the denoised point cloud data;

[0026] Screening out plane points to be retained from the surface point cloud data according to a comparison result of the first distance and a preset second distance;

[0027] The planar points are fused with the first dense point cloud to obtain fused point cloud data.

[0028] As an improvement to the above solution, color filtering is performed on the fused point cloud data to obtain target point cloud data, including:

[0029] Identifying a specified color in the fused point cloud data;

[0030] Delete the corresponding three-dimensional points of the specified color to obtain target point cloud data.

[0031] To achieve the above objectives, an embodiment of the present invention further provides a dense point cloud data generating device, comprising:

[0032] An edge extraction module is used to extract edges from a ground image and generate a first image containing edge information;

[0033] a pixel processing module, configured to identify the categories of all objects in the first image and assign a specified color to pixels corresponding to objects belonging to a specified category in the ground image, thereby obtaining a second image;

[0034] a sparse reconstruction module, configured to perform sparse reconstruction on the second image to obtain sparse point cloud data of the pose and scene of the second image;

[0035] a point cloud data generation module, configured to process the sparse point cloud data of the scene, the second image and its position and posture using at least two dense reconstruction methods respectively, to obtain at least two types of dense point cloud data;

[0036] A point cloud data fusion module is used to fuse at least two types of dense point cloud data to obtain fused point cloud data;

[0037] The point cloud data filtering module is used to perform color filtering on the fused point cloud data to obtain target point cloud data.

[0038] To achieve the above-mentioned objectives, an embodiment of the present invention also provides a dense point cloud data generation device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the dense point cloud data generation method as described in any of the above-mentioned embodiments is implemented.

[0039] To achieve the above-mentioned purpose, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the dense point cloud data generation method as described in any of the above embodiments.

[0040] Compared with the prior art, the dense point cloud data generation method, device, equipment and storage medium disclosed in the present invention, when densely reconstructing outdoor scenes, first performs edge extraction on the ground image to generate a first image containing edge information. Edge extraction can make the identification of targets of a specified category more accurate. Then, the targets of the specified category in the first image are identified, and the pixels corresponding to the targets of the specified category in the ground image are assigned to a specified color to obtain a second image. In the subsequent process, after edge extraction and pixel assignment, the interference of the specified category on the dense reconstruction algorithm can be greatly reduced. In addition, the pixels of the targets of the specified category are filled with a specified color. After obtaining the fused point cloud using the second image, these interfering points can be removed by color filtering without affecting other correct dense points, thereby obtaining a cleaner dense point cloud of the outdoor scene. At the same time, by fusing the dense point cloud data obtained by the two dense reconstruction methods, dense point cloud data with more complete structure and plane can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of a method for generating dense point cloud data provided by an embodiment of the present invention;

[0042] Figure 2 This is a structural block diagram of a dense point cloud data generating device provided by an embodiment of the present invention;

[0043] Figure 3 This is a structural block diagram of a dense point cloud data generation device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0045] See also Figure 1 , Figure 1 is a flow chart of a method for generating dense point cloud data provided by an embodiment of the present invention, the method comprising:

[0046] S1. Extracting edges from a ground image to generate a first image containing edge information;

[0047] S2. Identify the categories of all objects in the first image, and assign a specified color to pixels corresponding to objects belonging to a specified category, thereby obtaining a second image;

[0048] S3. Perform sparse reconstruction on the second image to obtain sparse point cloud data of the pose and scene of the second image;

[0049] S4. Processing the sparse point cloud data of the scene, the second image, and its pose using two dense reconstruction methods respectively to obtain two types of dense point cloud data;

[0050] S5. Fusing the two dense point cloud data to obtain fused point cloud data;

[0051] S6. Perform color filtering on the fused point cloud data to obtain target point cloud data.

[0052] Exemplarily, the ground image is a general term for all images captured from a ground perspective, using tools including, but not limited to, SLR cameras and mobile phones. For dense reconstruction of outdoor scenes, the input ground image is first edge extracted to generate a first image containing edge information. Edge extraction can more accurately identify targets of a specified category. Then, targets of the specified category are identified in the first image, and the pixels corresponding to the targets of the specified category in the ground image are assigned a specified color to obtain a second image. In subsequent processes, after edge extraction and pixel assignment, the interference of the specified category on the dense reconstruction algorithm can be greatly reduced. Furthermore, the pixels of the targets of the specified category are filled with a specified color. After obtaining a fused point cloud using the second image, color filtering can be used to remove these interfering points without affecting other correct dense points, thereby obtaining a cleaner dense point cloud for the outdoor scene. Furthermore, by fusing the dense point cloud data obtained by the two dense reconstruction methods, dense point cloud data with more complete structure and plane can be obtained.

[0053] Specifically, in step S1, the edge of the ground image is extracted to generate a first image containing edge information. Canny edge extraction is performed on the collected ground image data. In order to improve the integrity of the image semantic segmentation at the edge and prevent the ground object pixels that are connected to the specified category of target (such as the sky) from being mistakenly removed, the present invention first extracts the edge of the collected ground image I raw Perform Canny edge extraction. Specifically, the Canny algorithm uses a high threshold T H and low threshold T L To determine the strong edge and weak edge. Where strong edge = {pixel|M>T H}, weak edge = {pixel | T L ≤M≤T H Where M is the gradient magnitude, which satisfies the following formula:

[0054]

[0055] Among them, G x *I and G y *I represents the convolution results of image I and Sobel operator in horizontal and vertical directions respectively. Sobel operator G in horizontal direction x and the Sobel operator G in the vertical direction y Satisfies the following formula:

[0056]

[0057] After edge extraction, the first image set I with edge information recorded can be obtained. edge .

[0058] Furthermore, in order to improve the edge enhancement effect, the present invention uses the dilation operation in morphology to perform the dilation on the first image set I edge The edge information of all images in the image set is thickened. Then, the thickened edges are superimposed on the original image with black pixels to obtain the edge-enhanced first image set I enhanced .

[0059] Specifically, in step S2, identifying the categories of all objects in the first image includes: performing semantic segmentation on the first image to identify the categories of all objects in the first image.

[0060] Exemplarily, the edge-enhanced first image set I enhanced With the development of deep learning, image semantic segmentation has become a relatively mature technology, which can give different labels to pixels of different categories (such as sky, building, plant, ground, river, etc.) in the image. Using deep learning networks such as Efficient PS and SegFormer, semantic segmentation of each image can be quickly achieved. In the embodiment of the present invention, using I enhanced As the input of semantic segmentation. The first image is the ground image after edge enhancement and expansion, which is used as the input of semantic segmentation to obtain the semantic segmentation information of the image (pixel-by-pixel label). enhanced After semantic segmentation of all images in step S1, the original ground image (i.e., the ground image I without edge extraction in step S1) is segmented. raw ), assign the pixels corresponding to the target belonging to the specified category to the specified color according to the semantic information to obtain the second image.raw The pixels marked as sky are assigned white, and the pixels of other categories are not processed, thereby obtaining the second image set I without the sky. processed .

[0061] Furthermore, the embodiment of the present invention provides a pixel assignment method: the first image set I enhanced After semantic segmentation, a mask image is generated, and then the ground image I raw According to the semantic category information in the mask image, the pixels in the sky part are assigned a specified color (such as white).

[0062] In the dense reconstruction of outdoor scenes, the sky's pixels make up a significant portion of the image. The sky's long distance and weak texture make it a significant source of interference, easily leading to a large number of divergent outliers in the generated dense point cloud. To address this, the present invention employs an image semantic segmentation algorithm to remove this interference source from the data source, laying the foundation for generating a clean, dense point cloud.

[0063] Specifically, in step S3, the second image is sparsely reconstructed to obtain the posture of the second image and sparse point cloud data of the scene, including: extracting feature points of the second image; matching feature points between the second images based on the similarity between the image position information of the second image and the extracted feature point descriptors to obtain a matching relationship; generating a scene graph according to the matching relationship; and positioning the second image according to the scene graph to obtain the posture of the second image, and generating sparse point cloud data of the scene.

[0064] For example, for the second image set I processed Feature point extraction and matching are performed on all images in the scene. The feature points here can be traditional manual features. There are many methods for feature extraction, including but not limited to manual features SIFT (Scale-invariant feature transform), deep learning features super point, etc. Since many feature points can be extracted from each second image, the matching relationship between feature points of different images is unknown at the beginning, so feature matching is required. Feature point matching is a necessary step in the sparse reconstruction process, which is also called Structure From Motion (SFM). The sparse reconstruction process records the matching relationship of all second images in the scene. There will be "edges" between images with matching relationships, so that a graph can be formed. Therefore, a scene graph can be generated based on the matching relationships. Based on the scene graph, the second images can be positioned and posed using SFM technology to obtain the poses of all second images and generate a sparse point cloud model of the scene.

[0065] Specifically, in step S4, the two dense reconstruction methods are respectively used to process the sparse point cloud data of the scene, the second image and its posture to obtain two dense point cloud data, including: using a dense reconstruction method based on the traditional MVS algorithm to process the sparse point cloud data of the scene, the second image and its posture to obtain first dense point cloud data; using a dense reconstruction method based on the deep learning MVS algorithm to process the sparse point cloud data of the scene, the second image and its posture to obtain second dense point cloud data.

[0066] Exemplarily, dense reconstruction is performed on the sparse point cloud data of the scene, the second image, and its pose to obtain dense point cloud data. Here, dense reconstruction includes two types of algorithms: traditional and deep learning-based, and two dense point cloud results can be obtained. The two algorithms are described as follows:

[0067] 1) Use the traditional MVS (Multi-View Stereo) algorithm for dense reconstruction to obtain the first dense point cloud data P t . Traditional MVS algorithms usually include stereo matching, disparity map and depth map estimation, depth map fusion, point cloud generation and other steps. Among them, stereo matching determines the correspondence between feature points in images of different perspectives by comparing them. This step may involve calculating the similarity between image blocks to find the best match. Then, based on the results of stereo matching, the disparity of each pixel is calculated and converted into depth information. The depth map shows the distance of each pixel in the scene from the camera. Finally, the depth maps of multiple perspectives need to be fused into a complete point cloud. The first dense point cloud data P t .

[0068] It should be noted that MVS exploits the consistency between multiple viewpoints to recover a dense point cloud of the scene. It uses the camera pose parameters from SFM to capture richer information in the scene. Its main goal is to determine the most effective method to match corresponding points in different images, thereby increasing the density of the scene and enhancing the quality of reconstruction. MVS can be implemented through the following three main methods:

[0069] ① Voxel-based reconstruction; the voxel grid method reconstructs the 3D model by voxelizing the entire scene and assigning an occupancy value to each voxel;

[0070] ② Feature propagation: The feature propagation method generates surface slices and initial point clouds based on initial feature points, then projects the feature points onto the image and propagates them to the surrounding area;

[0071] ③ Depth map fusion; The depth map fusion method obtains a dense point cloud by selecting a reference image, estimating and optimizing the depth map.

[0072] Although traditional dense matching algorithms are highly robust, they still face significant challenges when dealing with large-scale and complex scenarios, especially in terms of computational efficiency and matching accuracy.

[0073] 2) Use the MVS algorithm based on deep learning for dense reconstruction to obtain the second dense point cloud data P d The MVS algorithm based on deep learning usually first uses the Feature Pyramid Network (FPN) to extract multi-scale deep image features, and then recovers the dense point cloud of the scene by aggregating the global context information of these features to obtain the second dense point cloud data P d .

[0074] It should be noted that stereo matching and multi-view stereo (MVS) based on deep learning are hot research directions in the field of computer vision. Traditional stereo matching algorithms require multiple steps, including feature extraction, matching, disparity calculation and optimization. The deep learning-based method can achieve end-to-end learning and learn the disparity map directly from the stereo image pair. MVSNet is the first network to apply deep learning to MVS. It estimates the depth map by learning image features and constructing the cost volume. Subsequent work, such as RMVSNet, PointMVSNet and P-MVSNet, has improved the memory efficiency and reconstruction quality of MVS by improving the network structure and introducing new regularization techniques. Stereo matching and MVS methods based on deep learning have made significant progress in the field of 3D reconstruction. By using image pyramid technology, the depth of weak texture areas can be better restored.

[0075] Specifically, after executing step S4, the method further includes: performing denoising processing on the first dense point cloud data to obtain denoised point cloud data; and performing plane feature calculation on the second dense point cloud data to obtain surface point cloud data.

[0076] For example, for the first dense point cloud data P t Perform denoising to obtain denoised point cloud data P s The existence of noise will affect the point cloud distance calculation in subsequent steps, so firstly, the first dense point cloud data P t Perform denoising to remove some outliers that are far away from surrounding points. For example, if the distance threshold T is given D1 and quantity threshold T N , for the first dense point cloud data P t For each point in the , with this point as the center, if the distance threshold T D1 The sum of all points in is less than T N , then the point is considered to be an external point and is removed, otherwise it is retained, and the denoised point cloud is recorded as the denoised point cloud data Ps .

[0077] For example, for the second dense point cloud data P d Perform plane feature calculation and extract surface point cloud data P p In the outdoor scene reconstruction task, planar structures such as walls and windows are common weak texture areas. In order to fully utilize the reconstruction advantages of the deep learning MVS algorithm for these areas, the present invention extracts the planar point cloud of the dense point cloud by calculating the flatness of the point cloud. Specifically, for the second dense point cloud data P d For each point, given a certain range of neighborhood, all three-dimensional points in the neighborhood are recorded as a set P = {p1, p2, ..., p n}, where n is the number of all points in the set, p i represents the i-th point, i=1,2...n, p i The coordinates of (x i ,y i ,z i ). Then, calculate the mean μ of all points in the set P in the x, y, and z directions respectively. x 、μ y 、μ z , satisfying the following formula:

[0078]

[0079] Based on the above results, the covariance matrix Cov is constructed to satisfy the following formula:

[0080]

[0081] It should be noted that, assuming A is an n-order matrix, that is, A is the covariance matrix in the above formula (4), if there exists a number λ and an n-dimensional non-zero vector X such that AX = λX (X≠0), then λ is called an eigenvalue of A. It can be concluded that: X is the eigenvector of A corresponding to the eigenvalue λ. Based on this, by performing eigenvalue decomposition on the covariance matrix, three eigenvalues ​​λ1, λ2, and λ3 can be obtained, and they satisfy: λ1≥λ2≥λ3≥0.

[0082] Then, the flatness P λ The calculation process satisfies the following formula:

[0083]

[0084] It should be noted that P λ The range of λ is [0,1]. In the plane feature, λ3 should be significantly smaller than the other two eigenvalues. Therefore, P λ The larger the value is, the higher the flatness of the target point is. Given a plane threshold T P , we can satisfy Pλ >T P All points are considered as plane points, as surface point cloud data P p .

[0085] Specifically, in step S5, the two dense point cloud data are fused to obtain fused point cloud data, including: for each reference point in the surface point cloud data, calculating the first distance from all reference points to the target point in the denoised point cloud data; screening out plane points to be retained from the surface point cloud data based on the comparison result of the first distance and a preset second distance; and fusing the plane points with the first dense point cloud to obtain fused point cloud data.

[0086] For example, the surface point cloud data P is calculated p Each reference point in the denoised point cloud data P s The first distance to the nearest point (target point) in the image, which involves the search for the nearest neighbor point. Generally, the denoised point cloud data P s Construct a KD tree index, and then you can easily find the surface point cloud data P p Each point in the denoised point cloud data P s The closest point distance is recorded as the first distance. If the first distance of any reference point is greater than the second distance T D2 , then keep the reference point, otherwise delete the reference point. Generally, when T D2 <T D1 (The first dense point cloud data P t When the distance threshold used in the point cloud denoising step is used, a relatively complete fusion result can be obtained. The plane points finally retained are the same as the first dense point cloud data P t Fusion is performed by directly merging the two point clouds into one point cloud to obtain the fused point cloud data P m .

[0087] In this embodiment of the present invention, the dense point cloud obtained by the traditional MVS algorithm has better recovery and more complete structure in areas with rich textures. In contrast, the deep learning-based MVS algorithm, by utilizing global information from images of varying resolutions, can better restore depth information in areas with weak textures, such as walls. Therefore, the present invention combines the advantages of both MVS algorithms and achieves a more complete dense reconstruction through point cloud fusion.

[0088] Specifically, in step S6, color filtering is performed on the fused point cloud data to obtain target point cloud data, including: identifying a specified color in the fused point cloud data; and deleting three-dimensional points corresponding to the specified color to obtain target point cloud data.

[0089] For example, the fused point cloud data P mPerform color filtering to obtain the final target point cloud data. In the edge area of ​​the object, due to the sudden change in depth, erroneous depth estimation results are likely to occur, and burrs will appear at the edge, especially in the part where the ground object meets the sky. Thanks to the present invention, the sky pixels are processed into a specified color (such as pure white) in advance, and the corresponding three-dimensional points of the specified color can be easily removed through color filtering. Taking the specified color as white as an example, the three-dimensional points with a color value of white (or nearly white) are directly removed, so as to obtain a cleaner dense point cloud as the final result. Specifically, given a filtering threshold T color , the filtered points satisfy the following constraints:

[0090] C R ≥T color and C G ≥T color and C B ≥T color (6);

[0091] Among them, (C R ,C G ,C B ) are the RGB color values ​​of the point, and the range is [0,255]. The closer these three values ​​are to 255, the closer the color is to white. Therefore, T color The value range of T is also [0,255]. color The larger the value, the fewer 3D points will be filtered out. The specific value depends on the color value of the ground objects in the actual scene. Due to the influence of viewing angle and lighting conditions, even if the ground objects are actually pure white, their images will usually not be pure white. Therefore, in general, when selecting a suitable threshold color T color Finally, color filtering does not affect the correct ground features.

[0092] Compared with the prior art, the dense point cloud data generation method disclosed in the embodiment of the present invention performs edge extraction and enhancement on the input image, and then fills the specified category with a specified color through semantic segmentation. The edge extraction and enhancement of the image can make the identification of targets of the specified category more accurate, and protect the buildings from being mistakenly identified as targets of the specified category. Then, in the subsequent processes, the image without the target of the specified category is used as input, which can greatly reduce the interference of the target of the specified category on the dense reconstruction algorithm. Thanks to the filling of the pixels of the target of the specified category with the specified color, after obtaining the fused point cloud, the present invention can remove these interfering pixels through color filtering without affecting other correct dense points, thereby obtaining a cleaner dense point cloud of the outdoor scene.

[0093] In addition, based on the understanding of the two major categories of dense reconstruction algorithms, traditional and deep learning-based, the advantages of the two algorithms are integrated to generate a more complete dense point cloud. In contrast, the traditional algorithm has a better reconstruction effect on structural areas with richer textures, while the deep learning algorithm can better restore the depth of weak texture areas. Therefore, based on the same sparse reconstruction result, the present invention aligns the traditional and deep learning-based dense reconstructions respectively to ensure that they are in the same coordinate system. Then, by calculating and extracting the planarity of the deep learning dense point cloud, denoising and distance calculation and fusion of the traditional dense point cloud, a dense point cloud with complete structure and plane is obtained.

[0094] Compared to existing algorithms, this method removes the sky from ground images through edge-enhanced semantic segmentation, reducing its interference with dense point cloud generation. Furthermore, color filtering removes near-white outliers that are mistakenly reconstructed around buildings, resulting in a cleaner dense point cloud. Furthermore, this method leverages the respective strengths of traditional and deep learning-based dense reconstruction algorithms, generating a dense point cloud with both complete planar and structural integrity through techniques such as planarity calculation and extraction, point cloud denoising, and fusion.

[0095] See also Figure 2 , Figure 2 is a structural block diagram of a dense point cloud data generating apparatus 100 provided in an embodiment of the present invention, wherein the dense point cloud data generating apparatus 100 comprises:

[0096] The edge extraction module 11 is used to extract the edge of the ground image and generate a first image containing edge information;

[0097] a pixel processing module 12 for identifying the categories of all objects in the first image and assigning a specified color to pixels corresponding to objects belonging to a specified category in the ground image to obtain a second image;

[0098] a sparse reconstruction module 13, configured to perform sparse reconstruction on the second image to obtain sparse point cloud data of the pose and scene of the second image;

[0099] a point cloud data generating module 14 for processing the sparse point cloud data of the scene, the second image and its position and posture using at least two dense reconstruction methods respectively to obtain at least two types of dense point cloud data;

[0100] a point cloud data fusion module 15 for fusing at least two types of dense point cloud data to obtain fused point cloud data;

[0101] The point cloud data filtering module 16 is used to perform color filtering on the fused point cloud data to obtain target point cloud data.

[0102] Specifically, the pixel processing module 12 is configured to perform semantic segmentation on the first image to identify categories of all objects in the first image.

[0103] Specifically, the sparse reconstruction module 13 is specifically used to: extract feature points from the second image; match feature points between the second images based on the similarity between the image position information of the second image and the extracted feature point descriptors to obtain a matching relationship; generate a scene graph based on the matching relationship; and perform pose positioning on the second image based on the scene graph to obtain the pose of the second image, and generate sparse point cloud data of the scene.

[0104] Specifically, the point cloud data generation module 14 is specifically used to: use a dense reconstruction method based on the traditional MVS algorithm to process the sparse point cloud data of the scene, the second image and its posture to obtain first dense point cloud data; use a dense reconstruction method based on the deep learning MVS algorithm to process the sparse point cloud data of the scene, the second image and its posture to obtain second dense point cloud data.

[0105] Specifically, the point cloud data generating module 14 is further configured to: perform denoising on the first dense point cloud data to obtain denoised point cloud data; and perform plane feature calculation on the second dense point cloud data to obtain surface point cloud data.

[0106] Specifically, the point cloud data fusion module 15 is specifically used to: calculate the first distance from all reference points to the target point in the denoised point cloud data for each reference point in the surface point cloud data; filter out the plane points that need to be retained from the surface point cloud data based on the comparison result of the first distance and the preset second distance; and fuse the plane points with the first dense point cloud to obtain fused point cloud data.

[0107] Specifically, the point cloud data filtering module 16 is specifically used to: identify a designated color in the fused point cloud data; and delete the three-dimensional points corresponding to the designated color to obtain target point cloud data.

[0108] It is worth noting that the working process of each module in the dense point cloud data generating device described in the embodiment of the present invention can refer to the flowchart of the dense point cloud data generating method described in the above embodiment, and will not be repeated here.

[0109] See also Figure 3 , Figure 32 is a block diagram of a dense point cloud data generation device 200 provided in an embodiment of the present invention. The dense point cloud data generation device 200 includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, the steps of each of the above-mentioned dense point cloud data generation method embodiments are implemented.

[0110] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in the dense point cloud data generating device 200.

[0111] The dense point cloud data generating device 200 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will appreciate that the schematic diagram is merely an example of the dense point cloud data generating device 200 and does not limit the dense point cloud data generating device 200. The dense point cloud data generating device 200 may include more or fewer components than shown in the diagram, or may combine certain components or different components. For example, the dense point cloud data generating device 200 may further include input and output devices, network access devices, buses, and the like.

[0112] The processor 21 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor. The processor 21 is the control center of the dense point cloud data generation device 200, and connects various parts of the entire dense point cloud data generation device 200 using various interfaces and lines.

[0113] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements the various functions of the dense point cloud data generation device 200 by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 can mainly include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory 22 can include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0114] If the modules / units integrated in the dense point cloud data generation device 200 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the above-mentioned embodiment methods by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor 21, it can implement the steps of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium.

[0115] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for generating dense point cloud data, characterized in that: include: Perform edge extraction on the ground image to generate a first image containing edge information; Identifying the categories of all objects in the first image, and assigning a specified color to pixels corresponding to objects belonging to a specified category in the ground image, to obtain a second image; Performing sparse reconstruction on the second image to obtain sparse point cloud data of the pose and scene of the second image; Using two dense reconstruction methods to process the sparse point cloud data of the scene, the second image and its pose, respectively, to obtain two types of dense point cloud data; Fuse the two dense point cloud data to obtain fused point cloud data; Color filtering is performed on the fused point cloud data to obtain target point cloud data.

2. The method for generating dense point cloud data according to claim 1, wherein: The identifying the categories of all objects in the first image includes: Perform semantic segmentation on the first image to identify categories of all objects in the first image.

3. The method for generating dense point cloud data according to claim 1, wherein: The sparse reconstruction of the second image to obtain sparse point cloud data of the pose and scene of the second image includes: performing feature point extraction on the second image; matching the feature points of the second images based on the similarity between the image position information of the second images and the extracted feature point descriptors to obtain a matching relationship; Generate a scene graph based on the matching relationship; According to the scene graph, the second image is positioned and posed to obtain the pose of the second image, and sparse point cloud data of the scene is generated.

4. The method for generating dense point cloud data according to claim 1, wherein: The two dense reconstruction methods are used to process the sparse point cloud data of the scene, the second image and its posture respectively to obtain two types of dense point cloud data, including: Processing the sparse point cloud data of the scene, the second image and its pose using a dense reconstruction method based on a traditional MVS algorithm to obtain first dense point cloud data; The sparse point cloud data of the scene, the second image and its pose are processed using a dense reconstruction method based on a deep learning MVS algorithm to obtain second dense point cloud data.

5. The method for generating dense point cloud data according to claim 4, wherein: After obtaining the two types of dense point cloud data, the method further includes: Performing denoising on the first dense point cloud data to obtain denoised point cloud data; Plane feature calculation is performed on the second dense point cloud data to obtain surface point cloud data.

6. The method for generating dense point cloud data according to claim 5, wherein: The fusing of the two dense point cloud data to obtain fused point cloud data includes: For each reference point in the surface point cloud data, calculating a first distance from all reference points to a target point in the denoised point cloud data; Screening out plane points to be retained from the surface point cloud data according to a comparison result of the first distance and a preset second distance; The planar points are fused with the first dense point cloud to obtain fused point cloud data.

7. The method for generating dense point cloud data according to claim 1, wherein: The color filtering of the fused point cloud data to obtain target point cloud data includes: Identifying a specified color in the fused point cloud data; Delete the corresponding three-dimensional points of the specified color to obtain target point cloud data.

8. A dense point cloud data generating device, characterized in that: include: An edge extraction module is used to extract edges from a ground image and generate a first image containing edge information; a pixel processing module, configured to identify the categories of all objects in the first image and assign a specified color to pixels corresponding to objects belonging to a specified category in the ground image, thereby obtaining a second image; a sparse reconstruction module, configured to perform sparse reconstruction on the second image to obtain sparse point cloud data of the pose and scene of the second image; a point cloud data generation module, configured to process the sparse point cloud data of the scene, the second image and its position and posture using at least two dense reconstruction methods respectively, to obtain at least two types of dense point cloud data; A point cloud data fusion module is used to fuse at least two types of dense point cloud data to obtain fused point cloud data; The point cloud data filtering module is used to perform color filtering on the fused point cloud data to obtain target point cloud data.

9. A dense point cloud data generation device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the method for generating dense point cloud data according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the dense point cloud data generating method according to any one of claims 1 to 8.

Images