Point cloud data processing method, device, equipment and storage medium

Abstract

The present application discloses a point cloud data processing method, device, equipment and storage medium, the method comprising: obtaining a set of two-dimensional images to be reconstructed, determining multiple groups of feature matching pairs in the two-dimensional image set based on the two-dimensional feature points in each two-dimensional image; wherein each group of feature matching pairs corresponds to a three-dimensional space point, and the feature matching pairs include multiple matching two-dimensional feature points; meshing the two-dimensional image according to a preset number of feature points, determining the two-dimensional feature points in each grid, and determining the three-dimensional space points corresponding to the grid according to the three-dimensional space points corresponding to the two-dimensional feature points; selecting the target three-dimensional space points of the grid from the three-dimensional space points corresponding to the grid as point cloud data for constructing a three-dimensional scene according to the number of observations of the three-dimensional space points and the preset internal point conditions, and the number of observations is equal to the number of two-dimensional feature points corresponding to the three-dimensional space points. Through the above technical means, the problems of low construction efficiency and poor construction quality of the existing three-dimensional reconstruction methods are solved.

Description

Technical Field

[0001] The present application relates to the field of space digitization technology, and in particular to a point cloud data processing method, device, equipment and storage medium. Background Art

[0002] Spatial digitization technology is based on the image information collected by the camera to construct a real three-dimensional scene from a two-dimensional image. The quality and efficiency of the three-dimensional scene construction are closely related to the spatial distribution and quantity of point cloud data. Evenly distributed point cloud data and appropriate quantity will improve construction performance.

[0003] The current 3D reconstruction method is to determine the matching feature points between each image, determine the real position of the corresponding spatial point through the matching feature points, and construct the 3D scene based on the spatial points. However, a large number of spatial points will be determined based on the feature points, and the uniform distribution of the spatial points cannot be guaranteed, resulting in low efficiency and poor quality in the construction of the 3D scene. Summary of the invention

[0004] The embodiments of the present application provide a point cloud data processing method, apparatus, device and storage medium, which solve the problem that the three-dimensional reconstruction method in the prior art constructs a three-dimensional scene based on a large number of unevenly distributed spatial points, resulting in low construction efficiency and poor construction quality, thereby improving the construction efficiency and construction quality of the three-dimensional scene.

[0005] In a first aspect, an embodiment of the present application provides a point cloud data processing method, comprising:

[0006] Acquire a set of two-dimensional images to be reconstructed, and determine multiple groups of feature matching pairs in the two-dimensional image set based on the two-dimensional feature points in each two-dimensional image; wherein each group of feature matching pairs corresponds to a three-dimensional space point, and the feature matching pair includes multiple matching two-dimensional feature points;

[0007] Dividing the two-dimensional image into grids according to a preset number of feature points, determining the two-dimensional feature points in each grid, and determining the three-dimensional space points corresponding to the grids according to the three-dimensional space points corresponding to the two-dimensional feature points;

[0008] According to the number of observations of the three-dimensional space point and the preset internal point condition, the target three-dimensional space point of the grid is selected from the three-dimensional space points corresponding to the grid as point cloud data for constructing a three-dimensional scene, and the number of observations is equal to the number of two-dimensional feature points corresponding to the three-dimensional space point.

[0009] In a second aspect, an embodiment of the present application provides a point cloud data processing device, comprising:

[0010] A data acquisition module is configured to acquire a three-dimensional space point and a corresponding plurality of two-dimensional feature points, each of the two-dimensional feature points being distributed in a corresponding two-dimensional image;

[0011] A grid division module is configured to divide the two-dimensional image into grids according to a preset number of feature points, determine the two-dimensional feature points in each grid, and determine the three-dimensional space points corresponding to the grids according to the three-dimensional space points corresponding to the two-dimensional feature points;

[0012] The data filtering module is configured to select the target three-dimensional space point of the grid from the three-dimensional space points corresponding to the grid as point cloud data for constructing a three-dimensional scene based on the number of observations of the three-dimensional space point and a preset internal point condition, and the number of observations is equal to the number of two-dimensional feature points corresponding to the three-dimensional space point.

[0013] In a third aspect, an embodiment of the present application provides a point cloud data processing device, including:

[0014] One or more processors; a storage device storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the point cloud data processing method as described in the first aspect.

[0015] In a fourth aspect, an embodiment of the present application provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the point cloud data processing method as described in the first aspect.

[0016] This application divides the two-dimensional image evenly into grids, selects a three-dimensional space point corresponding to a two-dimensional feature point from each grid as the target three-dimensional space point, so as to obtain evenly distributed point cloud data and improve the quality of three-dimensional scene construction. The three-dimensional space point with the highest number of observations and satisfying the internal point condition is used as the target three-dimensional space point of the corresponding grid, and the outliers are eliminated while retaining the optimal space point, thereby improving the accuracy of the point cloud data and the quality of three-dimensional scene construction. The amount of point cloud data is limited by the expected number of feature points to achieve filtering of a large number of three-dimensional space points, thereby reducing the scale of solving the three-dimensional reconstruction problem and improving the efficiency of three-dimensional scene construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of a point cloud data processing method provided in an embodiment of the present application;

[0018] Figure 2 is a flow chart of determining feature matching pairs in a two-dimensional image set provided by an embodiment of the present application;

[0019] Figure 3 is a schematic diagram of a two-dimensional image provided in an embodiment of the present application;

[0020] Figure 4is a flow chart of dividing three-dimensional space points by grids provided in an embodiment of the present application;

[0021] Figure 5 is a flow chart for selecting a three-dimensional space point provided in an embodiment of the present application;

[0022] Figure 6 is a flow chart for determining that a three-dimensional space point satisfies a preset interior point condition provided by an embodiment of the present application;

[0023] Figure 7 is a flow chart of another point cloud data processing method provided in an embodiment of the present application;

[0024] Figure 8 is a flow chart of supplementing a target space point set provided by an embodiment of the present application;

[0025] Fig. 9 It is a structural schematic diagram of a point cloud data processing device provided in an embodiment of the present application;

[0026] Fig.10 It is a structural schematic diagram of a point cloud data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0028] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0029] The point cloud data processing method provided in the embodiment of the present application can be executed by a point cloud data processing device, which can be implemented by software and / or hardware, and can be composed of two or more physical entities, or one physical entity. For example, the point cloud data processing device can be a device with strong computing power such as a server, or it can be an intelligent device for collecting image data.

[0030] The point cloud data processing device is installed with at least one type of operating system, wherein the operating system includes but is not limited to Android system, Linux system and Windows system. The point cloud data processing device can install at least one application based on the operating system, and the application can be an application that comes with the operating system, or an application downloaded from a third-party device or server. In the embodiment, the point cloud data processing device at least has an application that can execute the point cloud data processing method, so the point cloud data processing device can also be the application itself.

[0031] For ease of understanding, this embodiment is described by taking a server as an example of a subject that executes the point cloud data processing method.

[0032] In one embodiment, after the camera collects multiple frames of image data of the surveying area, the image data corresponding to the surveying area is packaged and uploaded to the server via a cellular network. After the server receives the image data of the operating area, the image is sorted according to the acquisition timestamp of each frame of the image to obtain an image sequence. The feature points of each frame of the image in the image sequence are extracted by the SIFI algorithm, and the feature points of adjacent frames are matched to obtain a feature matching point set. The feature points that are mismatched in the feature matching point set are eliminated by a random sampling consistency algorithm. The feature matching point set after eliminating the mismatched feature points is triangulated to obtain the spatial points corresponding to the feature matching point set. A three-dimensional scene is constructed based on all the spatial points determined by the image sequence. However, each feature matching point set will determine a spatial point, and the image sequence can determine a large number of feature matching point sets. Therefore, the image sequence will determine a large number of spatial points, and the uniform distribution of the spatial points cannot be guaranteed. Excessive spatial points constituting a three-dimensional scene will not only reduce the construction accuracy of the three-dimensional scene, but also increase the complexity of the three-dimensional reconstruction solution problem, affecting the reconstruction efficiency and accuracy. In addition, the spatial points determined by the feature matching point set may belong to external points, which will reduce the construction accuracy of the three-dimensional scene.

[0033] To solve the above problems, the present application provides a point cloud data processing method to improve the construction efficiency and quality of three-dimensional scenes.

[0034] Figure 1 A flowchart of a point cloud data processing method provided by an embodiment of the present application is given. Figure 1 , the point cloud data processing method specifically includes:

[0035] S110, obtaining a set of two-dimensional images to be reconstructed, and determining a plurality of groups of feature matching pairs in the two-dimensional image set based on the two-dimensional feature points in each two-dimensional image; wherein each group of feature matching pairs corresponds to a three-dimensional space point, and the feature matching pair includes a plurality of matching two-dimensional feature points;

[0036] The set of two-dimensional images to be reconstructed includes multiple two-dimensional images, and the feature matching pair refers to a set of matching two-dimensional feature points in multiple two-dimensional images, and multiple two-dimensional feature points in the feature matching pair belong to different two-dimensional images. The two-dimensional feature points in the two-dimensional image are extracted by SIFI, and the two-dimensional feature points of different two-dimensional images are matched to determine the matching two-dimensional feature points in the multiple two-dimensional images. The corresponding three-dimensional space points are triangulated according to the multiple matching two-dimensional feature points.

[0037] In one embodiment, Figure 2 1 is a flow chart of determining feature matching pairs in a two-dimensional image set provided by an embodiment of the present application. Figure 2 As shown, the step of determining feature matching pairs in a two-dimensional image set specifically includes S1101-S1102:

[0038] S1101 . Divide a two-dimensional image set into a plurality of to-be-matched image sets, each of which includes at least two frames of two-dimensional images.

[0039] The two-dimensional image set can be divided into multiple image sets to be matched based on the position information or acquisition time information of each two-dimensional image. The image set to be matched includes multiple frames of two-dimensional images, and there is a certain overlap requirement between the multiple frames of images in the image set to be matched.

[0040] S1102, extracting two-dimensional feature points of each two-dimensional image in the two-dimensional image set, and determining feature matching pairs corresponding to the image set to be matched based on the two-dimensional feature points of multiple frames of two-dimensional images in the image set to be matched.

[0041] The matching relationship between multiple frames of images in the to-be-matched image set is determined based on the extracted two-dimensional feature points, and the matching two-dimensional feature points are taken as a set of feature matching pairs.

[0042] For example, the two-dimensional images in the two-dimensional image set can be sorted in order according to the acquisition timestamps of each two-dimensional image in the two-dimensional image set to obtain an image sequence corresponding to the two-dimensional image set, and the two-dimensional image corresponds to the image frame in the image sequence. According to the order of the image sequence, multiple adjacent continuous image frames are selected as the image set to be matched, such as two adjacent two-dimensional image frames can be selected as a set of images to be matched; the feature points of each frame image in the image sequence are extracted by SIFI in Represents the kth SIFI feature point of the tth frame image in the image sequence. Calculate the feature point of the tth frame image The feature descriptor and the feature points of the t+1th frame image The distance between the feature descriptors is determined based on the distance between the feature points Matching feature points And the feature points as well as Stored in the feature matching point set. Further, calculate the feature points The feature descriptor and the feature points of the t+2th frame image The distance between the feature descriptors is determined based on the distance between the feature points Matching feature points And the feature points Store it in the feature matching point set. Repeat the above matching steps until the adjacent frames do not include matching feature points or the last frame image. After distance matching feature points, a feature matching point set including multiple feature points is obtained. The feature points that are mismatched in the feature matching point set are eliminated through a random sampling consistency algorithm to obtain the corresponding feature matching pairs.

[0043] In another embodiment, the position information of each two-dimensional image in the two-dimensional image set can also be obtained, and multiple frames of two-dimensional images within a preset distance range are obtained based on the position information to divide the two-dimensional image set into multiple sets of images to be matched, and the feature points of each frame of the image to be matched are extracted through SIFI. in Indicates the kth SIFI feature point of the tth frame image in the image sequence. Then, according to the distance between the two-dimensional feature points of each frame image, it is determined whether the two-dimensional feature points of each frame image match based on the distance, and the matching two-dimensional feature points are used as feature matching pairs.

[0044] Furthermore, a triangulation calculation is performed based on multiple two-dimensional feature points in the feature matching pair to determine the three-dimensional space point P corresponding to the feature matching pair. i , P i Represents the i-th 3D spatial point determined based on the image sequence. All 3D spatial points are stored in a spatial point set, and the set is obtained. Where (x i ,y i ,z i ) represents a three-dimensional space point P i The three-dimensional space coordinates of Represents the two-dimensional feature points corresponding to the three-dimensional space points, that is, the feature matching pairs.

[0045] S120, dividing the two-dimensional image into grids according to a preset number of feature points, determining the two-dimensional feature points in each grid, and determining the three-dimensional space points corresponding to the grids according to the three-dimensional space points corresponding to the two-dimensional feature points.

[0046] The preset number of feature points refers to the expected number of three-dimensional space points corresponding to each two-dimensional image for constructing a three-dimensional scene. It can be understood that the number of three-dimensional space points corresponding to each two-dimensional image is relatively large, and some three-dimensional space points are filtered to retain the three-dimensional space points that meet the expected number as point cloud data for constructing a three-dimensional scene, thereby reducing the complexity of constructing the three-dimensional scene.

[0047] Furthermore, evenly distributed point cloud data can improve the construction accuracy of three-dimensional scenes. Therefore, the three-dimensional space can be divided into multiple evenly distributed subspaces, and one or more three-dimensional space points can be selected from the evenly distributed subspaces to obtain evenly distributed point cloud data. Figure 3 is a schematic diagram of a two-dimensional image provided in an embodiment of the present application. Figure 3As shown, the two-dimensional image 10 can be regarded as a projection image of the three-dimensional space 20, and the two-dimensional image 10 is divided into uniformly distributed grids 11. The grids 11 can be regarded as projections of each subspace 21 in the three-dimensional space. When selecting two-dimensional feature points A, B and C from the grid 11, it is equivalent to selecting three-dimensional space points A', B' and C' correspondingly from the subspace. Therefore, the two-dimensional image can be divided into multiple uniformly distributed grids, and the three-dimensional space points can be divided into subspaces corresponding to each grid, and uniformly distributed point cloud data can be selected from the subspaces corresponding to the grids.

[0048] In one embodiment, the two-dimensional image can be evenly divided into m grids, where m is equal to the desired number, and then a three-dimensional space point is selected from the subspace corresponding to each grid as point cloud data. In another embodiment, the two-dimensional image can be evenly divided into m / k grids, where k is a positive integer, and then k three-dimensional space points are selected from the subspace corresponding to each grid as point cloud data. Both of the above embodiments can limit the number of point cloud data used to construct a three-dimensional scene and ensure uniform distribution of point cloud data.

[0049] This embodiment is described by taking the example of dividing a two-dimensional image into m grids and selecting a three-dimensional space point from the subspace corresponding to each grid. Figure 4 1 is a flow chart of dividing three-dimensional space points by grids provided in an embodiment of the present application. Figure 4 As shown, the step of dividing the three-dimensional space points by gridding specifically includes S1201-S1203:

[0050] S1201. Divide the two-dimensional image into a plurality of grids evenly according to a preset number of feature points, and determine the number of grids in each row and the number of grids in each column of the two-dimensional image.

[0051] For example, assuming that the size of the two-dimensional image is w×h, w is the width of the two-dimensional image, h is the height of the two-dimensional image, and the number of preset feature points is Q, then the side length of each grid is The number of grids in each row of the two-dimensional image is nr=w / G, and the number of grids in each column is nc=h / G.

[0052] S1202: Determine the two-dimensional feature points located in the grid according to the number of grids in each row, the number of grids in each column, and the coordinates of the two-dimensional feature points.

[0053] Exemplarily, according to the two-dimensional feature points corresponding to each three-dimensional space point Determine the two-dimensional feature points distributed in the two-dimensional image, and determine the grid position corresponding to the two-dimensional feature point according to the coordinates of the two-dimensional feature point and the number of grids in each row and column. Assume that the coordinates of the two-dimensional feature point are (x, y), and its corresponding grid position (c, r) = (x / nc, y / nc), where (c, r) represents the grid in the rth row and cth column in the two-dimensional image, x / nc is rounded up, and y / nc is rounded up. It can be understood that the projection point of the three-dimensional space point projected onto the two-dimensional image is the two-dimensional feature point used to construct the three-dimensional space point. Since not all two-dimensional feature points in the two-dimensional image can be used to construct three-dimensional space points, and these two-dimensional feature points that cannot be used to construct three-dimensional space points are mismatched points, it is not necessary to consider dividing these two-dimensional feature points.

[0054] S1203. Determine the three-dimensional space point located in the subspace corresponding to the grid according to the three-dimensional space point corresponding to the two-dimensional feature point.

[0055] Exemplarily, according to the grid position corresponding to the two-dimensional feature point, the two-dimensional feature point located in the same grid is determined, and the three-dimensional space point corresponding to the two-dimensional feature point in the same grid is divided into a subspace point set P corresponding to the grid. (t ,c,r) In, P (t,c,r) represents the subspace point set corresponding to the grid of the rth row and the cth column in the tth frame image, which includes the three-dimensional feature points corresponding to the two-dimensional feature points located in the grid. In one embodiment,

[0056] S130. According to the number of observations of the three-dimensional space point and the preset internal point condition, the target three-dimensional space point of the grid is selected from the three-dimensional space points corresponding to the grid as point cloud data for constructing the three-dimensional scene, and the number of observations is equal to the number of two-dimensional feature points corresponding to the three-dimensional space point.

[0057] Exemplarily, the subspace corresponding to some grids includes multiple three-dimensional space points. Due to the limitation of the expected number, the optimal three-dimensional space point can be selected from the subspace, and the optimal three-dimensional space point can be used as the point cloud data for constructing the three-dimensional scene. In one embodiment, the preset inlier condition refers to the condition that needs to be met when a three-dimensional space point belongs to the inlier of a three-dimensional scene. The number of observations refers to the number of two-dimensional images in which the three-dimensional space point is observed in the image sequence, which is equal to the number of two-dimensional feature points used to construct the three-dimensional space point. The higher the number of observations, the more two-dimensional feature points are used to construct the three-dimensional space point, the higher the matching degree of the two-dimensional feature points, and the higher the accuracy of the three-dimensional space point. This embodiment selects the three-dimensional space point with the highest accuracy and belonging to the inlier from the same subspace based on the number of observations of the three-dimensional space point and whether the preset inlier condition is met. In this embodiment, Figure 5 : is a flowchart of selecting a three-dimensional space point provided by an embodiment of the present application. Figure 5As shown, the step of selecting a three-dimensional space point specifically includes S1301-S1303:

[0058] S1301. Sort the three-dimensional space points corresponding to the grids from high to low according to the corresponding observation times, and determine the sorting of the three-dimensional space points corresponding to the grids.

[0059] Exemplarily, the statistical subspace point set P (t,c,r) The number of two-dimensional feature points corresponding to each three-dimensional spatial point in the 3D space is obtained, and the three-dimensional spatial points are sorted from high to low according to the number of observations. For example, the three-dimensional feature point This indicates that the number of observations of this three-dimensional feature point is 4.

[0060] S1302. Determine whether the three-dimensional space point satisfies a preset interior point condition based on a plurality of two-dimensional feature points corresponding to the three-dimensional space point.

[0061] In one embodiment, Figure 6 1 is a flow chart of determining whether a three-dimensional space point satisfies a preset interior point condition provided by an embodiment of the present application. Figure 6 As shown, the step of determining that a three-dimensional space point meets a preset interior point condition specifically includes S13021-S13023:

[0062] S13021. Pair the feature descriptors of multiple two-dimensional feature points corresponding to the three-dimensional space points in pairs to obtain multiple pairs of feature descriptors.

[0063] For example, from a three-dimensional space point Three or more feature descriptors are randomly selected from the corresponding two-dimensional feature points and paired with each other to obtain three or more pairs of feature descriptors. Selection and As a pair of feature descriptors, and As a pair of feature descriptors, and As a pair of feature descriptors.

[0064] S13022. Normalize each pair of feature descriptors and calculate the cosine similarity of each pair of feature descriptors.

[0065] For example, since the feature descriptor of SIFI is a floating-point descriptor, it is usually measured using Euclidean distance, and it is difficult to use the threshold method to screen the matching feature points. Therefore, after the feature descriptor is normalized, the cosine value measurement method is used to limit the matching degree between the feature points to the same dimension, which is convenient for screening by the threshold method. The value of cosine similarity is limited to [-1,1]. -1 means that the two two-dimensional feature points are completely dissimilar, and 1 means that the two two-dimensional feature points are completely similar. Therefore, the closer the cosine similarity of a pair of feature descriptors is to 1, the more similar the two two-dimensional feature points are. Assuming that the normalized pair of feature descriptors are f1 and f2, the cosine similarity calculation formula between the two is as follows:

[0066]

[0067] Among them, cos<f1,f2> is the cosine similarity between f1 and f2, ||f1|| is the modulus of feature descriptor f1, and ||f2|| is the modulus of feature descriptor f2. Exemplarily, the similarity of each pair of feature descriptors is calculated according to the above calculation formula.

[0068] S13023. When it is determined that the cosine similarity of each pair of feature descriptors is equal to or greater than a preset similarity threshold, it is determined that the three-dimensional space point meets a preset interior point condition.

[0069] Among them, the similarity threshold refers to the minimum value of the cosine similarity of the corresponding feature descriptors when two two-dimensional feature points are feature matching points. When the cosine similarity of a pair of feature descriptors is equal to or greater than the similarity threshold, it indicates that the two two-dimensional feature points corresponding to the pair of feature descriptors are feature matching points; when the cosine similarity of a pair of feature descriptors is less than the similarity threshold, it indicates that the two two-dimensional feature points corresponding to the pair of feature descriptors are not feature matching points. In this embodiment, when the cosine similarities of three or more pairs of feature descriptors corresponding to a three-dimensional space point are equal to or greater than the similarity threshold, it indicates that the two-dimensional feature points constructing the three-dimensional space point are feature matching points, then it is determined that there is no mismatch in the two-dimensional feature points corresponding to the three-dimensional space point, and then the three-dimensional space point is determined to be an inner point of the three-dimensional scene. Correspondingly, when the cosine similarity of a pair of feature descriptors is less than the similarity threshold, it indicates that the two two-dimensional feature points corresponding to the pair of feature descriptors are not feature matching points. It is then determined that the two-dimensional feature points corresponding to the three-dimensional space point are mismatched, and the authenticity of the three-dimensional space point determined based on the mismatched two-dimensional feature points is also questionable. Therefore, the three-dimensional space point is determined to be an external point of the three-dimensional scene.

[0070] S1303: Select at least one three-dimensional space point with the highest ranking and satisfying a preset interior point condition from the three-dimensional space points corresponding to the grid as the target three-dimensional space point of the grid.

[0071] Exemplarily, the three-dimensional space point is determined in descending order to determine whether it satisfies the preset inner point condition, and when it is determined that the three-dimensional space point satisfies the preset inner point condition, the three-dimensional space point is determined to be the target three-dimensional space point in the subspace corresponding to the grid. The target three-dimensional space point is point cloud data used to construct a three-dimensional scene.

[0072] Furthermore, the target three-dimensional space points in the subspace corresponding to each grid in the two-dimensional image are collected to obtain the target space point set P corresponding to the two-dimensional image. t = {P (t,i,j) ,…,P (t,c,r)}, P t represents the target space point set of the t-th frame image, P (t,i,j) Represents the target three-dimensional space point in the subspace corresponding to the grid in the i-th column and the j-th row. The three-dimensional space points in the target space point set corresponding to each two-dimensional image in the image sequence are collected to obtain point cloud data for constructing a three-dimensional scene, so that a high-precision three-dimensional scene can be quickly constructed through the point cloud data in the future.

[0073] In summary, the point cloud data processing method provided in the embodiment of the present application divides the two-dimensional image evenly into grids, selects a three-dimensional space point corresponding to a two-dimensional feature point from each grid as the target three-dimensional space point, so as to obtain evenly distributed point cloud data and improve the construction quality of the three-dimensional scene. The three-dimensional space point with the highest number of observations and satisfying the internal point condition is used as the target three-dimensional space point of the corresponding grid, and the outliers are eliminated while retaining the optimal space point, thereby improving the accuracy of the point cloud data and the construction quality of the three-dimensional scene. The amount of point cloud data is limited by the expected number of feature points to achieve filtering of a large number of three-dimensional space points, thereby reducing the scale of solving the three-dimensional reconstruction problem and improving the construction efficiency of the three-dimensional scene.

[0074] Based on the above embodiments, Figure 7 is a flow chart of another point cloud data processing method provided by an embodiment of the present application. Figure 7 As shown, the point cloud data processing method includes:

[0075] S210 , obtaining a three-dimensional space point and a plurality of corresponding two-dimensional feature points, each of which is distributed in a corresponding two-dimensional image.

[0076] S220, dividing the two-dimensional image into grids according to a preset number of feature points, determining the two-dimensional feature points in each grid, and determining the three-dimensional space points corresponding to the grids according to the three-dimensional space points corresponding to the two-dimensional feature points.

[0077] S230. According to the number of observations of the three-dimensional space point and the preset internal point condition, the target three-dimensional space point of the grid is selected from the three-dimensional space points corresponding to the grid as point cloud data for constructing the three-dimensional scene, and the number of observations is equal to the number of two-dimensional feature points corresponding to the three-dimensional space point.

[0078] Exemplarily, steps S210 - S230 may refer to steps S110 - S130 .

[0079] In this embodiment, when it is determined that a three-dimensional space point does not satisfy the interior point condition, the three-dimensional space point is removed from the corresponding subspace point set to avoid re-determining whether the three-dimensional space point satisfies the interior point condition when the target three-dimensional space point is subsequently supplemented.

[0080] S240 , determining the number of target three-dimensional space points of the two-dimensional image according to the target three-dimensional space points of each grid in the two-dimensional image.

[0081] Exemplarily, the target three-dimensional space points in the subspace corresponding to each grid in the two-dimensional image are collected to obtain a target space point set P corresponding to the two-dimensional image. t Then, the target space point set P corresponding to the two-dimensional image is counted t The number of target 3D space points within.

[0082] S250. When the number of target three-dimensional space points of the two-dimensional image is less than the preset number of feature points, at least one target three-dimensional space point is selected from the remaining three-dimensional space points of each grid of the two-dimensional image to make the number of target three-dimensional space points of the two-dimensional image equal to the preset number of feature points.

[0083] For example, due to the randomness of the distribution of three-dimensional space points, the subspace corresponding to some grids may not include three-dimensional space points, or the three-dimensional space points of the subspace corresponding to some grids are all external points, making the target space point set P of the two-dimensional image t The three-dimensional space points of the grid are not included. At this time, the target space point set P t The number of target three-dimensional space points in will be less than the expected number. t When the number of target three-dimensional space points in the three-dimensional space corresponding to the two-dimensional image is less than the expected number, the point cloud data used to construct the three-dimensional scene in the three-dimensional space corresponding to the two-dimensional image is too small, resulting in low accuracy of the constructed three-dimensional scene. Therefore, this embodiment proposes to select target three-dimensional space points from the remaining three-dimensional space points in the three-dimensional space corresponding to the two-dimensional image, so that the target space point set P t The number of target three-dimensional space points within is equal to the expected number.

[0084] In one embodiment, Figure 8 is a flow chart of supplementing the target space point set provided by the embodiment of the present application. Figure 8As shown, the steps of supplementing the target space point set specifically include 2501-2502:

[0085] 2501. Sort the remaining three-dimensional space points of each grid of the two-dimensional image from high to low according to the number of observations to determine the order of the three-dimensional space points.

[0086] 2502. From the remaining three-dimensional space points of each grid of the two-dimensional image, select at least one three-dimensional space point with the highest ranking and satisfying a preset interior point condition as a target three-dimensional space point of the two-dimensional image.

[0087] Exemplarily, the remaining three-dimensional space points of each grid of the two-dimensional image refer to the three-dimensional space points in all grids except those that have been determined as target three-dimensional space points. The remaining three-dimensional space points of all grids are collected, and the number of observations of the remaining three-dimensional space points is obtained. The remaining three-dimensional space points are sorted from high to low according to the number of observations, and it is determined whether the remaining three-dimensional space points meet the inner point condition based on the sorting from high to low. If the highest ranked remaining three-dimensional space point meets the inner point condition, the three-dimensional space point is used as the target three-dimensional space point, and it is determined whether the second highest ranked remaining three-dimensional space point meets the inner point condition, until n target three-dimensional space points are selected, where n is equal to the difference between the expected number and the number of target three-dimensional space points in the target space point set.

[0088] In summary, the point cloud data processing method provided in the embodiment of the present application divides the two-dimensional image evenly into grids, selects a three-dimensional space point corresponding to a two-dimensional feature point from each grid as the target three-dimensional space point, so as to obtain evenly distributed point cloud data and improve the construction quality of the three-dimensional scene. The three-dimensional space point with the highest number of observations and satisfying the internal point condition is used as the target three-dimensional space point of the corresponding grid, and the outliers are eliminated while retaining the optimal space point, thereby improving the accuracy of the point cloud data and the construction quality of the three-dimensional scene. The amount of point cloud data is limited by the expected number of feature points to filter a large number of three-dimensional space points, thereby reducing the scale of solving the three-dimensional reconstruction problem and improving the construction efficiency of the three-dimensional scene. By supplementing the expected number of target three-dimensional space points, the three-dimensional space corresponding to each two-dimensional image just retains the expected number of target three-dimensional space points, thereby reducing the scale of solving the three-dimensional reconstruction problem while ensuring the accuracy of the three-dimensional reconstruction.

[0089] Based on the above embodiments, Fig. 9 A schematic diagram of the structure of a point cloud data processing device provided in an embodiment of the present application. Fig. 9 The point cloud data processing device provided in this embodiment specifically includes: a data acquisition module 21, a grid division module 22 and a data filtering module 23.

[0090] The data acquisition module is configured to acquire a set of two-dimensional images to be reconstructed, and determine a plurality of sets of feature matching pairs in the set of two-dimensional images based on two-dimensional feature points in each two-dimensional image; wherein each set of feature matching pairs corresponds to a three-dimensional space point, and the feature matching pair includes a plurality of matching two-dimensional feature points;

[0091] A grid division module is configured to divide the two-dimensional image into grids according to a preset number of feature points, determine the two-dimensional feature points in each grid, and determine the three-dimensional space points corresponding to the grids according to the three-dimensional space points corresponding to the two-dimensional feature points;

[0092] The data filtering module is configured to select the target three-dimensional space points of the grid from the three-dimensional space points corresponding to the grid as point cloud data for constructing a three-dimensional scene based on the number of observations of the three-dimensional space points and preset internal point conditions, and the number of observations is equal to the number of two-dimensional feature points corresponding to the three-dimensional space points.

[0093] On the basis of the above embodiments, the point cloud data processing device also includes: a target point quantity counting module, configured to determine the number of target three-dimensional space points of the two-dimensional image based on the target three-dimensional space points of each grid in the two-dimensional image; a target point supplement module, configured to obtain the number of target three-dimensional space points corresponding to the two-dimensional image, when the number of target three-dimensional space points of the two-dimensional image is less than the preset number of feature points, select at least one target three-dimensional space point from the remaining three-dimensional space points of each grid of the two-dimensional image, so that the number of target three-dimensional space points of the two-dimensional image is equal to the preset number of feature points.

[0094] Based on the above embodiment, the data acquisition module includes: an image sequence acquisition unit, configured to divide the two-dimensional image set into multiple image sets to be matched, and the image set to be matched includes at least two frames of two-dimensional images; a feature point matching unit, configured to extract the two-dimensional feature points of each two-dimensional image in the two-dimensional image set, and determine the feature matching pairs corresponding to the image set to be matched based on the two-dimensional feature points of multiple frames of two-dimensional images in the image set to be matched.

[0095] Based on the above embodiment, the grid division module includes: a grid determination unit, configured to divide the two-dimensional image into multiple grids evenly according to a preset number of feature points, and determine the number of grids in each row and the number of grids in each column of the two-dimensional image; a feature point positioning unit, configured to determine the two-dimensional feature points located in the grid according to the number of grids in each row, the number of grids in each column and the coordinates of the two-dimensional feature points.

[0096] Based on the above embodiment, the data filtering module includes: a grid space point sorting unit, configured to sort the three-dimensional space points corresponding to the grid from high to low according to the corresponding number of observations, and determine the sorting of each three-dimensional space point corresponding to the grid; an inner point determination unit, configured to determine whether the three-dimensional space point meets the preset inner point condition based on multiple two-dimensional feature points corresponding to the three-dimensional space point; a target point selection unit, configured to select at least one three-dimensional space point with the highest sorting and meeting the preset inner point condition from the three-dimensional space points corresponding to the grid as the target three-dimensional space point of the grid.

[0097] On the basis of the above embodiment, the inner point determination unit includes: a feature pairing subunit, configured to pair the feature descriptors of multiple two-dimensional feature points corresponding to the three-dimensional space point in pairs to obtain multiple pairs of feature descriptors; a similarity calculation subunit, configured to normalize each pair of feature descriptors and calculate the cosine similarity of each pair of feature descriptors; an inner point determination subunit, configured to determine that when the cosine similarity of each pair of feature descriptors is equal to or greater than a preset similarity threshold, it is determined that the three-dimensional space point meets the preset inner point condition.

[0098] On the basis of the above embodiment, the target point completion module includes: a remaining spatial point sorting unit, configured to sort the remaining three-dimensional spatial points of each grid of the two-dimensional image from high to low according to the number of observations, and determine the sorting of the three-dimensional spatial points; a target point completion unit, configured to select at least one three-dimensional spatial point with the highest sorting and satisfying a preset internal point condition from the remaining three-dimensional spatial points of each grid of the two-dimensional image as the target three-dimensional spatial point of the two-dimensional image.

[0099] As mentioned above, the point cloud data processing device provided in the embodiment of the present application divides the two-dimensional image evenly through a grid, selects a three-dimensional space point corresponding to a two-dimensional feature point from each grid as a target three-dimensional space point, so as to obtain evenly distributed point cloud data and improve the construction quality of the three-dimensional scene. The three-dimensional space point with the highest number of observations and satisfying the internal point condition is used as the target three-dimensional space point of the corresponding grid, and the outliers are eliminated while retaining the optimal space point, thereby improving the accuracy of the point cloud data and the construction quality of the three-dimensional scene. The amount of point cloud data is limited by the expected number of feature points to filter a large number of three-dimensional space points, thereby reducing the scale of solving the three-dimensional reconstruction problem and improving the construction efficiency of the three-dimensional scene. By supplementing the expected number of target three-dimensional space points, the three-dimensional space corresponding to each two-dimensional image just retains the expected number of target three-dimensional space points, thereby reducing the scale of solving the three-dimensional reconstruction problem while ensuring the accuracy of the three-dimensional reconstruction.

[0100] The point cloud data processing device provided in the embodiment of the present application can be used to execute the point cloud data processing method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0101] Fig.10 is a schematic diagram of a point cloud data processing device provided in an embodiment of the present application, with reference to Fig.10 The point cloud data processing device includes: a processor 31, a memory 32, a communication device 33, an input device 34 and an output device 35. The number of processors 31 in the point cloud data processing device can be one or more, and the number of memories 32 in the point cloud data processing device can be one or more. The processor 31, memory 32, communication device 33, input device 34 and output device 35 of the point cloud data processing device can be connected via a bus or other means.

[0102] The memory 32, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the point cloud data processing method of any embodiment of the present application (for example, the data acquisition module 21, the grid division module 22 and the data filtering module 23 in the point cloud data processing device). The memory 32 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the device, etc. In addition, the memory 32 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0103] The communication device 33 is used for data transmission.

[0104] The processor 31 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 32, that is, implements the above-mentioned point cloud data processing method.

[0105] The input device 34 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 35 may include a display device such as a display screen.

[0106] The point cloud data processing device provided above can be used to execute the point cloud data processing method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0107] An embodiment of the present application also provides a storage medium containing computer executable instructions, which are used to execute a point cloud data processing method when executed by a computer processor. The point cloud data processing method includes: obtaining a set of two-dimensional images to be reconstructed, and determining multiple groups of feature matching pairs in the two-dimensional image set based on the two-dimensional feature points in each two-dimensional image; wherein each group of feature matching pairs corresponds to a three-dimensional space point, and the feature matching pairs include multiple matching two-dimensional feature points; gridding the two-dimensional image according to a preset number of feature points, determining the two-dimensional feature points in each grid, and determining the three-dimensional space points corresponding to the grid according to the three-dimensional space points corresponding to the two-dimensional feature points; and selecting the target three-dimensional space points of the grid from the three-dimensional space points corresponding to the grid as point cloud data for constructing a three-dimensional scene according to the number of observations of the three-dimensional space points and a preset internal point condition, and the number of observations is equal to the number of two-dimensional feature points corresponding to the three-dimensional space points.

[0108] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system, which is connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media residing in different locations (for example, in different computer systems connected by a network). The storage medium may store program instructions (for example, embodied as a computer program) that can be executed by one or more processors.

[0109] Of course, the storage medium containing computer executable instructions provided in the embodiment of the present application, whose computer executable instructions are not limited to the above point cloud data processing method, can also execute related operations in the point cloud data processing method provided in any embodiment of the present application.

[0110] The point cloud data processing device, storage medium and point cloud data processing equipment provided in the above embodiments can execute the point cloud data processing method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the point cloud data processing method provided in any embodiment of the present application.

[0111] The above are only preferred embodiments of the present application and the technical principles used. The present application is not limited to the specific embodiments herein, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not deviate from the protection scope of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A point cloud data processing method, characterized in that: include: Acquire a set of two-dimensional images to be reconstructed, and determine a plurality of sets of feature matching pairs in the set of two-dimensional images based on two-dimensional feature points in each two-dimensional image; wherein each set of feature matching pairs corresponds to a three-dimensional space point, and the feature matching pairs include a plurality of matching two-dimensional feature points; Dividing the two-dimensional image into grids according to a preset number of feature points, determining the two-dimensional feature points in each grid, and determining the three-dimensional space points corresponding to the grids according to the three-dimensional space points corresponding to the two-dimensional feature points; According to the number of observations of the three-dimensional space point and the preset internal point condition, the target three-dimensional space point of the grid is selected from the three-dimensional space points corresponding to the grid as point cloud data for constructing the three-dimensional scene, the number of observations is equal to the number of two-dimensional feature points corresponding to the three-dimensional space point, and the preset internal point condition is the condition satisfied when the three-dimensional space point belongs to the internal point of the three-dimensional scene.

2. The point cloud data processing method according to claim 1, characterized in that: After selecting the target three-dimensional space point of the grid from the three-dimensional space points corresponding to the grid as point cloud data for constructing the three-dimensional scene, the method further includes: Determining the number of target three-dimensional space points of the two-dimensional image according to the target three-dimensional space points of each grid in the two-dimensional image; Obtain the number of target three-dimensional space points corresponding to the two-dimensional image. When the number of target three-dimensional space points of the two-dimensional image is less than the preset number of feature points, select at least one target three-dimensional space point from the remaining three-dimensional space points of each grid of the two-dimensional image to make the number of target three-dimensional space points of the two-dimensional image equal to the preset number of feature points.

3. The point cloud data processing method according to claim 1, characterized in that: The determining of a plurality of groups of feature matching pairs in a two-dimensional image set based on the two-dimensional feature points in each two-dimensional image comprises: Dividing the two-dimensional image set into a plurality of to-be-matched image sets, each of which includes at least two frames of two-dimensional images; The two-dimensional feature points of each two-dimensional image in the two-dimensional image set are extracted, and the feature matching pairs corresponding to the image set to be matched are determined based on the two-dimensional feature points of multiple frames of two-dimensional images in the image set to be matched.

4. The point cloud data processing method according to claim 1, characterized in that: The step of dividing the two-dimensional image into grids according to the preset number of feature points and determining the two-dimensional feature points in each grid includes: Dividing the two-dimensional image into a plurality of grids evenly according to the preset number of feature points, and determining the number of grids in each row and the number of grids in each column of the two-dimensional image; The two-dimensional feature points located in the grids are determined according to the number of grids in each row, the number of grids in each column, and the coordinates of the two-dimensional feature points.

5. The point cloud data processing method according to claim 1, characterized in that: The step of selecting the target three-dimensional space point of the grid from the three-dimensional space points corresponding to the grid as point cloud data for constructing the three-dimensional scene according to the number of observations of the three-dimensional space point and the preset interior point condition comprises: Sort the three-dimensional space points corresponding to the grids from high to low according to the corresponding observation times, and determine the sorting of the three-dimensional space points corresponding to the grids; Determining whether the three-dimensional space point satisfies a preset interior point condition according to a plurality of two-dimensional feature points corresponding to the three-dimensional space point; At least one three-dimensional space point with the highest ranking and satisfying a preset interior point condition is selected from the three-dimensional space points corresponding to the grid as the target three-dimensional space point of the grid.

6. The point cloud data processing method according to claim 5, characterized in that: The determining, according to the two-dimensional feature point corresponding to the three-dimensional space point, whether the three-dimensional space point satisfies a preset interior point condition comprises: Pairing the feature descriptors of the multiple two-dimensional feature points corresponding to the three-dimensional space point in pairs to obtain multiple pairs of feature descriptors; Normalize each pair of feature descriptors and calculate the cosine similarity of each pair of feature descriptors; When it is determined that the cosine similarity of each pair of feature descriptors is equal to or greater than a preset similarity threshold, it is determined that the three-dimensional space point meets the preset interior point condition.

7. The point cloud data processing method according to claim 2, characterized in that: The selecting at least one target three-dimensional space point from the remaining three-dimensional space points of each grid of the two-dimensional image comprises: Sorting the remaining three-dimensional space points of each grid of the two-dimensional image from high to low according to the number of observations to determine the order of the three-dimensional space points; From the remaining three-dimensional space points of each grid of the two-dimensional image, at least one three-dimensional space point with the highest ranking and satisfying a preset interior point condition is selected as the target three-dimensional space point of the two-dimensional image.

8. A point cloud data processing device, characterized in that: include: A data acquisition module is configured to acquire a set of two-dimensional images to be reconstructed, and determine a plurality of sets of feature matching pairs in the set of two-dimensional images based on two-dimensional feature points in each two-dimensional image; wherein each set of feature matching pairs corresponds to a three-dimensional space point, and the feature matching pairs include a plurality of matching two-dimensional feature points; A grid division module is configured to divide the two-dimensional image into grids according to a preset number of feature points, determine the two-dimensional feature points in each grid, and determine the three-dimensional space points corresponding to the grids according to the three-dimensional space points corresponding to the two-dimensional feature points; The data filtering module is configured to select the target three-dimensional space point of the grid from the three-dimensional space points corresponding to the grid as point cloud data for constructing a three-dimensional scene based on the number of observations of the three-dimensional space point and a preset inner point condition, wherein the number of observations is equal to the number of two-dimensional feature points corresponding to the three-dimensional space point, and the preset inner point condition is a condition satisfied when the three-dimensional space point belongs to an inner point of the three-dimensional scene.

9. A point cloud data processing device, characterized in that: include: one or more processors; A storage device stores one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the point cloud data processing method as described in any one of claims 1-7.

10. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to execute the point cloud data processing method as described in any one of claims 1 to 7 when executed by a computer processor.

Images