Plane detection method, electronic equipment and computer readable medium

Abstract

The invention discloses a plane detection method, electronic equipment and a computer readable medium, and the method comprises the steps: segmenting an initial three-dimensional point cloud of a to-be-detected image into point connected domains according to a preset distance threshold value; projecting the three-dimensional points in the point connected domains to the to-be-detected image to obtain two-dimensional point clouds corresponding to the point connected domains; constructing triangular patches corresponding to the point connected domains according to a preset triangulation algorithm and the two-dimensional point clouds corresponding to the point connected domains; generating patch connected domains according to the triangular patches corresponding to the point connected domains; projecting the two-dimensional points in the patch connected domains to the to-be-detected image to obtain target three-dimensional point clouds corresponding to the patch connected domains; and performing fitting processing on each target three-dimensional point cloud to obtain a plane detection result. The method can effectively eliminate pseudo planes which cannot correspond to the real world, greatly improves the plane detection speed, saves calculation resources, is suitable for non-dense point clouds, and detects planes as many as possible.

Description

Technical Field

[0001] This application relates to the field of augmented reality technology, and specifically, to a plane detection method, an electronic device, and a computer-readable medium. Background Art

[0002] With the continuous development of augmented reality (AR) technology, plane detection algorithms have received increasing attention. Plane detection is a key algorithm in AR technology, which enables the device to recognize planes in the real world, such as floors, tables, and walls. Through plane detection, AR applications can place virtual objects on the planes in the real world and make them blend naturally with the surrounding environment, thus creating a more realistic and immersive experience.

[0003] However, the current plane detection algorithms have many drawbacks, such as being prone to generating pseudo planes, having slow running speeds, wasting computing resources, and having poor effects on non-dense point clouds. Summary of the Invention

[0004] This application aims to solve one of the technical problems in the related art to a certain extent. For this purpose, this application provides a plane detection method, an electronic device, and a computer-readable medium.

[0005] As the first aspect of this application, a plane detection method is provided, where the method includes:

[0006] Segment the initial three-dimensional point cloud of the image to be detected into point connected domains according to a preset distance threshold;

[0007] Project the three-dimensional points in each of the point connected domains onto the image to be detected to obtain two-dimensional point clouds corresponding to each of the point connected domains;

[0008] Construct triangular patches corresponding to each of the point connected domains according to a preset triangulation algorithm and the two-dimensional point clouds corresponding to each of the point connected domains;

[0009] Generate patch connected domains according to the triangular patches corresponding to each of the point connected domains;

[0010] Project the two-dimensional points in each of the patch connected domains onto the image to be detected to obtain target three-dimensional point clouds corresponding to each of the patch connected domains;

[0011] Perform fitting processing on each of the target three-dimensional point clouds respectively to obtain plane detection results.

[0012] Optionally, the generating patch connected domains according to the triangular patches corresponding to each of the point connected domains includes:

[0013] According to a preset area threshold, perform screening and merging processing on the triangular patches corresponding to each of the point connected regions respectively to obtain the processed patches corresponding to each of the point connected regions;

[0014] Determine any two processed patches with overlapping edges among the processed patches corresponding to each of the point connected regions as having an adjacency relationship, and construct an adjacency matrix of the processed patches;

[0015] Calculate the normal vectors of each of the processed patches;

[0016] Generate a patch connected region according to the breadth first search algorithm, the processed patches whose normal vectors match the gravity direction, and the adjacency matrix of the processed patches.

[0017] Optionally, after calculating the normal vectors of each of the processed patches and before generating a patch connected region according to the breadth first search algorithm, the processed patches whose normal vectors match the gravity direction, and the adjacency matrix of the processed patches, the method further includes:

[0018] Back project the multiple two-dimensional points included in the processed patch onto the to-be-detected image to obtain multiple three-dimensional points, and calculate the centroid of the processed patch;

[0019] Determine a target direction according to the pose information of the imaging device corresponding to the to-be-detected image and the centroid;

[0020] In the case where the direction of the normal vector of the processed patch is inconsistent with the target direction, adjust the direction of the normal vector of the processed patch to the target direction.

[0021] Optionally, the step of performing screening and merging processing on the triangular patches corresponding to each of the point connected regions respectively according to a preset area threshold to obtain the processed patches corresponding to each of the point connected regions includes:

[0022] For any one of the point connected regions, determine any two triangular patches with overlapping edges corresponding thereto as having an adjacency relationship, and construct an adjacency matrix of the triangular patches;

[0023] Determine all triangular patches with an area smaller than the preset area threshold corresponding to the current point connected region as patches to be merged;

[0024] According to the adjacency matrix of the triangular patches, merge each of the patches to be merged with its adjacent patches to be merged until the area of the merged patch is not less than the preset area threshold; wherein, the types of the processed patches include the merged patches and the triangular patches corresponding to the current point connected region with an area not less than the preset area threshold.

[0025] Optionally, in the step of calculating the normal vectors of the processed patches, the normal vector of any triangular patch corresponding to the processed patch is calculated according to the following formula:

[0026]

[0027] In formula (1), n represents the normal vector of any triangular patch, and p1, p2, and p3 represent three three-dimensional points obtained by back-projecting the three two-dimensional points included in any triangular patch onto the image to be detected.

[0028] Optionally, the step of segmenting the initial three-dimensional point cloud of the image to be detected into point-connected regions according to a preset distance threshold includes:

[0029] Determine that any two three-dimensional points in the initial three-dimensional point cloud with a distance less than the preset distance threshold have an adjacency relationship, and construct an adjacency matrix of three-dimensional points;

[0030] According to the breadth-first search algorithm and the adjacency matrix of three-dimensional points, segment the initial three-dimensional point cloud of the image to be detected into point-connected regions; wherein, the distance between any two three-dimensional points in any point-connected region is less than the preset distance threshold, and the distance between any two three-dimensional points from different point-connected regions is not less than the preset distance threshold.

[0031] Optionally, in the step of projecting the three-dimensional points in each point-connected region onto the image to be detected to obtain the two-dimensional point cloud corresponding to each point-connected region, the two-dimensional point is obtained through the following formula:

[0032]

[0033] In formula (2), represents the two-dimensional point, represents the three-dimensional point, (f x , f y ) represents the focal length corresponding to the image to be detected, and (cx, cy) represents the principal point of the imaging device corresponding to the image to be detected.

[0034] Optionally, the step of respectively performing fitting processing on each target three-dimensional point cloud to obtain a plane detection result includes:

[0035] For any target three-dimensional point cloud, perform the random fitting step according to a preset number of executions to obtain a plurality of fitting planes;

[0036] From the plurality of fitting planes, determine the fitting plane with the largest number of points as the target plane:

[0037] According to a preset fitting algorithm, perform fitting processing on the three-dimensional points in the target plane to obtain the plane detection result;

[0038] Among them, the random fitting step includes:

[0039] Randomly select three three-dimensional points from any one of the target three-dimensional point clouds;

[0040] According to the preset fitting algorithm, perform fitting on the selected three three-dimensional points to obtain a fitting plane;

[0041] Determine other three-dimensional points located on the fitting plane from any one of the target three-dimensional point clouds;

[0042] According to the determined other three-dimensional points and the selected three three-dimensional points, calculate the number of points on the fitting plane.

[0043] As a second aspect of the present application, there is provided an electronic device, where the electronic device includes:

[0044] One or more processors;

[0045] A memory, on which one or more computer programs are stored. When the one or more computer programs are executed by the one or more processors, the one or more processors implement the plane detection method as described in the first aspect of the present application.

[0046] As a third aspect of the present application, there is provided a computer-readable medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the plane detection method as described in the first aspect of the present application.

[0047] The plane detection method provided by the embodiments of the present application divides the initial three-dimensional point cloud of the image to be detected into point connectivity domains according to a preset distance threshold, projects the three-dimensional points in each point connectivity domain onto the image to be detected to obtain two-dimensional point clouds corresponding to each point connectivity domain, constructs triangular patches corresponding to each point connectivity domain according to a preset triangulation algorithm and the two-dimensional point clouds corresponding to each point connectivity domain, generates patch connectivity domains according to the triangular patches corresponding to each point connectivity domain, projects the two-dimensional points in each patch connectivity domain onto the image to be detected to obtain target three-dimensional point clouds corresponding to each patch connectivity domain, and performs fitting processing on each target three-dimensional point cloud respectively to obtain a plane detection result. It can effectively eliminate pseudo planes that cannot correspond to the real world, greatly improve the speed of plane detection, save computing resources, be applicable to non-dense point clouds, and detect planes as many as possible. Description of the Drawings

[0048] The following further describes the present application with reference to the drawings:

[0049] Figure 1 It is a flowchart of an implementation manner of the plane detection method provided by an embodiment of the present application;

[0050] Figure 2 It is a flowchart of another implementation manner of the plane detection method provided by an embodiment of the present application;

[0051] Figure 3 It is a flowchart of yet another implementation manner of the plane detection method provided by an embodiment of the present application;

[0052] Figure 4 It is a flowchart of still another implementation manner of the plane detection method provided by an embodiment of the present application;

[0053] Figure 5 It is a flowchart of another implementation manner of the plane detection method provided by an embodiment of the present application;

[0054] Figure 6 It is a flowchart of yet another implementation manner of the plane detection method provided by an embodiment of the present application;

[0055] Figure 7a It is a flowchart of an implementation manner of merging triangular patches provided by an embodiment of the present application;

[0056] Figure 7b It is a flowchart of an implementation manner of segmenting point connected components provided by an embodiment of the present application;

[0057] Figure 8 It is a module diagram of an implementation manner of an electronic device provided by an embodiment of the present application;

[0058] Figure 9 It is a schematic diagram of a computer-readable medium provided by an embodiment of the present application.

[0059] Description of reference numerals

[0060] 101: Processor 102: Memory

[0061] 103: I / O interface 104: Bus Detailed implementation manners

[0062] The embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. Based on the embodiments in the implementation manners, it is intended to explain the present application and should not be construed as a limitation to the present application.

[0063] As used herein, the phrase "one embodiment" or "example" or "instance" means that a particular feature, structure, or characteristic described in connection with the embodiment itself can be included in at least one embodiment disclosed in this application. The appearances of the phrase "in one embodiment" in various positions in the specification do not necessarily refer to the same embodiment.

[0064] Current plane detection algorithms have many drawbacks, such as being prone to generating false planes, having slow running speeds, wasting computing resources, and having poor performance on non-dense point clouds.

[0065] For example, there is a plane detection algorithm that first performs Random Sample Consensus (RANSAC) and Principal Component Analysis (PCA), then performs triangular meshing, and organizes the triangular patches into a quadtree for subsequent plane merging, resulting in a large number of random trials and errors, thus leading to a relatively slow running speed. Moreover, since this plane detection algorithm does not consider the spatial neighborhood relationship of the three-dimensional point cloud, it will also fit a large number of false planes.

[0066] There is another plane detection algorithm that uses the method of local region growth to detect planes in unordered dense point clouds. This plane detection algorithm only targets dense point clouds, has poor performance on non-dense point clouds, and has a long running time, making it not suitable for high-frame-rate scenarios in AR.

[0067] There is also a plane detection algorithm that structures the point cloud into an octree, then randomly explores each point and uses PCA for plane fitting. Although this plane detection algorithm analyzes the spatial neighborhood relationship between three-dimensional point clouds, its running speed is slow.

[0068] In response to this, the inventors of the present application propose a brand-new plane detection method to overcome the many drawbacks of existing plane detection algorithms, such as being prone to generating false planes, having slow running speeds, wasting computing resources, and having poor performance on non-dense point clouds.

[0069] As a first aspect of the embodiments of the present application, there is provided a plane detection method, as Figure 1 shown, the method may include:

[0070] Step S110, dividing the initial three-dimensional point cloud of the image to be detected into point connected domains according to a preset distance threshold;

[0071] Step S120, projecting the three-dimensional points in each of the point connected domains onto the image to be detected to obtain two-dimensional point clouds corresponding to the respective point connected domains;

[0072] Step S130: Construct triangular patches corresponding to each of the point connected regions according to a preset triangulation algorithm and the two-dimensional point cloud corresponding to each of the point connected regions.

[0073] Step S140: Generate patch connected regions according to the triangular patches corresponding to each of the point connected regions.

[0074] Step S150: Project the two-dimensional points within each of the patch connected regions onto the image to be detected to obtain the target three-dimensional point cloud corresponding to each of the patch connected regions.

[0075] Step S160: Perform fitting processing on each of the target three-dimensional point clouds respectively to obtain a plane detection result.

[0076] Among them, the embodiments of the present application do not make special limitations on how to obtain the initial three-dimensional point cloud of the image to be detected. For example, a three-dimensional point cloud output by a Simultaneous Localization and Mapping (SLAM) algorithm for the image to be detected can be obtained.

[0077] Among them, the embodiments of the present application do not make special limitations on the preset distance threshold. For example, it can be determined according to the scene in the image to be detected and set to 0.5 meters, 1 meter, 2 meters, etc. according to the scene.

[0078] Among them, the embodiments of the present application do not make special limitations on the preset triangulation algorithm. For example, the preset triangulation algorithm can be the Delaunay algorithm.

[0079] Among them, the embodiments of the present application do not make special limitations on how to perform fitting processing on each of the target three-dimensional point clouds respectively. For example, fitting processing can be performed through the RANSAC algorithm and the PCA algorithm.

[0080] The inventors of the present application propose that current plane detection algorithms do not consider that the initial three-dimensional point cloud is obtained by triangulation. However, the three-dimensional point cloud of the image to be detected must be the surface point cloud of the real world. Therefore, the two-dimensional point cloud obtained by projecting the initial three-dimensional point cloud of the image to be detected onto the image to be detected implies the three-dimensional structure of the surface point cloud. By using triangulation again, this implicit three-dimensional structure of the surface point cloud can be roughly obtained, which is used to understand the spatial domain relationship of the three-dimensional point cloud subsequently and avoid generating pseudo planes.

[0081] The inventors of the present application further propose that although the two-dimensional point cloud formed by projecting the initial three-dimensional point cloud onto an image implies the three-dimensional structure of the surface point cloud, it still lacks information in one dimension. This results in the situation that although two points may be very close on the image to be detected, in fact, they have a large gap in the depth direction of the three-dimensional space. To address this, by segmenting the initial three-dimensional point cloud of the image to be detected into point-connected regions according to a preset distance threshold and removing such connection relationships in advance before triangulation, it can be ensured that the triangular patches constructed by triangulation are truly connected meshes, avoiding the generation of pseudo planes.

[0082] The inventors of the present application further propose that according to the triangular patches corresponding to each point-connected region, a patch-connected region is generated. By utilizing the three-dimensional structure of the surface point cloud implied by the triangular patches, candidate planes are roughly obtained. Finally, the two-dimensional points within each patch-connected region are projected onto the image to be detected to obtain the target three-dimensional point cloud corresponding to each patch-connected region. Fitting is performed on the target three-dimensional point cloud after removing the pseudo planes, ensuring that the fitting algorithm uses the three-dimensional point cloud information for fitting, rather than using the implied surface point cloud information obtained after projecting the initial three-dimensional point cloud onto the image to be detected, thereby ensuring the accuracy of plane detection.

[0083] In addition, segmenting the initial three-dimensional point cloud into point-connected regions and constructing the two-dimensional point cloud into triangular patches according to a preset triangulation algorithm is equivalent to pre-segmenting the points in the three-dimensional space in advance, and finally performing fitting on each target three-dimensional point cloud. This can greatly reduce the number of fitting processes, thereby greatly improving the speed of plane detection, saving computing resources, and being applicable to non-dense point clouds and detecting as many planes as possible.

[0084] The plane detection method provided by the embodiments of the present application segments the initial three-dimensional point cloud of the image to be detected into point-connected regions according to a preset distance threshold, projects the three-dimensional points within each point-connected region onto the image to be detected to obtain the two-dimensional point cloud corresponding to each point-connected region, constructs the triangular patches corresponding to each point-connected region according to a preset triangulation algorithm and the two-dimensional point cloud corresponding to each point-connected region, generates a patch-connected region according to the triangular patches corresponding to each point-connected region, projects the two-dimensional points within each patch-connected region onto the image to be detected to obtain the target three-dimensional point cloud corresponding to each patch-connected region, and performs fitting processing on each target three-dimensional point cloud respectively to obtain the plane detection result. It can effectively remove the pseudo planes that cannot correspond to the real world, greatly improve the speed of plane detection, save computing resources, be applicable to non-dense point clouds, and detect as many planes as possible.

[0085] The inventors of the present application further propose that the reason why the current plane detection algorithm is not ideal is also related to the failure to eliminate planes in non-gravity directions and the failure to consider the impact of point cloud noise on the plane recall rate. In this regard, by screening and merging small triangular patches, the impact of point cloud noise on the plane recall rate can be eliminated, thereby improving the plane recall rate. By calculating the normal vectors of the processed patches and screening out the processed patches whose normal vectors match the gravity direction, planes in non-gravity directions can be eliminated.

[0086] Correspondingly, in some embodiments, generating a patch connectivity domain based on the triangular patches corresponding to each point connectivity domain (i.e., involved in step S140) may include: Figure 2 as shown below:

[0087] Step S210: Screen and merge the triangular patches corresponding to each point connectivity domain according to a preset area threshold to obtain the processed patches corresponding to each point connectivity domain;

[0088] Step S220: Determine that any two processed patches with overlapping edges among the processed patches corresponding to each point connectivity domain have an adjacency relationship, and construct an adjacency matrix of the processed patches;

[0089] Step S230: Calculate the normal vectors of each processed patch;

[0090] Step S240: Generate a patch connectivity domain according to the breadth-first search algorithm, the processed patches whose normal vectors match the gravity direction, and the adjacency matrix of the processed patches.

[0091] Among them, the present application embodiment does not make special limitations on the preset area threshold. For example, it can be determined according to the scene in the image to be detected. When the scene is larger, the preset area threshold can be larger accordingly.

[0092] By calculating the normal vectors to screen the patches and eliminating planes in non-gravity directions, only horizontal patches whose normal vectors match the gravity direction are used, which further improves the effect of plane detection. And since non-horizontal planes are quickly eliminated, a large amount of plane fitting time is also saved, thereby improving the speed of plane detection.

[0093] Since the initial three-dimensional point cloud is a point cloud given based on image features, if the feature points at a certain location in the image are relatively dense, the point cloud generated at this location may also be relatively dense. When three three-dimensional points are relatively close, the normal vectors of the three three-dimensional points are mainly determined by the noise of the three-dimensional points. This results in a large difference between the normal vector and the gravity direction even though the three three-dimensional points are indeed in the same horizontal plane, thereby leading to a low plane recall rate. By screening and merging the triangular patches with smaller areas, it is ensured that all the processed patches are patches with larger areas, avoiding the problem that the calculated value of the normal vector of the triangular patches with smaller areas has a large error due to their point cloud noise, eliminating the influence of point cloud noise on the plane recall rate, improving the plane recall rate, and thus being beneficial to improving the effect of plane detection.

[0094] The inventors of the present application further propose that the direction of the normal vector can also be aligned with the optical center of the imaging device by adjusting the positive and negative signs of the normal vector. Correspondingly, in some embodiments, after calculating the normal vectors of each of the processed patches (i.e., those involved in step S230), and before generating the patch connectivity domain according to the breadth-first search algorithm, the processed patches whose normal vectors match the gravity direction, and the adjacency matrix of the processed patches (i.e., those involved in step S240), as Figure 3 shown, the method may further include:

[0095] Step S310, back-projecting the multiple two-dimensional points included in the processed patch onto the to-be-detected image to obtain multiple three-dimensional points, and calculating the centroid of the processed patch;

[0096] Step S320, determining a target direction according to the pose information of the imaging device corresponding to the to-be-detected image and the centroid;

[0097] Step S330, in the case where the direction of the normal vector of the processed patch is inconsistent with the target direction, adjusting the direction of the normal vector of the processed patch to the target direction.

[0098] Among them, the direction of the normal vector is represented by its positive and negative signs. When the positive and negative signs of the normal vector are inconsistent with the determined target direction, the positive and negative signs of the normal vector can be adjusted to be consistent with the target direction.

[0099] Among them, the present application embodiment does not make specific limitations on how to specifically determine the target direction according to the pose information of the imaging device and the centroid of the patch. For example, taking to represent the pose information of the imaging device in the world coordinate system, and using t to represent the translation component of Among them, p1, p2, and p3 represent three three-dimensional points obtained by back-projecting the three two-dimensional points included in the triangular patch onto the image to be detected. It is determined that the orientation of the optical center relative to the triangular patch is t–p, that is, the target direction is t–p.

[0100] In some embodiments, according to the preset area threshold, the triangular patches corresponding to each of the point-connected regions are respectively screened and merged to obtain the processed patches corresponding to each of the point-connected regions (i.e., step S210), as Figure 4 shown, may include:

[0101] Step S410, for any one of the point-connected regions, determine any two triangular patches with overlapping edges corresponding thereto as having an adjacency relationship, and construct an adjacency matrix of the triangular patches;

[0102] Step S420, determine all the triangular patches corresponding to the current point-connected region and having an area smaller than the preset area threshold as patches to be merged;

[0103] Step S430, according to the adjacency matrix of the triangular patches, merge each of the patches to be merged with its adjacent patches to be merged until the area of the merged patch is not less than the preset area threshold; wherein, the types of the processed patches include the merged patches and the triangular patches corresponding to the current point-connected region and having an area not less than the preset area threshold.

[0104] Among them, the embodiments of the present application do not make special limitations on how steps S420 and S430 are specifically executed. For example, it can also be executed in a manner similar to the breadth-first search algorithm. Specifically, traverse each triangular patch corresponding to the current point-connected region, and use the triangular patches with an area smaller than the preset area threshold as patches to be merged; traverse the first patch to be merged; for the patch to be merged currently being traversed, add it to the merge list, and traverse each of its adjacent triangular patches according to the adjacency matrix of the triangular patches, and also add the adjacent triangular patches with an area smaller than the preset area threshold to the merge list until the total area of the triangular patches in the merge list is not less than the preset area threshold. At this time, merge the triangular patches in the merge list; traverse the next patch to be merged, and repeat the above steps until all the patches to be merged have been traversed.

[0105] As Figure 7a shown, it is a schematic diagram of an implementation manner of merging triangular patches provided by the embodiments of the present application. There are a total of 3 adjacent triangular patches in a certain connected region and their areas are all smaller than the preset area threshold, then these 3 adjacent triangular patches are merged to obtain a large polygon patch as the processed patch. It should be noted that Figure 7aFor exemplary illustration only, the number of merged triangular patches is not limited to 3, the number of triangular patches within the connected region is not limited to 3, and the positional relationship between the triangular patches within the connected region is not limited to being adjacent in sequence.

[0106] In some embodiments, in the step of calculating the normal vectors of the processed patches, the normal vector of any triangular patch corresponding to the processed patch is calculated according to the following formula:

[0107]

[0108] In formula (1), n represents the normal vector of the any triangular patch, and p1, p2, and p3 represent three three-dimensional points obtained by back-projecting the three two-dimensional points included in the any triangular patch onto the image to be detected.

[0109] Among them, in the case where the type of the processed patch is a triangular patch corresponding to the current point connected region and having an area not less than the preset area threshold, its normal vector can be directly calculated according to the above formula (1). In the case where the type of the processed patch is the merged patch, the processed patch corresponds to multiple triangular patches before processing. The normal vector of each triangular patch corresponding to the processed patch can be calculated according to the above formula (1), and then the average value of these normal vectors is calculated as the normal vector of the processed patch.

[0110] In some embodiments, the initial three-dimensional point cloud of the image to be detected is segmented into point connected regions according to the preset distance threshold (i.e., the one involved in step S110), as Figure 5 shown, may include:

[0111] Step S510, determining that any two three-dimensional points in the initial three-dimensional point cloud with a distance less than the preset distance threshold have an adjacency relationship, and constructing an adjacency matrix of the three-dimensional points;

[0112] Step S520, segmenting the initial three-dimensional point cloud of the image to be detected into point connected regions according to the breadth-first search algorithm and the adjacency matrix of the three-dimensional points; wherein, the distance between any two three-dimensional points in any one of the point connected regions is less than the preset distance threshold, and the distance between any two three-dimensional points from different point connected regions is not less than the preset distance threshold.

[0113] Among them, the embodiments of the present application do not make special limitations on how to specifically execute step S510. For example, two three-dimensional points in the initial three-dimensional point cloud can be traversed pairwise. When the distance between the two three-dimensional points is less than the preset distance threshold, it is determined that the two three-dimensional points have an adjacency relationship, and finally an adjacency matrix of the three-dimensional points is constructed.

[0114] Among them, the embodiments of the present application do not make special limitations on how to specifically execute step S520. For example, each three-dimensional point in the initial three-dimensional point cloud can be traversed; the currently traversed three-dimensional point is marked as a traversed point and added to the queue; a traversed point is taken out from the queue, and each adjacent three-dimensional point is traversed according to the adjacency matrix of the three-dimensional point. When the traversed adjacent three-dimensional point has not been traversed, it is marked as a traversed point and added to the queue. When the traversed adjacent three-dimensional point has been traversed, it is ignored, and the next adjacent three-dimensional point is continued to be traversed; the above steps are repeatedly executed until the queue is empty, and the point connectivity domain is constructed based on the three-dimensional points that have been traversed currently and added to the set of point connectivity domains; continue to traverse the next three-dimensional point in the initial three-dimensional point cloud until each three-dimensional point in the initial three-dimensional point cloud has been traversed.

[0115] As Figure 7b shown, it is a schematic diagram of an implementation manner for segmenting point connectivity domains provided by the embodiments of the present application. It can be seen that the three-dimensional points in the initial three-dimensional point cloud are segmented into two point connectivity domains. There are 3 three-dimensional points in connectivity domain 1 and 5 three-dimensional points in connectivity domain 2. It should be noted that Figure 7b only for exemplary illustration, the number of connectivity domains is not limited to 2, and the number of three-dimensional points in the connectivity domain is not limited to 3 and 5 either, which are all determined by the actual situation of the image to be detected.

[0116] In some embodiments, in the step of projecting the three-dimensional points in each of the point connectivity domains onto the image to be detected to obtain the two-dimensional point cloud corresponding to each of the point connectivity domains (i.e., the step involved in S120), the two-dimensional points are obtained through the following formula:

[0117]

[0118] In formula (2), represents the two-dimensional point, represents the three-dimensional point, (f x , f y ) represents the focal length corresponding to the image to be detected, and (cx, cy) represents the principal point of the imaging device corresponding to the image to be detected.

[0119] In some embodiments, for each of the target three-dimensional point clouds, performing a fitting process to obtain a plane detection result (i.e., the step involved in S160), as Figure 6 shown, may include:

[0120] Step S610, for any one of the target three-dimensional point clouds, performing a random fitting step according to a preset number of executions to obtain a plurality of fitting planes;

[0121] Step S620, determining the fitting plane with the largest number of points as the target plane from the plurality of fitting planes:

[0122] Step S630: Perform fitting processing on the three-dimensional points in the target plane according to a preset fitting algorithm to obtain the plane detection result.

[0123] Among them, the random fitting step (i.e., the one involved in step S610) may include:

[0124] Step S710: Randomly select three three-dimensional points from any one of the target three-dimensional point clouds;

[0125] Step S720: Perform fitting on the selected three three-dimensional points according to the preset fitting algorithm to obtain a fitting plane;

[0126] Step S730: Determine other three-dimensional points in any one of the target three-dimensional point clouds that are located on the fitting plane;

[0127] Step S740: Calculate the number of points on the fitting plane according to the determined other three-dimensional points and the selected three three-dimensional points.

[0128] Among them, the embodiments of the present application do not make special limitations on the preset number of executions. For example, it can be set according to the number of three-dimensional points in the target three-dimensional point cloud.

[0129] Among them, the embodiments of the present application do not make special limitations on the preset fitting algorithm. For example, the preset fitting algorithm can be the PCA algorithm.

[0130] As the second aspect of the embodiments of the present application, an electronic device is provided. As Figure 8 shown, the electronic device includes:

[0131] One or more processors 101;

[0132] A memory 102, on which one or more computer programs are stored. When the one or more computer programs are executed by the one or more processors 101, the one or more processors 101 implement the plane detection method provided in the first aspect of the embodiments of the present application.

[0133] The electronic device may further include one or more I / O interfaces 103, which are connected between the processor 101 and the memory 102 and are configured to implement information interaction between the processor 101 and the memory 102.

[0134] Among them, the processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU), etc.; the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically such as SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read / write interface) is connected between the processor and the memory and can implement information interaction between the processor and the memory, including but not limited to a data bus (Bus), etc.

[0135] In some embodiments, the processor 101, the memory 102, and the I / O interface 103 are interconnected through a bus 104 and further connected to other components of the computing device.

[0136] As the third aspect of the embodiments of the present application, as Figure 9 shown, a computer-readable medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the plane detection method provided in the first aspect of the embodiments of the present application is implemented.

[0137] Those of ordinary skill in the art can understand that to implement all or part of the processes in the above-described embodiment methods, it can be completed by instructing relevant hardware through a computer program. Accordingly, the computer program can be stored in a non-volatile computer-readable storage medium, and when the computer program is executed, the methods of any of the above embodiments can be implemented. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the embodiments of the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0138] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Those skilled in the art should understand that the present application includes but is not limited to the content described in the drawings and the above specific implementation manner. Any modification that does not deviate from the functional and structural principles of the present application will be included in the scope of the claims.

Claims

1. A planar detection method, characterized in that, The method includes: Segmenting the initial three-dimensional point cloud of the image to be detected into point connected regions according to a preset distance threshold; Projecting the three-dimensional points in each of the point connected regions onto the image to be detected to obtain two-dimensional point clouds corresponding to the point connected regions; Constructing triangular patches corresponding to the point connected regions according to a preset triangulation algorithm and the two-dimensional point clouds corresponding to the point connected regions; Generating patch connected regions according to the triangular patches corresponding to the point connected regions; Projecting the two-dimensional points in each of the patch connected regions onto the image to be detected to obtain target three-dimensional point clouds corresponding to the patch connected regions; Performing fitting processing on each of the target three-dimensional point clouds respectively to obtain a plane detection result.

2. The method according to claim 1, characterized in that, The generating patch connected regions according to the triangular patches corresponding to the point connected regions includes: Performing screening and merging processing on the triangular patches corresponding to the point connected regions respectively according to a preset area threshold to obtain processed patches corresponding to the point connected regions; Determining that any two processed patches with overlapping edges in the processed patches corresponding to the point connected regions have an adjacency relationship, and constructing an adjacency matrix of the processed patches; Calculating the normal vectors of the processed patches; Generating patch connected regions according to the breadth-first search algorithm, the processed patches whose normal vectors match the gravity direction, and the adjacency matrix of the processed patches.

3. The method according to claim 2, wherein After calculating the normal vectors of the processed patches and before generating patch connected regions according to the breadth-first search algorithm, the processed patches whose normal vectors match the gravity direction, and the adjacency matrix of the processed patches, the method further includes: Back-projecting the multiple two-dimensional points included in the processed patches onto the image to be detected to obtain multiple three-dimensional points, and calculating the centroid of the processed patches; Determining a target direction according to the pose information of the imaging device corresponding to the image to be detected and the centroid; In the case where the direction of the normal vector of the processed patch is inconsistent with the target direction, adjusting the direction of the normal vector of the processed patch to the target direction.

4. The method according to claim 2, wherein The performing screening and merging processing on the triangular patches corresponding to the point connected regions respectively according to a preset area threshold to obtain processed patches corresponding to the point connected regions includes: For any one of the point connected regions, determining that any two triangular patches with overlapping edges corresponding to it have an adjacency relationship, and constructing an adjacency matrix of the triangular patches; Determining all triangular patches corresponding to the current point connected region with an area smaller than the preset area threshold as patches to be merged; According to the adjacency matrix of the triangular patches, merging each of the patches to be merged with its adjacent patches to be merged until the area of the merged patch is not less than the preset area threshold; wherein, the types of the processed patches include the merged patches and the triangular patches corresponding to the current point connected region with an area not less than the preset area threshold.

5. The method according to claim 4, wherein In the step of calculating the normal vectors of the processed patches, the normal vector of any triangular patch corresponding to the processed patch is calculated according to the following formula: In formula (1), n represents the normal vector of any one of the triangular patches, and p1, p2, and p3 represent three three-dimensional points obtained by back-projecting the three two-dimensional points included in any one of the triangular patches onto the image to be detected.

6. The method according to claim 1, characterized in that The step of segmenting the initial three-dimensional point cloud of the image to be detected into point-connected regions according to a preset distance threshold includes: Determining that any two three-dimensional points in the initial three-dimensional point cloud with a distance less than the preset distance threshold have an adjacency relationship, and constructing an adjacency matrix of the three-dimensional points; Segmenting the initial three-dimensional point cloud of the image to be detected into point-connected regions according to the breadth-first search algorithm and the adjacency matrix of the three-dimensional points; wherein, the distance between any two three-dimensional points in any one of the point-connected regions is less than the preset distance threshold, and the distance between any two three-dimensional points from different point-connected regions is not less than the preset distance threshold.

7. The method according to claim 1, wherein In the step of projecting the three-dimensional points in each of the point-connected regions onto the image to be detected to obtain two-dimensional point clouds corresponding to each of the point-connected regions, the two-dimensional points are obtained through the following formula: In formula (2), represents a two-dimensional point, represents a three-dimensional point, (f x , f y ) represents the focal length corresponding to the image to be detected, and (cx, cy) represents the principal point of the imaging device corresponding to the image to be detected.

8. The method according to claim 1, characterized in that, The step of respectively performing fitting processing on each of the target three-dimensional point clouds to obtain a plane detection result includes: For any one of the target three-dimensional point clouds, performing a random fitting step according to a preset number of executions to obtain a plurality of fitting planes; Determining the fitting plane with the largest number of points as the target plane from the plurality of fitting planes: Performing fitting processing on the three-dimensional points in the target plane according to a preset fitting algorithm to obtain the plane detection result; Wherein, the random fitting step includes: Randomly selecting three three-dimensional points from any one of the target three-dimensional point clouds; Performing fitting on the selected three three-dimensional points according to the preset fitting algorithm to obtain a fitting plane; Determining other three-dimensional points located in the fitting plane from any one of the target three-dimensional point clouds; Calculating the number of points of the fitting plane according to the determined other three-dimensional points and the selected three three-dimensional points.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; A memory storing one or more computer programs thereon, which when executed by the one or more processors, cause the one or more processors to implement the plane detection method according to any one of claims 1-8.

10. A computer-readable medium having a computer program stored thereon, characterized in that, The computer program, when executed by the processor, implements the plane detection method according to any one of claims 1-8.

Images