A plane detection method, device and equipment based on three-dimensional point cloud and storage medium

Abstract

This application discloses a planar detection method, apparatus, device, and storage medium based on 3D point clouds. The related embodiments can be applied to various scenarios such as maps and intelligent transportation to improve planar detection efficiency. The method includes: dividing the 3D sparse point set to be processed into a candidate point subset and multiple insertion point subsets; if the number of points in the insertion point subset is greater than a first point quantity threshold, a temporary plane is generated using random sampling; the plane parameters are iteratively updated to obtain an updated plane; any two updated planes that meet the plane merging conditions are merged to obtain a merged updated plane; if the number of points in the updated candidate point set is greater than a second point quantity threshold, and the total number of merged updated planes is less than a plane threshold, candidate planes are generated based on the points in the updated candidate point set; and all candidate planes, merged updated planes, and remaining unmerged updated planes are summarized into a target plane set.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a planar detection method, apparatus, device, and storage medium based on three-dimensional point clouds. Background Technology

[0002] With the rapid development of Internet technology, Augmented Reality (AR) technology based on network technology is being widely used in more and more fields, such as short videos, games, UI interaction, mechanical assembly, equipment maintenance and structural display. Among them, AR virtual object placement is adopted by many applications as a new interaction method, and plane detection is an important capability of AR. Therefore, how to perform plane detection more accurately and quickly is a hot research topic.

[0003] Most current planar detection schemes are based on RGB or RGB-D images. RGB image planar detection uses algorithms such as image segmentation and deep learning to perform planar detection and segmentation on two-dimensional images to obtain pixel-level planar representations. RGB-D image planar detection, due to the presence of depth D information, can perform planar detection in three-dimensional space. It combines 2D pixel information to assist in planar segmentation and detection, thereby obtaining the three-dimensional plane of the current image frame.

[0004] However, RGB image plane detection can only process two-dimensional information, and the plane exists only in pixel space, which cannot be directly applied in AR scenes. RGB-D image plane detection requires an RGB-D camera, which is difficult to meet the hardware requirements in actual detection, resulting in poor versatility. Summary of the Invention

[0005] This application provides a planar detection method, apparatus, device, and storage medium based on 3D point clouds. It is used to fully acquire the planes that the 3D sparse point set to be processed may form in the scene by using a random sampling algorithm. It can be directly and widely applied to planar detection in AR scenes without the need for complex calculations such as planar detection segmentation or deep learning on 2D images, thus enabling fast and efficient planar detection.

[0006] One embodiment of this application provides a planar detection method based on 3D point clouds, including:

[0007] Obtain the set of 3D sparse points to be processed, which includes multiple points and the spatial coordinates of each point;

[0008] Based on the spatial coordinates corresponding to each point, the 3D sparse point set to be processed is divided into a candidate point subset and multiple insertion point subsets. Each insertion point subset describes the set of points that can be inserted into the plane, and the candidate point subset describes the set of points that cannot be inserted into the plane.

[0009] For each subset of points to be inserted, if the number of points in the subset is greater than the first threshold, a temporary plane is generated by random sampling. The plane parameters of the current plane with the subset of points to be inserted are iteratively updated to obtain the updated plane.

[0010] Merge any two updated planes that meet the plane merging conditions to obtain a merged updated plane;

[0011] Update the candidate point subset to the candidate point set. If the number of points in the updated candidate point set is greater than the second point number threshold, and the total number of currently merged and updated planes is less than the plane threshold, then generate candidate planes based on the points in the updated candidate point set.

[0012] All candidate planes, merged and updated planes, and remaining unmerged updated planes are summarized to obtain the target plane set corresponding to the 3D sparse point set to be processed.

[0013] Another aspect of this application provides a planar detection device based on 3D point clouds, comprising:

[0014] The acquisition unit is used to acquire the three-dimensional sparse point set to be processed, wherein the three-dimensional sparse point set to be processed includes multiple points and the spatial coordinates corresponding to each point;

[0015] The processing unit is used to divide the 3D sparse point set to be processed into a candidate point subset and multiple insertion point subsets based on the spatial coordinates corresponding to each point. Each insertion point subset is used to describe the set of points that can be inserted into the plane, and the candidate point subset is used to describe the set of points that cannot be inserted into the plane.

[0016] The processing unit is also used to generate a temporary plane by random sampling for each subset of points to be inserted. If the number of points in the subset of points to be inserted is greater than the first threshold, the plane parameters of the current plane with the subset of points to be inserted inserted are iteratively updated to obtain the updated plane.

[0017] The processing unit is also used to merge any two update planes that meet the plane merging conditions to obtain a merged update plane;

[0018] The processing unit is also used to update the candidate point subset to the candidate point set. If the number of points in the updated candidate point set is greater than the second point number threshold, and the total number of currently merged and updated planes is less than the plane threshold, then a candidate plane is generated based on the points in the updated candidate point set.

[0019] The determination unit is used to summarize all candidate planes, merged and updated planes, and the remaining unmerged updated planes to obtain the target plane set corresponding to the three-dimensional sparse point set to be processed.

[0020] In one possible design, in another implementation of the embodiments of this application, the processing unit may specifically be used for:

[0021] For the current plane that has been inserted with a subset of points to be inserted, three points are randomly selected in each iteration to generate a temporary plane corresponding to the current iteration;

[0022] Calculate the first distance between each point in the current plane and the temporary plane corresponding to the current iteration;

[0023] Based on the first distance, determine the set of interior points of the temporary plane corresponding to the current iteration;

[0024] Based on the set of interior points of the temporary plane corresponding to the current iteration, the plane parameters of the current plane are iteratively updated to obtain the updated plane.

[0025] In one possible design, in another implementation of the embodiments of this application, the processing unit may specifically be used for:

[0026] Based on the set of interior points of the temporary plane corresponding to the current iteration, plane fitting is performed to obtain the plane fitting parameters corresponding to the temporary plane;

[0027] The plane fitting parameters are iteratively optimized to obtain the temporary optimized plane corresponding to the temporary plane.

[0028] Based on the number of points in the optimization plane corresponding to each iteration, the update plane is selected from all the temporary optimization planes.

[0029] In one possible design, in another implementation of the embodiments of this application, the processing unit may specifically be used for:

[0030] Calculate the second distance between each point in the current plane and the temporary optimization plane;

[0031] Points corresponding to the second distance that is less than or equal to the distance threshold are identified as points within the optimization plane corresponding to the temporary optimization plane, so as to count the number of points within the optimization plane corresponding to the temporary optimization plane.

[0032] The temporary optimization plane corresponding to the largest number of points in the optimization plane is determined as the update plane.

[0033] In one possible design, in another implementation of the embodiments of this application, the processing unit may specifically be used for:

[0034] Obtain all points within the temporary optimization plane and project them onto the temporary optimization plane to form the largest outer contour polygon.

[0035] Calculate the vertex centers of the polygon and the projection centers of all points in the optimization plane onto the temporary optimization plane;

[0036] Based on the vertex center and projection center, abnormal vertices are selected from all vertices of the polygon and removed to obtain the update plane and the optimized vertex set of the update plane.

[0037] Based on the optimized vertex set, the optimized outer contour polygon and the area of ​​the optimized outer contour polygon corresponding to the updated plane are recalculated, and the plane information corresponding to the updated plane is calculated, wherein the plane information includes at least the plane normal vector.

[0038] In one possible design, in another implementation of the embodiments of this application, the processing unit may specifically be used for:

[0039] Calculate the angle between the plane normal vectors corresponding to any two update planes;

[0040] Based on the optimized outer contour polygon and the area of ​​the optimized outer contour polygon corresponding to each updated plane, calculate the overlapping area and the area intersection-union ratio of the outer contour polygons after the mutual projection of any two updated planes.

[0041] When the plane merging conditions are that the included angle is less than the included angle threshold, the overlapping area is greater than the area threshold, and the area intersection-to-union ratio is greater than the intersection-to-union ratio threshold, any two updated planes that meet the plane merging conditions are merged to obtain the merged updated plane.

[0042] In one possible design, in another implementation of the embodiments of this application,

[0043] The processing unit is also used to calculate the area of ​​the outer contour polygon corresponding to each merged update plane;

[0044] The processing unit is also used to compare the area of ​​the outer contour polygon corresponding to each merged update plane and the area of ​​the optimized outer contour polygon corresponding to the remaining unmerged update planes with the second area threshold, respectively.

[0045] The processing unit is also used to remove merged update planes with an area smaller than the second area threshold or the remaining unmerged update planes to obtain the optimal set of update planes.

[0046] The determination unit is also used to summarize all candidate planes and the optimal update plane set to obtain the target plane set corresponding to the three-dimensional sparse point set to be processed.

[0047] In one possible design, in another implementation of the embodiments of this application, the processing unit may specifically be used for:

[0048] Merge all points on any two update planes that meet the plane merging conditions to obtain the merged plane point set;

[0049] Based on the merged plane point set, plane fitting is performed to obtain the merged plane fitting parameters;

[0050] The fitting parameters of the merged plane are iteratively optimized to obtain the merged updated plane.

[0051] In one possible design, in another implementation of the embodiments of this application,

[0052] The determining unit is also used to determine, for the update plane, the points corresponding to the second distance greater than the distance threshold as the external points corresponding to the update plane, so as to summarize and obtain the set of external points corresponding to the update plane;

[0053] The processing unit is also used to add the set of outliers to the total set of candidate points in order to update the total set of candidate points.

[0054] In one possible design, in another implementation of the embodiments of this application, the processing unit may specifically be used for:

[0055] The point corresponding to the first distance that is less than or equal to the distance threshold is determined as the interior point of the temporary plane corresponding to the current iteration;

[0056] Summarize the interior points of the temporary plane corresponding to the current iteration to obtain the set of interior points of the temporary plane corresponding to the current iteration.

[0057] In one possible design, in another implementation of the embodiments of this application, the processing unit may specifically be used for:

[0058] In each iteration, three points are randomly selected from the updated candidate point set to generate a temporary candidate plane.

[0059] Calculate the fourth distance between each candidate point in the updated candidate point set and the temporary candidate plane;

[0060] Based on the fourth distance, determine the set of interior points of the temporary candidate plane;

[0061] Based on the set of interior points of the temporary candidate plane corresponding to each iteration, select the temporary candidate plane with the largest number of interior points;

[0062] If the total number of interior points of the temporary candidate plane with the largest number of interior points is greater than the interior point threshold, then the temporary candidate plane with the largest number of interior points is determined as a valid candidate plane.

[0063] The effective candidate planes are iteratively optimized by optimizing the plane parameters to obtain the candidate planes.

[0064] In one possible design, in another implementation of the embodiments of this application, the processing unit may specifically be used for:

[0065] If the current number of planes is not zero, then calculate the third distance between each point in the 3D sparse point set to be processed and each current plane;

[0066] Based on the third distance, the insertion points for each current plane are selected from the set of sparse points to be processed in the three-dimensional plane, resulting in a subset of candidate points and a subset of points to be inserted for each current plane.

[0067] In one possible design, in another implementation of the embodiments of this application, the processing unit may specifically be used for:

[0068] The third distance between each point and each current plane is compared with a distance threshold to obtain the comparison result;

[0069] If the comparison result is greater than the distance threshold, the point corresponding to the third distance is determined as a candidate point, and all candidate points are summarized to obtain a subset of candidate points;

[0070] If the comparison result is less than or equal to the distance threshold, the point corresponding to the third distance is determined as the insertion point of the current plane, so as to obtain the subset of points to be inserted corresponding to each current plane.

[0071] This application also provides a computer device, including: a memory, a processor, and a bus system;

[0072] The memory is used to store programs;

[0073] The processor implements the methods described above when executing a program in memory;

[0074] Bus systems are used to connect memory and processor to enable communication between them.

[0075] Another aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described above.

[0076] As can be seen from the above technical solutions, the embodiments of this application have the following beneficial effects:

[0077] By acquiring the 3D sparse point set to be processed, and based on the spatial coordinates of each point, the 3D sparse point set to be processed is divided into a candidate point subset and multiple insertion point subsets. Then, if the number of points in the insertion point subset is greater than the first point quantity threshold, a temporary plane is generated by random sampling. The plane parameters of the current plane with the insertion point subset are iteratively updated to obtain an updated plane. Any two updated planes that meet the plane merging conditions are merged to obtain a merged updated plane. Then, the candidate point subset is updated to the candidate point set. If the number of points in the updated candidate point set is greater than the second point quantity threshold, and the total number of merged updated planes is less than the plane threshold, a candidate plane is generated based on the points in the updated candidate point set. All candidate planes, merged updated planes, and the remaining unmerged updated planes are summarized to obtain the target plane set corresponding to the 3D sparse point set to be processed. The above method enables the generation of temporary planes based on the sparse 3D point set to be processed by random sampling. The plane parameters are iteratively updated to obtain the updated plane. The planes are then merged in combination with the plane merging conditions to further update the planes. Candidate planes are generated based on the points in the updated candidate point set. This method fully acquires the planes that the sparse 3D point set to be processed may form in the scene. It can be directly and widely applied to plane detection in AR scenes without the need for complex calculations such as plane detection and segmentation or deep learning on 2D images, thus enabling fast and efficient plane detection. Attached Figure Description

[0078] Figure 1 This is a schematic diagram of the architecture of the image data control system in an embodiment of this application;

[0079] Figure 2 This is a flowchart of one embodiment of the planar detection method based on 3D point clouds in this application.

[0080] Figure 3 This is a flowchart of another embodiment of the planar detection method based on 3D point clouds in this application;

[0081] Figure 4 This is a flowchart of another embodiment of the planar detection method based on 3D point clouds in this application;

[0082] Figure 5 This is a flowchart of another embodiment of the planar detection method based on 3D point clouds in this application;

[0083] Figure 6 This is a flowchart of another embodiment of the planar detection method based on 3D point clouds in this application;

[0084] Figure 7This is a flowchart of another embodiment of the planar detection method based on 3D point clouds in this application;

[0085] Figure 8 This is a flowchart of another embodiment of the planar detection method based on 3D point clouds in this application;

[0086] Figure 9 This is a flowchart of another embodiment of the planar detection method based on 3D point clouds in this application;

[0087] Figure 10 This is a flowchart of another embodiment of the planar detection method based on 3D point clouds in this application;

[0088] Figure 11 This is a flowchart of another embodiment of the planar detection method based on 3D point clouds in this application;

[0089] Figure 12 This is a flowchart of another embodiment of the planar detection method based on 3D point clouds in this application;

[0090] Figure 13 This is a flowchart of another embodiment of the planar detection method based on 3D point clouds in this application;

[0091] Figure 14 This is a flowchart of another embodiment of the planar detection method based on 3D point clouds in this application;

[0092] Figure 15 This is a schematic diagram illustrating the principle of a planar detection method based on 3D point clouds in an embodiment of this application;

[0093] Figure 16 This is a schematic diagram of a planar anomaly far point removal method based on three-dimensional point clouds in an embodiment of this application;

[0094] Figure 17 This is a schematic diagram of one embodiment of the planar detection device based on three-dimensional point clouds in this application;

[0095] Figure 18 This is a schematic diagram of one embodiment of the computer device described in this application. Detailed Implementation

[0096] This application provides a planar detection method, apparatus, device, and storage medium based on 3D point clouds. It is used to fully acquire the planes that the 3D sparse point set to be processed may form in the scene by using a random sampling algorithm. It can be directly and widely applied to planar detection in AR scenes without the need for complex calculations such as planar detection segmentation or deep learning on 2D images, thus enabling fast and efficient planar detection.

[0097] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0098] To facilitate understanding, some terms or concepts involved in the embodiments of this application will be explained first.

[0099] 1. Augmented Reality (AR): This is a technology that cleverly integrates virtual information with the real world. It widely uses various technologies such as multimedia, 3D modeling, real-time detection and registration, intelligent interaction, and sensing to simulate and apply computer-generated virtual information such as text, images, 3D models, music, and videos to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

[0100] It is understood that in the specific embodiments of this application, data related to sparse three-dimensional point sets, plane thresholds, etc. are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0101] It is understandable that the planar detection method based on 3D point clouds disclosed in this application involves cloud technology, which will be further introduced below. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize data computing, storage, processing, and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology applied based on the cloud computing business model. It can form a resource pool, which can be used on demand, flexibly and conveniently. Cloud computing technology will become an important support. The back-end services of technical network systems require a large amount of computing and storage resources, such as video websites, image websites, and more portal websites. With the rapid development and application of the Internet industry, in the future, every item may have its own identification mark, which will need to be transmitted to the back-end system for logical processing. Data of different levels will be processed separately, and various industry data will all require strong system support, which can only be achieved through cloud computing.

[0102] Cloud computing is a computing model that distributes computing tasks across a resource pool composed of a large number of computers, enabling various application systems to access computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, the resources in the "cloud" are infinitely scalable, readily available, on-demand, expandable, and pay-as-you-go.

[0103] As a provider of fundamental cloud computing capabilities, a cloud resource pool (referred to as a cloud platform, generally called an IaaS (Infrastructure as a Service) platform) is established. Various types of virtual resources are deployed in the resource pool for external customers to choose from. The cloud resource pool mainly includes: computing devices (virtualized machines containing operating systems), storage devices, and network devices.

[0104] Based on logical function, a PaaS (Platform as a Service) layer can be deployed on top of the IaaS (Infrastructure as a Service) layer, and a SaaS (Software as a Service) layer can be deployed on top of the PaaS layer. Alternatively, SaaS can be directly deployed on top of IaaS. PaaS is a platform for running software, such as databases and web containers. SaaS refers to various types of transaction software, such as web portals and bulk SMS senders. Generally speaking, SaaS and PaaS are upper layers compared to IaaS.

[0105] Secondly, cloud security refers to the collective term for security software, hardware, users, organizations, and security cloud platforms based on cloud computing business models. Cloud security integrates emerging technologies and concepts such as parallel processing, grid computing, and unknown virus behavior detection. Through a large network of clients, it monitors abnormal software behavior on the network, obtains the latest information on Trojans and malware on the Internet, sends it to the server for automatic analysis and processing, and then distributes solutions for viruses and Trojans to each client.

[0106] Secondly, cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology and distributed storage file systems to bring together a large number of storage devices of various types (storage devices are also called storage nodes) in the network to work together and jointly provide data storage and transaction access functions to the outside world.

[0107] Currently, the storage method in storage systems is as follows: Logical volumes are created, and during creation, physical storage space is allocated to each logical volume. This physical storage space may consist of a single storage device or the disks of several storage devices. Clients store data on a logical volume, which means storing the data on the file system. The file system divides the data into many parts, each part being an object. Each object contains not only the data but also additional information such as a data identifier (ID, ID entity). The file system writes each object to the physical storage space of that logical volume and records the storage location information of each object. Therefore, when a client requests access to data, the file system can allow the client to access the data based on the storage location information of each object.

[0108] The process by which a storage system allocates physical storage space to a logical volume is as follows: the physical storage space is pre-divided into strips according to the capacity estimate of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of Redundant Array of Independent Disks (RAID). A logical volume can be understood as a strip, thus allocating physical storage space to the logical volume.

[0109] It should be understood that the planar detection method based on 3D point clouds provided in this application can be applied to various scenarios, including but not limited to artificial intelligence, cloud technology, maps, and intelligent transportation. It is used to perform planar detection through sparse 3D point cloud sets to obtain the planes existing in the scene, so as to be applied to scenarios such as interior design scenarios, exterior architectural design scenarios, game scene design scenarios, and intelligent map navigation.

[0110] To address the aforementioned problems, this application proposes a planar detection method based on 3D point clouds, which is applied to... Figure 1 Please refer to the image data control system shown. Figure 1 , Figure 1 This is a schematic diagram of the architecture of the image data control system in an embodiment of this application, such as... Figure 1 As shown, the server obtains the set of 3D sparse points to be processed provided by the terminal device. Based on the spatial coordinates of each point, the set of 3D sparse points to be processed is divided into a subset of candidate points and multiple subsets of points to be inserted. Then, if the number of points in the subset of points to be inserted is greater than the first threshold, a temporary plane is generated by random sampling. The plane parameters of the current plane with the subset of points to be inserted are iteratively updated to obtain an updated plane. Any two updated planes that meet the plane merging conditions are merged to obtain a merged updated plane. Then, the candidate point subset is updated to the total set of candidate points. If the number of points in the updated total set of candidate points is greater than the second threshold and the total number of merged updated planes is less than the plane threshold, a candidate plane is generated based on the points in the updated total set of candidate points. All candidate planes, merged updated planes, and the remaining unmerged updated planes are summarized to obtain the target plane set corresponding to the set of 3D sparse points to be processed. The above method enables the generation of temporary planes based on the sparse 3D point set to be processed by random sampling. The plane parameters are iteratively updated to obtain the updated plane. The planes are then merged in combination with the plane merging conditions to further update the planes. Candidate planes are generated based on the points in the updated candidate point set. This method fully acquires the planes that the sparse 3D point set to be processed may form in the scene. It can be directly and widely applied to plane detection in AR scenes without the need for complex calculations such as plane detection and segmentation or deep learning on 2D images, thus enabling fast and efficient plane detection.

[0111] Understandable, Figure 1 Only one type of terminal device is shown in the diagram. In real-world scenarios, many more types of terminal devices can participate in the data processing. These include, but are not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, and in-vehicle terminals. The specific number and types depend on the actual scenario and are not limited here. Furthermore, Figure 1 The diagram shows one server, but in real-world scenarios, multiple servers can be involved, especially in scenarios involving multi-model training and interaction. The number of servers depends on the specific scenario and is not limited here.

[0112] It should be noted that in this embodiment, the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal devices and servers can be directly or indirectly connected via wired or wireless communication, and terminal devices and servers can be connected to form a blockchain network; this application does not impose any limitations on this.

[0113] Based on the above introduction, the planar detection method based on 3D point clouds in this application will be described below. Please refer to [link / reference]. Figure 2 One embodiment of the planar detection method based on 3D point clouds in this application includes:

[0114] In step S101, a three-dimensional sparse point set to be processed is obtained, wherein the three-dimensional sparse point set to be processed includes multiple points and the spatial coordinates corresponding to each point;

[0115] In this embodiment, when it is necessary to perform planar detection on the target scene, a batch of three-dimensional sparse points in the current target scene can be obtained, namely the set of three-dimensional sparse points to be processed, so that the planar reconstruction can be performed based on the set of three-dimensional sparse points to be processed.

[0116] The sparse point set to be processed is a batch of sparse point 3D point clouds detected in the scene. A point cloud is a dataset of points in space, which can represent 3D shapes or objects, and is usually acquired by a 3D scanner. The position of each point in the point cloud is described by a set of Cartesian coordinates (X, Y, Z), and some may contain color information (R, G, B) or object reflective surface intensity information. A sparse point 3D point cloud refers to a massive set of points with a relatively small number of points and a relatively large distance between points, usually obtained using a 3D coordinate measuring machine. The target scene can specifically represent AR animation production scenes, AR game environments or map scene design, architectural design or intelligent map navigation design, etc., and can also represent other scenes, without specific limitations here. The sparse point set to be processed includes multiple points and the spatial coordinates corresponding to each point, which are a set of Cartesian coordinates (X, Y, Z).

[0117] Specifically, obtaining the set of sparse points to be processed can be achieved by scanning the target scene (such as a planar detection scene of an AR indoor floor) using a vision-based simultaneous localization and mapping system (VSLAM / VISLAM), and outputting a batch of sparse point 3D point clouds, which is the set of sparse points to be processed.

[0118] In step S102, based on the spatial coordinates corresponding to each point, the three-dimensional sparse point set to be processed is divided into a candidate point subset and multiple insertion point subsets. Each insertion point subset is used to describe the set of points that can be inserted into the plane, and the candidate point subset is used to describe the set of points that cannot be inserted into the plane.

[0119] In this embodiment, after obtaining the 3D sparse point set to be processed, based on the spatial coordinates corresponding to each point, it is possible to initially determine which points in the 3D sparse point set to be processed can be inserted into a plane and which cannot be inserted into a plane. Points that can be inserted into a plane refer to points that may belong to the current plane. That is, the 3D sparse point set to be processed can be divided into a subset of candidate points and multiple subsets of points to be inserted, so that the detected planes can be updated based on the subsets of points to be inserted, thereby more accurately detecting the planes existing in the target scene and improving the accuracy of plane detection.

[0120] Each subset of points to be inserted describes the set of points that can be inserted into the plane, i.e., the set of points that may belong to the plane. The subset of candidate points describes the set of points that cannot be inserted into the plane, i.e., the set of points that do not belong to the plane.

[0121] Specifically, after obtaining the set of 3D sparse points to be processed, we can first detect or determine whether the number of planes is zero. If it is zero, it can be understood that this is the first time to perform plane detection and no plane has been detected yet. That is, the points that can be inserted into the plane are empty, that is, the subset of points to be inserted is empty. Therefore, the set of 3D sparse points to be processed can be used as a subset of candidate points.

[0122] Furthermore, if the current number of planes is not zero, such as Figure 15 As shown, it is determined whether the 3D sparse point set to be processed has points that can be inserted into the existing plane. If so, the points that can be inserted into the existing plane can be determined as the insertion points of the current plane, and the points that cannot be inserted into the existing plane can be determined as candidate points. That is, the 3D sparse point set to be processed is divided into a candidate point subset and multiple insertion point subsets. If not, the insertion point subset can be considered empty, that is, the 3D sparse point set to be processed is determined as a candidate point subset.

[0123] Furthermore, to determine which points in the 3D sparse point set to be processed can be inserted into the existing plane, specifically, the distance between each point in the 3D sparse point set to be processed and each current plane can be calculated using the point-to-plane distance formula. Based on this distance, the insertion points for each current plane are selected from the 3D sparse point set to be processed to obtain a subset of points to be inserted for each current plane. Then, the remaining points are used as candidate points to obtain a subset of candidate points.

[0124] The general equation of the plane is shown in formula (1) below:

[0125] (1);

[0126] Where A, B, C, and D are known constants, which are plane parameters of the plane, and A, B, and C are not all zero at the same time.

[0127] The formula for the distance from a point to a surface can be specifically shown in formula (2) below:

[0128] (2)

[0129] in, Let be the spatial coordinates of the point.

[0130] In step S103, for each subset of points to be inserted, if the number of points in the subset is greater than the first threshold, a temporary plane is generated by random sampling, and the plane parameters of the current plane with the subset of points to be inserted are iteratively updated to obtain the updated plane.

[0131] In this embodiment, when the subset of points to be inserted is not empty and the number of points in the subset of points to be inserted is greater than the first threshold, a temporary plane can be generated by random sampling. Then, based on the temporary plane, the plane parameters of the current plane with the subset of points to be inserted can be iteratively updated to obtain the updated plane, thereby more accurately detecting the planes existing in the target scene and improving the accuracy of plane detection.

[0132] The first point quantity threshold is set according to actual application requirements. It is used to determine whether the number of points inserted into the existing plane has caused a change in the state of the current plane, thereby determining whether the current plane that has inserted a subset of points to be inserted needs to be updated.

[0133] Specifically, when the subset of points to be inserted is not empty, and the number of points in the subset is greater than the first threshold, it can be understood that the height or size of the current plane into which the subset of points to be inserted has changed. Therefore, in order to detect the plane state more accurately, this embodiment can be based on, as follows: Figure 15The plane update module shown in the figure iteratively updates the plane parameters of the current plane with the inserted subset of points. Specifically, it can iteratively update the current plane with the inserted subset of points according to the preset number of iterations. In each iteration, three points are randomly selected from all the points of the current plane. Since three non-collinear points determine a plane in space, after selecting three random points, it can first determine whether the three selected points are collinear. If they are collinear, the next iteration is stopped. If they are not collinear, the spatial coordinates of the three selected points are substituted into the above formula (1) for calculation to obtain the temporary plane corresponding to the three selected points.

[0134] Furthermore, according to the above formula (2), the distance between each point in the current plane and the temporary plane corresponding to the current iteration can be calculated respectively. Then, based on the distance, the points belonging to the temporary plane can be determined, that is, the interior points of the temporary plane. Then, the set of interior points of the temporary plane can be obtained by summarizing. Furthermore, based on the set of interior points of the temporary plane corresponding to the current iteration, the plane parameters of the current plane can be iteratively updated. After completing all iterations, the updated plane, the plane parameters of the updated plane, and the interior points belonging to the updated plane can be determined.

[0135] In step S104, any two updating planes that meet the plane merging conditions are merged to obtain a merged updating plane;

[0136] In this embodiment, since multiple updated planes may belong to one plane after the plane state changes, after obtaining all the updated planes, any two updated planes that meet the plane merging conditions can be merged to obtain a merged updated plane, which can further update and maintain the updated plane, thereby improving the accuracy of plane detection.

[0137] The plane merging condition is set according to the actual application requirements. In this embodiment, it can be set as follows: the angle between the normal vectors of the two planes is less than the angle threshold, and the polygon obtained by projecting the outer contour polygon of plane 1 onto another plane 2 has an overlap area greater than the area threshold with the outer contour polygon of plane 2, and the area intersection-union ratio is greater than the intersection-union ratio threshold. The polygon projected from plane 2 onto plane 1 also needs to meet the above conditions.

[0138] Specifically, such as Figure 15 As shown, after obtaining all the update planes, it is possible to base it on, as follows: Figure 15The multi-plane maintenance module, as shown, traverses all planes and determines whether the plane information of any two traversed planes meets the above plane merging conditions. If they do not meet the conditions, they cannot be merged; if they do meet the conditions, that is, the two planes belong to one plane, they can be merged. This process continues until all planes that meet the above plane merging conditions are merged, thus obtaining all merged update planes and the remaining update planes that cannot be merged.

[0139] In step S105, the candidate point subset is updated to the candidate point set. If the number of points in the updated candidate point set is greater than the second point number threshold, and the total number of currently merged and updated planes is less than the plane threshold, then a candidate plane is generated based on the points in the updated candidate point set.

[0140] In this embodiment, a subset of candidate points is added to the total set of candidate points to update the total set of candidate points. If the number of points in the updated total set of candidate points is greater than the second point quantity threshold, and the total number of the current merged update plane and the remaining unmerged update plane is less than the plane threshold, then new plane detection can be performed on the points in the updated total set of candidate points to obtain a new plane, i.e., a candidate plane.

[0141] The plane threshold is set according to actual application requirements and is not specifically limited here. It is used to determine whether the number of planes for plane detection in the 3D sparse point set to be processed is sufficient. The second point quantity threshold is set according to actual application requirements and is not specifically limited here. It is used to determine whether the points in the total set of candidate points are enough to generate a plane.

[0142] Specifically, such as Figure 15 As shown, after obtaining the merged update planes and the remaining unmerged update planes, we can first determine whether the total number of merged update planes and the remaining unmerged update planes has reached the maximum number of planes. If it has, the candidate planes are empty; if not, we can determine whether the total number of points in the updated candidate point set is greater than the second point number threshold. If it is less, the candidate planes are empty; if it is greater, we can proceed based on... Figure 15 The initial plane detection module, as shown, generates a new plane. That is, it can use the above-mentioned random sampling algorithm to generate candidate planes from the points in the updated candidate point set.

[0143] In step S106, all candidate planes, merged and updated planes, and the remaining unmerged updated planes are summarized to obtain the target plane set corresponding to the three-dimensional sparse point set to be processed.

[0144] Specifically, after obtaining all candidate planes, merged updated planes, and remaining unmerged updated planes, all candidate planes, merged updated planes, and remaining unmerged updated planes can be summarized to obtain the target plane set detected based on the three-dimensional sparse point set to be processed.

[0145] It should be noted that after obtaining all candidate planes, merging and updating planes, and the remaining unmerged updated planes, this embodiment can also perform expired 3D point cleanup, count the remaining 3D point cloud, that is, the remaining points in the total set of candidate points after generating candidate planes, and obtain the latest update time of each of these remaining points. Then, calculate the difference between the latest update time of each of these remaining points and the current time. If the difference is greater than the time threshold, it can be understood that the point has expired, and then the 3D point is deleted.

[0146] In this application embodiment, a plane detection method based on 3D point cloud is provided. Through the above method, a temporary plane can be generated by random sampling based on the 3D sparse point set to be processed. The plane parameters are iteratively updated to obtain an updated plane. The plane is then merged in combination with the plane merging condition to further update the plane. Candidate planes are generated based on the points in the updated candidate point set. This achieves full acquisition of the planes that the 3D sparse point set to be processed may form in the scene. It can be directly and widely applied to plane detection in AR scenes. Moreover, it does not require complex calculations such as plane detection segmentation or deep learning on 2D images, thus enabling fast and efficient plane detection.

[0147] Optionally, in the above Figure 2 Based on the corresponding embodiments, in another optional embodiment of the planar detection method based on 3D point clouds provided in this application, such as... Figure 3 As shown, in step S103, if the number of points in the subset of points to be inserted is greater than the first threshold, a temporary plane is generated by random sampling. The plane parameters of the current plane with the subset of points to be inserted are iteratively updated to obtain the updated plane, including:

[0148] In step S301, for the current plane that has inserted a subset of points to be inserted, three points are randomly selected in each iteration to generate a temporary plane corresponding to the current iteration;

[0149] In step S302, the first distance between each point in the current plane and the temporary plane corresponding to the current iteration is calculated.

[0150] In step S303, based on the first distance, the set of interior points of the temporary plane corresponding to the current iteration is determined;

[0151] In step S304, the plane parameters of the current plane are iteratively updated based on the set of interior points of the temporary plane corresponding to the current iteration to obtain the updated plane.

[0152] In this embodiment, after obtaining the subset of points to be inserted corresponding to each current plane, the subset of points to be inserted can be assigned to the current plane. Then, the current plane can be iteratively updated. In each iteration, three points are randomly selected to generate a temporary plane corresponding to the current iteration, and the first distance between each point in the current plane and the temporary plane corresponding to the current iteration is calculated. Then, based on the first distance, the set of interior points of the temporary plane corresponding to the current iteration can be determined, and the plane parameters of the current plane can be iteratively updated based on the set of interior points of the temporary plane corresponding to the current iteration to obtain an updated plane, thereby more accurately detecting the planes existing in the target scene and improving the accuracy of plane detection.

[0153] Specifically, after obtaining the subset of points to be inserted corresponding to each current plane, if the subset of points to be inserted is not empty, the subset of points to be inserted can be assigned to the current plane. If the number of points in the subset of points to be inserted is greater than the first threshold, it can be understood that the height or size of the current plane with the subset of points to be inserted may have changed. Therefore, in order to detect the plane state more accurately, this embodiment can be based on, as follows: Figure 15 The plane update module shown iteratively updates the plane parameters of the current plane with the inserted subset of points. Specifically, it iteratively updates the current plane with the inserted subset of points according to a preset number of iterations. In each iteration, three points are randomly selected from all points on the current plane. Since three non-collinear points determine a plane in space, after selecting three random points, it can be determined whether the three selected points are collinear. If they are collinear, the next iteration is stopped. If they are not collinear, the spatial coordinates of the three selected points are substituted into the above formula (1) for calculation to obtain the temporary plane corresponding to the three selected points. The number of iterations is set according to the actual application requirements and is not specifically limited here.

[0154] Further, according to the above formula (2), the distance between each point in the current plane and the temporary plane corresponding to the current iteration can be calculated, i.e., the first distance. Then, based on the first distance, the points that can belong to the temporary plane can be determined, i.e., the interior points of the temporary plane, so as to obtain the set of interior points of the temporary plane. Further, based on the set of interior points of the temporary plane corresponding to the current iteration, plane fitting processing can be performed to obtain the plane fitting parameters corresponding to the temporary plane. The plane fitting parameters can be iteratively optimized to obtain the temporary optimized plane corresponding to the temporary plane. Then, after completing all iterations, the updated plane can be selected from all the temporary optimized planes based on the number of interior points of the optimized plane corresponding to the temporary optimized plane in each iteration, and the updated plane, the plane parameters of the updated plane, and the interior points belonging to the updated plane can be determined.

[0155] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment of the planar detection method based on 3D point clouds provided in this application, such as... Figure 4 As shown, step S304 iteratively updates the plane parameters of the current plane based on the set of interior points of the temporary plane corresponding to the current iteration, to obtain the updated plane, including:

[0156] In step S401, based on the set of interior points of the temporary plane corresponding to the current iteration, plane fitting processing is performed to obtain the plane fitting parameters corresponding to the temporary plane;

[0157] In step S402, the plane fitting parameters are iteratively optimized to obtain the temporary optimized plane corresponding to the temporary plane;

[0158] In step S403, based on the number of points in the optimization plane corresponding to the temporary optimization plane in each iteration, the update plane is selected from all the temporary optimization planes.

[0159] In this embodiment, after obtaining the set of interior points of the temporary plane corresponding to the current iteration, in order to better determine the change in the height or size of the current plane and more accurately detect the state of the plane, this embodiment can perform plane fitting processing based on the set of interior points of the temporary plane corresponding to the current iteration to obtain the plane fitting parameters corresponding to the temporary plane. The plane fitting parameters are then iteratively optimized to obtain the temporary optimized plane corresponding to the temporary plane. Then, based on the number of interior points of the optimized plane corresponding to the temporary optimized plane in each iteration, the updated plane can be selected from all the temporary optimized planes, thereby more accurately detecting the planes existing in the target scene and improving the accuracy of plane detection.

[0160] Specifically, after obtaining the set of interior points of the temporary plane corresponding to the current iteration, the least squares method can be used to perform plane fitting on the interior points of the set of interior points of the temporary plane corresponding to the current iteration to obtain the plane fitting parameters (such as the parameter values ​​of A, B, C, and D in the general equation of the plane). The plane fitting parameters can then be iteratively optimized. Specifically, optimization algorithms (such as stochastic gradient descent) can be used to iteratively optimize the plane parameters to obtain the temporary optimized plane corresponding to the temporary plane. Then, after completing all iterations, the plane with the most interior points can be selected from all the temporary optimized planes based on the number of interior points of the optimized plane corresponding to each iteration. This plane is then used as the update plane, and the update plane, its plane parameters, and the interior points belonging to the update plane are determined.

[0161] Optionally, in the above Figure 4 Based on the corresponding embodiments, in another optional embodiment of the planar detection method based on 3D point clouds provided in this application, such as... Figure 5 As shown, step S403, based on the number of points within the temporary optimization plane corresponding to each iteration, selects the update plane from all temporary optimization planes, including:

[0162] In step S501, the second distance between each point in the current plane and the temporary optimized plane is calculated.

[0163] In step S502, the points corresponding to the second distance that is less than or equal to the distance threshold are determined as points in the optimization plane corresponding to the temporary optimization plane, so as to count the number of points in the optimization plane corresponding to the temporary optimization plane.

[0164] In step S503, the temporary optimization plane corresponding to the number of points in the optimization plane with the largest value is determined as the update plane.

[0165] In this embodiment, after obtaining the temporary optimized plane corresponding to the temporary plane, the second distance between each point in the current plane and the temporary optimized plane can be calculated. Points with a second distance less than or equal to the distance threshold are determined as points within the optimized plane corresponding to the temporary optimized plane. The number of points within the optimized plane corresponding to the temporary optimized plane is then counted. Finally, the temporary optimized plane corresponding to the largest number of points within the optimized plane is determined as the update plane, thereby more accurately detecting the planes existing in the target scene and improving the accuracy of plane detection.

[0166] Specifically, after obtaining the temporary optimization plane corresponding to the temporary plane, the distance between each point in the current plane and the temporary optimization plane can be calculated according to the above formula (2), i.e., the second distance. Then, based on the second distance, it can be determined which points can belong to the temporary optimization plane. Specifically, the points corresponding to the second distance that are less than or equal to the distance threshold can be determined as the points in the optimization plane corresponding to the temporary optimization plane, so as to obtain the set of points in the temporary plane and thus count the number of points in the optimization plane corresponding to the temporary optimization plane. Then, the temporary optimization plane corresponding to the largest number of points in the optimization plane can be determined as the update plane. The distance threshold is set according to the actual application requirements and is not specifically limited here.

[0167] It is understandable that after obtaining the update plane and determining the plane parameters of the update plane and the interior points belonging to the update plane, the points corresponding to the second distance greater than the distance threshold can be identified as the exterior points corresponding to the update plane. By summarizing these exterior point sets, the exterior point set can be updated to the candidate point set, so that the candidate plane can be generated based on the candidate point set in the future.

[0168] Optionally, in the above Figure 5 Based on the corresponding embodiments, in another optional embodiment of the planar detection method based on 3D point clouds provided in this application, such as... Figure 6 As shown, step S503 determines the temporary optimization plane corresponding to the number of points in the optimization plane with the largest numerical value as the update plane, including:

[0169] In step S601, all points in the optimization plane in the temporary optimization plane are obtained, and the maximum outer contour polygon projected onto the temporary optimization plane is obtained.

[0170] In step S602, the vertex centers of the polygon and the projection centers of all points in the optimization plane onto the temporary optimization plane are calculated.

[0171] In step S603, based on the vertex center and the projection center, abnormal vertices are selected from all vertices of the polygon and removed to obtain the update plane and the optimized vertex set of the update plane.

[0172] In step S604, based on the optimized vertex set, the optimized outer contour polygon and the area of ​​the optimized outer contour polygon corresponding to the updated plane are recalculated, and the plane information corresponding to the updated plane is calculated, wherein the plane information includes at least the plane normal vector.

[0173] In this embodiment, after obtaining the temporary optimization plane corresponding to the largest number of points in the optimization plane, all points in the temporary optimization plane can be projected onto the temporary optimization plane to obtain the largest outer contour polygon. Then, the vertex center of the polygon and the projection center of all points in the optimization plane on the temporary optimization plane are calculated. Based on the vertex center and the projection center, abnormal vertices are filtered out from all vertices of the polygon and removed to obtain the updated plane and the set of optimized vertices of the updated plane. Then, based on the set of optimized vertices, the area of ​​the optimized outer contour polygon corresponding to the updated plane and the plane information corresponding to the updated plane can be recalculated. By filtering out points in the temporary optimization plane that are far from the center of the points in the plane, i.e. the projection center, i.e. abnormal vertices, abnormal vertices can be removed to prevent abnormal vertices (i.e. noise points) from increasing the plane area, thereby improving the accuracy of plane detection to a certain extent.

[0174] The plane information includes the plane normal vector and the current plane geometric center.

[0175] Specifically, after obtaining the temporary optimization plane corresponding to the largest number of points in the optimization plane, all points in the temporary optimization plane can be projected onto the temporary optimization plane. Then, based on the Convex Hull Algorithm, or other contour calculation methods (no specific restrictions are imposed here), the maximum outer contour polygon projected onto the temporary optimization plane (e.g., ...) is calculated. Figure 16 (The polygon shown on the left).

[0176] Furthermore, after obtaining the largest outer contour polygon, the vertex centers of the polygon can be obtained by summing and averaging the spatial coordinates of each vertex in the largest outer contour polygon. Simultaneously, obtain the points projected onto the temporary optimization plane, i.e., the projection points (e.g., ...). Figure 16 By examining the vertices and interior points of the polygon shown on the left, the coordinates of the projection point can be obtained. These coordinates can then be summed and averaged to determine the projection center. .

[0177] Furthermore, based on the vertex center and projection center, abnormal vertices can be filtered out from all vertices of the polygon. Specifically, this can be done by judging each vertex in the polygon with the largest outer contour. Does it meet the following abnormal conditions:

[0178] 1) The angles between the vertex and its two adjacent points on the left and right are all less than the angle threshold;

[0179] 2) Each vertex satisfy ;

[0180] The angle threshold is set according to actual application requirements and is not specifically limited here. Dis represents the Euclidean distance between two points, i.e. For each vertex With vertex center The Euclidean distance between them Represent each vertex With the projection center The Euclidean distance between them. Wherein, This represents the preset distance threshold, which is set according to the actual application requirements. It can usually be set to 1.5 or other specific values, and there are no specific restrictions here.

[0181] Furthermore, if a vertex in the maximum outer contour polygon satisfies the above two conditions, it can be understood that the vertex is far from the center of the interior point of the temporary optimization plane, and is therefore an abnormal vertex that needs to be removed. After all vertices are evaluated, the updated plane is obtained (e.g., ...). Figure 16 The polygon shown on the right corresponds to the plane, and the optimized vertex set for updating the plane (such as...) Figure 16 (All vertices of the polygon shown on the right).

[0182] Furthermore, after obtaining the update plane and its optimized vertex set, the optimized outer contour polygon corresponding to the update plane can be recalculated (e.g., ...). Figure 16 The calculation involves determining the area of ​​the polygon shown on the right (and the area of ​​the optimized outer contour polygon), as well as calculating the planar information corresponding to the updated plane. Specifically, calculating the area of ​​the optimized outer contour polygon can be done using the shoelace formula, a triangulation algorithm combined with Heron's formula, or other algorithms; no specific restrictions are imposed here. Calculating the planar information corresponding to the updated plane can be done based on the spatial coordinates of each vertex in the optimized vertex set, and according to the mathematical definitions of the geometric center and normal vector, the geometric center and normal vector of the plane can be calculated.

[0183] Optionally, in the above Figure 6 Based on the corresponding embodiments, in another optional embodiment of the planar detection method based on 3D point clouds provided in this application, such as... Figure 7 As shown, step S104 merges any two updated planes that meet the plane merging conditions to obtain a merged updated plane, including:

[0184] In step S701, the angle between the plane normal vectors corresponding to any two update planes is calculated.

[0185] In step S702, based on the optimized outer contour polygon and the area of ​​the optimized outer contour polygon corresponding to each updated plane, the overlapping area and the area intersection-union ratio of the outer contour polygons after the mutual projection of any two updated planes are calculated.

[0186] In step S703, when the plane merging conditions are that the included angle is less than the included angle threshold, the overlapping area is greater than the area threshold, and the area intersection-to-union ratio is greater than the intersection-to-union ratio threshold, any two updated planes that meet the plane merging conditions are merged to obtain the merged updated plane.

[0187] In this embodiment, after obtaining all the update planes, the angle between the plane normal vectors corresponding to any two update planes is calculated. Based on the area of ​​the optimized outer contour polygon corresponding to each update plane, the overlapping area and the area intersection-union ratio of the outer contour polygons after the mutual projection of any two update planes are calculated. Two planes that meet the plane merging conditions are selected if the angle is less than the angle threshold, the overlapping area is greater than the area threshold, and the area intersection-union ratio is greater than the intersection-union ratio threshold. Then, any two update planes that meet the plane merging conditions are merged to obtain the merged update plane.

[0188] Specifically, such as Figure 15 As shown, after obtaining all the update planes, it is possible to base it on, as follows: Figure 15 The multi-plane maintenance module shown traverses all planes and determines whether the plane information of any two traversed planes meets the above-mentioned plane merging conditions. The plane merging conditions are set according to actual application requirements. In this embodiment, they can be set as follows: the angle between the normal vectors of the two planes is less than the angle threshold, and the polygon obtained by projecting the outer contour polygon of plane 1 onto another plane 2 has an overlap area greater than the area threshold and the area intersection-union ratio is greater than the intersection-union ratio threshold. In addition, the overlap area between the polygon projected from the outer contour polygon of plane 2 onto plane 1 and the outer contour polygon of plane 1 is greater than the area threshold and the area intersection-union ratio is greater than the intersection-union ratio threshold.

[0189] Furthermore, based on the conventional angle calculation formula, the angle between the plane normal vectors corresponding to any two planes is calculated. At the same time, the optimized outer contour polygons of any two planes (such as plane 1 and plane 2) are projected onto each other to obtain the polygon projected from the optimized outer contour polygon of plane 1 onto another plane 2, and the polygon projected from the optimized outer contour polygon of plane 2 onto plane 1. Based on the shoelace formula, the overlapping area and area intersection-union ratio of the polygon projected from the optimized outer contour polygon of plane 1 onto another plane 2 and the outer contour polygon of plane 2 are calculated, as well as the overlapping area and area intersection-union ratio of the polygon projected from the optimized outer contour polygon of plane 2 onto plane 1 and the outer contour polygon of plane 1.

[0190] Furthermore, the calculated included angle is compared with the included angle threshold, the overlapping area is compared with the area threshold, and the area intersection-union ratio is compared with the intersection-union ratio threshold. If the comparison results do not meet the above plane merging conditions, it can be considered that the two planes do not belong to the same plane and cannot be merged; if the comparison results meet the above plane merging conditions, it can be considered that the two planes belong to the same plane and can be merged. This process continues until all planes that meet the above plane merging conditions are merged, thus obtaining all merged updated planes and the remaining updated planes that cannot be merged.

[0191] Optionally, in the above Figure 7 Based on the corresponding embodiments, in another optional embodiment of the planar detection method based on 3D point clouds provided in this application, such as... Figure 8 As shown, in step S703, when the plane merging conditions are that the included angle is less than the included angle threshold, the overlapping area is greater than the area threshold, and the area intersection-to-union ratio is greater than the intersection-to-union ratio threshold, any two updated planes that meet the plane merging conditions are merged to obtain the merged updated plane. After that, the method further includes: steps S801 to S803; and step S106 includes: step S804.

[0192] In step S801, the area of ​​the outer contour polygon corresponding to each merged update plane is calculated;

[0193] In step S802, the area of ​​the outer contour polygon corresponding to each merged update plane and the area of ​​the optimized outer contour polygon corresponding to the remaining unmerged update planes are compared with the second area threshold respectively.

[0194] In step S803, merged update planes with areas smaller than the second area threshold or remaining unmerged update planes are removed to obtain the optimal update plane set.

[0195] In step S804, all candidate planes and the optimal update plane set are summarized to obtain the target plane set corresponding to the three-dimensional sparse point set to be processed.

[0196] In this embodiment, after obtaining all the merged update planes, the area of ​​the outer contour polygon corresponding to each merged update plane can be calculated. The area of ​​the outer contour polygon corresponding to each merged update plane, as well as the area of ​​the optimized outer contour polygon corresponding to the remaining unmerged update planes, are compared with the second area threshold. Then, merged update planes with areas smaller than the second area threshold or the remaining unmerged update planes are removed to obtain the optimal update plane set. All candidate planes and the optimal update plane set are summarized to obtain the target plane set corresponding to the three-dimensional sparse point set to be processed. By removing planes with excessively small areas, interference from noise planes can be avoided.

[0197] Specifically, after obtaining all the merged update planes, for each merged update plane, the interior points of each merged update plane can be counted, and the interior points of the merged update plane can be projected onto the merged update plane to obtain the outer contour polygon corresponding to the merged update plane. Then, the area of ​​the outer contour polygon corresponding to each merged update plane can be calculated based on the shoelace formula.

[0198] Furthermore, the area of ​​the outer contour polygon corresponding to each merged update plane, and the area of ​​the optimized outer contour polygon corresponding to the remaining unmerged update planes, are compared with a second area threshold. If the area is greater than or equal to the second area threshold, the merged update plane or the remaining unmerged update plane is considered a valid plane and is retained to obtain the optimal update plane set. Conversely, if the area is less than the second area threshold, the merged update plane or the remaining unmerged update plane is considered an invalid plane and is discarded, along with the 3D point cloud belonging to that plane. Then, all candidate planes and the optimal update plane set are summarized to obtain the target plane set corresponding to the 3D sparse point set to be processed. The second area threshold is set according to actual application requirements and is not specifically limited here.

[0199] Understandably, if no new 3D points are inserted into the current plane for a long time, that is, if the current plane does not acquire the corresponding subset of points to be inserted for a long time, the current plane can be considered an invalid plane, and the current plane will be discarded, along with the 3D point cloud belonging to the current plane.

[0200] Optionally, in the above Figure 2 Based on the corresponding embodiments, in another optional embodiment of the planar detection method based on 3D point clouds provided in this application, such as... Figure 9As shown, step S104 merges any two updated planes that meet the plane merging conditions to obtain a merged updated plane, including:

[0201] In step S901, all points on any two update planes that meet the plane merging conditions are merged to obtain a merged plane point set;

[0202] In step S902, plane fitting is performed based on the merged plane point set to obtain the merged plane fitting parameters;

[0203] In step S903, the fitting parameters of the merging plane are iteratively optimized to obtain the merging update plane.

[0204] In this embodiment, if any two updated planes meet the plane merging condition, all points on the two updated planes can be merged to obtain a merged plane point set. Based on the merged plane point set, plane fitting processing is performed to obtain merged plane fitting parameters. The merged plane fitting parameters are then iteratively optimized to obtain the merged updated plane, ensuring that the plane state of the merged updated plane is obtained accurately, thereby improving the plane detection accuracy to a certain extent.

[0205] Specifically, if any two update planes meet the plane merging condition, all points on the two update planes can be merged to obtain a merged plane point set. Based on the merged plane point set, the least squares method can be used to perform plane fitting on the points in the merged plane point set to obtain the merged plane fitting parameters (such as the parameter values ​​of A, B, C, and D in the general equation of the plane). The merged plane fitting parameters can then be iteratively optimized using optimization algorithms (e.g., stochastic gradient descent) to obtain the merged update plane.

[0206] Optionally, in the above Figure 5 Based on the corresponding embodiments, in another optional embodiment of the planar detection method based on 3D point clouds provided in this application, such as... Figure 10 As shown, after determining the temporary optimization plane corresponding to the largest number of points in the optimization plane as the update plane in step S503, the method further includes:

[0207] In step S1001, for the update plane, the points corresponding to the second distance greater than the distance threshold are determined as the external points corresponding to the update plane, so as to obtain the set of external points corresponding to the update plane.

[0208] In step S1002, the set of external points is added to the total set of candidate points to update the total set of candidate points.

[0209] In this embodiment, after obtaining the updated plane, points corresponding to the second distance greater than the distance threshold can be identified as the outer points corresponding to the updated plane, so as to obtain the set of outer points corresponding to the updated plane. The set of outer points is then added to the candidate point set to update the candidate point set, so that the candidate plane can be generated based on the new candidate point set in the future.

[0210] Specifically, after obtaining the update plane and determining its plane parameters and the interior points belonging to the update plane, the points corresponding to the second distance greater than the distance threshold can be identified as the exterior points corresponding to the update plane. This allows for the aggregation of the exterior point set of the update plane. Then, this exterior point set can be updated to the candidate point set so that candidate planes can be generated based on the candidate point set in the future.

[0211] Optionally, in the above Figure 3 Based on the corresponding embodiments, in another optional embodiment of the planar detection method based on 3D point clouds provided in this application, such as... Figure 11 As shown, step S303, based on the first distance, determines the set of interior points of the temporary plane corresponding to the current iteration, including:

[0212] In step S1101, the point corresponding to the first distance that is less than or equal to the distance threshold is determined as the interior point of the temporary plane corresponding to the current iteration;

[0213] In step S1102, the interior points of the temporary plane corresponding to the current iteration are summarized to obtain the set of interior points of the temporary plane corresponding to the current iteration.

[0214] Specifically, after obtaining the first distance between each point and the temporary plane corresponding to the current iteration, the first distance can be compared with a distance threshold. Then, points with a first distance less than or equal to the distance threshold are considered to be points of the temporary plane corresponding to the current iteration, and thus determined as interior points of the temporary plane corresponding to the current iteration. The interior points of the temporary plane corresponding to the current iteration are then summarized to obtain the set of interior points of the temporary plane corresponding to the current iteration.

[0215] Optionally, in the above Figure 2 Based on the corresponding embodiments, in another optional embodiment of the planar detection method based on 3D point clouds provided in this application, such as... Figure 12 As shown, step S105 generates a candidate plane based on the points in the updated candidate point set, including:

[0216] In step S1201, three points are randomly selected from the updated set of candidate points in each iteration to generate a temporary candidate plane;

[0217] In step S1202, the fourth distance between each candidate point in the updated candidate point set and the temporary candidate plane is calculated.

[0218] In step S1203, the set of interior points of the temporary candidate plane is determined based on the fourth distance;

[0219] In step S1204, based on the set of interior points of the temporary candidate plane corresponding to each iteration, the temporary candidate plane with the largest number of interior points is selected;

[0220] In step S1205, if the total number of interior points of the temporary candidate plane with the largest number of interior points is greater than the interior point threshold, then the temporary candidate plane with the largest number of interior points is determined as a valid candidate plane.

[0221] In step S1206, the plane parameters of the effective candidate planes are iteratively optimized to obtain the candidate planes.

[0222] In this embodiment, after obtaining the updated candidate point set, three points are randomly selected from the updated candidate point set in each iteration to generate a temporary candidate plane. Then, the fourth distance between each candidate point in the updated candidate point set and the temporary candidate plane is calculated. Based on the fourth distance, the set of interior points of the temporary candidate plane is determined. Then, based on the set of interior points of the temporary candidate plane corresponding to each iteration, the temporary candidate plane with the largest number of interior points is selected. Then, if the total number of interior points of the temporary candidate plane with the largest number of interior points is greater than the interior point threshold, the temporary candidate plane with the largest number of interior points is determined as a valid candidate plane. The plane parameters of the valid candidate plane are iteratively optimized to obtain a candidate plane.

[0223] Specifically, such as Figure 15 As shown, after obtaining the merged update planes and the remaining unmerged update planes, we can first determine whether the total number of merged update planes and the remaining unmerged update planes has reached the maximum number of planes. If it has, the candidate planes are empty; if not, we can determine whether the total number of points in the updated candidate point set is greater than the second point number threshold. If it is less, the candidate planes are empty; if it is greater, we can proceed based on... Figure 15The initial plane detection module, as shown, generates a new plane. Specifically, it randomly selects three points from the updated candidate point set for each iteration, according to a preset number of iterations. Since three non-collinear points determine a plane in space, after selecting three random points, it can first determine whether the three selected points are collinear. If they are collinear, the iteration stops; otherwise, the spatial coordinates of the three selected points are substituted into formula (1) above for calculation to obtain the temporary candidate plane corresponding to the three selected points. The number of iterations is set according to actual application requirements and is not specifically limited here.

[0224] Furthermore, according to the above formula (2), the distance between each candidate point in the updated candidate point set and the temporary candidate plane can be calculated, i.e., the fourth distance. Based on the fourth distance, the points that can belong to the temporary candidate plane can be determined, i.e., the interior points of the temporary candidate plane, so as to obtain the set of interior points of the temporary candidate plane.

[0225] Furthermore, after the iteration is completed, the plane with the most interior points in the set of interior points of the temporary candidate planes during the iteration is recorded, that is, the temporary candidate plane with the largest number of interior points. If the number of interior points of the temporary candidate plane with the largest number of interior points is greater than the interior point threshold, then the temporary candidate plane with the largest number of interior points is considered to be valid, that is, the temporary candidate plane with the largest number of interior points is determined as a valid candidate plane.

[0226] Furthermore, the effective candidate plane can be optimized using least squares to update the plane parameters, and an optimization algorithm can be used to iteratively optimize the parameters to obtain the candidate optimization plane corresponding to the effective candidate plane. Then, the distance between each point in the candidate point set and the candidate optimization plane is calculated, and the points with distances less than or equal to the distance threshold are determined as interior points corresponding to the candidate optimization plane, that is, the number of interior points corresponding to the candidate optimization plane can be counted. At the same time, the points with distances greater than the distance threshold are determined as exterior points corresponding to the candidate optimization plane and put back into the candidate point set.

[0227] Furthermore, after obtaining the candidate optimization plane and the corresponding interior points, all interior points in the candidate optimization plane can be projected onto the candidate optimization plane, and the maximum outer contour polygon projected onto the candidate optimization plane can be calculated based on the Convex Hull Algorithm or other contour calculation methods, without specific restrictions here.

[0228] Furthermore, after obtaining the maximum outer contour polygon, the vertex center of the polygon can be obtained by summing and averaging the spatial coordinates of each vertex in the maximum outer contour polygon. At the same time, the points projected onto the candidate optimization plane, i.e., the projection points, can be obtained by summing and averaging the coordinates of the projection points to obtain the projection center.

[0229] Furthermore, based on the vertex center and projection center, abnormal vertices can be filtered out from all vertices of the polygon. Specifically, this can be done by judging whether each vertex in the maximum outer contour polygon meets the above abnormal conditions. If a vertex in the maximum outer contour polygon meets the above abnormal conditions, it can be understood that the vertex is far from the interior center of the candidate optimization plane and is an abnormal vertex, which needs to be removed. After all vertices are judged, the candidate plane to be verified and the set of candidate vertices of the candidate plane to be verified are obtained.

[0230] Furthermore, after obtaining the candidate plane to be verified and the set of candidate vertices of the candidate plane to be verified, the outer contour polygon and its area corresponding to the candidate plane to be verified can be recalculated, as well as the plane information corresponding to the candidate plane to be verified. Then, the area of ​​the outer contour polygon can be calculated and compared with the area threshold. If the comparison result is that the area of ​​the outer contour polygon is greater than the area threshold, the candidate plane to be verified can be considered a valid plane, and the candidate plane to be verified can be determined as a candidate plane. Furthermore, abnormal vertex detection can be performed on the candidate plane again and it can be removed to maintain the detection accuracy of the candidate plane. Conversely, if the comparison result is that the area of ​​the outer contour polygon is less than the area threshold, the candidate plane to be verified can be considered an invalid plane, that is, no candidate plane is generated.

[0231] Optionally, in the above Figure 2 Based on the corresponding embodiments, in another optional embodiment of the planar detection method based on 3D point clouds provided in this application, such as... Figure 13 As shown, step S102 divides the 3D sparse point set to be processed into a candidate point subset and multiple insertion point subsets based on the spatial coordinates corresponding to each point, including:

[0232] In step S1301, if the current number of planes is not zero, the third distance between each point in the three-dimensional sparse point set to be processed and each current plane is calculated respectively.

[0233] In step S1302, based on the third distance, insertion points for each current plane are selected from the set of sparse points to be processed in the three-dimensional plane, resulting in a subset of candidate points and a subset of points to be inserted for each current plane.

[0234] In this embodiment, if the current number of planes is not zero, the third distance between each point in the three-dimensional sparse point set to be processed and each current plane is calculated respectively. Based on the third distance, the insertion point of each current plane is selected from the three-dimensional sparse point set to be processed, so as to divide the three-dimensional sparse point set to be processed into a candidate point subset and a point subset to be inserted corresponding to each current plane.

[0235] Specifically, after obtaining the set of 3D sparse points to be processed, we can first detect or determine whether the current number of planes is zero. If it is zero, it can be understood that this is the first time to perform plane detection and no plane has been detected yet. That is, the points that can be inserted into the plane are empty, i.e. the subset of points to be inserted is empty. Therefore, the set of 3D sparse points to be processed can be used as a subset of candidate points and updated to the total set of candidate points so that new plane detection can be performed based on the points in the total set of candidate points in the future.

[0236] Furthermore, if the current number of planes is not zero, such as Figure 15 As shown, to determine whether there are points in the 3D sparse point set to be processed that can be inserted into the existing plane, it can be done by calculating the third distance between each point in the 3D sparse point set to be processed and each current plane. If the third distance is less than or equal to the distance threshold, then the point can be considered to belong to the current plane, that is, there are points that can be inserted into the current plane. Therefore, these points can be separated from the 3D sparse point set to be processed and used as a subset of points to be inserted for each current plane. If the third distance is greater than the distance threshold, then the point can be considered not to belong to the current plane. Therefore, these points can be separated from the 3D sparse point set to be processed and used as a subset of candidate points.

[0237] Optionally, in the above Figure 13 Based on the corresponding embodiments, in another optional embodiment of the planar detection method based on 3D point clouds provided in this application, such as... Figure 14 As shown, step S1302, based on the third distance, filters the insertion points for each current plane from the set of sparse points to be processed in the three-dimensional space, obtaining a subset of candidate points and a subset of points to be inserted for each current plane, including:

[0238] In step S1401, the third distance between each point and each current plane is compared with a distance threshold to obtain a comparison result;

[0239] In step S1402, if the comparison result is greater than the distance threshold, the point corresponding to the third distance is determined as a candidate point, and all candidate points are summarized to obtain a subset of candidate points;

[0240] In step S1403, if the comparison result is less than or equal to the distance threshold, the point corresponding to the third distance is determined as the insertion point of the current plane, so as to obtain the subset of insertion points corresponding to each current plane.

[0241] In this embodiment, if the current number of planes is not zero, such as Figure 15 As shown, to determine whether there are points in the 3D sparse point set to be processed that can be inserted into the existing plane, it can be done by calculating the third distance between each point in the 3D sparse point set to be processed and each current plane. Then, the third distance between each point and each current plane is compared with a distance threshold. If the comparison result is that the third distance is greater than the distance threshold, it can be considered that the point does not belong to the current plane. Therefore, these points can be separated from the 3D sparse point set to be processed to obtain a subset of candidate points. Conversely, if the comparison result is that the third distance is less than or equal to the distance threshold, it can be considered that the point belongs to the current plane, that is, there are points that can be inserted into the current plane. Therefore, these points can be separated from the 3D sparse point set to be processed to obtain a subset of points to be inserted for each current plane.

[0242] The planar detection device based on 3D point clouds in this application is described in detail below. Please refer to [link / reference]. Figure 17 , Figure 17 This is a schematic diagram of one embodiment of the planar detection device based on 3D point clouds in this application. The planar detection device 20 based on 3D point clouds includes:

[0243] The acquisition unit 201 is used to acquire a set of three-dimensional sparse points to be processed, wherein the set of three-dimensional sparse points to be processed includes multiple points and the spatial coordinates corresponding to each point.

[0244] The processing unit 202 is used to divide the three-dimensional sparse point set to be processed into a candidate point subset and multiple insertion point subsets based on the spatial coordinates corresponding to each point. Each insertion point subset is used to describe the set of points that can be inserted into the plane, and the candidate point subset is used to describe the set of points that cannot be inserted into the plane.

[0245] The processing unit 202 is also used to, for each subset of points to be inserted, if the number of points in the subset of points to be inserted is greater than the first threshold, generate a temporary plane by random sampling, and iteratively update the plane parameters of the current plane that has inserted the subset of points to be inserted to obtain the updated plane.

[0246] Processing unit 202 is also used to merge any two updating planes that meet the plane merging conditions to obtain a merged updating plane;

[0247] The processing unit 202 is also used to update the candidate point subset to the candidate point set. If the number of points in the updated candidate point set is greater than the second point number threshold, and the total number of currently merged and updated planes is less than the plane threshold, then a candidate plane is generated based on the points in the updated candidate point set.

[0248] The determination unit 203 is used to summarize all candidate planes, merged update planes, and remaining unmerged update planes to obtain the target plane set corresponding to the three-dimensional sparse point set to be processed.

[0249] Optionally, in the above Figure 17 Based on the corresponding embodiments, in another embodiment of the planar detection device based on three-dimensional point clouds provided in this application, the processing unit 202 can specifically be used for:

[0250] For the current plane that has been inserted with a subset of points to be inserted, three points are randomly selected in each iteration to generate a temporary plane corresponding to the current iteration;

[0251] Calculate the first distance between each point in the current plane and the temporary plane corresponding to the current iteration;

[0252] Based on the first distance, determine the set of interior points of the temporary plane corresponding to the current iteration;

[0253] Based on the set of interior points of the temporary plane corresponding to the current iteration, the plane parameters of the current plane are iteratively updated to obtain the updated plane.

[0254] Optionally, in the above Figure 17 Based on the corresponding embodiments, in another embodiment of the planar detection device based on three-dimensional point clouds provided in this application, the processing unit 202 can specifically be used for:

[0255] Based on the set of interior points of the temporary plane corresponding to the current iteration, plane fitting is performed to obtain the plane fitting parameters corresponding to the temporary plane;

[0256] The plane fitting parameters are iteratively optimized to obtain the temporary optimized plane corresponding to the temporary plane.

[0257] Based on the number of points in the optimization plane corresponding to each iteration, the update plane is selected from all the temporary optimization planes.

[0258] Optionally, in the above Figure 17 Based on the corresponding embodiments, in another embodiment of the planar detection device based on three-dimensional point clouds provided in this application, the processing unit 202 can specifically be used for:

[0259] Calculate the second distance between each point in the current plane and the temporary optimization plane;

[0260] Points corresponding to the second distance that is less than or equal to the distance threshold are identified as points within the optimization plane corresponding to the temporary optimization plane, so as to count the number of points within the optimization plane corresponding to the temporary optimization plane.

[0261] The temporary optimization plane corresponding to the largest number of points in the optimization plane is determined as the update plane.

[0262] Optionally, in the above Figure 17 Based on the corresponding embodiments, in another embodiment of the planar detection device based on three-dimensional point clouds provided in this application, the processing unit 202 can specifically be used for:

[0263] Get all points in the temporary optimization plane and project them onto the update plane to form the largest outer contour polygon.

[0264] Calculate the vertex centers of the polygon and the projection centers of all points in the optimization plane onto the update plane;

[0265] Based on the vertex center and projection center, abnormal vertices are selected from all vertices of the polygon and removed to obtain the update plane and the optimized vertex set of the update plane.

[0266] Based on the optimized vertex set, the optimized outer contour polygon and the area of ​​the optimized outer contour polygon corresponding to the updated plane are recalculated, and the plane information corresponding to the updated plane is calculated, wherein the plane information includes at least the plane normal vector.

[0267] Optionally, in the above Figure 17 Based on the corresponding embodiments, in another embodiment of the planar detection device based on three-dimensional point clouds provided in this application, the processing unit 202 can specifically be used for:

[0268] Calculate the angle between the plane normal vectors corresponding to any two update planes;

[0269] Based on the optimized outer contour polygon and the area of ​​the optimized outer contour polygon corresponding to each updated plane, calculate the overlapping area and the area intersection-union ratio of the outer contour polygons after the mutual projection of any two updated planes.

[0270] When the plane merging conditions are that the included angle is less than the included angle threshold, the overlapping area is greater than the area threshold, and the area intersection-to-union ratio is greater than the intersection-to-union ratio threshold, any two updated planes that meet the plane merging conditions are merged to obtain the merged updated plane.

[0271] Optionally, in the above Figure 17 Based on the corresponding embodiments, in another embodiment of the planar detection device based on 3D point clouds provided in this application,

[0272] The processing unit 202 is also used to calculate the area of ​​the outer contour polygon corresponding to each merged update plane;

[0273] The processing unit 202 is further configured to compare the area of ​​the outer contour polygon corresponding to each merged update plane and the area of ​​the optimized outer contour polygon corresponding to the remaining unmerged update planes with the second area threshold, respectively.

[0274] The processing unit 202 is also used to remove merged update planes with an area smaller than the second area threshold or the remaining unmerged update planes to obtain the optimal set of update planes.

[0275] The determining unit 203 is also used to summarize all candidate planes and the optimal update plane set to obtain the target plane set corresponding to the three-dimensional sparse point set to be processed.

[0276] Optionally, in the above Figure 17 Based on the corresponding embodiments, in another embodiment of the planar detection device based on three-dimensional point clouds provided in this application, the processing unit 202 can specifically be used for:

[0277] Merge all points on any two update planes that meet the plane merging conditions to obtain the merged plane point set;

[0278] Based on the merged plane point set, plane fitting is performed to obtain the merged plane fitting parameters;

[0279] The fitting parameters of the merged plane are iteratively optimized to obtain the merged updated plane.

[0280] Optionally, in the above Figure 17 Based on the corresponding embodiments, in another embodiment of the planar detection device based on 3D point clouds provided in this application,

[0281] The determining unit 203 is also used to determine, for the update plane, the points corresponding to the second distance greater than the distance threshold as the external points corresponding to the update plane, so as to summarize and obtain the set of external points corresponding to the update plane;

[0282] The processing unit 202 is also used to add the set of outliers to the total set of candidate points in order to update the total set of candidate points.

[0283] Optionally, in the above Figure 17 Based on the corresponding embodiments, in another embodiment of the planar detection device based on three-dimensional point clouds provided in this application, the processing unit 202 can specifically be used for:

[0284] The point corresponding to the first distance that is less than or equal to the distance threshold is determined as the interior point of the temporary plane corresponding to the current iteration;

[0285] Summarize the interior points of the temporary plane corresponding to the current iteration to obtain the set of interior points of the temporary plane corresponding to the current iteration.

[0286] Optionally, in the above Figure 17 Based on the corresponding embodiments, in another embodiment of the planar detection device based on three-dimensional point clouds provided in this application, the processing unit 202 can specifically be used for:

[0287] In each iteration, three points are randomly selected from the updated candidate point set to generate a temporary candidate plane.

[0288] Calculate the fourth distance between each candidate point in the updated candidate point set and the temporary candidate plane;

[0289] Based on the fourth distance, determine the set of interior points of the temporary candidate plane;

[0290] Based on the set of interior points of the temporary candidate plane corresponding to each iteration, select the temporary candidate plane with the largest number of interior points;

[0291] If the total number of interior points of the temporary candidate plane with the largest number of interior points is greater than the interior point threshold, then the temporary candidate plane with the largest number of interior points is determined as a valid candidate plane.

[0292] The effective candidate planes are iteratively optimized by optimizing the plane parameters to obtain the candidate planes.

[0293] Optionally, in the above Figure 17 Based on the corresponding embodiments, in another embodiment of the planar detection device based on three-dimensional point clouds provided in this application, the processing unit 202 can specifically be used for:

[0294] If the current number of planes is not zero, then calculate the third distance between each point in the 3D sparse point set to be processed and each current plane;

[0295] Based on the third distance, the insertion points for each current plane are selected from the set of sparse points to be processed in the three-dimensional plane, resulting in a subset of candidate points and a subset of points to be inserted for each current plane.

[0296] Optionally, in the above Figure 17 Based on the corresponding embodiments, in another embodiment of the planar detection device based on three-dimensional point clouds provided in this application, the processing unit 202 can specifically be used for:

[0297] The third distance between each point and each current plane is compared with a distance threshold to obtain the comparison result;

[0298] If the comparison result is greater than the distance threshold, the point corresponding to the third distance is determined as a candidate point, and all candidate points are summarized to obtain a subset of candidate points;

[0299] If the comparison result is less than or equal to the distance threshold, the point corresponding to the third distance is determined as the insertion point of the current plane, so as to obtain the subset of points to be inserted corresponding to each current plane.

[0300] This application also provides a schematic diagram of another computer device, such as... Figure 18 As shown, Figure 18 This is a schematic diagram of a computer device structure provided in an embodiment of this application. The computer device 300 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 310 (e.g., one or more processors) and a memory 320, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 331 or data 332. The memory 320 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the computer device 300. Furthermore, the CPU 310 may be configured to communicate with the storage media 330 and execute the series of instruction operations in the storage media 330 on the computer device 300.

[0301] Computer device 300 may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input / output interfaces 360, and / or one or more operating systems 333, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0302] The aforementioned computer device 300 is also used to perform, for example Figures 2 to 14 The steps in the corresponding embodiments.

[0303] Another aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements... Figures 2 to 14 The steps in the method described in the illustrated embodiment.

[0304] Another aspect of this application provides a computer program product comprising a computer program, which, when executed by a processor, implements as follows: Figures 2 to 14 The steps in the method described in the illustrated embodiment.

[0305] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0306] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0307] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0308] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0309] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A planar detection method based on 3D point clouds, characterized in that, include: Obtain a set of three-dimensional sparse points to be processed, wherein the set of three-dimensional sparse points to be processed includes multiple points and the spatial coordinates corresponding to each point; Based on the spatial coordinates corresponding to each point, the three-dimensional sparse point set to be processed is divided into a candidate point subset and multiple insertion point subsets. Each insertion point subset is used to describe a set of points that can be inserted into the plane, and the candidate point subset is used to describe a set of points that cannot be inserted into the plane. For each subset of points to be inserted, if the number of points in the subset is greater than the first threshold, a temporary plane is generated by random sampling, and the plane parameters of the current plane into which the subset of points to be inserted are iteratively updated to obtain the updated plane. Merge any two updated planes that meet the plane merging conditions to obtain a merged updated plane; The subset of candidate points is updated to the total set of candidate points. If the number of points in the updated total set of candidate points is greater than the second threshold, and the total number of merged and updated planes is less than the plane threshold, then a candidate plane is generated based on the points in the updated total set of candidate points. All the candidate planes, the merged and updated planes, and the remaining unmerged updated planes are summarized to obtain the target plane set corresponding to the three-dimensional sparse point set to be processed.

2. The method according to claim 1, characterized in that, If the number of points in the subset of points to be inserted is greater than the first threshold, a temporary plane is generated by random sampling, and the plane parameters of the current plane with the subset of points to be inserted are iteratively updated to obtain an updated plane, including: For the current plane into which the subset of points to be inserted has been inserted, three points are randomly selected in each iteration to generate a temporary plane corresponding to the current iteration; Calculate the first distance between each point in the current plane and the temporary plane corresponding to the current iteration; Based on the first distance, determine the set of interior points of the temporary plane corresponding to the current iteration; Based on the set of interior points of the temporary plane corresponding to the current iteration, the plane parameters of the current plane are iteratively updated to obtain the updated plane.

3. The method according to claim 2, characterized in that, The step of iteratively updating the plane parameters of the current plane based on the set of interior points of the temporary plane corresponding to the current iteration to obtain the updated plane includes: Based on the set of interior points of the temporary plane corresponding to the current iteration, plane fitting is performed to obtain the plane fitting parameters corresponding to the temporary plane; The plane fitting parameters are iteratively optimized to obtain the temporary optimized plane corresponding to the temporary plane; The updated plane is selected from all temporary optimized planes based on the number of points in the optimized plane corresponding to each iteration.

4. The method according to claim 3, characterized in that, The step of selecting the updated plane from all temporary optimization planes based on the number of points within the temporary optimization plane corresponding to each iteration includes: Calculate the second distance between each point in the current plane and the temporary optimized plane; Points corresponding to the second distance that is less than or equal to the distance threshold are identified as points within the optimization plane corresponding to the temporary optimization plane, so as to count the number of points within the optimization plane corresponding to the temporary optimization plane; The temporary optimization plane corresponding to the largest number of points in the optimization plane is determined as the updated plane.

5. The method according to claim 4, characterized in that, The step of determining the temporary optimization plane corresponding to the number of points in the optimization plane with the largest numerical value as the updated plane includes: Obtain all points within the optimization plane in the temporary optimization plane and project them onto the maximum outer contour polygon on the temporary optimization plane; Calculate the vertex centers of the polygon and the projection centers of all points in the optimization plane onto the temporary optimization plane; Based on the vertex center and the projection center, abnormal vertices are selected from all vertices of the polygon and removed to obtain the update plane and the optimized vertex set of the update plane. Based on the optimized vertex set, the optimized outer contour polygon corresponding to the updated plane and the area of ​​the optimized outer contour polygon are recalculated, and the plane information corresponding to the updated plane is calculated, wherein the plane information includes at least the plane normal vector.

6. The method according to claim 5, characterized in that, The step of merging any two updated planes that meet the plane merging conditions to obtain a merged updated plane includes: Calculate the angle between the plane normal vectors corresponding to any two of the updated planes; Based on the optimized outer contour polygon corresponding to each of the updated planes and the area of ​​the optimized outer contour polygon, calculate the overlapping area and the area intersection-union ratio of the outer contour polygons after projecting any two of the updated planes onto each other. When the plane merging conditions are that the included angle is less than the included angle threshold, the overlapping area is greater than the area threshold, and the area intersection-to-union ratio is greater than the intersection-to-union ratio threshold, any two updated planes that meet the plane merging conditions are merged to obtain the merged updated plane.

7. The method according to claim 6, characterized in that, When the plane merging conditions are that the included angle is less than an included angle threshold, the overlapping area is greater than an area threshold, and the area intersection-union ratio is greater than an intersection-union ratio threshold, any two updated planes that meet the plane merging conditions are merged to obtain the merged updated plane. The method further includes: Calculate the area of ​​the outer contour polygon corresponding to each of the merged update planes; The area of ​​the outer contour polygon corresponding to each of the merged update planes, and the area of ​​the optimized outer contour polygon corresponding to the remaining unmerged update planes, are compared with the second area threshold, respectively. The merged update planes with an area smaller than the second area threshold or the remaining unmerged update planes are removed to obtain the optimal set of update planes; The step of summing up all the candidate planes, the merged and updated planes, and the remaining unmerged updated planes to obtain the target plane set corresponding to the three-dimensional sparse point set to be processed includes: The set of all candidate planes and the set of optimal update planes are summarized to obtain the set of target planes corresponding to the set of sparse points to be processed in the three-dimensional space.

8. The method according to claim 1, characterized in that, The step of merging any two updated planes that meet the plane merging conditions to obtain a merged updated plane includes: The process involves merging all points on any two update planes that meet the plane merging conditions to obtain a merged plane point set. Based on the merged plane point set, plane fitting processing is performed to obtain the merged plane fitting parameters; The merging plane fitting parameters are iteratively optimized to obtain the merging update plane.

9. The method according to claim 4, characterized in that, After determining the temporary optimization plane corresponding to the number of points in the optimization plane with the largest numerical value as the updated plane, the method further includes: For the updated plane, points corresponding to the second distance greater than the distance threshold are determined as the outer points corresponding to the updated plane, and the sets of outer points corresponding to the updated plane are obtained by summarizing them. The set of outliers is added to the total set of candidate points to update the total set of candidate points.

10. The method according to claim 2, characterized in that, The step of determining the set of interior points of the temporary plane corresponding to the current iteration based on the first distance includes: The point corresponding to the first distance, which is less than or equal to the distance threshold, is determined as the interior point of the temporary plane corresponding to the current iteration; The interior points of the temporary plane corresponding to the current iteration are summarized to obtain the set of interior points of the temporary plane corresponding to the current iteration.

11. The method according to claim 1, characterized in that, The step of generating a candidate plane based on the updated candidate point set includes: In each iteration, three points are randomly selected from the updated set of candidate points to generate a temporary candidate plane. Calculate the fourth distance between each candidate point in the updated candidate point set and the temporary candidate plane; Based on the fourth distance, determine the set of interior points of the temporary candidate plane; Based on the set of interior points of the temporary candidate plane corresponding to each iteration, the temporary candidate plane with the largest number of interior points is selected. If the total number of interior points of the temporary candidate plane with the largest number of interior points is greater than the interior point threshold, then the temporary candidate plane with the largest number of interior points is determined as a valid candidate plane. The candidate planes are obtained by iteratively optimizing their plane parameters.

12. The method according to claim 1, characterized in that, Based on the spatial coordinates corresponding to each point, the set of 3D sparse points to be processed is divided into a candidate point subset and multiple point subsets to be inserted, including: If the current number of planes is not zero, then calculate the third distance between each point in the three-dimensional sparse point set to be processed and each of the current planes; Based on the third distance, insertion points for each current plane are selected from the set of sparse points to be processed in the three-dimensional plane, resulting in the candidate point subset and the insertion point subset corresponding to each current plane.

13. The method according to claim 12, characterized in that, The step of filtering insertion points for each current plane from the set of sparse points to be processed based on the third distance, to obtain the candidate point subset and the insertion point subset corresponding to each current plane, includes: The third distance between each point and each of the current planes is compared with a distance threshold to obtain the comparison result; If the comparison result is greater than the distance threshold, then the point corresponding to the third distance is determined as the candidate point, and all the candidate points are summarized to obtain the candidate point subset; If the comparison result is less than or equal to the distance threshold, then the point corresponding to the third distance is determined as the insertion point of the current plane, so as to obtain the subset of insertion points corresponding to each current plane.

14. A planar detection device based on three-dimensional point clouds, characterized in that, include: The acquisition unit is used to acquire a set of three-dimensional sparse points to be processed, wherein the set of three-dimensional sparse points to be processed includes multiple points and the spatial coordinates corresponding to each point. The processing unit is configured to divide the three-dimensional sparse point set to be processed into a candidate point subset and multiple insertion point subsets based on the spatial coordinates corresponding to each point. Each insertion point subset is used to describe a set of points that can be inserted into a plane, and the candidate point subset is used to describe a set of points that cannot be inserted into a plane. The processing unit is further configured to, for each subset of points to be inserted, if the number of points in the subset of points to be inserted is greater than the first threshold number of points, generate a temporary plane by random sampling, and iteratively update the plane parameters of the current plane into which the subset of points to be inserted has been inserted, to obtain an updated plane. The processing unit is also used to merge any two updating planes that meet the plane merging conditions to obtain a merged updating plane; The processing unit is further configured to update the candidate point subset to the candidate point set. If the number of points in the updated candidate point set is greater than the second point number threshold, and the total number of the current merged and updated planes is less than the plane threshold, then a candidate plane is generated based on the points in the updated candidate point set. The determining unit is used to summarize all the candidate planes, the merged and updated planes, and the remaining unmerged updated planes to obtain the target plane set corresponding to the three-dimensional sparse point set to be processed.

15. The apparatus according to claim 14, characterized in that, The processing unit is specifically used for: For the current plane into which the subset of points to be inserted has been inserted, three points are randomly selected in each iteration to generate a temporary plane corresponding to the current iteration; Calculate the first distance between each point in the current plane and the temporary plane corresponding to the current iteration; Based on the first distance, determine the set of interior points of the temporary plane corresponding to the current iteration; Based on the set of interior points of the temporary plane corresponding to the current iteration, the plane parameters of the current plane are iteratively updated to obtain the updated plane.

16. The apparatus according to claim 15, characterized in that, The processing unit is specifically used for: Based on the set of interior points of the temporary plane corresponding to the current iteration, plane fitting is performed to obtain the plane fitting parameters corresponding to the temporary plane; The plane fitting parameters are iteratively optimized to obtain the temporary optimized plane corresponding to the temporary plane; The updated plane is selected from all temporary optimized planes based on the number of points in the optimized plane corresponding to each iteration.

17. The apparatus according to claim 16, characterized in that, The processing unit is specifically used for: Calculate the second distance between each point in the current plane and the temporary optimized plane; Points corresponding to the second distance that is less than or equal to the distance threshold are identified as points within the optimization plane corresponding to the temporary optimization plane, so as to count the number of points within the optimization plane corresponding to the temporary optimization plane; The temporary optimization plane corresponding to the largest number of points in the optimization plane is determined as the updated plane.

18. The apparatus according to claim 17, characterized in that, The processing unit is specifically used for: Obtain all points within the optimization plane in the temporary optimization plane and project them onto the maximum outer contour polygon on the temporary optimization plane; Calculate the vertex centers of the polygon and the projection centers of all points in the optimization plane onto the temporary optimization plane; Based on the vertex center and the projection center, abnormal vertices are selected from all vertices of the polygon and removed to obtain the update plane and the optimized vertex set of the update plane. Based on the optimized vertex set, the optimized outer contour polygon corresponding to the updated plane and the area of ​​the optimized outer contour polygon are recalculated, and the plane information corresponding to the updated plane is calculated, wherein the plane information includes at least the plane normal vector.

19. The apparatus according to claim 18, characterized in that, The processing unit is specifically used for: Calculate the angle between the plane normal vectors corresponding to any two of the updated planes; Based on the optimized outer contour polygon corresponding to each of the updated planes and the area of ​​the optimized outer contour polygon, calculate the overlapping area and the area intersection-union ratio of the outer contour polygons after projecting any two of the updated planes onto each other. When the plane merging conditions are that the included angle is less than the included angle threshold, the overlapping area is greater than the area threshold, and the area intersection-to-union ratio is greater than the intersection-to-union ratio threshold, any two updated planes that meet the plane merging conditions are merged to obtain the merged updated plane.

20. The apparatus according to claim 19, characterized in that, The processing unit is also used to calculate the area of ​​the outer contour polygon corresponding to each of the merged update planes; The processing unit is further configured to compare the area of ​​the outer contour polygon corresponding to each of the merged update planes, and the area of ​​the optimized outer contour polygon corresponding to the remaining unmerged update planes, with a second area threshold, respectively. The processing unit is further configured to remove merged update planes with an area smaller than the second area threshold or the remaining unmerged update planes to obtain the optimal set of update planes. The determining unit is further configured to summarize all the candidate planes and the optimal update plane set to obtain the target plane set corresponding to the three-dimensional sparse point set to be processed.

21. The apparatus according to claim 14, characterized in that, The processing unit is specifically used for: Merge all points on any two update planes that meet the plane merging conditions to obtain the merged plane point set; Based on the merged plane point set, plane fitting processing is performed to obtain the merged plane fitting parameters; The merging plane fitting parameters are iteratively optimized to obtain the merging update plane.

22. The apparatus according to claim 17, characterized in that, The determining unit is further configured to, for the updated plane, determine the points corresponding to the second distance greater than the distance threshold as the external points corresponding to the updated plane, so as to summarize and obtain the set of external points corresponding to the updated plane; The processing unit is further configured to add the set of external points to the total set of candidate points in order to update the total set of candidate points.

23. The apparatus according to claim 15, characterized in that, The processing unit is specifically used for: The point corresponding to the first distance, which is less than or equal to the distance threshold, is determined as the interior point of the temporary plane corresponding to the current iteration; The interior points of the temporary plane corresponding to the current iteration are summarized to obtain the set of interior points of the temporary plane corresponding to the current iteration.

24. The apparatus according to claim 14, characterized in that, The processing unit is specifically used for: In each iteration, three points are randomly selected from the updated set of candidate points to generate a temporary candidate plane. Calculate the fourth distance between each candidate point in the updated candidate point set and the temporary candidate plane; Based on the fourth distance, determine the set of interior points of the temporary candidate plane; Based on the set of interior points of the temporary candidate plane corresponding to each iteration, the temporary candidate plane with the largest number of interior points is selected. If the total number of interior points of the temporary candidate plane with the largest number of interior points is greater than the interior point threshold, then the temporary candidate plane with the largest number of interior points is determined as a valid candidate plane. The candidate planes are obtained by iteratively optimizing their plane parameters.

25. The apparatus according to claim 14, characterized in that, The processing unit is specifically used for: If the current number of planes is not zero, then calculate the third distance between each point in the three-dimensional sparse point set to be processed and each of the current planes; Based on the third distance, insertion points for each current plane are selected from the set of sparse points to be processed in the three-dimensional plane, resulting in the candidate point subset and the insertion point subset corresponding to each current plane.

26. The apparatus according to claim 25, characterized in that, The processing unit is specifically used for: The third distance between each point and each of the current planes is compared with a distance threshold to obtain the comparison result; If the comparison result is greater than the distance threshold, then the point corresponding to the third distance is determined as the candidate point, and all the candidate points are summarized to obtain the candidate point subset; If the comparison result is less than or equal to the distance threshold, then the point corresponding to the third distance is determined as the insertion point of the current plane, so as to obtain the subset of insertion points corresponding to each current plane.

27. A computer device comprising a memory, a processor, and a bus system, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 13; The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

28. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 13.

29. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 13.

Images