Sparse Point-Based Plane Extraction Method, System, and Electronic Device

Abstract

A plane extraction method based on sparse points, its system and electronic device. The plane extraction method based on sparse points includes the steps of: performing spatial point extraction processing on the acquired image information to obtain a plurality of spatial points in the world coordinate system; constructing one or more candidate planes based on four of the spatial points randomly selected from all the spatial points; calculating the plane covariance for each of the candidate planes respectively to use the candidate plane that meets the preset plane screening condition as the plane to be confirmed; and outputting the plane to be confirmed that meets the preset output condition as the finally extracted plane by calculating the singular value of the plane matrix for all the planes to be confirmed.

Description

Technical Field

[0001] The present invention relates to the technical field of SLAM, and in particular to a method and system for extracting planes based on sparse points and an electronic device. Background Art

[0002] At present, augmented reality (AR) applications based on mobile devices have become a reality and are highly favored by major mobile device manufacturers and users. Since it is necessary to determine the anchor points of virtual objects based on planes when fusing virtual objects into the real environment and for multi-person interaction, and then render virtual objects at the determined anchor points, plane detection technology is extremely important.

[0003] There are many existing plane detection methods: The first plane detection method is a deep learning-based method. First, the neural network information and plane segmentation information of the image are obtained, and then the above information is input into the network model to output the plane detection result; the second plane detection method is to first process multiple images based on a slam system to obtain a dense point cloud, and then perform plane detection through the point cloud data to obtain the global plane.

[0004] However, the problem with the first method above is that different plane regions usually cannot be correctly segmented through the network model, especially in a building scene with multiple planes. It is even more impossible to correctly segment different plane regions; the second method is that since the dense point cloud is obtained by extracting feature points from multiple images, and the dense point cloud contains a large number of feature points, this method has a high computational complexity, high hardware requirements, and high costs, which restricts its scope of use. However, the computing power of mobile devices is limited. Using an algorithm with a higher complexity for plane detection is slow and the user experience is poor. Summary of the Invention

[0005] One advantage of the present invention is to provide a method and system for extracting planes based on sparse points and an electronic device, which can not only quickly detect planes but also remove the detection interference of false planes.

[0006] Another advantage of the present invention is to provide a method and system for extracting planes based on sparse points and an electronic device. In one embodiment of the present invention, the computational complexity of the method for extracting planes based on sparse points is relatively low, with low hardware requirements and low costs, which particularly matches the computing power of mobile devices and improves the user's comfort experience.

[0007] Another advantage of the present invention is to provide a plane extraction method based on sparse points, its system and electronic device. Wherein, in an embodiment of the present invention, the plane extraction method based on sparse points can determine a plane region based on the singular values of a plane matrix and a plane covariance matrix, which helps to improve the accuracy of plane extraction.

[0008] Another advantage of the present invention is to provide a plane extraction method based on sparse points, its system and electronic device. Wherein, in an embodiment of the present invention, the plane extraction method based on sparse points can screen inliers of a plane based on the absolute median difference, which helps to improve the accuracy of inliers in the plane.

[0009] Another advantage of the present invention is to provide a plane extraction method based on sparse points, its system and electronic device. Wherein, in an embodiment of the present invention, the plane extraction method based on sparse points can perform a small-scale expansion outward from the center point of the plane region, which helps to improve the system robustness and reduce the randomness of Random Sample Consensus (RANSAC).

[0010] Another advantage of the present invention is to provide a plane extraction method based on sparse points, its system and electronic device. In order to achieve the above advantages, in the present invention, there is no need to adopt a complex structure and a large amount of computation, and the requirements for software and hardware are low. Therefore, the present invention successfully and effectively provides a solution, not only providing a plane extraction method based on sparse points, its system and electronic device, but also increasing the practicability and reliability of the plane extraction method based on sparse points, its system and electronic device.

[0011] In order to achieve at least one of the above advantages or other advantages and purposes, the present invention provides a plane extraction method based on sparse points, including the steps of:

[0012] Performing spatial point extraction processing on the acquired image information to obtain a plurality of spatial points in the world coordinate system;

[0013] Based on four of the spatial points randomly selected from all the spatial points, constructing one or more candidate planes;

[0014] Performing plane covariance calculation on all the candidate planes respectively to use the candidate planes that meet the preset plane screening conditions as planes to be confirmed; and

[0015] By performing singular value calculation of the plane matrix on all the planes to be confirmed, outputting the planes to be confirmed that meet the preset output conditions as the finally extracted planes.

[0016] According to an embodiment of the present invention, the acquired image information is a binocular image.

[0017] According to an embodiment of the present invention, the step of performing spatial point extraction processing on the acquired image information to obtain a plurality of spatial points in the world coordinate system includes the steps of:

[0018] Extract features from the binocular images to obtain left-eye feature points and right-eye feature points;

[0019] Perform feature point matching on the left-eye feature points and right-eye feature points by the left and right optical flow tracking method to obtain left and right eye matching point pairs; and

[0020] Perform triangulation calculation on the left and right eye matching point pairs to obtain the coordinate values of the plurality of spatial points in the world coordinate system.

[0021] According to an embodiment of the present invention, the step of constructing one or more candidate planes based on four of the spatial points randomly selected from all the spatial points includes the steps of:

[0022] Randomly select four of the spatial points from all the spatial points by the random sample consensus method to construct a 4×4 plane matrix;

[0023] Decompose the 4×4 plane matrix by the singular value decomposition method to calculate the internal parameters of the initial candidate plane;

[0024] By calculating the distance between the remaining spatial points and the initial candidate plane, determine the spatial points with a distance less than the first threshold as the inliers of the initial candidate plane, and reconstruct the candidate plane matrix according to the inliers of the initial candidate plane and the four selected spatial points; and

[0025] Detect whether the ratio of the minimum singular value to the maximum singular value of the candidate plane matrix is less than the second threshold by a detection function. If so, use the plane corresponding to the candidate plane matrix as the candidate plane.

[0026] According to an embodiment of the present invention, the first threshold is a predetermined multiple of the distance corresponding to the median of the distances from all the remaining spatial points to the initial candidate plane arranged in ascending order.

[0027] According to an embodiment of the present invention, the step of detecting whether the ratio of the minimum singular value to the maximum singular value of the candidate plane matrix is less than the second threshold by a detection function. If so, using the plane corresponding to the candidate plane matrix as the candidate plane includes the steps of:

[0028] Judge whether the number of inliers of the initial candidate plane is greater than the first preset inlier number;

[0029] If so, detect whether the ratio of the minimum singular value to the maximum singular value of the candidate plane matrix is less than the second threshold through a detection function;

[0030] If so, determine whether the included angle between the normal vector of the plane corresponding to the candidate plane matrix and the reference normal vector is less than the included angle threshold; and

[0031] If so, confirm the plane corresponding to the candidate plane matrix as the candidate plane.

[0032] According to an embodiment of the present invention, the preset plane screening condition is:

[0033]

[0034] wherein: minEigValue, medEigValue, and maxEigValue are respectively implemented as the minimum singular value, median singular value, and maximum singular value of the covariance matrix of the candidate plane; K 1 and K 2 are both preset parameters.

[0035] According to an embodiment of the present invention, the preset output condition is:

[0036] The ratio of the minimum singular value to the maximum singular value of the plane matrix corresponding to the plane to be confirmed is less than the third threshold, and the included angle between the normal vector of the plane to be confirmed and the reference normal vector is the smallest.

[0037] According to an embodiment of the present invention, the plane extraction method based on sparse points further includes the steps of:

[0038] Perform local expansion processing on the plane to be confirmed to obtain an expanded plane, and use the expanded plane to replace the plane to be confirmed for subsequent processing.

[0039] According to an embodiment of the present invention, the step of performing local expansion processing on the plane to be confirmed to obtain an expanded plane and using the expanded plane to replace the plane to be confirmed for subsequent processing includes the steps of:

[0040] Successively calculate whether the distance between the spatial points from the center of the field of view from near to far and the plane to be confirmed is less than the distance threshold. If so, use the corresponding spatial points as the inliers of the plane to be confirmed;

[0041] Count the inliers of the plane to be confirmed until the number of inliers of the plane to be confirmed reaches the second preset number of inliers; and

[0042] Based on all the inliers in the plane to be confirmed, reconstruct and calculate the plane matrix to obtain the internal parameters of the expanded plane.

[0043] According to an embodiment of the present invention, the distance threshold MDP is implemented as:

[0044] MDP = median(D) + s * (k * median(d i - median(D))), D = {d 1 , d 2 , …, d n}

[0045] where: d i is the distance from the i-th spatial point to the plane to be confirmed; median is the median; k and s are preset parameters.

[0046] According to another aspect of the present invention, the present invention further provides a plane extraction system based on sparse points, including: communicatively connected to each other:

[0047] A spatial point extraction module, configured to perform spatial point extraction processing on the acquired image information to obtain a plurality of spatial points in the world coordinate system;

[0048] A candidate plane construction module, configured to construct one or more candidate planes based on four of the spatial points randomly selected from all the spatial points;

[0049] A covariance calculation module, configured to calculate the plane covariance of all the candidate planes respectively, and use the candidate planes that meet the preset plane screening conditions as the planes to be confirmed; and

[0050] A plane output module, configured to output the planes to be confirmed that meet the preset output conditions as the finally extracted planes by calculating the singular values of the plane matrices of all the planes to be confirmed.

[0051] According to an embodiment of the present invention, the candidate plane component module includes: communicatively connected to each other: a random selection module, configured to randomly select four of the spatial points from all the spatial points by the random sample consensus method to construct a 4 * 4 plane matrix; a singular value decomposition module, configured to decompose the 4 * 4 plane matrix by the singular value decomposition method to calculate the internal parameters of the initial candidate plane; an inlier confirmation module, configured to determine the spatial points with a distance less than the first threshold as the inliers of the initial candidate plane by calculating the distance between the remaining spatial points and the initial candidate plane, and reconstruct the candidate plane matrix according to the inliers of the initial candidate plane and the four selected spatial points; and a detection module, configured to detect whether the ratio of the minimum singular value to the maximum singular value of the candidate plane matrix is less than the second threshold, and if so, use the plane corresponding to the candidate plane matrix as the candidate plane.

[0052] According to an embodiment of the present invention, the sparse point-based plane extraction system further includes a local expansion module, where the local expansion module is used to perform local expansion processing on the plane to be confirmed to obtain an expanded plane, and the expanded plane is used to replace the plane to be confirmed for subsequent processing.

[0053] According to an embodiment of the present invention, the local expansion module includes: a distance calculation module communicatively connected to each other, which is used to sequentially calculate whether the distance between the spatial points from the center of the field of view from near to far and the plane to be confirmed is less than a distance threshold. If so, the corresponding spatial points are used as the inliers of the plane to be confirmed; an inlier statistics module, which is used to count the inliers of the plane to be confirmed until the number of inliers of the plane to be confirmed reaches a second preset number of inliers; and a matrix construction module, which is used to reconstruct and calculate a plane matrix based on all the inliers in the plane to be confirmed to obtain the internal parameters of the expanded plane.

[0054] According to another aspect of the present invention, the present invention further provides an electronic device, including:

[0055] At least one processor for executing instructions; and

[0056] A memory communicatively connected to the at least one processor, where the memory has at least one instruction, and the instruction is executed by the at least one processor so that the at least one processor executes some or all of the steps in the sparse point-based plane extraction method, where the sparse point-based plane extraction method includes the steps of:

[0057] Performing spatial point extraction processing on the acquired image information to obtain a plurality of spatial points in the world coordinate system;

[0058] Based on four of the spatial points randomly selected from all the spatial points, constructing one or more candidate planes;

[0059] Performing plane covariance calculation on all the candidate planes respectively to use the candidate planes that meet the preset plane screening conditions as the planes to be confirmed; and

[0060] By performing singular value calculation of the plane matrix on all the planes to be confirmed, outputting the planes to be confirmed that meet the preset output conditions as the finally extracted planes.

[0061] Through the understanding of the subsequent description and the drawings, the further objectives and advantages of the present invention will be fully reflected.

[0062] These and other objects, features, and advantages of the present invention will be fully embodied in the following detailed description, the drawings, and the claims. Description of the Drawings

[0063] Figure 1 is a schematic flowchart of a plane extraction method based on sparse points according to an embodiment of the present invention.

[0064] Figure 2 shows a schematic flowchart of one of the steps of the plane extraction method based on sparse points according to the above embodiment of the present invention.

[0065] Figure 3 shows a schematic flowchart of the second step of the plane extraction method based on sparse points according to the above embodiment of the present invention.

[0066] Figure 4 shows a schematic flowchart of the third step of the plane extraction method based on sparse points according to the above embodiment of the present invention.

[0067] Figure 5 shows an example of the plane extraction method based on sparse points according to the above embodiment of the present invention.

[0068] Figure 6 is a schematic block diagram of a plane extraction system based on sparse points according to an embodiment of the present invention.

[0069] Figure 7 shows a schematic block diagram of an electronic device according to an embodiment of the present invention. Detailed Embodiment

[0070] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments in the following description are only examples, and those skilled in the art can think of other obvious variations. The basic principles defined in the following description of the present invention can be applied to other embodiments, variations, improvements, equivalent solutions, and other technical solutions that do not depart from the spirit and scope of the present invention.

[0071] In the present invention, the term "a" in the claims and the specification should be understood as "one or more". That is, in one embodiment, the number of an element can be one, and in another embodiment, the number of the element can be multiple. Unless it is clearly indicated in the disclosure of the present invention that the number of the element is only one, the term "a" cannot be understood as being unique or single, and the term "a" cannot be understood as a limitation on the number.

[0072] In the description of the present invention, it should be understood that "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through a medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0073] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0074] Although there are many plane detection algorithms at this stage, such as ARCore, ARKit, etc., they all have many deficiencies or shortcomings. Exemplarily, the overall algorithm scheme of an existing plane detection method is divided into the following steps: first, the current frame image of the environmental scene captured by the binocular camera is divided, and the feature points corresponding to each sub-block obtained by the division are detected; secondly, according to the three-dimensional coordinate information of the feature points in the world coordinate system, the sparse point data of the current frame image is obtained; finally, the current frame image is subjected to plane detection based on the sparse point data to obtain the optimal effective plane of the current frame image. The specific process of the above-mentioned existing plane detection method is as follows:

[0075] The first step is to segment the image collected by the binocular camera, extract features from each sub-image obtained after segmentation, and use fast feature point detection to select the corner point with the highest Harris response in the sub-block as the first feature point; then, use the extreme matching method to track left and right to obtain matching point pairs; at the same time, set the center of the left eye image to the origin, set the pixels with Euclidean distance greater than a certain threshold to the second state, and the rest to the first state; then, triangulate the matching point pairs, obtain the camera pose from SLAM, and calculate the three-dimensional point coordinates in the world coordinate system through coordinate transformation.

[0076] Step 2: Based on the sparse point cloud calculated previously, perform plane detection on the current frame to obtain the optimal effective plane of the current frame image. The most effective plane is defined as the plane with the most feature points. First, randomly extract three feature points from the sparse points through the RANSAC method to determine whether they are collinear; if they are not collinear, determine the plane according to these three points, and determine the normal vector and the number of the first inliers of the current plane, where the normal vector can use the eigenvector corresponding to the minimum eigenvalue among all inliers in the current plane according to the covariance matrix of all inliers in the plane as the normal vector of the plane; if the angle between its normal vector and the reference normal vector is less than a certain threshold, calculate the inliers according to this plane model, where the inliers can be determined that the inliers are less than 2 cm according to the distance from the point to the plane, and it is considered an inlier, and the number of these inliers should be greater than the number of inliers in the plane constructed previously; then, recalculate the plane normal vector, and at the same time, adjust the plane iteration times according to the ratio of the inliers in the plane to all points; otherwise, restart selecting 3D points to construct the plane; finally, determine whether it is an effective plane according to the angle between the current normal vector and the reference normal vector. If they are collinear, re-extract the feature points; repeatedly iterate to construct the plane and select the optimal plane.

[0077] Step 3: First, randomly determine a seed point in the optimal plane and set a radius; then, determine the neighboring points of the seed point in the optimal plane, and then set the neighboring points as the starting points to determine their neighboring points until the end, and the end is the feature point without neighboring points; finally, calculate the number of seed points. If the number of seed points meets the preset threshold, output an effective plane, and restart the seed selection among the remaining inliers in the optimal plane until all inliers in the plane are traversed and multiple effective planes are output.

[0078] However, although the above-mentioned existing plane detection methods can extract planes, there are large randomness in the extracted planes, and the limit matching is greatly affected by light, which may lead to inaccurate triangulation and thus affect plane extraction.

[0079] Therefore, the present application provides a plane extraction method, system and electronic device based on sparse points, which can not only quickly detect planes but also remove the detection of false planes, and is particularly suitable for the application scenarios and computing power limitations of mobile devices.

[0080] Schematic method

[0081] Referring to Figures 1 to 4 as shown in the accompanying drawings of the specification, a plane extraction method based on sparse points according to an embodiment of the present invention is illustrated. Specifically, as Figure 1 shown, the plane extraction method based on sparse points may include the steps:

[0082] S100: Extract spatial points from the acquired image information to obtain multiple spatial points in the world coordinate system;

[0083] S200: Based on four of the spatial points randomly selected from all the spatial points, construct one or more candidate planes;

[0084] S300: Calculate the plane covariance for each of the candidate planes respectively to use the candidate planes that meet the preset plane screening conditions as the planes to be confirmed; and

[0085] S400: By calculating the singular values of the plane matrices for all the planes to be confirmed, output the planes to be confirmed that meet the preset output conditions as the finally extracted planes.

[0086] It should be noted that the plane extraction method based on sparse points in this application constructs the candidate planes based on four of the spatial points randomly selected from all the spatial points, rather than three spatial points. Since the plane matrix constructed by four spatial points necessarily has the following characteristics: when the four spatial points are not in the same plane, the rank of the plane matrix must be greater than 3, and if the rank of the plane matrix is equal to 3, then one of the four spatial points must be linearly related to the other three spatial points. Therefore, the plane extraction method based on sparse points in this application can greatly improve the accuracy of plane detection by using the above characteristics. At the same time, the plane extraction method based on sparse points in this application also screens the candidate planes through plane covariance calculation and further improves the accuracy of plane detection by using the singular value calculation of the plane matrix to remove the detection of false planes.

[0087] Preferably, the image information in this application can be but is not limited to being implemented as an image acquired by binocular vision. That is to say, the image information is a binocular image, where the binocular image includes a left-eye image and a right-eye image.

[0088] More specifically, as Figure 2 shown, step S100 of the plane extraction method based on sparse points in this application may include the steps:

[0089] S110: Extract features from the binocular image to obtain left-eye feature points and right-eye feature points;

[0090] S120: Match the left-eye feature points and right-eye feature points through the left and right optical flow tracking method to obtain left and right matching point pairs; and

[0091] S130: Perform triangulation calculation on the left and right matching point pairs to obtain the coordinate values of the multiple spatial points in the world coordinate system.

[0092] Exemplarily, in the above-mentioned step S110 of the present application: It is possible but not limited to use the shi-Thomas algorithm to extract features from the binocular images, and remove the feature points at the image edges, so as to remove the feature points with relatively serious distortion, so as not to affect the calculation of the coordinate values (i.e., 3D coordinates) of the feature points in the world coordinate system.

[0093] In the above-mentioned step S120 of the present application: It is possible but not limited to first calculate the left-eye feature points with the pre-calibrated relative pose of the left and right eyes to obtain the initial coordinates on the right eye; then use optical flow tracking for feature point matching to obtain the left and right eye matching point pairs.

[0094] Preferably, the step S110 may further include the step of: screening the left and right eye matching point pairs according to preset point pair screening conditions to determine the correct matching point pairs.

[0095] More preferably, the preset point pair screening conditions can be implemented as calculating the distance difference between the right-eye coordinates of the optical flow tracking and the previously calculated initial coordinates respectively, sorting them from small to large, selecting the distance of the median of all distance differences multiplied by a coefficient as the threshold, and the point pairs with a distance less than this threshold are considered to pass the preliminary screening, and then the point pairs passing the preliminary screening are subjected to a secondary screening, and the point pairs with the difference in the left and right eye abscissas less than a certain threshold and the difference in the ordinates also less than a certain threshold are considered to be accurate, that is, the final left and right eye matching point pairs are screened out.

[0096] In the step S130 of the present application: Triangulate the matched left and right eye matching point pairs, calculate the depth value under the left eye, and calculate the reprojection error. When the reprojection error is less than the pre-set threshold, it is considered that the depth value is good, and the coordinate values of multiple space points in the world coordinate system are calculated through the pose provided by SLAM.

[0097] It should be noted that although in the attached Figure 2 And in the above description, taking the obtained image information being implemented as binocular images as an example to elaborate on the features and advantages of the method for extracting a plane based on sparse points of the present invention, those skilled in the art can understand that the attached Figure 2 And the binocular images disclosed in the above description are only examples, which do not constitute a limitation on the content and scope of the present invention. For example, in other examples of the method for extracting a plane based on sparse points, the obtained image information can also be but not limited to being implemented as images containing depth information such as RGBD images and the like.

[0098] According to the above embodiments of the present application, such as Figure 3As shown, step S200 of the sparse point-based plane extraction method may include the steps:

[0099] S210: Randomly select four of all the spatial points by the Random Sample Consensus (RANSAC) method to construct a 4×4 plane matrix;

[0100] S220: Decompose the 4×4 plane matrix by the Singular Value Decomposition (SVD) method to calculate the internal parameters of the initial candidate plane;

[0101] S230: By calculating the distances between the remaining spatial points and the initial candidate plane, determine the inliers of the initial candidate plane as the spatial points with distances less than a first threshold, and reconstruct a candidate plane matrix based on the inliers of the initial candidate plane and the four selected spatial points; and

[0102] S240: Detect whether the ratio of the minimum singular value to the maximum singular value of the candidate plane matrix is less than a second threshold through a detection function. If so, take the plane corresponding to the candidate plane matrix as the candidate plane.

[0103] Exemplarily, as Figure 5 shown, in step S200: First, randomly select 4 spatial points by the RANSAC (Random Sample Consensus) method to construct a plane matrix A 4×4 , and the fourth column of matrix A is filled with 1; then, decompose matrix A using SVD (Singular Value Decomposition), take the eigenvector corresponding to the minimum singular value in matrix A as the normal vector of the initial candidate plane, and calculate the internal parameters of the initial candidate plane (including the normal vector and the center point); after that, for the remaining spatial points, determine the inliers as those with the distance from the point to the plane less than the first threshold T; then, mainly detect whether the ratio of the singular values of the plane matrix (i.e., the minimum singular value / maximum singular value) is less than the second threshold through a detection function. If so, save the candidate plane; finally, repeat the iteration until all candidate planes are constructed.

[0104] It should be noted that the purpose of constructing matrix A is that if the four spatial points are not in a plane, the rank of matrix A must not be 3, and if the rank of matrix A is 3, then one of the four spatial points is related to the other points. Therefore, the sparse point-based plane extraction method can utilize this characteristic to greatly improve the accuracy of plane detection.

[0105] Preferably, the first threshold T may but is not limited to be implemented as a preset multiple of the distance corresponding to the median of the distances from all the remaining spatial points to the initial candidate plane. It can be understood that step S230 of the present application screens the inliers of the plane based on the absolute median difference, which helps to improve the screening accuracy of the inliers of the plane.

[0106] More preferably, as Figure 5 shown, step S240 may include the steps of: determining whether the number of inliers of the initial candidate plane is greater than a first preset inlier quantity; if so, further detecting whether the pW (i.e., the minimum singular value / maximum singular value) of the candidate plane matrix is less than the second threshold through a detection function PlaneTest; if so, determining whether the included angle between the normal vector of the plane corresponding to the candidate plane matrix and the reference normal vector is less than an included angle threshold; if so, then determining the plane corresponding to the candidate plane matrix as the candidate plane to complete the construction of the candidate plane.

[0107] It is worth mentioning that, as Figure 5 shown, in step S300 of the plane extraction method based on sparse points of the present invention: the preset plane screening condition may but is not limited to be implemented as:

[0108]

[0109] wherein: minEigValue, medEigValue, and maxEigValue are respectively implemented as the minimum singular value, median singular value, and maximum singular value of the covariance matrix of the candidate plane; K 1 and K 2 are both preset parameters.

[0110] As Figure 5 shown, in step S400 of the plane extraction method based on sparse points of the present invention: the preset output condition may but is not limited to be implemented as:

[0111] The ratio of the minimum singular value to the maximum singular value of the plane matrix corresponding to the plane to be confirmed is less than a third threshold, and the included angle between the normal vector of the plane to be confirmed and the reference normal vector is the smallest.

[0112] Preferably, the third threshold may but is not limited to be equal to the second threshold.

[0113] It should be noted that in other examples of this application, the preset output condition can also be implemented as: the ratio of the minimum singular value to the maximum singular value of the plane matrix corresponding to the plane to be confirmed is less than a third threshold, and the angle between the normal vector of the plane to be confirmed and the reference normal vector is less than a preset angle threshold (or the angle between the normal vector of the plane to be confirmed and the reference normal vector is among the two or more smallest).

[0114] Preferably, the reference normal vector can be, but is not limited to, implemented as the normal vector of a horizontal plane or the normal vector of a vertical plane.

[0115] It is worth mentioning that as Figure 1 shown, in an embodiment of this application, between the step S300 and the step S400 of the plane extraction method based on sparse points, the method may further include the step of:

[0116] S500: Perform local expansion processing on the plane to be confirmed to obtain an expanded plane, and use the expanded plane to replace the plane to be confirmed for subsequent processing.

[0117] More specifically, as Figure 4 shown, the step S500 of the plane extraction method based on sparse points may include the steps of:

[0118] S510: Sequentially calculate whether the distance between the spatial points from the center of the field of view from near to far and the plane to be confirmed is less than a distance threshold. If so, use the corresponding spatial point as an inlier of the plane to be confirmed;

[0119] S520: Count the inliers of the plane to be confirmed until the number of inliers of the plane to be confirmed reaches a second preset inlier number; and

[0120] S530: Based on all the inliers in the plane to be confirmed, reconstruct and calculate the plane matrix to obtain the internal parameters of the expanded plane.

[0121] Preferably, the distance threshold MDP can be, but is not limited to, implemented as:

[0122] MDP = median(D) + s * (k * median(d i - median(D))), D = {d 1 , d 2 , …, d n}

[0123] where: d i is the distance between the i-th spatial point and the plane to be confirmed; median is the median; k and s are preset parameters.

[0124] It should be noted that the second preset number of inlier points can be, but is not limited to, implemented as 70 to 100.

[0125] Schematic system

[0126] Referring to the accompanying drawings of the specification Figure 6 As shown, a sparse point-based plane extraction system 600 according to an embodiment of the present invention is illustrated. Specifically, as Figure 6 shown, the sparse point-based plane extraction system 600 includes communicatively connected to each other: a spatial point extraction module 610 for performing spatial point extraction processing on the acquired image information to obtain a plurality of spatial points in the world coordinate system; a candidate plane construction module 620 for constructing one or more candidate planes based on four of the spatial points randomly selected from all the spatial points; a covariance calculation module 630 for calculating the plane covariance of all the candidate planes respectively to use the candidate planes that meet the preset plane screening conditions as the planes to be confirmed; and a plane output module 640 for outputting the planes to be confirmed that meet the preset output conditions as the finally extracted planes by calculating the singular values of the plane matrices of all the planes to be confirmed.

[0127] More specifically, as Figure 6 shown, the candidate plane construction module 620 includes communicatively connected to each other: a random selection module 621 for randomly selecting four of the spatial points from all the spatial points by the random sample consensus method to construct a 4×4 plane matrix; a singular value decomposition module 622 for decomposing the 4×4 plane matrix by the singular value decomposition method to calculate the internal parameters of the initial candidate plane; an inlier confirmation module 623 for determining the spatial points with a distance less than the first threshold as the inliers of the initial candidate plane by calculating the distance between the remaining spatial points and the initial candidate plane, and reconstructing a candidate plane matrix according to the inliers of the initial candidate plane and the four selected spatial points; and a detection module 624 for detecting whether the ratio of the minimum singular value to the maximum singular value of the candidate plane matrix is less than the second threshold, and if so, using the plane corresponding to the candidate plane matrix as the candidate plane.

[0128] It should be noted that in the above embodiment of the present application, as Figure 6 shown, the sparse point-based plane extraction system 600 further includes a local expansion module 650, where the local expansion module 650 is used to perform local expansion processing on the planes to be confirmed to obtain expanded planes, and use the expanded planes to replace the planes to be confirmed for subsequent processing.

[0129] In an example of the present application, asFigure 6 As shown, the local expansion module 650 includes the following components communicatively connected to each other: a distance calculation module 651, which is configured to sequentially calculate whether the distance between a spatial point from the center of the field of view from near to far and the plane to be confirmed is less than a distance threshold. If so, the corresponding spatial point is regarded as an inlier of the plane to be confirmed; an inlier statistics module 652, which is configured to count the inliers of the plane to be confirmed until the number of inliers of the plane to be confirmed reaches a second preset number of inliers; and a matrix construction module 653, which is configured to reconstruct and calculate a plane matrix based on all the inliers in the plane to be confirmed to obtain the internal parameters of the expanded plane.

[0130] Schematic electronic device

[0131] Next, refer to Figure 7 to describe an electronic device according to an embodiment of the present invention. As Figure 7 shown, the electronic device 90 includes one or more processors 91 and a memory 92.

[0132] The processor 91 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 90 to perform desired functions. In other words, the processor 91 includes one or more physical devices configured to execute instructions. For example, the processor 91 may be configured to execute instructions as part of the following: one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, transform the states of one or more components, achieve technical effects, or otherwise obtain desired results.

[0133] The processor 91 may include one or more processors configured to execute software instructions. As a supplement or replacement, the processor 91 may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. The processors of the processor 91 may be single-core or multi-core, and the instructions executed thereon may be configured for serial, parallel, and / or distributed processing. The various components of the processor 91 may optionally be distributed on two or more separate devices, which may be located remotely and / or configured for collaborative processing. Aspects of the processor 91 may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration.

[0134] The memory 92 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage media, and the processor 11 may run the program instructions to implement some or all of the steps in the above-described schematic method of the present invention, and / or other desired functions.

[0135] In other words, the memory 92 includes one or more physical devices configured to hold machine-readable instructions executable by the processor 91 to implement the methods and processes described herein. When implementing these methods and processes, the state of the memory 92 may be transformed (e.g., to hold different data). The memory 92 may include removable and / or built-in devices. The memory 92 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-ray Disc, etc.), semiconductor memory (e.g., RAM, EPROM, EEPROM, etc.), and / or magnetic memory (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), and so on. The memory 92 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location-addressable, file-addressable, and / or content-addressable devices.

[0136] It can be understood that the memory 92 includes one or more physical devices. However, aspects of the instructions described herein may alternatively be propagated by a communication medium (e.g., electromagnetic signals, optical signals, etc.) not held by a physical device for a limited duration. Aspects of the processor 91 and the memory 92 may be integrated together into one or more hardware logic components. These hardware logic components may include, for example, field programmable gate arrays (FPGA), program and application specific integrated circuits (PASIC / ASIC), program and application specific standard products (PSSP / ASSP), system on a chip (SOC), and complex programmable logic devices (CPLD).

[0137] In one example, as Figure 7As shown, the electronic device 90 may further include an input device 93 and an output device 94, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown). For example, the input device 93 may be, for example, a camera module for collecting image data or video data, etc. Also, for example, the input device 93 may include one or more user input devices such as a keyboard, a mouse, a touch screen, or a game controller, or be docked with them. In some embodiments, the input device 93 may include or be docked with a selected natural user input (NUI) component. Such a component may be integrated or peripheral, and the transduction and / or processing of input actions may be processed on-board or off-board. Example NUI components may include a microphone for language and / or speech recognition; an infrared, color, stereo display, and / or depth camera for machine vision and / or gesture recognition; a head tracker, an eye tracker, an accelerometer, and / or a gyroscope for motion detection and / or intention recognition; and an electric field sensing component for evaluating brain activity and / or body movement; and / or any other suitable sensor.

[0138] The output device 94 may output various information to the outside, including classification results, etc. The output device 94 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0139] Of course, the electronic device 90 may further include the communication device, wherein the communication device may be configured to communicatively couple the electronic device 90 with one or more other computer devices. The communication device may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem may be configured to communicate via a wireless telephone network or a wired or wireless local area network or wide area network. In some embodiments, the communication device may allow the electronic device 90 to send messages to other devices and / or receive messages from other devices via a network such as the Internet.

[0140] It will be understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples should not be considered restrictive, as many variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. Thus, the various actions shown and / or described may be performed in the order shown and / or described, in other orders, in parallel, or omitted. Similarly, the order of the above processes may be changed.

[0141] Of course, for simplicity, Figure 7Only some of the components related to the present invention in the electronic device 90 are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, according to specific application scenarios, the electronic device 90 may further include any other appropriate components.

[0142] It should also be noted that in the devices, equipment and methods of the present invention, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations shall be regarded as equivalent solutions of the present invention.

[0143] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0144] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the drawings are only examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functions and structural principles of the present invention have been demonstrated and illustrated in the embodiments, and the embodiments of the present invention can have any deformation or modification without departing from the said principles.

Claims

1. Plane extraction method based on sparse points, characterized in that, it includes the steps of: Performing spatial point extraction processing on the acquired image information to obtain a plurality of spatial points in the world coordinate system; Based on four of the spatial points randomly selected from all the spatial points, constructing one or more candidate planes; Calculating the plane covariance for each of the candidate planes respectively to use the candidate planes that meet the preset plane screening conditions as the planes to be confirmed; and By calculating the singular values of the plane matrices for all the planes to be confirmed, outputting the planes to be confirmed that meet the preset output conditions as the finally extracted planes.

2. The plane extraction method based on sparse points according to claim 1, wherein, the acquired image information is a binocular image.

3. The plane extraction method based on sparse points according to claim 2, wherein, the step of performing spatial point extraction processing on the acquired image information to obtain a plurality of spatial points in the world coordinate system includes the steps of: Performing feature extraction on the binocular image to obtain left-eye feature points and right-eye feature points; Performing feature point matching on the left-eye feature points and right-eye feature points by the left and right optical flow tracking method to obtain left and right eye matching point pairs; and Performing triangulation calculation on the left and right eye matching point pairs to obtain the coordinate values of the plurality of spatial points in the world coordinate system.

4. The plane extraction method based on sparse points according to any one of claims 1 to 3, wherein, the step of constructing one or more candidate planes based on four of the spatial points randomly selected from all the spatial points includes the steps of: Randomly selecting four of the spatial points from all the spatial points by the random sample consensus method to construct a 4*4 plane matrix; Decomposing the 4*4 plane matrix by the singular value decomposition method to calculate the internal parameters of the initial candidate plane; By calculating the distances between the remaining spatial points and the initial candidate plane, determining the spatial points with distances less than the first threshold as the inliers of the initial candidate plane, and reconstructing the candidate plane matrix according to the inliers of the initial candidate plane and the four selected spatial points; and By detecting whether the ratio of the minimum singular value to the maximum singular value of the candidate plane matrix is less than the second threshold, if so, using the plane corresponding to the candidate plane matrix as the candidate plane.

5. The plane extraction method based on sparse points according to claim 4, wherein, the first threshold is a preset multiple of the distance corresponding to the median of the distances from all the remaining spatial points to the initial candidate plane arranged from small to large.

6. The plane extraction method based on sparse points according to claim 5, wherein, the step of detecting whether the ratio of the minimum singular value to the maximum singular value of the candidate plane matrix is less than the second threshold by the detection function, if so, using the plane corresponding to the candidate plane matrix as the candidate plane includes the steps of: Judging whether the number of inliers of the initial candidate plane is greater than the first preset inlier number; If so, detect whether the ratio of the minimum singular value to the maximum singular value of the candidate plane matrix is less than the second threshold through a detection function; If so, determine whether the included angle between the normal vector of the plane corresponding to the candidate plane matrix and the reference normal vector is less than the included angle threshold; And If so, confirm the plane corresponding to the candidate plane matrix as the candidate plane.

7. The plane extraction method based on sparse points according to any one of claims 1 to 3, Wherein, The preset plane screening condition is: Where: minEigValue, medEigValue and maxEigValue are respectively implemented as the minimum singular value, median singular value and maximum singular value of the covariance matrix of the candidate plane; K1 and K2 are both preset parameters.

8. The plane extraction method based on sparse points according to any one of claims 1 to 3, Wherein, The preset output condition is: The ratio of the minimum singular value to the maximum singular value of the plane matrix corresponding to the plane to be confirmed is less than the third threshold, and the included angle between the normal vector of the plane to be confirmed and the reference normal vector is the smallest.

9. The plane extraction method based on sparse points according to any one of claims 1 to 3, further comprising the steps of: Perform local expansion processing on the plane to be confirmed to obtain an expanded plane, and use the expanded plane to replace the plane to be confirmed for subsequent processing.

10. The plane extraction method based on sparse points according to claim 9, Wherein, The step of performing local expansion processing on the plane to be confirmed to obtain an expanded plane and using the expanded plane to replace the plane to be confirmed for subsequent processing includes the steps of: Successively calculate whether the distance between the spatial points from the center of the field of view from near to far and the plane to be confirmed is less than the distance threshold. If so, use the corresponding spatial points as the inliers of the plane to be confirmed; Count the inliers of the plane to be confirmed until the number of inliers of the plane to be confirmed reaches the second preset inlier number; And Based on all the inliers in the plane to be confirmed, reconstruct and calculate the plane matrix to obtain the internal parameters of the expanded plane.

11. The plane extraction method based on sparse points according to claim 10, Wherein, The distance threshold MDP is implemented as: MDP = median(D) + s * (k * median(||di - median(D)||)), D = {d 1 , d 2 , …, d n} where: d i is the distance from the i-th spatial point to the plane to be confirmed; median is the median; k and s are preset parameters.

12. A plane extraction system based on sparse points, Characterized in that, Including components communicatively connected to each other: A spatial point extraction module for performing spatial point extraction processing on the acquired image information to obtain a plurality of spatial points in the world coordinate system; A candidate plane construction module for constructing one or more candidate planes based on four of the spatial points randomly selected from all the spatial points; A covariance calculation module for calculating the plane covariance of all the candidate planes respectively, and using the candidate planes that meet the preset plane screening conditions as the planes to be confirmed; And A plane output module for outputting the planes to be confirmed that meet the preset output conditions as the finally extracted planes by calculating the singular values of the plane matrices of all the planes to be confirmed.

13. The plane extraction system based on sparse points as described in claim 12, wherein, the candidate plane construction module includes: a randomly selecting module communicatively connected to each other, configured to randomly select four of all the spatial points by the random sample consensus method to construct a 4×4 plane matrix; a singular value decomposition module, configured to decompose the 4×4 plane matrix by the singular value decomposition method to calculate the internal parameters of the initial candidate plane; an inlier confirmation module, configured to determine, by calculating the distances between the remaining spatial points and the initial candidate plane, the spatial points with distances less than a first threshold as the inliers of the initial candidate plane, and reconstruct a candidate plane matrix according to the inliers of the initial candidate plane and the four selected spatial points; and a detection module, configured to detect whether the ratio of the minimum singular value to the maximum singular value of the candidate plane matrix is less than a second threshold by a detection function, and if so, take the plane corresponding to the candidate plane matrix as the candidate plane.

14. The plane extraction system based on sparse points as described in claim 12 or 13, further comprising a local expansion module, wherein the local expansion module is configured to perform local expansion processing on the plane to be confirmed to obtain an expanded plane, and use the expanded plane to replace the plane to be confirmed for subsequent processing.

15. The plane extraction system based on sparse points as described in claim 14, wherein, the local expansion module includes: a distance calculation module communicatively connected to each other, configured to sequentially calculate whether the distances between the spatial points from the center of the field of view from near to far and the plane to be confirmed are less than a distance threshold, and if so, take the corresponding spatial points as the inliers of the plane to be confirmed; an inlier statistics module, configured to count the inliers of the plane to be confirmed until the number of inliers of the plane to be confirmed reaches a second preset number of inliers; and a matrix construction module, configured to reconstruct and calculate a plane matrix based on all the inliers in the plane to be confirmed to obtain the internal parameters of the expanded plane.

16. An electronic device, characterized in that it includes: at least one processor for executing instructions; and a memory communicatively connected to the at least one processor, wherein the memory has at least one instruction, and the instruction is executed by the at least one processor to cause the at least one processor to execute some or all of the steps in the plane extraction method based on sparse points, and the plane extraction method based on sparse points includes the steps of: performing spatial point extraction processing on the acquired image information to obtain a plurality of spatial points in the world coordinate system; constructing one or more candidate planes based on four of all the spatial points randomly selected; performing plane covariance calculation on all the candidate planes respectively to take the candidate planes that meet the preset plane screening conditions as the planes to be confirmed; and outputting the planes to be confirmed that meet the preset output conditions as the finally extracted planes by calculating the singular values of the plane matrices of all the planes to be confirmed.

Images