Plane Expansion Method Based on Sparse Point Cloud, Its System and Electronic Device

Abstract

A plane expansion method based on sparse point cloud, its system and electronic device. The plane expansion method based on sparse point cloud includes the steps of: selecting a local region of interest on the current frame image according to the geometric center of the projected points of all existing spatial points belonging to the current plane in the previous frame image on the current frame image; extracting spatial points from the current frame image according to the local region of interest to obtain spatial points to be confirmed; screening all the existing spatial points and the spatial points to be confirmed in the current plane to obtain spatial points to be updated belonging to the current plane; and recalculating plane information of the current plane based on the spatial points to be updated to obtain a newly expanded plane.

Description

Technical Field

[0001] The present invention relates to the technical field of planar expansion, and in particular to a planar expansion method, system and electronic device based on sparse point cloud. Background Art

[0002] In the AR scenario, it is often necessary to construct a three-dimensional space model to achieve the function of augmented reality, which makes it an important prerequisite to determine an effective space coordinate system in advance to affect the effect of subsequent modeling algorithms. However, since the physical space scene is generally complex, randomly determining a coordinate system often increases the complexity of subsequent modeling steps and affects the user's visual experience. This requires selecting a specific shape area in the physical space to determine the space coordinate system. As a common shape feature, the plane has the advantages of simple structure and good adaptability, and is suitable as the reference of the space coordinate system.

[0003] The planes detected through plane detection in different physical spaces need to be further updated with information, so as to ensure that the initial coordinate system can be continuously updated and synchronized with the current user's vision. Currently, the commonly used plane expansion schemes are all based on point cloud information, and there are certain limitations for devices that cannot be equipped with depth sensors or have high requirements for SLAM real-time performance. For example, the basic implementation method of the currently commonly used plane expansion technology is to obtain the point cloud information of the current physical space according to an external depth sensor or a binocular camera, and use algorithms such as region growing, RANSAC screening, and Hough transform to implement the construction and update of the plane coordinate system.

[0004] However, although this technology can relatively easily extract plane information because the point cloud can provide sufficient space information, and can also be applicable to the application of multi-plane detection, it still has some defects. For example, for plane expansion under a depth sensor, in order to ensure sufficient space information, the existing plane expansion scheme needs to input dense point cloud information, and processing this information takes a lot of time. In some lightweight AR devices, in order to ensure visual synchronization, a very high frame rate must be guaranteed, making this dense point cloud scheme infeasible. For the coefficient point cloud plane expansion under a binocular camera, the existing plane expansion scheme needs to input the spatial feature points extracted by SLAM and perform triangulation operations to obtain the spatial information of these point clouds. Although this scheme greatly speeds up the running time of the algorithm because it uses spatially sparse feature points, the real-time performance of the SLAM system requires that the extracted feature points are often controlled within a certain number, and these points generally cannot meet the minimum point number requirements for plane expansion after screening, thus causing the expansion to stop and being difficult to continue. Summary of the Invention

[0005] One advantage of the present invention is to provide a plane expansion method, system and electronic device based on sparse point cloud, which can solve the real-time update of the plane coordinate system in different physical spaces.

[0006] Another advantage of the present invention is to provide a plane expansion method, system and electronic device based on sparse point cloud. In one embodiment of the present invention, the plane expansion method based on sparse point cloud can, in an augmented reality scenario, enable the plane coordinate system detected in the initial state to still quickly update information as the scenario changes.

[0007] Another advantage of the present invention is to provide a plane expansion method, system and electronic device based on sparse point cloud. In one embodiment of the present invention, the plane expansion method based on sparse point cloud can, for a device without a configured depth sensor, update the plane coordinate system while ensuring the running time.

[0008] Another advantage of the present invention is to provide a plane expansion method, system and electronic device based on sparse point cloud. In one embodiment of the present invention, the plane expansion method based on sparse point cloud does not require a dense point cloud as input, thus avoiding a complex calculation process and reducing the time overhead.

[0009] Another advantage of the present invention is to provide a plane expansion method, system and electronic device based on sparse point cloud. In one embodiment of the present invention, the plane expansion method based on sparse point cloud can no longer solely use the feature points input by SLAM as the judgment information, but instead re-extract feature points in the current local image window and use binocular image information to solve for the corresponding spatial points, so as to eliminate the interruption of plane expansion caused by the over-sparsity of the input feature points.

[0010] Another advantage of the present invention is to provide a plane expansion method, system and electronic device based on sparse point cloud. In one embodiment of the present invention, the plane expansion method based on sparse point cloud does not select the entire image, but instead selects a range with a certain window size from the center of the plane, which speeds up the feature point extraction speed and also ensures the natural gradual change of plane expansion.

[0011] Another advantage of the present invention is to provide a plane expansion method, system and electronic device based on sparse point cloud. In one embodiment of the present invention, the plane expansion method based on sparse point cloud can use the 3D information generated by the distance from the spatial point to the plane and the 2D information generated by the homography matrix calculated from the points in the front and back frame planes to jointly constrain the candidate points from the geometric space, so as to obtain a more accurate inlier judgment.

[0012] Another advantage of the present invention is to provide a plane expansion method based on sparse point cloud, its system and electronic device. In one embodiment of the present invention, the plane expansion method based on sparse point cloud can further screen those spaces that cannot be eliminated in the geometric space by using the color and distance features of these sparse points, so as to reduce the misjudgment of plane points caused by factors such as incorrect triangulation results.

[0013] Another advantage of the present invention is to provide a plane expansion method based on sparse point cloud, its system and electronic device. In one embodiment of the present invention, the implementation of the plane expansion method based on sparse point cloud is simple and has strong applicability, which is of great significance in reducing algorithm time and improving algorithm accuracy, and has great application value for plane expansion algorithms based on binocular vision or monocular inertial SLAM, and is expected to be applied in fields such as augmented reality and autonomous driving.

[0014] Another advantage of the present invention is to provide a plane expansion method based on sparse point cloud, its system and electronic device. In one embodiment of the present invention, the plane expansion method based on sparse point cloud does not use the dense point cloud obtained by a depth sensor as input, but continuously extracts spatial feature points through a binocular camera, thus eliminating the impact of excessive point cloud computing volume on real-time performance.

[0015] Another advantage of the present invention is to provide a plane expansion method based on sparse point cloud, its system and electronic device. In order to achieve the above advantages, in the present invention, there is no need to adopt a complex structure and a large amount of computation, and the requirements for software and hardware are low. Therefore, the present invention successfully and effectively provides a solution, not only providing a plane expansion method based on sparse point cloud, its system and electronic device, but also increasing the practicability and reliability of the plane expansion method based on sparse point cloud, its system and electronic device.

[0016] To achieve at least one of the above advantages or other advantages and purposes, the present invention provides a plane expansion method based on sparse point cloud, including the steps of:

[0017] Select a local region of interest on the current frame image according to the geometric center of the points projected on the current frame image by all the existing spatial points belonging to the current plane in the previous frame image;

[0018] Extract spatial points from the current frame image according to the local region of interest to obtain the spatial points to be confirmed;

[0019] Screen all the existing spatial points and the spatial points to be confirmed in the current plane to obtain the spatial points to be updated belonging to the current plane; and

[0020] Recalculate the plane information for the current plane based on the space points to be updated to obtain a newly expanded plane.

[0021] According to an embodiment of the present application, the local region of interest is a rectangular region centered on the geometric center and having a size smaller than the current frame image.

[0022] According to an embodiment of the present application, the step of extracting space points from the current frame image according to the local region of interest to obtain space points to be confirmed includes the steps of:

[0023] Perform corner detection on the local region of interest of the current frame image to extract sparse feature points from the local region of interest;

[0024] By using the left and right camera optical flow tracking method, solve for the feature points on the right eye image that match the feature points on the left eye image to obtain a pair of matching left and right eye feature points; and

[0025] According to the normalized coordinates and relative poses corresponding to the left and right cameras, perform triangulation calculation on the pair of left and right eye feature points to solve for the spatial coordinates of the landmark as the initial space points.

[0026] According to an embodiment of the present application, the step of extracting space points from the current frame image according to the local region of interest to obtain space points to be confirmed further includes the steps of:

[0027] Verify the reprojection error of the initial space points to delete the space points with an error exceeding the threshold, and use the remaining initial space points as the space points to be confirmed.

[0028] According to an embodiment of the present application, the step of screening all existing space points and the space points to be confirmed in the current plane to obtain the space points to be updated belonging to the current plane includes the steps of:

[0029] Delete the space points to be confirmed with a space distance greater than the first distance threshold according to the space distance between the space points to be confirmed and the current plane to obtain the space points after 3D screening;

[0030] Project the points that are tracked in the current frame image and do not belong to the current plane in the previous frame image onto the current frame image through the homography matrix to solve for the plane distance between the projection points and the corresponding observed points, and delete the projection points with a plane distance greater than the second distance threshold to obtain the space points after 2D screening; and

[0031] Cluster processing is performed based on the grayscale differences and distances between all the existing spatial points, the 3D filtered spatial points, and the 2D filtered spatial points in the current plane and the center point of the current plane, so as to remove the points that do not belong to the same class as the center point and obtain the spatial points to be updated belonging to the current plane.

[0032] According to an embodiment of the present application, the homography matrix is calculated based on the points belonging to the current plane in the previous frame image that are tracked on the current frame image.

[0033] According to another aspect of the present application, the present application further provides a plane expansion system based on sparse point clouds, including: communicatively connected to each other:

[0034] A local ROI extraction module, configured to extract a local region of interest on the current frame image according to the geometric center of the points projected on the current frame image for all the existing spatial points belonging to the current plane in the previous frame image;

[0035] A spatial point extraction module, configured to perform spatial point extraction on the current frame image according to the local region of interest to obtain the spatial points to be confirmed;

[0036] A spatial point filtering module, configured to filter all the existing spatial points and the spatial points to be confirmed in the current plane to obtain the spatial points to be updated belonging to the current plane; and

[0037] A plane expansion module, configured to recalculate the plane information of the current plane based on the spatial points to be updated to obtain a newly expanded plane.

[0038] According to an embodiment of the present application, the local region of interest is a rectangular region centered on the geometric center and with a size smaller than the current frame image.

[0039] According to an embodiment of the present application, the spatial point extraction module includes a feature point extraction module, a feature point matching module, and a triangulation processing module that are communicatively connected to each other. The feature point extraction module is configured to perform corner detection on the local region of interest of the current frame image to extract sparse feature points from the local region of interest. The feature point matching module is configured to solve the feature points matched on the right-eye image for the feature points on the left-eye image through the left and right camera optical flow tracking method to obtain the matched left and right-eye feature point pairs. The triangulation processing module is configured to perform triangulation calculation on the left and right-eye feature point pairs according to the normalized coordinates and relative poses corresponding to the left and right cameras to solve the spatial coordinates of the waypoint as the initial spatial point.

[0040] According to an embodiment of the present application, the spatial point extraction module further includes a reprojection error verification module for verifying the reprojection error of the initial spatial points to delete the spatial points with errors exceeding the threshold, and taking the remaining initial spatial points as the spatial points to be confirmed.

[0041] According to an embodiment of the present application, the spatial point screening module includes a 3D screening module, a 2D screening module, and a clustering screening module that are communicatively connected to each other. The 3D screening module is configured to delete the spatial points to be confirmed with a spatial distance greater than the first distance threshold according to the spatial distance between the spatial points to be confirmed and the current plane, so as to obtain the spatially screened spatial points. The 2D screening module is configured to project the points that are tracked in the current frame image and do not belong to the current plane in the previous frame image onto the current frame image through a homography matrix to solve the planar distance between the projected points and the corresponding observed points, and delete the projected points with a planar distance greater than the second distance threshold, so as to obtain the 2D screened spatial points. The clustering screening module is configured to perform clustering processing according to the gray difference and distance between all the existing spatial points in the current plane, the 3D screened spatial points, and the 2D screened spatial points and the center point of the current plane, so as to eliminate the points that do not belong to the same category as the center point and obtain the spatial points to be updated belonging to the current plane.

[0042] According to another aspect of the present application, the present application further provides an electronic device, including:

[0043] At least one processor for executing instructions; and

[0044] A memory communicatively connected to the at least one processor, wherein the memory has at least one instruction, and the instruction is executed by the at least one processor so that the at least one processor executes some or all of the steps in the planar expansion method based on sparse point clouds, and the planar expansion method based on sparse point clouds includes the steps of:

[0045] Select a local region of interest on the current frame image according to the geometric center of the points projected on the current frame image for all the existing spatial points belonging to the current plane in the previous frame image;

[0046] Extract spatial points from the current frame image according to the local region of interest to obtain the spatial points to be confirmed;

[0047] Screen all the existing spatial points and the spatial points to be confirmed in the current plane to obtain the spatial points to be updated belonging to the current plane; and

[0048] Recalculate the plane information of the current plane based on the spatial points to be updated to obtain a newly expanded plane.

[0049] The further objects and advantages of the present invention will be fully realized upon understanding the subsequent description and the accompanying drawings.

[0050] These and other objects, features, and advantages of the present invention will be fully realized through the following detailed description, the accompanying drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a schematic flowchart of a method for planar expansion based on sparse point cloud according to an embodiment of the present invention.

[0052] Figure 2 shows a schematic flowchart of one of the steps of the method for planar expansion based on sparse point cloud according to the above embodiment of the present invention.

[0053] Figure 3 shows a schematic flowchart of the second step of the method for planar expansion based on sparse point cloud according to the above embodiment of the present invention.

[0054] Figure 4 shows a schematic diagram of the principle of solving spatial points in the method for planar expansion based on sparse point cloud according to the above embodiment of the present invention.

[0055] Figure 5 shows an application schematic diagram of the method for planar expansion based on sparse point cloud according to the above embodiment of the present invention.

[0056] Figure 6 is a schematic block diagram of a system for planar expansion based on sparse point cloud according to an embodiment of the present invention.

[0057] Figure 7 shows a schematic block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0058] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles defined in the following description can be applied to other embodiments, variations, improvements, equivalent embodiments, and other technical solutions that do not depart from the spirit and scope of the present invention.

[0059] In the present invention, the term "a" in the claims and the specification should be understood as "one or more". That is, in one embodiment, the number of an element can be one, while in other embodiments, the number of the element can be multiple. Unless it is clearly indicated in the disclosure of the present invention that the number of the element is only one, the term "a" cannot be understood as being unique or single, and the term "a" cannot be understood as a limitation on the number.

[0060] In the description of this specification, the descriptions with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0061] Currently, existing planar expansion technical solutions are basically based on point cloud information, and there are certain limitations for devices that cannot be equipped with depth sensors or have high requirements for SLAM real-time performance. Therefore, in order to solve the real-time update of the planar coordinate system (i.e., planar expansion) in different physical spaces, the present application provides a planar expansion method, system, and electronic device based on sparse point clouds, so that in an augmented reality scenario, the planar coordinate system detected in the initial state can still be quickly updated with the change of the scenario. Especially for devices without depth sensors, the update of the planar coordinate system can be achieved while ensuring the running time. The present application reextracts the feature points in the current local window, obtains their spatial information through triangulation, and further eliminates the points that do not meet the planar requirements through methods such as the distance from a point to a plane, homography matrix, and sparse feature clustering, and determines the points in the final plane, so as to recalculate the relevant information of the current planar coordinate system, that is, to achieve real-time planar expansion.

[0062] Schematic method

[0063] Referring to the accompanying drawings of the specification Figures 1 to 3 As shown, a planar expansion method based on sparse point clouds according to an embodiment of the present invention is illustrated. Specifically, as Figure 1 shown, the planar expansion method based on sparse point clouds may include the steps:

[0064] S100: Select a local region of interest on the current frame image according to the geometric center of the projection points of all the existing spatial points belonging to the current plane in the previous frame image on the current frame image;

[0065] S200: Extract spatial points from the current frame image according to the local region of interest to obtain spatial points to be confirmed;

[0066] S300: Screen all the existing spatial points and the spatial points to be confirmed in the current plane to obtain spatial points to be updated belonging to the current plane; and

[0067] S400: Recalculate the plane information of the current plane based on the spatial points to be updated to obtain a newly expanded plane.

[0068] It should be noted that since the plane expansion method based on sparse point cloud in this application selects a local region of interest (i.e., local ROI) on the current frame image, thereby limiting the image range for subsequent feature point extraction to ensure the uniformity of newly added feature points, the plane expansion method based on sparse point cloud in this application not only speeds up the feature point extraction speed but also ensures the natural gradient of plane expansion. In addition, compared with the existing plane expansion scheme based on depth sensors, the plane expansion method based on sparse point cloud in this application does not require a dense point cloud as input, thereby avoiding complex calculation processes and reducing time overhead.

[0069] Preferably, in order to further reduce the time overhead of the plane expansion method based on sparse point cloud and improve real-time performance, both the previous frame image and the current frame image in the plane expansion method based on sparse point cloud in this application are key optical frame images.

[0070] More specifically, in step S100 of the plane expansion method based on sparse point cloud, the local region of interest is preferably implemented as a rectangular region centered on the geometric center and with a size smaller than the current frame image. For example, the local region of interest can be centered on the geometric center, with a width of 0.3 to 0.5 times the total number of columns of the current frame image and a height of 0.3 to 0.5 times the total number of rows of the current frame image.

[0071] In addition, the calculation model of the geometric center is as follows:

[0072]

[0073] Where: and are the abscissa and ordinate of the geometric center on the current frame image respectively; x i and y iThey are respectively the abscissa and ordinate of the i-th projection point on the current frame image; N is the total number of the projection points.

[0074] According to the above embodiments of the present application, taking the input images as left and right eye images as an example, in order to solve the problem that the plane expansion is interrupted due to too few feature points caused by unstable SLAM tracking, in the step S200 of the plane expansion method based on sparse point cloud of the present application, the input left and right eye images are first subjected to feature extraction and optical flow tracking to obtain the corresponding feature point pairs of the left and right eyes; then the coordinate information of the 3D points in the camera coordinate system is obtained through triangulation, and finally, the 3D point information in the world coordinate system is calculated through the input SLAM pose information to complete the extraction of the spatial points. Of course, in other examples of the present invention, the input images can also but are not limited to be implemented as TOF images, which can directly obtain the required sparse spatial points through feature point extraction.

[0075] Specifically, as Figure 2 shown, the step S200 of the plane expansion method based on sparse point cloud of the present application may include the steps:

[0076] S210: Perform corner detection on the local region of interest of the current frame image to extract sparse feature points from the local region of interest;

[0077] S220: Solve the feature points matching the feature points on the left eye image on the right eye image through the left and right camera optical flow tracking method to obtain the matching left and right eye feature point pairs; and

[0078] S230: Perform triangulation calculation on the left and right eye feature point pairs according to the normalized coordinates and relative poses corresponding to the left and right cameras to solve the spatial coordinates of the landmark points as the initial spatial points.

[0079] More specifically, in the step S210 of the plane expansion method based on sparse point cloud, the Shi-Tomasi operator is selected to perform corner detection on the image in the local region of interest, so as to ensure that while controlling the maximum number of feature points in the field of view within a certain range to ensure the sparsity of the feature points, a certain distance is also maintained between adjacent feature points.

[0080] In the step S220 of the plane expansion method based on sparse point cloud, the coordinates of the feature points of the left eye image on the right eye image are obtained through the pyramid optical flow tracking method, and at the same time, the tracking point pairs with too large distance changes are deleted to ensure obtaining more accurate feature point pairs.

[0081] The step S230 of the plane expansion method based on sparse point cloud is mainly used to calculate the spatial point coordinates required for plane expansion. Specifically, during implementation, the normalized coordinates and relative poses corresponding to the left and right cameras are mainly used to solve the spatial coordinates of the road signs. For example, as Figure 4 shown, substituting the normalized coordinates of the current left and right cameras into the reprojection equation of the camera d r X r = d l R r-l X l + T r-l , where d r and d l represent the depths of the left and right cameras respectively; Xr and Xl represent the coordinate values of the spatial point in the left and right camera coordinate systems respectively; R r-l represents the rotation matrix between the left and right cameras; T r-l represents the translation vector between the left and right cameras.

[0082] It should be noted that in an example of the present application, the plane expansion method based on sparse point cloud of the present application can directly use the initial spatial points obtained through the step S230 as the spatial points to be confirmed for subsequent screening and calculation. In another example of the present application, as Figure 2 shown, in order to delete some error points and improve the plane expansion accuracy, the step S200 of the plane expansion method based on sparse point cloud of the present application may further include the steps of:

[0083] S240: Perform reprojection error verification on the initial spatial points to delete the spatial points with errors exceeding the threshold, and use the remaining initial spatial points as the spatial points to be confirmed.

[0084] Exemplarily, in the step S240 of the plane expansion method based on sparse point cloud of the present application, the spatial points in the left camera coordinate system are projected into the right camera coordinate system and compared with the observed coordinates tracked by the right camera, and the points with errors exceeding the threshold are deleted, so as to complete the reprojection error verification to further screen out the error points.

[0085] According to the above embodiments of the present application, as Figure 3 shown, the step S300 of the plane expansion method based on sparse point cloud of the present application may include the steps of:

[0086] S310: According to the spatial distance between the spatial points to be confirmed and the current plane, delete the spatial points to be confirmed with a spatial distance greater than the first distance threshold to obtain the 3D screened spatial points;

[0087] S320: Project the points that are tracked on the current frame image and are not in the current plane on the previous frame image onto the current frame image through the homography matrix to solve the planar distance between the projected points and the corresponding observed points, and delete the spatial projected points whose planar distance is greater than the second distance threshold to obtain the 2D selected spatial points; and

[0088] S330: Perform clustering processing based on the gray - level differences and distances between all existing spatial points in the current plane, the 3D selected spatial points, and the 2D selected spatial points and the center point of the current plane, so as to eliminate the points that do not belong to the same class as the center point, and obtain the spatial points to be updated that belong to the current plane.

[0089] Preferably, the homography matrix is calculated through the points that are tracked on the current frame image and are in the current plane on the previous frame image. It can be understood that if the points in the physical space plane are observed simultaneously at different times, these points satisfy a 3×3 matrix constraint relationship.

[0090] It should be noted that step S300 of the planar expansion method based on sparse point cloud in this application mainly uses 3D information to select the points that belong to the current plane from a series of landmark points obtained by triangulation, and uses 2D information to select all the points that belong to the current plane, including the selection based on the homography matrix and the selection based on feature clustering. For example, in the specific implementation process, first, for the points that are tracked in the current frame image and are not in the current plane on the previous frame image, calculate the homography matrix according to the points that are tracked in the current frame image and are in the current plane on the previous frame image, and then use the homography matrix to project these points onto the current frame image. If the distance between the projected point and its observed point in the current frame image exceeds a certain threshold, it is considered that the point does not belong to the current plane and is eliminated, otherwise it is considered that the point belongs to the current plane and is retained. Secondly, after completing the 3D selection and the homography matrix selection, all the points that belong to the current plane are clustered according to their gray - level differences and distances from the plane center point, and the plane points that do not belong to the same class as the center point are eliminated, and the remaining points are used as the spatial points to be updated. It can be understood that the planar expansion method based on sparse point cloud in this application uses the 3D information generated by the distance from the spatial point to the plane and the 2D information generated by the homography matrix calculated from the points in the planes of the front and rear frames to jointly constrain the candidate points in the geometric space, and can obtain a more accurate inlier judgment. In addition, for those spatial points that cannot be eliminated in the geometric space, the color and distance features of these sparse points are used for further selection to reduce the misjudgment of plane points caused by factors such as incorrect triangulation results.

[0091] In the above embodiments of the present application, the step S400 of the method for expanding a plane based on a sparse point cloud is mainly used to update the parameters of the current plane in the current frame image according to the to-be-updated spatial points. Generally speaking, the plane equation can be expressed in the form of ax + by + cz + d = 0. By solving the plane equation, the parameters of the updated plane can be obtained (including the plane normal vector, the center point, the singular value ratio, the covariance, the sum of the singular values after the covariance SVD decomposition, the minimum singular value, the median singular value, the maximum singular value, and the angle between the normal vector and the horizontal normal vector in the inertial coordinate system), so as to realize the expansion of the plane.

[0092] It is worth mentioning that the plane expansion involved in the present application is a sub-module in the SLAM framework, which is mainly responsible for updating the initial plane coordinate system as the physical space changes. For example, the relationship between the plane expansion sub-module of the present application and the SLAM framework is as Figure 5 shown, and the SLAM framework mainly involves two main modules, SLAM and plane detection. The plane expansion scheme is to calculate the current sparse point cloud through the initial plane information given by the first plane detection and the key frame poses obtained by SLAM at each moment in plane detection, so as to realize the update of the plane coordinate system.

[0093] It should be noted that although the above embodiments of the present application take the binocular vision inertial navigation SLAM framework (i.e., the sensor is mainly a binocular camera) as an example to illustrate the advantages and features of the method for expanding a plane based on a sparse point cloud of the present application, it can be combined with other non-visual sensors such as imu to achieve the positioning and tracking effects. In particular, the applicable scope of the method for expanding a plane based on a sparse point cloud of the present application is not limited to this, and it can also be used in a monocular inertial navigation SLAM framework, etc.

[0094] Schematic system

[0095] Referring to Figure 6 as shown in the accompanying drawings of the specification, a plane expansion system based on a sparse point cloud according to an embodiment of the present invention is illustrated. Specifically, as Figure 6As shown, the planar expansion system 1 based on sparse point clouds may include components communicatively connected to each other: a local ROI extraction module 10, configured to extract a local region of interest on the current frame image according to the geometric center of the points projected on the current frame image by all the existing spatial points within the current plane in the previous frame image; a spatial point extraction module 20, configured to perform spatial point extraction on the current frame image according to the local region of interest to obtain spatial points to be confirmed; a spatial point screening module 30, configured to screen all the existing spatial points and the spatial points to be confirmed within the current plane to obtain the spatial points to be updated belonging to the current plane; and a planar expansion module 40, configured to recalculate the planar information of the current plane based on the spatial points to be updated to obtain a newly expanded plane.

[0096] It should be noted that the local region of interest is a rectangular region centered at the geometric center and with a size smaller than the current frame image.

[0097] In an example of the present application, as Figure 6 shown, the spatial point extraction module 20 includes a feature point extraction module 21, a feature point matching module 22, and a triangulation processing module 23 communicatively connected to each other. The feature point extraction module 21 is configured to perform corner detection on the local region of interest of the current frame image to extract sparse feature points from the local region of interest. The feature point matching module 22 is configured to solve for the feature points matched on the right-eye image of the feature points on the left-eye image by the left-right camera optical flow tracking method to obtain a pair of matched left-right eye feature points. The triangulation processing module 23 is configured to perform triangulation calculation on the pair of left-right eye feature points according to the normalized coordinates and relative poses corresponding to the left and right cameras to solve for the spatial coordinates of the waypoint as the initial spatial points.

[0098] In an example of the present application, as Figure 6 shown, the spatial point extraction module 20 further includes a reprojection error verification module 24, configured to perform reprojection error verification on the initial spatial points to delete the spatial points with errors exceeding the threshold and use the remaining initial spatial points as the spatial points to be confirmed.

[0099] In an example of the present application, as Figure 6As shown, the spatial point selection module 30 includes a 3D selection module 31, a 2D selection module 32, and a clustering selection module 33 that are communicatively connected to each other. The 3D selection module 31 is configured to delete the to-be-confirmed spatial points whose spatial distance from the current plane is greater than a first distance threshold based on the spatial distance between the to-be-confirmed spatial points and the current plane, so as to obtain the 3D selected spatial points. The 2D selection module 32 is configured to project the points that are tracked in the current frame image and do not belong to the current plane in the previous frame image onto the current frame image through a homography matrix to solve the plane distance between the projected points and the corresponding observed points, and delete the projected points whose plane distance is greater than a second distance threshold, so as to obtain the 2D selected spatial points. The clustering selection module 33 is configured to perform clustering processing based on the grayscale difference and distance between all the existing spatial points, the 3D selected spatial points, and the 2D selected spatial points in the current plane and the center point of the current plane, so as to eliminate the points that do not belong to the same class as the center point and obtain the to-be-updated spatial points belonging to the current plane.

[0100] Schematic electronic device

[0101] Next, refer to Figure 7 to describe an electronic device according to an embodiment of the present invention. As Figure 7 shown, the electronic device 90 includes one or more processors 91 and a memory 92.

[0102] The processor 91 may be a central processing unit (CPU) or other form of processing unit having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 90 to perform desired functions. In other words, the processor 91 includes one or more physical devices configured to execute instructions. For example, the processor 91 may be configured to execute instructions that are part of the following: one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, transform the state of one or more components, achieve technical effects, or otherwise obtain desired results.

[0103] The processor 91 may include one or more processors configured to execute software instructions. As a supplement or alternative, the processor 91 may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. The processors of the processor 91 may be single-core or multi-core, and the instructions executed thereon may be configured for serial, parallel, and / or distributed processing. The various components of the processor 91 may optionally be distributed over two or more separate devices, which may be located remotely and / or configured for cooperative processing. Aspects of the processor 91 may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration.

[0104] The memory 92 may include one or more computing program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computing program instructions may be stored on the computer-readable storage media, and the processor 11 may run the program instructions to implement some or all of the steps of the above-described illustrative method of the present invention, as well as / or other desired functions.

[0105] In other words, the memory 92 includes one or more physical devices configured to store machine-readable instructions executable by the processor 91 to implement the methods and processes described herein. When implementing these methods and processes, the state of the memory 92 may be transformed (e.g., storing different data). The memory 92 may include removable and / or built-in devices. The memory 92 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-ray Disc, etc.), semiconductor memory (e.g., RAM, EPROM, EEPROM, etc.), and / or magnetic memory (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), and so on. The memory 92 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location-addressable, file-addressable, and / or content-addressable devices.

[0106] It will be appreciated that the memory 92 includes one or more physical devices. However, aspects of the instructions described herein may alternatively be propagated by a communication medium (e.g., electromagnetic signals, optical signals, etc.) that is not held by a physical device for a finite duration. Aspects of the processor 91 and the memory 92 may be integrated together into one or more hardware logic components. These hardware logic components may include, for example, field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASIC / ASICs), program and application specific standard products (PSSP / ASSPs), systems on a chip (SOCs), and complex programmable logic devices (CPLDs).

[0107] In one example, as Figure 7 shown, the electronic device 90 may further include an input device 93 and an output device 94, which are interconnected by a bus system and / or other forms of connection means (not shown). For example, the input device 93 may be, for example, a camera module for collecting image data or video data, etc. Again, the input device 93 may include one or more user input devices such as a keyboard, a mouse, a touch screen, or a game controller, or be docked therewith. In some embodiments, the input device 93 may include or be docked with a selected natural user input (NUI) component. Such an element may be integrated or peripheral, and the transduction and / or processing of the input action may be processed on-board or off-board. Example NUI components may include a microphone for speech and / or voice recognition; an infrared, color, stereo display, and / or depth camera for machine vision and / or gesture recognition; a head tracker, an eye tracker, an accelerometer, and / or a gyroscope for motion detection and / or intent recognition; and an electric field sensing component for evaluating brain activity and / or body movement; and / or any other suitable sensor.

[0108] The output device 94 may output various information to the outside, including classification results, etc. The output device 94 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0109] Of course, the electronic device 90 may further include the communication device, where the communication device may be configured to communicatively couple the electronic device 90 with one or more other computer devices. The communication device may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem may be configured to communicate via a wireless telephone network or a wired or wireless local area network or wide area network. In some embodiments, the communication device may allow the electronic device 90 to send messages to other devices and / or receive messages from other devices via a network such as the Internet.

[0110] It will be understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples should not be considered restrictive as many variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, the various acts shown and / or described may be performed in the shown and / or described order, in other orders, in parallel, or omitted. Similarly, the order of the above processes may be changed.

[0111] Of course, for simplicity, Figure 7 only some of the components of the electronic device 90 related to the present invention are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, depending on the specific application, the electronic device 90 may further include any other appropriate components.

[0112] It should also be noted that in the devices, equipment and methods of the present invention, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent solutions of the present invention.

[0113] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.

[0114] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the drawings are only examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functions and structural principles of the present invention have been demonstrated and illustrated in the embodiments, and without departing from the said principles, the embodiments of the present invention can have any deformation or modification.

Claims

1. A plane expansion method based on sparse point cloud. It is characterized in that Includes steps: Selecting a local region of interest on the current frame image according to the geometric centers of the points projected on the current frame image by all existing spatial points in the previous frame image and belonging to the current plane; Extracting spatial points from the current frame image according to the local region of interest to obtain spatial points to be confirmed; All existing spatial points and the spatial points to be confirmed in the current plane are deleted to obtain the spatial points to be updated that belong to the current plane; as well as The plane information of the current plane is recalculated based on the spatial point to be updated to obtain a new extended plane.

2. The plane expansion method based on sparse point cloud as claimed in claim 1, in, The local region of interest is a rectangular region centered on the geometric center and smaller in size than the current frame image.

3. The plane expansion method based on sparse point cloud as claimed in claim 2, in, The step of extracting spatial points from the current frame image according to the local region of interest to obtain spatial points to be confirmed comprises the steps of: Performing corner point detection on the local region of interest of the current frame image to extract sparse feature points from the local region of interest; By using the left and right camera optical flow tracking method, the feature points on the left eye image are solved to match the feature points on the right eye image, so as to obtain the matching left and right eye feature point pairs; as well as According to the normalized coordinates and relative poses corresponding to the left and right cameras, the left and right feature point pairs are triangulated to solve the spatial coordinates of the landmark points as the initial spatial points.

4. The plane expansion method based on sparse point cloud as claimed in claim 3, in, The step of extracting spatial points from the current frame image according to the local region of interest to obtain spatial points to be confirmed further comprises the steps of: The reprojection error verification is performed on the initial spatial points to delete the spatial points whose errors exceed a threshold, and the remaining initial spatial points are used as the spatial points to be confirmed.

5. The plane expansion method based on sparse point cloud as claimed in any one of claims 1 to 4, in, The step of deleting all existing spatial points and the spatial points to be confirmed in the current plane to obtain the spatial points to be updated belonging to the current plane comprises the steps of: According to the spatial distance between the spatial point to be confirmed and the current plane, the spatial point to be confirmed whose spatial distance is greater than a first distance threshold is deleted to obtain the spatial point after 3D selection; Projecting the points tracked on the current frame image and not belonging to the current plane on the previous frame image to the current frame image through the homography matrix to solve the plane distance between the projection point and the corresponding observation point, and deleting the projection point whose plane distance is greater than the second distance threshold to obtain the 2D selected space point; as well as Clustering is performed based on the grayscale difference and distance between all the existing spatial points in the current plane, the spatial points after 3D selection, and the spatial points after 2D selection and the center point of the current plane to eliminate points that do not belong to the same class as the center point, and obtain the spatial points to be updated that belong to the current plane.

6. The plane expansion method based on sparse point cloud according to claim 5, wherein, the homography matrix is calculated by the points tracked on the current frame image that belong to the current plane on the previous frame image.

7. A plane expansion system based on sparse point cloud, characterized in that it includes components communicatively connected to each other: a local ROI extraction module, configured to extract a local region of interest on the current frame image according to the geometric center of the points projected on the current frame image by all the existing spatial points belonging to the current plane in the previous frame image; a spatial point extraction module, configured to perform spatial point extraction on the current frame image according to the local region of interest to obtain spatial points to be confirmed; a spatial point screening module, configured to screen all the existing spatial points in the current plane and the spatial points to be confirmed to obtain the spatial points to be updated belonging to the current plane; and a plane expansion module, configured to recalculate the plane information of the current plane based on the spatial points to be updated to obtain a newly expanded plane.

8. The plane expansion system based on sparse point cloud according to claim 7, wherein, the local region of interest is a rectangular region centered at the geometric center and with a size smaller than the current frame image.

9. The plane expansion system based on sparse point cloud according to claim 8, wherein, the spatial point extraction module includes a feature point extraction module, a feature point matching module, and a triangulation processing module communicatively connected to each other. The feature point extraction module is configured to perform corner detection on the local region of interest of the current frame image to extract sparse feature points from the local region of interest; the feature point matching module is configured to solve the feature points matched on the right-eye image for the feature points on the left-eye image by the left-right camera optical flow tracking method to obtain the matched left-right eye feature point pairs; the triangulation processing module is configured to perform triangulation calculation on the left-right eye feature point pairs according to the normalized coordinates and relative poses corresponding to the left and right cameras to solve the spatial coordinates of the waypoints as the initial spatial points.

10. The plane expansion system based on sparse point cloud according to claim 9, wherein, the spatial point extraction module further includes a reprojection error verification module, configured to perform reprojection error verification on the initial spatial points to delete the spatial points with errors exceeding the threshold, and use the remaining initial spatial points as the spatial points to be confirmed.

11. The plane expansion system based on sparse point cloud according to any one of claims 7 to 10, wherein, the spatial point screening module includes a 3D screening module, a 2D screening module, and a clustering screening module communicatively connected to each other. The 3D screening module is configured to delete the spatial points to be confirmed with a spatial distance greater than the first distance threshold according to the spatial distance between the spatial points to be confirmed and the current plane to obtain the spatially screened spatial points; ​ The 2D screening module is configured to project, by means of a homography matrix, points that are tracked on the current frame image and that are not within the current plane on the previous frame image onto the current frame image to solve the planar distance between the projected points and the corresponding observed points, and delete the projected points with a planar distance greater than a second distance threshold to obtain the spatial points after 2D screening; the clustering and screening module is configured to perform clustering processing based on the gray difference and distance between all the existing spatial points within the current plane, the spatial points after 3D screening, and the spatial points after 2D screening and the center point of the current plane, so as to remove the points that do not belong to the same class as the center point and obtain the spatial points to be updated that belong to the current plane.

12. An electronic device, characterized in that it includes: at least one processor for executing instructions; and a memory communicatively connected to the at least one processor, wherein the memory has at least one instruction, and the instruction is executed by the at least one processor to cause the at least one processor to execute all steps in the planar expansion method based on sparse point clouds, and the planar expansion method based on sparse point clouds includes the steps of: selecting a local region of interest on the current frame image according to the geometric center of the points projected on the current frame image by all the existing spatial points within the current plane in the previous frame image; performing spatial point extraction on the current frame image according to the local region of interest to obtain the spatial points to be confirmed; screening all the existing spatial points within the current plane and the spatial points to be confirmed to obtain the spatial points to be updated that belong to the current plane; and recalculating the planar information of the current plane based on the spatial points to be updated to obtain a newly expanded plane.

Images