A method for automatic recognition and precise positioning of safety boundaries, planes, and obstacles

Abstract

The present invention discloses a method for automatically identifying and precisely positioning a safety boundary, plane, and obstacle, comprising the following steps: S1, the backend main module calls a plane function and interacts with a sparse plane module; S2, the backend main module activates the relocalization detection and interacts with the loop detection module, and outputs environmental point cloud and obstacle recognition information externally; S3, after the relocalization detection is activated, the loop detection module determines loop key frames; S4, the sparse plane module is updated according to whether there is new point cloud data, and outputs plane recognition information externally; S5, a safety boundary used by a user is determined according to plane recognition and environmental point cloud. Through a framework based on the automatic identification of safety boundaries, planes, and obstacles, the present invention can be effectively used to automatically identify safety boundaries in virtual reality experiences, detect horizontal and vertical planes in the environment where the user is located, and identify obstacles in the environment, improving the user experience performance and immersion feeling.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and particularly to a method for automatically identifying and precisely positioning a safety boundary, a plane, and an obstacle. Background Art

[0002] With the development of computer vision, virtual reality (VR) and augmented reality (AR) glasses, robotic arms, smartphones, and robots can all serve as terminals to provide users with computing resources for interaction and positioning. With the rise of the metaverse, users need to better perceive the surrounding world during game experiences or interactive entertainment. At the same time, simultaneous localization and mapping (SLAM) or visual odometry (VO) algorithms can position devices, but they cannot identify or optimize planes to give users a better real experience. In virtual reality experiences, when users connect virtual reality and augmented reality in see-through mode, they need to manually define a safety boundary and identify the ground plane. This method cannot identify obstacles in the environment, and there are often situations where the edges of the recognized objects bounce up and down, the obstacles do not fit the displayed point cloud information, the ground plane bounces up and down, or the ground plane flies away during the experience, greatly affecting the user experience. Traditional SLAM methods mainly observe point features, line features, and plane features during the optimization solution, but the extraction of plane features is very time-consuming during the tracking process, greatly increasing the power consumption of the system. Anchor points are usually used to assist in positioning by setting fixed objects, but during the user experience, human body occlusion, dynamic objects, and the field of view will block the information of the anchor points, making it impossible to perform efficient assisted positioning. Therefore, we propose a method for automatically identifying and precisely positioning a safety boundary, a plane, and an obstacle. Summary of the Invention

[0003] The purpose of the present invention is to provide a method for automatically identifying and precisely positioning a safety boundary, a plane, and an obstacle to solve the problems raised in the above background art.

[0004] To achieve the above purpose, the present invention provides the following technical solution: A method for automatically identifying and precisely positioning a safety boundary, a plane, and an obstacle, including the following steps:

[0005] S1. The backend main module calls the plane function and interacts with the sparse plane module;

[0006] S2. The backend main module activates the relocalization detection and interacts with the loop detection module, and outputs environmental point cloud and obstacle recognition information externally;

[0007] S3. After the loop detection module activates the relocalization detection, it determines the loop key frames;

[0008] S4. The sparse plane module updates according to whether there is new point cloud data and outputs plane recognition information externally;

[0009] S5. Determine the safety boundary used by the user based on plane recognition and environmental point cloud, and continuously update the safety boundary.

[0010] Preferably, the specific implementation methods of S1 and S2 are as follows: The backend main thread obtains new key frames from the input interface, then selects candidate key frames. After passing through the candidate key frames, preprocessing, preprocessing sliding window data, updating the pose in the sliding window, local map matching, adding matching observations, merging map points, updating the map point discovery rate, map point culling, and updating the reference frame are performed in sequence. Then, it enters the judgment of whether a new frame is needed. After judging that it is a new frame, inserting a new frame into the map, updating the convergence state of the visual inertial odometer, creating new map points, and performing local BA (bundle adjustment) optimization are carried out. Then, the key frame culling judgment is performed. After passing the key frame culling judgment, the plane function and the activation of relocalization judgment are called, and the information is passed to the loop detection module and the sparse plane module. Then, pose processing and the backend call the callback function, and the pose and the point cloud map are output externally.

[0011] Preferably, the specific implementation method of S3 is as follows: After the loop detection module is activated by the backend main thread for relocalization detection, it judges the loop new key frames. If they are new key frames, it enters the loop judgment module. The loop judgment module inserts new key frames into the database and indexes images in the loop after starting, then obtains the loop detection results and processes the inlier loop, and performs the inlier loop judgment. If the inlier loop judgment is passed, the loop detection is corrected; otherwise, the untracked loop is directly processed. In the case of a loop, the pose information can be effectively updated, and when the user returns to the origin after running for a period of time, the error can be effectively reduced, improving the user experience performance.

[0012] Preferably, the specific implementation method of S4 is as follows: After the sparse plane module is called by the backend main thread for the plane function, it judges whether there is new point cloud data, then processes the map obtained by the backend main thread, creates map planes according to the map, obtains triangulated points and creates a 3D mesh (mesh), then clusters planes through the mesh, separates all plane information in the environment where the user is located from the mesh, then updates the connection index inside all planes through the mesh, redefines the planes and associates with the old planes, and then outputs the recognized plane information in the environment where the user is located.

[0013] Preferably, the specific implementation method of S5 is as follows: after the sparse plane module outputs the horizontal plane and the vertical plane, the user can select one or more planes as safety boundaries according to the plane information. At this time, in a hierarchical environment, the user can experience in a multi-story environment. Then, the system feeds back the selected plane as an anchor to the framework for automatic recognition of safety boundaries, planes, and obstacles. Then, the best anchor is selected as the reference of the world coordinate system to switch the reference of the world coordinate system determined in the initial state to the selected anchor. Then, the map coordinates and pose coordinates are updated. Then, all the selected planes are added to the state estimation observation to improve the accuracy of pose optimization and output a more accurate high-frequency pose.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0015] 1. The present invention can be effectively used in virtual reality experiences to automatically recognize safety boundaries, detect horizontal and vertical planes in the user's environment, and identify obstacles in the environment, improving the user experience performance and immersion.

[0016] 2. The present invention creates point cloud information, establishes a local map, and performs pose optimization in the main thread through the method of sparse point cloud map and plane optimization. In the loop detection thread, it can identify the positions or repeated areas that have been walked through to further optimize the loop area and improve the quality of the sparse point cloud map. The plane thread performs the establishment, association, and optimization of planes.

[0017] 3. The sparse point cloud map maintained by the present invention can identify the edges of obstacles and objects within the current field of view in perspective mode, enhancing the user's ability to perceive the surrounding world.

[0018] 4. The plane optimization method proposed by the present invention can identify horizontal and vertical planes, can help users identify the ground, desktop, and wall surface, further enhancing the user's ability to perceive the surrounding environment. In addition, it can expand the user experience environment from the original local environment to a multi-story environment, improving the user experience performance.

[0019] 5. The present invention can use the plane selected by the user as an anchor to switch the world coordinates from the initial position to the anchor position, using the plane where the anchor is located as the reference for user experience and positioning to improve the stability of the user experience.

[0020] 6. The present invention can add the optimized stable plane and anchor as the result of observation to the state optimization to improve the positioning accuracy of the SLAM system, further enhancing the stability and robustness of the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a framework diagram for automatic recognition of safety boundaries, planes, and obstacles of the present invention;

[0022] Figure 2 This is the flowchart of the main backend module of the present invention;

[0023] Figure 3 This is the flowchart of the loop detection module of the present invention;

[0024] Figure 4 This is the flowchart of the sparse plane module of the present invention;

[0025] Figure 5 This is the flowchart for improving the stability performance of the user experience of the plane in the present invention. Specific embodiments

[0026] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0027] Please refer to Figures 1-5 , a method for automatic identification and precise positioning of safety boundaries, planes, and obstacles, including the following steps:

[0028] S1. The main backend module calls the plane function and interacts with the sparse plane module;

[0029] S2. The main backend module activates the relocalization detection and interacts with the loop detection module, and outputs the environmental point cloud and obstacle recognition information externally;

[0030] S3. After the relocalization detection is activated, the loop detection module judges the loop key frames;

[0031] S4. The sparse plane module is updated according to whether there is new point cloud data, and outputs the plane recognition information externally;

[0032] S5. Determine the safety boundary used by the user according to the plane recognition and the environmental point cloud, and continuously update the safety boundary.

[0033] The specific implementation methods of S1 and S2 are as follows: The backend main thread obtains new key frames from the input interface, then selects candidate key frames. After passing through the candidate key frames, it performs preprocessing, preprocessing sliding window data, updating the pose in the sliding window, local map matching, adding matching observations, merging map points, updating the discovery rate of map points, removing map points and updating the reference frame, and then enters the judgment of whether a new frame is needed. After judging that it is a new frame, it inserts a new frame into the map, updates the convergence state of the visual inertial odometer, creates new map points and performs local BA (bundle adjustment) optimization, and then performs the judgment of key frame removal. After passing the key frame removal judgment, it calls the plane function and activates the relocalization judgment, transmits the information to the loop closure detection module and the sparse plane module, and then performs pose processing and the backend calls the callback function, and outputs the pose and the point cloud map externally.

[0034] The specific implementation method of S3 is as follows: After the loop closure detection module is activated by the backend main thread for relocalization detection, it judges the loop closure new key frame. If it is a new key frame, it enters the loop closure judgment module. The loop closure judgment module inserts a new key frame into the database and indexes the image in the loop closure after starting, and then obtains the loop closure detection result and processes the inlier loop closure. It performs the inlier loop closure judgment. If it passes the inlier loop closure judgment, it corrects the loop closure detection, otherwise it directly processes the un-tracked loop closure. In the case of a loop closure, it can effectively update the pose information, and when the user returns to the origin after running for a period of time, it can effectively reduce the error and improve the user experience performance.

[0035] The specific implementation method of S4 is as follows: After the sparse plane module is called by the backend main thread for the plane function, it judges whether there is new point cloud data, and then processes the map obtained by the backend main thread. It creates map planes according to the map, obtains the triangulated points and creates a 3D mesh (mesh), and then clusters the planes through the mesh, separates all the plane information in the environment where the user is located from the mesh, and then updates the connection index inside all the planes through the mesh, re-sets the planes and associates the old planes, and then outputs the recognized plane information in the environment where the user is located.

[0036] The specific implementation method of S5 is as follows: After the sparse plane module outputs the horizontal plane and the vertical plane, the user can select one or more planes as safety boundaries according to the plane information. At this time, in a hierarchical environment, the user can experience in a multi-story building environment. Then the system feeds back the selected plane as an anchor to the framework of automatic recognition of safety boundaries, planes and obstacles. Then it selects the best anchor as the world coordinate system benchmark and switches the world coordinate system benchmark determined in the initial state to the selected anchor, and then updates the map coordinates and pose coordinates. Then it adds all the selected planes to the state estimation observations, improves the accuracy of pose optimization, and outputs a more accurate high-frequency pose.

[0037] The present invention can be effectively used to automatically identify the safety boundary in virtual reality experiences, detect the horizontal and vertical planes in the environment where the user is located, and identify obstacles in the environment, improving the user experience performance and immersion. By means of the sparse point cloud map and plane optimization method, point cloud information is created, a local map is established, and pose optimization is carried out in the main thread. In the loop detection thread, the positions that have been walked through or repeated areas can be identified to further optimize the loop area and improve the quality of the sparse point cloud map. The plane thread performs the establishment, association, and optimization of planes. The maintained sparse point cloud map can identify the obstacles and the edges of objects within the current field of view in perspective mode, enhancing the user's ability to perceive the surrounding world. The proposed plane optimization method can identify the horizontal and vertical planes, and can help the user identify the ground, desktop, and wall surface, further enhancing the user's ability to perceive the surrounding environment. In addition, it can expand the user experience environment from the original local environment to a multi-story environment, improving the user experience performance. It can use the plane selected by the user as an anchor point to switch the world coordinates from the initial position to the anchor point position, and use the plane where the anchor point is located as the reference for user experience and positioning to improve the stability performance of the user experience. It can add the optimized stable plane and anchor point as the observation results to the state optimization to improve the positioning accuracy of the SLAM system, further enhancing the stability and robustness of the user experience.

[0038] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for automatically identifying and precisely positioning a safety boundary, plane, and obstacle, characterized in that: It includes the following steps: S1. The backend main module calls the plane function and interacts with the sparse plane module; S2. The backend main module activates the relocalization detection and interacts with the loop closure detection module, and outputs the environmental point cloud and obstacle recognition information externally; S3. After the relocalization detection is activated, the loop closure detection module determines the loop closure key frames; S4. The sparse plane module is updated according to whether there is new point cloud data, and outputs the plane recognition information externally; S5. Determine the safety boundary used by the user based on the plane recognition information and the environmental point cloud, and continuously update the safety boundary; The specific implementation methods of S1 and S2 are as follows: The backend main thread obtains a new key frame from the input interface, then selects candidate key frames. After passing the candidate key frames, preprocessing, preprocessing sliding window data, updating the pose in the sliding window, local map matching, adding matching observations, merging map points, updating the map point discovery rate, map point culling, and updating the reference frame are performed in sequence. Then, it enters the judgment of whether a new frame is needed. After judging that it is a new frame, a new frame is inserted into the map, the convergence state of the visual inertial odometer is updated, new map points are created, and local BA (bundle adjustment) optimization is performed. Then, the key frame culling judgment is carried out. After passing the key frame culling judgment, the plane function is called and the relocalization judgment is activated, and the information is transmitted to the loop closure detection module and the sparse plane module. Then, pose processing and the backend call the callback function, and the pose and the point cloud map are output externally; The specific implementation method of S3 is as follows: After the loop closure detection module is activated by the backend main thread for relocalization detection, it judges the loop closure new key frames. If it is a new key frame, it enters the loop closure judgment module. After the loop closure judgment module starts, it inserts a new key frame into the database and indexes the images in the loop closure, then obtains the loop closure detection result and processes the inlier loop closure, and performs the inlier loop closure judgment. If the inlier loop closure judgment is passed, the loop closure detection is corrected, otherwise, the un-tracked loop closure is directly processed; The specific implementation method of S4 is as follows: After the sparse plane module is called by the backend main thread for the plane function, it judges whether there is new point cloud data, and then processes the map obtained by the backend main thread. Creates a map plane according to the map, obtains the triangulated points and creates a 3D mesh. Then, clusters the planes through the mesh, and segments all the plane information in the environment where the user is located from the mesh. Then, updates the connection index inside all the planes through the mesh, re-sets the planes and associates the old planes, and finally outputs the recognized plane information in the environment where the user is located; The specific implementation method of S5 is as follows: After the sparse plane module outputs the horizontal plane and the vertical plane, the user selects one or more planes as the safety boundary according to the plane information. At this time, in a hierarchical environment, the user can experience in a multi-story building environment. Then the system feeds back the selected plane as an anchor point to the framework for automatic recognition of safety boundaries, planes, and obstacles. Then the best anchor point is selected as the world coordinate system benchmark to switch the world coordinate system benchmark determined in the initial state to the selected anchor point. Then the map coordinates and pose coordinates are updated. Then all the selected planes are added to the state estimation observation to improve the accuracy of pose optimization and output a more accurate high-frequency pose.

Images