Obstacle detection method and VR device

Abstract

The invention provides an obstacle detection method and VR equipment. The method is used for improving obstacle detection efficiency. Comprising the following steps: on the basis of real-time point cloud data of a current frame image, updating obstacle confidence of each voxel grid divided in an obstacle detection area; performing plane detection on the obstacle detection area by using the global point cloud data to obtain a plane in the obstacle detection area; performing convex hull detection on the plane to obtain boundary points of the plane; for any boundary point in any plane, determining a voxel grid where the boundary point is located based on the 3D position coordinates of the boundary point, and determining whether the boundary point is an obstacle vertex or not according to the obstacle confidence after updating of the voxel grid; if the number of the boundary points which are the vertex of the obstacle in the plane is greater than a specified number, determining the position of the obstacle in the plane according to the 3D position coordinates of each boundary point and the voxel grid where each boundary point is located; and obtaining the obstacle in the obstacle detection area through the obstacle position in each plane.

Description

Background Art

[0002] Existing 3D (three dimensional) obstacle detection solutions are mainly concentrated in the field of autonomous driving. For example, obstacle avoidance trajectory planning for unmanned vehicles and drones. Such solutions are mainly divided into two types: one is to restore depth using depth sensors such as lidar, and the other is to restore depth using stereo vision using multiple cameras.

[0003] However, due to factors such as high cost, depth sensors related to LiDAR are not currently applicable to the field of VR (Virtual Reality). Although stereoscopic vision solutions can be applied to obstacle detection in the VR field, 3D obstacle detection solutions of stereoscopic vision generally use neural networks for depth detection. Such solutions generally require at least two cameras to observe at similar horizontal viewing angles to facilitate epipolar alignment and have a large common viewing area. However, in the use scenario of VR, different cameras in VR headsets often have different viewing angles. For example, two cameras with a rotation angle difference of 90°. And the camera common viewing area on the VR headset is also much smaller than that of the vehicle-mounted equipment. If you want to use neural network obstacle detection on VR headsets, not only does it cost a lot to annotate the depth image data set, but once the lens structure changes, the original neural network model may no longer be applicable and needs to be retrained with new data. Therefore, the portability of such solutions is very poor, resulting in low efficiency in 3D obstacle detection. Therefore, a new obstacle detection method is urgently needed. Summary of the invention

[0004] The present application provides an obstacle detection method and VR device for detecting 3D obstacles in a VR scene. The method is independent of large-scale, high-quality annotation data, has strong versatility and portability, and improves the detection efficiency of 3D obstacles.

[0005] In a first aspect, an embodiment of the present application provides a method for detecting an obstacle, which is applied to a VR device. The method includes:

[0006] Based on the real-time point cloud data corresponding to the current frame image obtained, the obstacle confidence of each voxel grid divided in the obstacle detection area is updated to obtain the updated obstacle confidence of each voxel grid, wherein the obstacle confidence of any voxel grid is the probability that an obstacle exists in the voxel grid;

[0007] Performing plane detection on the obstacle detection area using global point cloud data to obtain at least one plane in the obstacle detection area, wherein any one plane is a horizontal plane or a vertical plane, and the global point cloud data includes real-time point cloud data corresponding to each frame image from the time when the VR device is turned on to the current time;

[0008] Performing convex hull detection on the at least one plane to obtain boundary points of the at least one plane;

[0009] For any boundary point in any plane, determine the voxel grid where the boundary point is located based on the 3D position coordinates of the boundary point, and determine whether the boundary point is an obstacle vertex according to the obstacle confidence after the voxel grid is updated; and

[0010] If the number of boundary points of the plane that are vertices of obstacles is greater than a specified number, determining the position of the obstacles in the plane according to the 3D position coordinates of the boundary points and the voxel grids where the boundary points are located;

[0011] The position of the obstacle in the obstacle detection area is obtained through the position of the obstacle in each plane.

[0012] A second aspect of the present application provides a VR device, including a processor and a memory, wherein the processor and the memory are connected via a bus;

[0013] The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program:

[0014] Based on the real-time point cloud data corresponding to the current frame image obtained, the obstacle confidence of each voxel grid divided in the obstacle detection area is updated to obtain the updated obstacle confidence of each voxel grid, wherein the obstacle confidence of any voxel grid is the probability that an obstacle exists in the voxel grid;

[0015] Performing plane detection on the obstacle detection area using global point cloud data to obtain at least one plane in the obstacle detection area, wherein any one plane is a horizontal plane or a vertical plane, and the global point cloud data includes real-time point cloud data corresponding to each frame image from the time when the VR device is turned on to the current time;

[0016] Performing convex hull detection on the at least one plane to obtain boundary points of the at least one plane;

[0017] For any boundary point in any plane, determine the voxel grid where the boundary point is located based on the 3D position coordinates of the boundary point, and determine whether the boundary point is an obstacle vertex according to the obstacle confidence after the voxel grid is updated; and

[0018] If the number of boundary points of the plane that are vertices of obstacles is greater than a specified number, determining the position of the obstacles in the plane according to the 3D position coordinates of the boundary points and the voxel grids where the boundary points are located;

[0019] The position of the obstacle in the obstacle detection area is obtained through the position of the obstacle in each plane.

[0020] According to a third aspect provided by an embodiment of the present invention, a computer storage medium is provided, wherein the computer storage medium stores a computer program, and the computer program is used to execute the method as described in the first aspect.

[0021] In the above embodiment of the present application, based on the real-time point cloud data corresponding to the current frame image acquired, the obstacle confidence of each voxel grid divided in the obstacle detection area is updated, and the obstacle detection area is plane-detected using the global point cloud data to obtain the plane in the obstacle detection area, and then the updated obstacle confidence of the voxel grid where each boundary point in the plane is located is used to determine whether there is an obstacle in the plane, and if so, the obstacle position in the plane is determined according to the 3D position coordinates of each boundary point and the voxel grid where each boundary point is located, and finally the obstacle position in the obstacle detection area is obtained through the obstacle position in each plane. Therefore, the 3D obstacle detection in the embodiment of the present application is performed by combining the real-time point cloud data with the global point cloud data, which does not rely on large-scale high-quality annotation data, has strong versatility and portability, and improves the detection efficiency of 3D obstacles. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0023] Figure 1 One of the flowcharts of the obstacle detection method provided in the embodiment of the present application is exemplarily shown;

[0024] Figure 2 A schematic diagram of a voxel grid provided by an embodiment of the present application is exemplarily shown;

[0025] Figure 3 The following is a schematic diagram of a process for determining an obstacle detection area provided in an embodiment of the present application;

[0026] Figure 4 A schematic diagram of a depth distribution histogram of global point cloud data provided by an embodiment of the present application is exemplarily shown;

[0027] Figure 5 The flowchart of updating the obstacle confidence of each voxel grid provided by the embodiment of the present application is exemplarily shown;

[0028] Figure 6 The following is a schematic diagram of a process for determining the position of an obstacle in a plane provided by an embodiment of the present application;

[0029] Figure 7 One of the schematic diagrams of the voxel grid obstacle provided by the embodiment of the present application is exemplarily shown;

[0030] Figure 8 The second schematic diagram of the voxel grid obstacle provided by the embodiment of the present application is exemplarily shown;

[0031] Fig. 9 The second flowchart of the obstacle detection method provided in the embodiment of the present application is exemplarily shown;

[0032] Fig.10 The following is a schematic diagram of an obstacle detection device provided in an embodiment of the present application;

[0033] Fig.11 The hardware structure diagram of the VR device provided in the embodiment of the present application is exemplified. DETAILED DESCRIPTION

[0034] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0035] Based on the exemplary embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of the disclosure can also constitute a complete implementation method separately.

[0036] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0037] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a series of components is not necessarily limited to those components clearly listed, but may include other components not clearly listed or inherent to these products or devices.

[0038] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0039] The idea of ​​the embodiments of the present application is summarized below.

[0040] Since the obstacle detection method in the VR field in the prior art mainly relies on a large amount of high-quality annotation data, the portability of the obstacle solution is poor, which reduces the detection efficiency of 3D obstacles. Therefore, the embodiment of the present application provides an obstacle detection method, which updates the obstacle confidence of each voxel grid divided in the obstacle detection area based on the real-time point cloud data corresponding to the current frame image obtained, and uses the global point cloud data to perform plane detection on the obstacle detection area to obtain the plane in the obstacle detection area, and then determines whether there is an obstacle in the plane according to the updated obstacle confidence of the voxel grid where each boundary point in the plane is located. If there is an obstacle, the obstacle position in the plane is determined according to the 3D position coordinates of each boundary point and the voxel grid where each boundary point is located. Finally, the obstacle position in the obstacle detection area is obtained by the obstacle position in each plane. Therefore, the embodiment of the present application uses real-time point cloud data combined with global point cloud data to detect 3D obstacles, does not rely on large-scale high-quality annotation data, has strong versatility and portability, and improves the detection efficiency of 3D obstacles.

[0041] Before introducing the obstacle detection method in the application in detail, the terms in the application are explained first:

[0042] Global point cloud data: Contains all real-time point cloud data generated by the VR device from the time it is turned on to the current time. The 3D feature points detected by the SLAM (Simultaneous Localization and Mapping) system in the VR device will be added to the global point cloud data and will exist in the global point cloud data for a long time until the VR device is turned off.

[0043] Real-time point cloud data: the sparse point cloud that can be observed in the current frame image, that is, the 3D feature points currently detected in the SLAM system.

[0044] The obstacle detection method in the present application is described in detail below with reference to the accompanying drawings.

[0045] like Figure 1 FIG. 1 is a flow chart of an obstacle detection method, which may include the following steps:

[0046] Step 101: Based on the acquired real-time point cloud data corresponding to the current frame image, the obstacle confidence of each voxel grid divided in the obstacle detection area is updated to obtain the updated obstacle confidence of each voxel grid, wherein the obstacle confidence of any voxel grid is the probability that an obstacle exists in the voxel grid;

[0047] The real-time point cloud data in the embodiment of the present application is obtained based on the SLAM algorithm. Determining the real-time point cloud data by the SLAM algorithm belongs to the prior art and will not be described in detail in the embodiment of the present application. In addition, the obstacle detection area is divided into voxel grids of fixed size in the embodiment of the present application. Figure 2 As shown in Figure 1, each rectangle in the obstacle detection area is a voxel grid. Figure 2 Only the voxel grid of the obstacle detection surface is shown in the figure. However, in the embodiment of the present application, not only the surface of the obstacle detection area is divided, but the entire obstacle detection area is divided, that is, the interior of the obstacle detection area is also divided into multiple voxel grids. Therefore, there are corresponding multiple voxel grids inside the obstacle detection area.

[0048] It should be noted that the division of the voxel grid in the embodiment of the present application is only for illustration, and does not limit the shape and division method of the voxel grid. The shape and division method of the voxel grid can be set according to actual conditions.

[0049] First, the method for determining the obstacle detection area in this application is introduced. Figure 3 As shown, it is a schematic diagram of a process for determining an obstacle detection area, which may include the following steps:

[0050] Step 301: obtaining a second target depth within a specified range according to a depth distribution histogram of the global point cloud data;

[0051] In one implementation, step 301 can be specifically implemented as follows: based on the number of 3D feature points corresponding to each depth in the depth histogram, sort the depths in the depth histogram in descending order according to the number of 3D feature points, select a previously specified number of depths, and determine the depth within a specified range among the previously specified number of depths as the second target depth.

[0052] Among them, the depth mentioned in the embodiment of the present application is the distance from the center point of the head display of the VR device, and the center point of the head display of the VR device is used as the coordinate origin in the embodiment of the application. If there are multiple depths within the specified range, the depth with the largest number of 3D feature points or the depth with the largest value is determined as the second target depth. And the specified number and the specified range in the embodiment of the present application can be set according to actual conditions, and the embodiment of the present application does not limit the specified number and the specified range here. The specified range in the embodiment of the present application is a specified depth range.

[0053] like Figure 4 As shown, it is a schematic diagram of the depth distribution histogram of the global point cloud data. From the figure, it can be seen that the number of 3D feature points in the global point cloud data corresponding to each depth. In the embodiment of the present application, the vertical coordinate of the 3D feature point is determined as the corresponding depth.

[0054] Step 302: obtaining the position of the ground plane in the obstacle detection area based on the 3D position coordinates of each 3D feature point corresponding to the second target depth;

[0055] In one embodiment, step 302 can be specifically implemented as follows: determining the 3D position coordinates of the 3D feature point with the largest horizontal position coordinate value among the 3D feature points as the 3D position coordinates of a vertex of the ground plane, and determining the 3D position coordinates of the 3D feature point with the smallest horizontal position coordinate value among the 3D feature points as the 3D position coordinates of a vertex of the ground plane, and determining the 3D position coordinates of the 3D feature point with the largest vertical position coordinate value among the 3D feature points as the 3D position coordinates of a vertex of the ground plane, and determining the 3D position coordinates of the 3D feature point with the smallest vertical position coordinate value among the 3D feature points as a vertex of the ground plane. The position of the ground plane is obtained according to the obtained 3D position coordinates of the four vertices.

[0056] Step 303: Obtain the obstacle detection area according to the horizontal interval position of the safety area set by the user and the position of the ground plane.

[0057] In one embodiment, step 303 can be specifically implemented as follows: according to the horizontal interval position of the safety area set by the user (including the position of the center point of the horizontal interval and the size of the horizontal interval), the horizontal interval of the obstacle detection area is obtained, and based on the position of the ground plane, the range of the obstacle detection area in the vertical direction is obtained.

[0058] Specifically, in the embodiment of the present application, the position center point of the safety zone is used as the center point of the obstacle detection zone, and the size of the horizontal interval of the safety zone is doubled as the size of the horizontal interval of the obstacle detection zone. And based on the depth of the ground plane, the range of the obstacle detection zone in the vertical direction is determined. For example, the depth of the ground plane (the vertical coordinate in the position of the ground plane) is subtracted from the first specified threshold to obtain the maximum range in the vertical direction, and the depth of the ground plane is subtracted from the second specified threshold to obtain the minimum range in the vertical direction, wherein the first specified threshold is less than the second specified threshold. The reason for doing so in the embodiment of the present application is that the detection of obstacles that are too close to the ground plane is meaningless, and they may be floating errors of feature points on the ground plane. The detection of obstacles that are too high from the ground plane is also meaningless, because the lights on the ceiling will not become dangerous targets that users may collide with. In this way, the amount of calculation can be reduced and the efficiency of obstacle detection can be improved.

[0059] It should be noted that the first specified threshold and the second specified threshold in the embodiment of the present application can be set according to actual conditions, and the embodiment of the present application does not limit the first specified threshold and the second specified threshold.

[0060] Next, the method for updating the obstacle confidence of each voxel grid is introduced, such as Figure 5 As shown, it is a schematic diagram of the process of updating the obstacle confidence of each voxel grid, which includes the following steps:

[0061] Step 501: for any voxel grid, obtaining a first parameter of the voxel grid based on the number of observation cameras of the voxel grid, wherein the observation camera is a camera that observes at least one 3D feature point in the voxel grid;

[0062] The real-time point cloud data in the embodiment of the present application includes multiple 3D feature points observed by multiple cameras in the VR device, and the number of the current frame images is the same as the number of cameras in the VR device. For example, if the VR device includes four cameras, the number of the current frame images obtained is four.

[0063] In one embodiment, step 501 can be specifically implemented as determining, for any 3D feature point in the voxel grid, a target current frame image corresponding to the 3D feature point, determining a camera that captures the target current frame image as an observation camera corresponding to the voxel grid, and if there are duplicate observation cameras in the observation cameras corresponding to the voxel grid, filtering the observation cameras corresponding to the voxel grid so that there are no duplicate observation cameras in the observation cameras corresponding to the voxel grid, and determining the number of observation cameras corresponding to the voxel grid as a first parameter of the voxel grid, wherein the target current frame image is an image having 2D feature points corresponding to the 3D feature points.

[0064] In the embodiment of the present application, when the SLAM algorithm extracts 3D feature points, it can be determined based on which image of each current frame image the 3D feature point is obtained, so the target current frame image corresponding to each 3D feature point can be directly obtained.

[0065] For example, voxel grid A includes three 3D feature points, namely 3D feature point 1, 3D feature point 2 and 3D feature point 3. Among them, the observation camera corresponding to 3D feature point 1 is camera a, the observation camera corresponding to 3D feature point 2 is camera b, and the observation camera corresponding to 3D feature point 3 is camera a. Then the observation cameras corresponding to the determined voxel grid A include: camera a, camera b and camera a. Since camera a is repeated, the redundant camera a is deleted, and the observation cameras corresponding to the determined voxel grid A include: camera a, camera b. Therefore, the first parameter corresponding to voxel grid A is 2.

[0066] Step 502: obtaining a second parameter of the voxel grid according to the number of 3D feature points in the voxel grid and the type of the voxel grid, wherein the type of the voxel grid is an edge voxel grid or a non-edge voxel grid;

[0067] In one embodiment, step 502 may be specifically implemented as follows: if the voxel grid is the edge voxel grid, setting the idle parameter of the voxel grid to a specified parameter; or if the voxel grid is the non-edge voxel grid, setting the idle parameter to the number of the 3D feature points.

[0068] The edge voxel grids in the embodiment of the present application are the voxel grids on the surface of the obstacle detection area, and the type of each voxel grid is determined when the division is performed.

[0069] It should be noted that the specified parameters in the embodiments of the present application are constants and can be set according to actual conditions, and the embodiments of the present application are not limited here.

[0070] Step 503: using the first parameter and the second parameter to update the obstacle confidence of the voxel grid, to obtain the updated obstacle confidence of the voxel grid.

[0071] In one embodiment, step 503 may be specifically implemented as follows: inputting the first parameter and the second parameter into a binary Bayesian filtering algorithm to obtain the obstacle confidence after the voxel grid is updated.

[0072] The binary Bayesian filtering algorithm used in the implementation of this application is an algorithm in the prior art, and the embodiments of this application will not be described in detail here.

[0073] Step 102: Performing plane detection on the obstacle detection area using the global point cloud data to obtain at least one plane in the obstacle detection area, wherein any one plane is a horizontal plane or a vertical plane, and the global point cloud data includes real-time point cloud data corresponding to each frame image from the time when the VR device is turned on to the current time;

[0074] The following describes the detection method of the horizontal plane and the vertical plane in the embodiment of the present application:

[0075] 1. Horizontal plane: According to the depth distribution histogram of the global point cloud data, at least one first target depth and each 3D feature point corresponding to the at least one first target depth are obtained, and for any first target depth, according to each 3D feature point corresponding to the first target depth, a horizontal plane corresponding to the first target depth is obtained, wherein the first target depth is a depth at which the number of 3D feature points in the depth distribution histogram is greater than a specified number.

[0076] In the embodiment of the present application, for any first target depth, the plane formed by the 3D feature points corresponding to the first target depth is the horizontal plane corresponding to the first target depth. The specified number in the embodiment of the present application can be set according to actual conditions, and the embodiment of the present application does not limit the specific value of the specified number.

[0077] 2. Vertical plane: Use the Hough detection algorithm to perform straight line detection on the ground plane of the obstacle detection area to obtain the position of the straight line in the ground plane of the obstacle detection area, and project each 3D feature point in the global point cloud data to obtain the projection position coordinates of each 3D feature point in the ground plane, use the position of the straight line and the projection position coordinates of each 3D feature point to obtain each target 3D feature point corresponding to the straight line, and obtain the vertical plane based on the each target 3D feature point.

[0078] In one embodiment, each target 3D feature point corresponding to the straight line is obtained by:

[0079] Based on the position of the straight line and the position coordinates of each 3D feature point in the ground plane, each 3D feature point located in the straight line is obtained, each 3D feature point located in the straight line is projected in the ground plane to obtain the projection position coordinates of each 3D feature point located in the straight line, and other 3D feature points except the 3D feature points located in the straight line are projected in the ground plane to obtain the projection position coordinates of the other 3D feature points. For any other 3D feature point, based on the projection position coordinates of the other 3D feature point and the projection position coordinates of each 3D feature point located in the straight line, the distance between the other 3D feature point and each 3D feature point located in the straight line is obtained. If there is a 3D feature point located in the straight line whose distance to the other 3D feature point is less than a specified distance, the other 3D feature point is determined as the target vertical 3D feature point, and the plane formed by each target vertical 3D feature point is determined as the vertical plane.

[0080] Step 103: performing convex hull detection on the at least one plane to obtain boundary points of the at least one plane;

[0081] The convex hull detection method in the embodiment of the present application is: to use the library function algorithm in Opencv. The specific convex hull detection method in the embodiment of the present application can be set according to actual conditions. The embodiment of the present application does not limit the convex hull detection method here.

[0082] Step 104: for any boundary point in any plane, determine the voxel grid where the boundary point is located based on the 3D position coordinates of the boundary point, and determine whether the boundary point is an obstacle vertex according to the obstacle confidence after the voxel grid is updated;

[0083] Each voxel grid in the embodiment of the present application has a pre-set plane equation, and the plane function is in the form of ax+by+cz+d=0. Then the 3D position coordinates of the boundary point can be substituted into the plane equation of each voxel grid. If the 3D position coordinates of the boundary point satisfy the plane equation of a certain voxel grid, it is determined that the boundary point is in the voxel grid. Otherwise, it is determined that the boundary point is not in the voxel grid.

[0084] In one embodiment, whether the boundary point is an obstacle vertex is determined in the following manner:

[0085] If the obstacle confidence of the voxel grid where the boundary point is located after being updated is greater than a specified threshold, the boundary point is determined to be an obstacle vertex; if the obstacle confidence of the voxel grid where the boundary point is located after being updated is not greater than the specified threshold, the boundary point is determined not to be an obstacle vertex.

[0086] It should be noted that the specified threshold in the embodiment of the present application can be set according to actual conditions, and the embodiment of the present application does not limit the specified threshold.

[0087] Step 105: if the number of boundary points of the plane that are obstacle vertices is greater than a specified number, determining the obstacle position in the plane according to the 3D position coordinates of the boundary points and the voxel grid where the boundary points are located;

[0088] It should be noted that the specified number in the embodiments of the present application can be set according to actual conditions, and the specified number is not limited in the embodiments of the present application.

[0089] like Figure 6 As shown, it is a schematic diagram of a process for determining the position of an obstacle in a plane, comprising the following steps:

[0090] Step 601: Determine the 3D position coordinates of the center points of other voxel grids in the plane except the voxel grids where the boundary points are located as the 3D position coordinates of the obstacle point;

[0091] like Figure 7 As shown, the voxel grids with dots in the figure represent voxel grids with obstacles. It can be seen from the figure that voxel grids a, voxel grids b and voxel grids c are in plane 1. However, in the image on the left, voxel grids a, voxel grids b and voxel grids c are not determined as obstacles. After using this solution, it can be seen from the right figure that voxel grids a, voxel grids b and voxel grids c are determined as obstacles.

[0092] Step 602: for any boundary point, determine the 3D position coordinates of the boundary point as the 3D position coordinates of the obstacle point in the voxel grid where the boundary point is located;

[0093] like Figure 8 As shown, voxel grid C and voxel grid d initially determine the center point of the voxel grid as the 3D position coordinate of the obstacle point, but the position of the obstacle point is corrected by the method in the embodiment of the present application, that is, the 3D position coordinate of the boundary point located in voxel grid c and voxel grid d is determined as the 3D position coordinate of the obstacle point. The accuracy of obstacle detection is guaranteed.

[0094] Step 603: Based on the 3D position coordinates of each obstacle point, obtain the position of the obstacle in the plane.

[0095] Right now Figure 8 As shown in the right figure, the position of the obstacle in the plane is obtained.

[0096] Step 106: Obtain obstacles in the obstacle detection area through the positions of obstacles in each plane.

[0097] Since the voxel grids with obstacles in each plane in the obstacle detection area will be filled, that is, Figure 7 and Figure 8 Therefore, after each voxel grid where an obstacle exists is filled, the detected obstacle is obtained and rendered and displayed.

[0098] In order to further connect the method in this application, the obstacle detection method in this application is introduced in combination with the usage scenario. Fig. 9 FIG. 1 is a flow chart of the obstacle detection method in the present application, which may include the following steps:

[0099] Step 901: for any voxel grid in the obstacle detection area, obtaining a first parameter of the voxel grid based on the number of observation cameras of the voxel grid, wherein the observation camera is a camera that observes at least one 3D feature point in the voxel grid;

[0100] Step 902: obtaining a second parameter of the voxel grid according to the number of 3D feature points in the voxel grid and the type of the voxel grid, wherein the type of the voxel grid is an edge voxel grid or a non-edge voxel grid;

[0101] Step 903: using the first parameter and the second parameter to update the obstacle confidence of the voxel grid, to obtain the updated obstacle confidence of the voxel grid, wherein the obstacle confidence of any voxel grid is the probability that an obstacle exists in the voxel grid;

[0102] Step 904: Performing plane detection on the obstacle detection area using the global point cloud data to obtain at least one plane in the obstacle detection area, wherein any one plane is a horizontal plane or a vertical plane, and the global point cloud data includes real-time point cloud data corresponding to each frame image from the time when the VR device is turned on to the current time;

[0103] Step 905: performing convex hull detection on the at least one plane to obtain boundary points of the at least one plane;

[0104] Step 906: determining the voxel grid where the boundary point is located based on the 3D position coordinates of the boundary point;

[0105] Step 907: Determine whether the updated obstacle confidence of the voxel grid where the boundary point is located is greater than a specified threshold, if so, execute step 908, if not, execute step 909;

[0106] Step 908: Determine the boundary point as an obstacle vertex;

[0107] Step 909: Determine that the boundary point is not an obstacle vertex;

[0108] Step 910: Determine the 3D position coordinates of the center points of other voxel grids except the voxel grids where the boundary points are located in the plane where the number of boundary points that are obstacle vertices is greater than a specified number as the 3D position coordinates of the obstacle point;

[0109] Step 911: for any boundary point, determine the 3D position coordinates of the boundary point as the 3D position coordinates of the obstacle point;

[0110] Step 912: Obtain the position of the obstacle in the plane based on the 3D position coordinates of each obstacle point;

[0111] Step 913: Obtain obstacles in the obstacle detection area through the positions of obstacles in each plane.

[0112] Based on the same inventive concept, the obstacle detection method disclosed above can also be implemented by an obstacle detection device. The effect of the obstacle detection device is similar to that of the above method, and will not be described in detail here.

[0113] Fig.10 The figure is a schematic diagram of the structure of an obstacle detection device according to an embodiment of the present disclosure.

[0114] like Fig.10 As shown, the obstacle detection device 1000 of the present disclosure may include an obstacle confidence update module 1010 , a plane detection module 1020 , a convex hull detection module 1030 , a judgment module 1040 , a plane obstacle detection module 1050 and an obstacle detection module 1060 .

[0115] The obstacle confidence updating module 1010 is used to update the obstacle confidence of each voxel grid divided in the obstacle detection area based on the real-time point cloud data corresponding to the current frame image acquired, and obtain the updated obstacle confidence of each voxel grid, wherein the obstacle confidence of any voxel grid is the probability that an obstacle exists in the voxel grid;

[0116] The plane detection module 1020 is used to perform plane detection on the obstacle detection area using the global point cloud data to obtain at least one plane in the obstacle detection area, wherein any one plane is a horizontal plane or a vertical plane, and the global point cloud data includes real-time point cloud data corresponding to each frame image from the start of the VR device to the current time;

[0117] A convex hull detection module 1030, configured to perform convex hull detection on the at least one plane to obtain boundary points of the at least one plane;

[0118] A judgment module 1040 is used to determine, for any boundary point in any plane, a voxel grid where the boundary point is located based on the 3D position coordinates of the boundary point, and determine whether the boundary point is an obstacle vertex according to the obstacle confidence after the voxel grid is updated;

[0119] A plane obstacle detection module 1050, configured to determine the position of the obstacle in the plane according to the 3D position coordinates of each boundary point and the voxel grid where each boundary point is located, if the number of boundary points that are obstacle vertices among the boundary points of the plane is greater than a specified number;

[0120] The obstacle detection module 1060 is used to obtain obstacles in the obstacle detection area through the positions of obstacles in each plane.

[0121] In one embodiment, the real-time point cloud data includes a plurality of 3D feature points observed by a plurality of cameras in the VR device;

[0122] The obstacle confidence updating module 1010 is specifically used for:

[0123] For any voxel grid, based on the number of observation cameras of the voxel grid, obtaining a first parameter of the voxel grid, wherein the observation camera is a camera that observes at least one 3D feature point in the voxel grid;

[0124] Obtaining a second parameter of the voxel grid according to the number of 3D feature points in the voxel grid and the type of the voxel grid, wherein the type of the voxel grid is an edge voxel grid or a non-edge voxel grid;

[0125] The obstacle confidence of the voxel grid is updated using the first parameter and the second parameter to obtain the updated obstacle confidence of the voxel grid.

[0126] In one embodiment, the number of the current frame images is the same as the number of cameras in the VR device;

[0127] The obstacle confidence updating module 1010 performs the step of obtaining the first parameter of the voxel grid based on the number of observation cameras in the voxel grid, which is specifically used for:

[0128] For any 3D feature point in the voxel grid, determine a target current frame image corresponding to the 3D feature point, determine a camera that shoots the target current frame image as an observation camera corresponding to the voxel grid, and if there are repeated observation cameras in the observation cameras corresponding to the voxel grid, filter the observation cameras corresponding to the voxel grid so that there are no repeated observation cameras in the observation cameras corresponding to the voxel grid, and determine the number of observation cameras corresponding to the voxel grid as a first parameter of the voxel grid, wherein the target current frame image is an image having 2D feature points corresponding to the 3D feature points;

[0129] The obstacle confidence updating module 1010 performs the step of obtaining the second parameter of the voxel grid according to the number of 3D feature points in the voxel grid and the type of the voxel grid, specifically for:

[0130] If the voxel grid is the edge voxel grid, setting the idle parameters of the voxel grid to the specified parameters; or,

[0131] If the voxel grid is the non-edge voxel grid, the idle parameter is set to the number of 3D feature points in the voxel grid.

[0132] In one embodiment, the plane detection module 1020 is specifically used to:

[0133] According to the depth distribution histogram of the global point cloud data, at least one first target depth and each 3D feature point corresponding to the at least one first target depth are obtained, and for any first target depth, according to each 3D feature point corresponding to the first target depth, a horizontal plane corresponding to the first target depth is obtained, wherein the first target depth is a depth at which the number of 3D feature points in the depth distribution histogram is greater than a specified number; and / or,

[0134] A Hough detection algorithm is used to perform straight line detection on the ground plane of the obstacle detection area to obtain the position of the straight line in the ground plane of the obstacle detection area, and each 3D feature point in the global point cloud data is projected to obtain the projection position coordinates of each 3D feature point in the ground plane, and each target 3D feature point corresponding to the straight line is obtained by using the position of the straight line and the projection position coordinates of each 3D feature point, and the vertical plane is obtained based on the each target 3D feature point.

[0135] In one embodiment, the determination module 1040 is specifically configured to:

[0136] For any voxel grid, if the 3D position coordinates of the boundary point satisfy the plane equation corresponding to the voxel grid, then the boundary point is determined to be in the voxel grid;

[0137] Otherwise, it is determined that the boundary point is not in the voxel grid.

[0138] In one embodiment, the determination module 1040 is specifically configured to:

[0139] If the updated obstacle confidence of the voxel grid where the boundary point is located is greater than a specified threshold, then the boundary point is determined to be an obstacle vertex; or,

[0140] If the updated obstacle confidence of the voxel grid where the boundary point is located is not greater than the specified threshold, it is determined that the boundary point is not an obstacle vertex.

[0141] In one embodiment, the obstacle detection module 1060 is specifically configured to:

[0142] Determine the 3D position coordinates of the center points of other voxel grids in the plane except the voxel grids where the boundary points are located as the 3D position coordinates of the obstacle point; and

[0143] For any boundary point, determining the 3D position coordinates of the boundary point as the 3D position coordinates of the obstacle point in the voxel grid where the boundary point is located;

[0144] Based on the 3D position coordinates of each obstacle point, the position of the obstacle in the plane is obtained.

[0145] In one embodiment, the apparatus further comprises:

[0146] The obstacle detection area determination module 1070 is configured to obtain the obstacle detection area by:

[0147] Obtaining a second target depth within a specified range according to a depth distribution histogram of the global point cloud data;

[0148] Obtaining a position of a ground plane in the obstacle detection area based on the 3D position coordinates of each 3D feature point corresponding to the second target depth;

[0149] The obstacle detection area is obtained according to the horizontal interval position of the safety area set by the user and the position of the ground plane.

[0150] After introducing an obstacle detection method and device according to an exemplary embodiment of the present invention, next, a VR device according to another exemplary embodiment of the present invention is introduced.

[0151] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0152] In some possible implementations, the VR device according to the present invention may include at least one processor and at least one computer storage medium. The computer storage medium stores program code, and when the program code is executed by the processor, the processor executes the steps of the obstacle detection method according to various exemplary embodiments of the present invention described above in this specification. For example, the processor may execute the following steps: Figure 1 Steps 101-106 shown in .

[0153] Refer to the following Fig.11 hereinafter, a VR device 1100 according to this embodiment of the present invention is described. Fig.11 The VR device 1100 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0154] like Fig.11 As shown, the VR device 1100 is presented in the form of a general VR device. The components of the VR device 1100 may include, but are not limited to: the at least one processor 1101, the at least one computer storage medium 1102, and a bus 1103 connecting different system components (including the computer storage medium 1102 and the processor 1101).

[0155] Bus 1103 represents one or more of several types of bus structures, including a computer storage media bus or computer storage media controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0156] Computer storage media 1102 may include readable media in the form of volatile computer storage media, such as random access computer storage media (RAM) 1121 and / or cache storage media 1122 , and may further include read-only computer storage media (ROM) 1123 .

[0157] The computer storage medium 1102 may also include a program / utility 1125 having a set (at least one) of program modules 1124, such program modules 1124 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0158] The VR device 1100 may also communicate with one or more external devices 1104 (e.g., keyboards, pointing devices, etc.), and may also communicate with one or more devices that enable a user to interact with the VR device 1100 and / or any device that enables the VR device 1100 to communicate with one or more other VR devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1105. Furthermore, the VR device 1100 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1106. As shown, the network adapter 1106 communicates with other modules for the VR device 1100 via a bus 1103. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the VR device 1100, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0159] In some possible implementations, various aspects of the obstacle detection method provided by the present invention may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the obstacle detection method according to various exemplary embodiments of the present invention described above in this specification.

[0160] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for detecting an obstacle, characterized in that: Applied to VR equipment, the method includes: Based on the real-time point cloud data corresponding to the current frame image obtained, the obstacle confidence of each voxel grid divided in the obstacle detection area is updated to obtain the updated obstacle confidence of each voxel grid, wherein the obstacle confidence of any voxel grid is the probability that an obstacle exists in the voxel grid; Performing plane detection on the obstacle detection area using global point cloud data to obtain at least one plane in the obstacle detection area, wherein any one plane is a horizontal plane or a vertical plane, and the global point cloud data includes real-time point cloud data corresponding to each frame image from the time when the VR device is turned on to the current time; Performing convex hull detection on the at least one plane to obtain boundary points of the at least one plane; For any boundary point in any plane, determine the voxel grid where the boundary point is located based on the 3D position coordinates of the boundary point, and determine whether the boundary point is an obstacle vertex according to the obstacle confidence after the voxel grid is updated; and If the number of boundary points that are obstacle vertices among the boundary points of the plane is greater than a specified number, the obstacle positions in the plane are determined according to the 3D position coordinates of the boundary points and the voxel grids where the boundary points are located; and the obstacles in the obstacle detection area are obtained through the obstacle positions in the planes.

2. The method according to claim 1, characterized in that The real-time point cloud data includes a plurality of 3D feature points observed by a plurality of cameras in the VR device; The updating of the obstacle confidence of each voxel grid divided in the obstacle detection area based on the real-time point cloud data corresponding to the current frame image obtained to obtain the updated obstacle confidence of each voxel grid includes: For any voxel grid, a first parameter of the voxel grid is obtained based on the number of observation cameras of the voxel grid, wherein the observation camera is a camera that observes at least one 3D feature point in the voxel grid; and Obtaining a second parameter of the voxel grid according to the number of 3D feature points in the voxel grid and the type of the voxel grid, wherein the type of the voxel grid is an edge voxel grid or a non-edge voxel grid; The obstacle confidence of the voxel grid is updated using the first parameter and the second parameter to obtain the updated obstacle confidence of the voxel grid.

3. The method according to claim 2, characterized in that The number of the current frame images is the same as the number of cameras in the VR device; The obtaining a first parameter of the voxel grid based on the number of observation cameras in the voxel grid comprises: For any 3D feature point in the voxel grid, determine a target current frame image corresponding to the 3D feature point, determine a camera that shoots the target current frame image as an observation camera corresponding to the voxel grid, and if there are repeated observation cameras in the observation cameras corresponding to the voxel grid, filter the observation cameras corresponding to the voxel grid so that there are no repeated observation cameras in the observation cameras corresponding to the voxel grid, and determine the number of observation cameras corresponding to the voxel grid as a first parameter of the voxel grid, wherein the target current frame image is an image having 2D feature points corresponding to the 3D feature points; The obtaining, according to the number of 3D feature points in the voxel grid and the type of the voxel grid, a second parameter of the voxel grid comprises: If the voxel grid is the edge voxel grid, setting the idle parameters of the voxel grid to the specified parameters; or, If the voxel grid is the non-edge voxel grid, the idle parameter is set to the number of 3D feature points in the voxel grid.

4. The method according to claim 1, characterized in that: The using the global point cloud data to perform plane detection on the obstacle detection area to obtain at least one plane in the obstacle detection area includes: According to the depth distribution histogram of the global point cloud data, at least one first target depth and each 3D feature point corresponding to the at least one first target depth are obtained, and for any first target depth, according to each 3D feature point corresponding to the first target depth, a horizontal plane corresponding to the first target depth is obtained, wherein the first target depth is a depth at which the number of 3D feature points in the depth distribution histogram is greater than a specified number; and / or, A Hough detection algorithm is used to perform straight line detection on the ground plane of the obstacle detection area to obtain the position of the straight line in the ground plane of the obstacle detection area, and each 3D feature point in the global point cloud data is projected to obtain the projection position coordinates of each 3D feature point in the ground plane, and each target 3D feature point corresponding to the straight line is obtained by using the position of the straight line and the projection position coordinates of each 3D feature point, and the vertical plane is obtained based on the each target 3D feature point.

5. The method according to claim 1, characterized in that The determining the voxel grid where the boundary point is located based on the 3D position coordinates of the boundary point includes: For any voxel grid, if the 3D position coordinates of the boundary point satisfy the plane equation corresponding to the voxel grid, then the boundary point is determined to be in the voxel grid; Otherwise, it is determined that the boundary point is not in the voxel grid.

6. The method according to claim 1, characterized in that The determining whether the boundary point is an obstacle vertex according to the obstacle confidence after the voxel grid is updated includes: If the updated obstacle confidence of the voxel grid where the boundary point is located is greater than a specified threshold, then the boundary point is determined to be an obstacle vertex; or, If the updated obstacle confidence of the voxel grid where the boundary point is located is not greater than the specified threshold, it is determined that the boundary point is not an obstacle vertex.

7. The method according to claim 1, characterized in that The determining the obstacle position in the plane according to the 3D position coordinates of each boundary point and the voxel grid where each boundary point is located includes: Determine the 3D position coordinates of the center points of other voxel grids in the plane except the voxel grids where the boundary points are located as the 3D position coordinates of the obstacle point; and For any boundary point, determining the 3D position coordinates of the boundary point as the 3D position coordinates of the obstacle point in the voxel grid where the boundary point is located; Based on the 3D position coordinates of each obstacle point, the position of the obstacle in the plane is obtained.

8. The method according to any one of claims 1 to 7, characterized in that: The obstacle detection area is obtained by: Obtaining a second target depth within a specified range according to a depth distribution histogram of the global point cloud data; Obtaining a position of a ground plane in the obstacle detection area based on the 3D position coordinates of each 3D feature point corresponding to the second target depth; The obstacle detection area is obtained according to the horizontal interval position of the safety area set by the user and the position of the ground plane.

9. A VR device, characterized in that: comprising a processor and a memory, wherein the processor and the memory are connected via a bus; The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program: Based on the real-time point cloud data corresponding to the current frame image obtained, the obstacle confidence of each voxel grid divided in the obstacle detection area is updated to obtain the updated obstacle confidence of each voxel grid, wherein the obstacle confidence of any voxel grid is the probability that an obstacle exists in the voxel grid; Performing plane detection on the obstacle detection area using global point cloud data to obtain at least one plane in the obstacle detection area, wherein any one plane is a horizontal plane or a vertical plane, and the global point cloud data includes real-time point cloud data corresponding to each frame image from the time when the VR device is turned on to the current time; Performing convex hull detection on the at least one plane to obtain boundary points of the at least one plane; For any boundary point in any plane, determine the voxel grid where the boundary point is located based on the 3D position coordinates of the boundary point, and determine whether the boundary point is an obstacle vertex according to the obstacle confidence after the voxel grid is updated; and If the number of boundary points of the plane that are vertices of obstacles is greater than a specified number, determining the position of the obstacles in the plane according to the 3D position coordinates of the boundary points and the voxel grids where the boundary points are located; The position of the obstacle in the obstacle detection area is obtained through the position of the obstacle in each plane.

10. The VR device according to claim 9, characterized in that: The real-time point cloud data includes a plurality of 3D feature points observed by a plurality of cameras in the VR device; The processor executes the updating of the obstacle confidence of each voxel grid divided in the obstacle detection area based on the real-time point cloud data corresponding to the current frame image acquired to obtain the obstacle confidence of each voxel grid after update, and is specifically configured as follows: For any voxel grid, a first parameter of the voxel grid is obtained based on the number of observation cameras of the voxel grid, wherein the observation camera is a camera that observes at least one 3D feature point in the voxel grid; and Obtaining a second parameter of the voxel grid according to the number of 3D feature points in the voxel grid and the type of the voxel grid, wherein the type of the voxel grid is an edge voxel grid or a non-edge voxel grid; The obstacle confidence of the voxel grid is updated using the first parameter and the second parameter to obtain the updated obstacle confidence of the voxel grid.

Images