Information processing device, information processing method, and program

Abstract

An information processing device (100) is provided with a control unit (130). The control unit (130) acquires first three-dimensional information relating to the probability that an object occupies a real space, and second three-dimensional information relating to the result of estimating the surface shape of the object. The control unit (130) classifies objects on the basis of the first three-dimensional information and floor surface information relating to a floor surface in the real space. The control unit (130) highlights the surface of the classified objects on the basis of the second three-dimensional information.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, many devices have been released that perform processing in response to user movements. For example, there are games in which a character displayed on the screen moves by synchronizing the character with the user's movements. In games where the user is constantly operating the device, as in these games, the user can become so engrossed in the operation that they lose awareness of their surroundings, which can lead to problems such as bumping into surrounding objects. In particular, when enjoying VR (Virtual Reality) content played while wearing an HMD (Head Mounted Display), the user may not be able to see their surroundings at all, increasing the risk of bumping into real objects.

[0003] As a technique for detecting surrounding objects, for example, a technique is known in which feature points are extracted from captured images acquired by a stereo camera and the object is recognized based on the spatial positions of the extracted feature points. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-33819 Summary of the Invention [Problem to be solved by the invention]

[0005] As mentioned above, users who enjoy VR content may not be able to see their surroundings at all, which increases the risk of bumping into real-world objects. Therefore, a system that notifies users of obstacles is desirable. This allows users to safely enjoy VR content by moving the object or moving around to avoid it.

[0006] Therefore, the present disclosure provides a mechanism that allows users to enjoy content more safely.

[0007] It should be noted that the above problem or object is merely one of multiple problems or objects that can be solved or achieved by multiple embodiments disclosed in this specification. [Means for solving the problem]

[0008] According to the present disclosure, there is provided an information processing device. The information processing device includes a control unit. The control unit acquires first three-dimensional information related to an occupancy probability of an object in a real space and second three-dimensional information related to an estimation result of a surface shape of the object. The control unit classifies the object based on the first three-dimensional information and floor surface information related to a floor surface of the real space. The control unit highlights the surface of the object classified based on the second three-dimensional information. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an overview of an information processing system according to the present disclosure. [Figure 2] FIG. 10 is a diagram for explaining an overview of an obstacle display process according to an embodiment of the present disclosure. [Figure 3] FIG. 10 is a diagram illustrating an example of second three-dimensional information according to an embodiment of the present disclosure. [Figure 4] FIG. 10 is a diagram for explaining an overview of a display suppression process according to an embodiment of the present disclosure. [Figure 5] 1 is a block diagram illustrating a configuration example of a terminal device according to a first embodiment of the present disclosure. [Figure 6] 1 is a block diagram illustrating an example configuration of an information processing device according to a first embodiment of the present disclosure. [Figure 7] FIG. 2 is a diagram for explaining an example of a voxel according to an embodiment of the present disclosure. [Figure 8] FIG. 2 is a block diagram illustrating an example configuration of a display control unit according to an embodiment of the present disclosure. [Figure 9]FIG. 2 is a block diagram illustrating a configuration example of an obstacle detection unit according to an embodiment of the present disclosure. [Figure 10] FIG. 10 is a diagram for explaining an example of an obstacle detected by a clustering processing unit according to an embodiment of the present disclosure. [Figure 11] FIG. 10 is a diagram for explaining mesh information with display labels according to an embodiment of the present disclosure. [Figure 12] FIG. 2 is a diagram illustrating a configuration example of a fake obstacle determination unit according to an embodiment of the present disclosure. [Figure 13] 1 is a diagram for explaining the relationship between a target voxel and a ranging range of a ranging device according to an embodiment of the present disclosure. FIG. [Figure 14] 1 is a diagram for explaining the relationship between a target voxel and a ranging range of a ranging device according to an embodiment of the present disclosure. FIG. [Figure 15] FIG. 10 is a diagram for explaining an example of a state transition of a target voxel according to an embodiment of the present disclosure. [Figure 16] 10 is a flowchart illustrating an example of the flow of an image generation process according to an embodiment of the present disclosure. [Figure 17] 10 is a flowchart illustrating an example of a flow of an obstacle division process according to an embodiment of the present disclosure. [Figure 18] 10 is a flowchart illustrating an example of a flow of a fake obstacle determination process according to an embodiment of the present disclosure. [Figure 19] 10 is a flowchart illustrating an example of the flow of a display image generation process according to an embodiment of the present disclosure. [Figure 20] FIG. 10 is a diagram for explaining an example of a real space according to a modified example of an embodiment of the present disclosure. [Figure 21] FIG. 1 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of an information processing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0011] Furthermore, in the present specification and drawings, specific values ​​may be used for explanation, but these values ​​are merely examples and other values ​​may be applied. Furthermore, in the present specification, the following reference documents may be used for explanation:

[0012] (References) [1]Angela Dai, et al. "ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans", CVPR 2018 [2]Margarita Grinvald, et al. “Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery”, IROS 2019 [3]Xianzhi Li, et al. “DNF-Net: a Deep Normal Filtering Network for Mesh Denoising”, IEEE Transactions on Visualization and Computer Graphics (TVCG) 2020 [4]S. Fleishman, et al. “Bilateral mesh denoising”, SIGGRAPH 2003 [5]Raul Mur-Artal, et al. “ORB-SLAM: A Versatile and Accurate Monocular SLAM System”, IEEE Transactions on Robotics, vol. 31, no. 5, pp. 1147-1163, October 2015 [6]B. Curless and M. Levoy “A volumetric method for building complex models from range images”, In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, SIGGRAPH ’96, pages 303-312, New York, NY, USA, 1996. ACM [7]William E. Lorensen, Harvey E. Cline: Marching Cubes: A high resolution 3D surface construction algorithm. In: Computer Graphics, Vol. 21, Nr. 4, July 1987 [8]Armin Hornung, et al. "OctoMap: An efficient probabilistic 3D mapping framework based on octrees." Autonomous robots 34.3 (2013): 189-206. [9]Ruwen Schnabel, et al. "Efficient RANSAC for point‐cloud shape detection." Computer graphics forum. Vol. 26. No. 2. Oxford, UK: Blackwell Publishing Ltd, 2007

[10] Kesheng Wu, et al. “Two Strategies to Speed up Connected Component Labeling Algorithms”, Published 2005, Mathematics, Computer Science, Lawrence Berkeley National Laboratory

[0013] One or more embodiments (including examples and modifications) described below can be implemented independently. However, at least a portion of the embodiments described below may be implemented in appropriate combination with at least a portion of another embodiment. These embodiments may include novel features that are different from one another. Therefore, these embodiments may contribute to solving different purposes or problems and may produce different effects.

[0014] <<1. Introduction>> <1.1. Overview of Information Processing System 1> 1 is a diagram illustrating an overview of an information processing system 1 according to the present disclosure. As shown in FIG. 1, the information processing system 1 includes an information processing device 100 and a terminal device 200.

[0015] The information processing device 100 and the terminal device 200 can communicate with each other via various wired or wireless networks. Note that any communication method can be used in the network, regardless of whether it is wired or wireless (for example, WiFi (registered trademark), Bluetooth (registered trademark), etc.).

[0016] Furthermore, the number of information processing devices 100 and terminal devices 200 included in the information processing system 1 is not limited to the number shown in Fig. 1, and more may be included. Furthermore, Fig. 1 illustrates a case in which the information processing system 1 includes the information processing device 100 and the terminal device 200, respectively, but is not limited to this. For example, the information processing device 100 and the terminal device 200 may be realized as a single device. For example, the functions of both the information processing device 100 and the terminal device 200 may be realized in a single device, such as a standalone HMD.

[0017] The terminal device 200 is a wearable device (eyewear device) such as a glasses-type HMD that the user U wears on his / her head.

[0018] An eyewear device applicable as the terminal device 200 may be a so-called see-through type head-mounted display (AR (Augmented Reality) glasses) that allows images in real space to pass through, or may be a goggle-type device (VR (Virtual Reality) goggles) that does not allow images in real space to pass through.

[0019] Furthermore, in the present disclosure, the terminal device 200 is not limited to being an HMD, and may be, for example, a tablet or smartphone held by the user U.

[0020] The information processing device 100 comprehensively controls the operation of the terminal device 200. The information processing device 100 is realized by, for example, processing circuits such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). Note that a detailed configuration of the information processing device 100 according to the present disclosure will be described later.

[0021] <1.2. Overview> <1.2.1. Overview of Obstacle Display Processing> Here, as described above, when the user U moves while wearing an HMD or the like, there is a possibility that the user U may bump into a real object.

[0022] Therefore, in order to protect the physical safety of the user U, the information processing device 100 controls the HMD so that the user U moves within a safe play area (allowable area) where the user U does not come into contact with real objects. Such a play area can be set by the user U, for example, before the start of a game. Alternatively, the play area can be identified by the information processing device 100 based on the sensing results of a sensor or the like mounted on the terminal device 200, for example.

[0023] For example, in FIG. 1, an area PA is identified as a play area where a user U can move and reach without bumping into obstacles. Note that the play area may be represented as a three-dimensional area, such as a combination of a dotted line PA1 shown on the floor and a wall PA2 extending perpendicularly from the dotted line PA1. Alternatively, the play area may be represented as a two-dimensional area of ​​the dotted line PA1. In this way, the play area may be set as a two-dimensional area or a three-dimensional area.

[0024] Here, as shown in FIG. 1, the area PA identified as the play area (hereinafter simply referred to as the play area PA) may include obstacles Ob1 and Ob2 (hereinafter simply referred to as obstacles Ob1 and Ob2).

[0025] When a user U sets a play area PA, even if obstacles Ob1 and Ob2 are present, there is a possibility that the play area PA will be set to include obstacles Ob1 and Ob2, for example, by setting the same play area PA as the last time the game was played.

[0026] Alternatively, even if a play area PA that does not include obstacles Ob1 and Ob2 is set or specified, obstacles Ob1 and Ob2 may be placed within the play area PA while the user U is playing the game.

[0027] In this way, when obstacles Ob1 and Ob2 exist within the play area PA, a mechanism is desired that notifies the user U of the presence of the obstacles Ob1 and Ob2. This allows the user U to move the obstacles Ob1 and Ob2 out of the play area PA or move while avoiding the obstacles Ob1 and Ob2, thereby ensuring the safety of the user U more reliably.

[0028] For example, conventionally, methods for detecting obstacles Ob are known, such as those in references [1] and [2]. For example, reference [1] discloses a method for segmenting each voxel of 3D information using a trained convolutional neural network (CNN). Also, reference [2] discloses a method for segmenting a 2D image and mapping it to 3D information. However, these methods require a large classifier and require long processing times. Therefore, a method for segmenting obstacles Ob using few resources and in a short time is desired.

[0029] Therefore, the information processing system 1 according to the present disclosure executes an obstacle display process to detect obstacles within the play area PA. Fig. 2 is a diagram for explaining an overview of the obstacle display process according to the embodiment of the present disclosure. The obstacle display process shown in Fig. 2 is executed by, for example, the information processing device 100.

[0030] 2, the information processing device 100 acquires first three-dimensional information (step S11). The first three-dimensional information is, for example, information regarding the occupancy probability of an object in a real space where a user U exists. An example of the first three-dimensional information is an occupancy grid map.

[0031] The information processing device 100 acquires floor surface information (step S12). The floor surface information is, for example, information about a floor surface in a real space.

[0032] The information processing device 100 classifies obstacles based on the first three-dimensional information and floor information (step S13). The information processing device 100 excludes information corresponding to the floor from the first three-dimensional information and classifies obstacles from the remaining information. When the first three-dimensional information is an occupancy grid map (hereinafter also referred to as an Occupancy Map), the information processing device 100 classifies obstacles by excluding voxels corresponding to the floor and clustering adjacent voxels among voxels whose state is occupied. Details of obstacle classification will be described later.

[0033] The information processing device 100 acquires second three-dimensional information (step S14). The second three-dimensional information is information about the surface shape of an object in the real space where the user U exists. The second three-dimensional information includes, for example, mesh data that defines a surface by a plurality of vertices and edges connecting the plurality of vertices.

[0034] The information processing device 100 highlights the surfaces of the classified obstacles (step S15). For example, the information processing device 100 highlights the surfaces of the obstacles by changing the display color of mesh data corresponding to the obstacles in the second three-dimensional information.

[0035] The information processing device 100 classifies obstacles Ob1 and Ob2 in the play area PA by executing an obstacle display process in the play area PA shown in Fig. 1, for example. The information processing device 100 highlights the classified obstacles Ob1 and Ob2.

[0036] This allows the information processing device 100 to notify the user U of the presence of the obstacles Ob1 and Ob2. As a result, the user U can remove the obstacles Ob1 and Ob2 or move around while avoiding the obstacles Ob1 and Ob2, allowing the user U to enjoy content (for example, a game) more safely.

[0037] 1, the number of obstacles Ob is two, but is not limited to this. The number of obstacles Ob may be one, or may be three or more.

[0038] <1.2.2. Overview of display suppression processing> While an obstacle is highlighted, a display image may be generated that shows an obstacle in a space where no obstacle exists due to noise, etc. Such an erroneously detected obstacle will be referred to as a false obstacle below.

[0039] For example, it is assumed that a display image is generated based on the second three-dimensional information including the mesh data described above. By using the mesh data in this way, the information processing device 100 can generate a smoother image than when using an occupancy map. On the other hand, unlike an occupancy map, the second three-dimensional information including mesh data does not have an "Unknown" state, and the previous data continues to be held until the next distance information is acquired.

[0040] 3 is a diagram illustrating an example of second three-dimensional information according to an embodiment of the present disclosure. As shown in the circled portion of FIG. 3, once noise occurs, the noise may be retained as second three-dimensional information for a long time and may continue to be presented to the user U as a fake obstacle. In this case, it does not matter whether the fake obstacle is highlighted or not.

[0041] Known techniques for preventing such false obstacles from being presented to the user U include those disclosed in references [3] and [4]. Reference [3] discloses a mesh denoising method using a trained DNN (Deep Neural Network). Reference [4] also discloses a mesh denoising method using a model base such as a bilateral filter. However, these methods suppress irregularities in relation to surrounding meshes. Therefore, it is necessary to suppress the display of false obstacles that are observed isolated from their surroundings due to erroneous depth observation.

[0042] Therefore, the information processing system 1 according to the present disclosure executes a display suppression process to suppress the display of the fake obstacle. Fig. 4 is a diagram for explaining an overview of the display suppression process according to the embodiment of the present disclosure. The display suppression process shown in Fig. 4 is executed by, for example, the information processing device 100.

[0043] 4, the information processing device 100 acquires first three-dimensional information (step S21). The first three-dimensional information is the same as the first three-dimensional information acquired in the obstacle display processing shown in FIG.

[0044] The information processing device 100 acquires floor surface information (step S22). The floor surface information is the same as the floor surface information acquired in the obstacle display processing shown in FIG.

[0045] The information processing device 100 detects an obstacle based on the first three-dimensional information and floor surface information (step S23). The information processing device 100 may detect an obstacle using the same method as that used to classify obstacles in the obstacle display process shown in Fig. 2, or may detect voxels in which the Occupancy Map, which is the first three-dimensional information, is Occupied (occupied state) as obstacles. The information processing device 100 may detect an obstacle using a method according to predetermined conditions used to determine whether an obstacle is a false obstacle, which will be described later.

[0046] The information processing device 100 determines whether the detected obstacles are fake obstacles (step S24). The information processing device 100 determines whether the detected obstacles are fake obstacles by determining an outlier rate according to predetermined conditions. For example, the information processing device 100 determines the outlier rate according to the size (number of voxels) of the detected obstacle. The information processing device 100 determines the outlier rate according to the proportion of voxels that are unknown (unobserved state) among the voxels surrounding the detected obstacle. The information processing device 100 determines the outlier rate according to a time change in the state of the second three-dimensional information (voxels). The information processing device 100 determines the outlier rate of the obstacle according to its height from the floor. Details of how the information processing device 100 determines the outlier rate will be described later.

[0047] The information processing device 100 acquires second three-dimensional information (step S25). The second three-dimensional information is the same as the second three-dimensional information acquired in the obstacle display processing shown in FIG.

[0048] The information processing device 100 suppresses the display of the false obstacle (step S26). For example, the information processing device 100 suppresses the display of the false obstacle by displaying the obstacle with a transparency according to the outlier rate.

[0049] This allows the information processing device 100 to notify the user U of the obstacle Ob with higher accuracy without displaying a fake obstacle. This allows the user U to enjoy content (for example, a game) more safely.

[0050] The information processing device 100 may execute both the obstacle display process and the display suppression process, or may execute at least one of them. When executing both the obstacle display process and the display suppression process, the information processing device 100 may omit one of the processes that overlap in both the obstacle display process and the display suppression process.

[0051] <<2. Information Processing System>> <2.1. Example of terminal device configuration> 5 is a block diagram showing a configuration example of the terminal device 200 according to the first embodiment of the present disclosure. As shown in FIG. 5, the terminal device 200 includes a communication unit 210, a sensor unit 220, a display unit 230, an input unit 240, and a control unit 250.

[0052] [Communications Department 210] The communication unit 210 transmits and receives information to and from other devices. For example, the communication unit 210 transmits a video playback request and the sensing result of the sensor unit 220 to the information processing device 100 under the control of the control unit 250. The communication unit 210 also receives the video to be played back from the information processing device 100.

[0053] [Sensor unit 220] The sensor unit 220 may include, for example, a camera (image sensor), a depth sensor, a microphone, an acceleration sensor, a gyroscope, a geomagnetic sensor, a GPS (Global Positioning System) receiver, etc. The sensor unit 220 may also include a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) that integrates these.

[0054] For example, the sensor unit 220 senses the position of the terminal device 200 in real space (or the position of a user U using the terminal device 200), the orientation or posture of the terminal device 200, or acceleration. The sensor unit 220 also senses depth information around the terminal device 200. If the sensor unit 220 includes a distance measuring device that senses depth information, the distance measuring device may be a stereo camera, a ToF (Time of Flight) distance image sensor, or the like.

[0055] [Display section 230] The display unit 230 displays an image under the control of the control unit 250. For example, the display unit 230 may have a right-eye display unit and a left-eye display unit (not shown). In this case, the right-eye display unit projects an image onto at least a partial area of ​​a right-eye lens (not shown) included in the terminal device 200 as a projection surface. The left-eye display unit projects an image onto at least a partial area of ​​a left-eye lens (not shown) included in the terminal device 200 as a projection surface.

[0056] Alternatively, if the terminal device 200 has goggle-type lenses, the display unit 230 may project an image onto at least a partial area of ​​the goggle-type lenses as a projection surface. The left-eye lens and the right-eye lens (or the goggle-type lenses) may be made of a transparent material such as resin or glass.

[0057] Alternatively, the display unit 230 may be configured as a non-transmissive display device. For example, the display unit 230 may be configured to include an LCD (Liquid Crystal Display) or an OLED (Organic Light Emitting Diode). In this case, images of the area in front of the user U captured by the sensor unit 220 (camera) may be sequentially displayed on the display unit 230. This allows the user U to view the scenery in front of the user U via the images displayed on the display unit 230.

[0058] [Input section 240] The input unit 240 may include a touch panel, buttons, levers, switches, etc. The input unit 240 accepts various inputs from the user U. For example, when an AI character is placed in a virtual space, the input unit 240 may accept an input from the user U to change the placement position of the AI ​​character, etc.

[0059] [Control unit 250] The control unit 250 performs overall control of the operation of the terminal device 200 using, for example, a CPU, a GPU (Graphics Processing Unit), and a RAM built into the terminal device 200. For example, the control unit 250 causes the display unit 230 to display a video received from the information processing device 100.

[0060] As an example, assume that the terminal device 200 receives a video. In this case, the control unit 250 causes the display unit 230 to display a part of the video that corresponds to the information on the position and posture of the terminal device 200 (or the user U, etc.) sensed by the sensor unit 220.

[0061] Furthermore, if the display unit 230 has a right-eye display unit and a left-eye display unit (not shown), the control unit 250 generates a right-eye image and a left-eye image based on the video received from the information processing device 100. Then, the control unit 250 causes the right-eye image to be displayed on the right-eye display unit, and the left-eye image to be displayed on the left-eye display unit. In this way, the control unit 250 allows the user U to view stereoscopic video.

[0062] Furthermore, the control unit 250 can perform various recognition processes based on the sensing results of the sensor unit 220. For example, the control unit 250 can recognize the behavior of the user U wearing the terminal device 200 (for example, the gestures of the user U, the movement of the user U, etc.) based on the sensing results.

[0063] <2.2. Configuration example of information processing device> 6 is a block diagram showing an example configuration of the information processing device 100 according to the first embodiment of the present disclosure. As shown in FIG. 6, the information processing device 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

[0064] [Communications Department 110] The communication unit 110 transmits and receives information to and from other devices. For example, the communication unit 110 transmits a video to be played back to the information processing device 100 under the control of the control unit 130. The communication unit 110 also receives a video playback request and sensing results from the terminal device 200.

[0065] [Storage section 120] The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM), a read only memory (ROM), or a flash memory, or a storage device such as a hard disk or an optical disk.

[0066] [Control unit 130] The control unit 130 performs overall control of the operation of the information processing device 100 using, for example, a CPU, a GPU (Graphics Processing Unit), and RAM built into the information processing device 100. For example, the control unit 130 is realized by a processor executing various programs stored in a storage device inside the information processing device 100 using RAM (Random Access Memory) or the like as a working area. Note that the control unit 130 may also be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array). The CPU, MPU, ASIC, and FPGA can all be considered as controllers.

[0067] As shown in FIG. 6, the control unit 130 includes an estimation unit 131, an integrated processing unit 132, and a display control unit 133. Each block (estimation unit 131 to display control unit 133) constituting the control unit 130 is a functional block indicating a function of the control unit 130. These functional blocks may be software blocks or hardware blocks. For example, each of the above-mentioned functional blocks may be a software module realized by software (including a microprogram), or may be a circuit block on a semiconductor chip (die). Of course, each functional block may be a processor or an integrated circuit. The functional blocks may be configured in any manner. The control unit 130 may be configured by functional units different from the above-mentioned functional blocks.

[0068] (Estimation part 131) The estimation unit 131 estimates the attitude (pose) of the terminal device 200 based on the sensing results acquired by the sensor unit 220 of the terminal device 200. For example, the estimation unit 131 acquires measurement results (e.g., acceleration and angular velocity; hereinafter also referred to as IMU information) of an IMU, which is an example of the sensor unit 220, and image capture results (hereinafter also referred to as camera images) of a camera. The estimation unit 131 estimates the camera pose using, for example, simultaneous localization and mapping (SLAM), a self-position estimation technique typified by Reference [5].

[0069] The estimation unit 131 estimates the self-position and orientation (hereinafter also referred to as the camera pose) and the direction of gravity of the terminal device 200 (a camera which is an example of the sensor unit 220) based on the acquired IMU information and camera image. The estimation unit 131 outputs the estimated camera pose and direction of gravity to the integration processing unit 132.

[0070] (Integration processing unit 132) The integrated processing unit 132 generates first and second three-dimensional information based on distance information (hereinafter also referred to as depth information), which is the distance measurement result of a distance measuring device, which is an example of the sensor unit 220, and the camera pose estimated by the estimation unit 131.

[0071] Here, the first three-dimensional information is, for example, an occupancy map. The second three-dimensional information is information including mesh data. Both the first and second three-dimensional information are information held in each voxel obtained by dividing a three-dimensional space (real space) into a voxel grid of finite width.

[0072] Fig. 7 is a diagram for explaining an example of voxels according to an embodiment of the present disclosure. Note that Fig. 7 shows an example in which a three-dimensional space is divided into 4 × 5 × 6 = 120 voxels, but the number of voxels is not limited to 120 and may be less than 120 or 121 or more.

[0073] 7, an embodiment of the present disclosure uses a world coordinate system in which the direction of gravity is the negative z-axis direction. That is, the voxel arrangements of the first and second three-dimensional information are aligned with the world coordinate system, and the voxel arrangements of the first and second three-dimensional information are the same.

[0074] (Mesh information) The integration processing unit 132 holds, for example, distance information from the object surface in each voxel shown in FIG. 7. The distance information is called, for example, a truncated signed distance field (TSDF). The TSDF is a well-known technology disclosed in, for example, reference [6]. The integration processing unit 132 updates the TSDF in the time direction based on the distance information.

[0075] The integration processing unit 132 also converts the distance information held for each voxel into a mesh by extracting an isosurface from the TSDF using, for example, the marching cube method disclosed in Reference [7]. Note that the conversion from the TSDF to a mesh may be performed by the integration processing unit 132 or by the display control unit 133, which will be described later.

[0076] In addition, in this disclosure, unless otherwise specified, the second three-dimensional information is assumed to be mesh data obtained by converting a TSDF into a mesh, but the second three-dimensional information may be a TSDF. Furthermore, the TSDF stored for each voxel in three-dimensional space is also referred to as TSDF information, and information obtained by converting each TSDF information into a mesh is also referred to as mesh information.

[0077] The integration processing unit 132 outputs the generated mesh information to the display control unit 133.

[0078] (Occupancy Map) The integration processing unit 132 generates an occupancy map that holds, for example, the occupancy probability of an object in each voxel, as shown in Fig. 7. The occupancy map is three-dimensional space information described in, for example, Reference [8].

[0079] Each voxel in the Occupancy Map is classified into the following three states by classifying the occupancy probability according to a threshold value. Occupied: Indicates that the voxel is occupied by an object (occupied state). Free: Indicates that the voxel is not occupied by an object and is empty space (unoccupied state). · Unknown: Indicates that due to lack of observations, it is not possible to determine whether the voxel is occupied by an object or not (unobserved state).

[0080] For example, suppose the range of possible values ​​of the occupancy probability held by each voxel is from "0" to "1." In this case, if the occupancy probability is greater than or equal to the threshold p occ The state of a voxel is defined as Occupied. Also, if the occupancy probability is greater than or equal to the threshold p free The state of a voxel that is below the threshold p occ or greater than threshold p free The state of a voxel that does not satisfy the following conditions is called Unknown.

[0081] The integration processing unit 132 updates the occupancy probability in the time direction based on the distance information, thereby generating an occupancy map. The integration processing unit 132 outputs the generated occupancy map to the display control unit 133.

[0082] Both the mesh information and the occupancy map described above are technologies for representing objects that exist in three-dimensional space, but each has its own advantages and disadvantages.

[0083] For example, mesh information can represent the detailed surface shape of an object by retaining distance information from the object surface. On the other hand, mesh information cannot retain unobserved states. Therefore, as described above, if incorrect shape information is retained in a voxel due to depth noise, the incorrect shape information will continue to be displayed unless a new depth is observed at that voxel.

[0084] On the other hand, the occupancy map cannot represent detailed surface shapes because voxels maintain unoccupied and occupied states, but it can maintain unobserved states. Therefore, even if incorrect information is maintained in a voxel due to depth noise, if no new depth is subsequently observed in the voxel, the state of the voxel transitions to Unknown. Furthermore, the information processing device 100 can use the occupancy map to determine the reliability of a given voxel and the existence section of an object from the states of voxels surrounding the given voxel (hereinafter also referred to as surrounding voxels).

[0085] (plane information) Returning to FIG. 6, the integration processing unit 132 generates floor surface information from the 3D information of the real space. The integration processing unit 132 detects the floor surface by, for example, calculating the maximum plane for the 3D information using RANSAC. Note that the calculation of the maximum plane using RANSAC can be performed using, for example, the technology described in Reference [9]. Note that the 3D information may include, for example, the above-mentioned occupancy map, mesh information, or a 3D point cloud obtained from distance information.

[0086] The integration processing unit 132 outputs the generated floor surface information to the display control unit 133.

[0087] (Display control unit 133) The display control unit 133 generates a video (display image) to be played on the terminal device 200. For example, the display control unit 133 generates a video of the surroundings of the user U using mesh information. At this time, the display control unit 133 generates the display image by highlighting obstacles in the play area PA, for example. Furthermore, the display control unit 133 generates the display image by suppressing the display of fake obstacles.

[0088] 8 is a block diagram showing an example configuration of the display control unit 133 according to an embodiment of the present disclosure. As shown in FIG. 8, the display control unit 133 includes an obstacle detection unit 1331, a fake obstacle determination unit 1332, and a display change unit 1333.

[0089] (Obstacle detection unit 1331) The obstacle detection unit 1331 detects obstacles by dividing obstacles that exist in the real space using floor surface information and an occupancy map. The obstacle detection unit 1331 generates mesh information with display labels by adding display information (display labels) corresponding to the detected obstacles to mesh information.

[0090] 9 is a block diagram showing an example configuration of the obstacle detection unit 1331 according to an embodiment of the present disclosure. The obstacle detection unit 1331 shown in FIG. 9 includes a voxel extraction unit 1331A, a clustering processing unit 1331B, and a display information providing unit 1331C.

[0091] (Voxel extraction unit 1331A) The voxel extraction unit 1331A uses floor surface information and an occupancy map to extract voxels that may become obstacles.

[0092] For example, the voxel extraction unit 1331A selects occupied voxels whose state is occupied from among the voxels in the occupancy map. The voxel extraction unit 1331A uses floor information to extract, from the selected occupied voxels, occupied voxels that are on the floor, as voxels that may become obstacles (hereinafter also referred to as voxels to be divided).

[0093] The voxel extraction unit 1331A outputs the extracted voxels to be divided to the clustering processing unit 1331B.

[0094] (Clustering processing unit 1331B) The clustering processing unit 1331B clusters the voxels to be divided based on the connectivity information of the voxels to be divided. The clustering processing unit 1331B classifies the voxels to be divided into connected voxel groups using, for example, the technique described in Reference

[10] .

[0095] The clustering processing unit 1331B looks at the series of voxels to be divided, determines that a cluster boundary has occurred where the series is broken, and assigns the same label to the voxels to be divided within the boundary.

[0096] In this way, the clustering processing unit 1331B classifies connected voxels to be divided as one obstacle, and assigns the same label (hereinafter also referred to as an obstacle label).

[0097] Generally, obstacles in the real space are often placed on the floor surface. The voxels to be divided are occupied voxels excluding the floor surface. Therefore, the clustering processing unit 1331B divides the voxels to be divided based on the connectivity information of the voxels to be divided excluding the floor surface, and the clustering processing unit 1331B can detect individual obstacles on the floor surface.

[0098] 10 is a diagram illustrating an example of obstacles detected by the clustering processing unit 1331B according to an embodiment of the present disclosure. As shown in Fig. 10, the clustering processing unit 1331B detects, for example, three circled obstacles by clustering the voxels to be divided, and assigns obstacle labels CL1 to CL3 to each of them.

[0099] Voxels assigned with obstacle labels CL1 to CL3 are connected via the floor surface, but by the voxel extraction unit 1331A extracting the voxels to be divided from which the floor surface has been removed, the clustering processing unit 1331B can detect obstacles more accurately.

[0100] Returning to Fig. 9, the clustering processing unit 1331B outputs the labeled occupancy map to which the obstacle labels CL have been added to the display information adding unit 1331C.

[0101] (Display information providing unit 1331C) The display information assigning unit 1331C assigns a display label to mesh data corresponding to voxels to which an obstacle label CL has been assigned, among the mesh information. The display information assigning unit 1331C generates mesh information with a display label by assigning a display label to the mesh data. Note that the mesh data corresponding to the voxels to which the obstacle label CL has been assigned is mesh data calculated from the TSDF information held in the voxels. The display label is display information used to change the display in subsequent processing.

[0102] The display information attached to the mesh data is not limited to a display label, and may be any information that can be used in the subsequent highlighting process, such as meta information.

[0103] Although the display information assigning unit 1331C assigns a display label to mesh data calculated from TSDF information, this is not limiting. The display information assigning unit 1331C may assign a display label to TSDF information. In this case, the display information assigning unit 1331C assigns a display label to TSDF information held in a voxel to which an obstacle label CL has been assigned, thereby generating mesh information with a display label.

[0104] 11 is a diagram illustrating mesh information with display labels according to an embodiment of the present disclosure. A display change unit 1333, which will be described later, generates a display image by highlighting mesh data to which display labels have been added.

[0105] As shown in FIG. 11, the display change unit 1333 generates a display image by highlighting the mesh areas M1 to M3 corresponding to the obstacle labels CL1 to CL3 (see FIG. 10).

[0106] In this way, the display information adding unit 1331C generates mesh information with display labels, and thus the information processing device 100 can generate a display image in which obstacles are highlighted.

[0107] Returning to Fig. 9, the display information providing unit 1331C outputs the labeled occupancy map and the mesh information with display labels to the fake obstacle determining unit 1332. The labeled occupancy map and the mesh information with display labels are also collectively referred to as obstacle information.

[0108] (Fake obstacle determination unit 1332) Returning to FIG. 8, the fake obstacle determination unit 1332 determines whether an object that is considered to be an obstacle in real space is a fake obstacle such as noise, based on floor surface information, the occupancy map, and obstacle information. The fake obstacle determination unit 1332 determines whether an obstacle is a fake obstacle by determining the outlier rate of each obstacle. For example, the fake obstacle determination unit 1332 determines the outlier rate of voxels whose state is occupied according to a predetermined condition.

[0109] 12 is a diagram illustrating a configuration example of the fake obstacle determination unit 1332 according to an embodiment of the present disclosure. As illustrated in FIG. 12, the fake obstacle determination unit 1332 includes an element number calculation unit 1332A, a spatial statistics calculation unit 1332B, a temporal statistics calculation unit 1332C, an outlier determination unit 1332D, and an outlier information assignment unit 1332K.

[0110] (Number of elements calculation unit 1332A) The element number calculation unit 1332A calculates the number of series of occupied voxels (the number of elements) using connectivity information of occupied voxels among the voxels in the occupancy map. That is, the element number calculation unit 1332A calculates the size of an object detected as an obstacle (occupied) in real space.

[0111] At this time, the element number calculation unit 1332A may calculate the number of elements by treating multiple obstacles connected via the floor surface as one obstacle, or may calculate the number of elements of occupied voxels excluding the floor surface.

[0112] The occupied voxels excluding the floor surface are the voxels to be divided as described above. Therefore, when calculating the number of elements of the occupied voxels excluding the floor surface, the element number calculation unit 1332A may count the number of elements of voxels to which the same obstacle label CL is assigned in the labeled occupancy map.

[0113] For example, if the information processing device 100 erroneously determines that an area without an obstacle is occupied due to noise, the size of the erroneously determined obstacle (false obstacle) is often smaller than the actual obstacle.

[0114] Therefore, by the element number calculation unit 1332A calculating the size of the obstacle (the number of elements of connected occupied voxels), the outlier determination unit 1332D at the subsequent stage can determine whether or not it is an obstacle.

[0115] At this time, it is desirable to detect an object close to the floor surface as an obstacle even if it is small in size, because obstacles close to the floor surface are likely to hinder the movement of the user U. On the other hand, obstacles that are far from the floor surface, such as near the ceiling, often do not hinder the movement of the user U. Therefore, it is desirable to detect objects that are far from the floor surface and that are relatively large in size as obstacles.

[0116] In other words, it is desirable to make it difficult for obstacles close to the floor to be determined as fake obstacles even if they are small in size. Also, it is desirable to make obstacles far from the floor be determined as fake obstacles if they are small in size. In this way, the criterion for determining whether an obstacle is a fake obstacle depends on the distance from the floor.

[0117] Therefore, the element number calculation unit 1332A calculates the height from the floor of the obstacle (connected occupied voxels) whose number of elements has been counted. The element number calculation unit 1332A calculates the minimum distance between the counted elements (voxels) and the floor as the floor height. Alternatively, the element number calculation unit 1332A may calculate the minimum distance between the counted elements and the floor as the floor height, or may calculate the average value as the floor height. Alternatively, the element number calculation unit 1332A may calculate the floor height of each voxel included in the obstacle.

[0118] The element number calculation unit 1332A outputs the calculated number of elements and height information relating to the height from the floor to the outlier determination unit 1332D.

[0119] (Spatial statistics calculation unit 1332B) The spatial statistics calculation unit 1332B calculates the ratio of voxels whose state is Unknown (hereinafter also referred to as spatial Unknown ratio) among voxels surrounding an object (for example, occupied voxels).

[0120] The spatial statistics calculation unit 1332B obtains the state of the 3x3x3 voxels (hereinafter also referred to as surrounding voxels) surrounding the occupied voxel from the occupancy map, and calculates the ratio of unknown voxels to the surrounding voxels as the spatial unknown ratio.

[0121] When a voxel is determined to be occupied due to noise, there is a high possibility that the surroundings of the voxel are not occupied voxels, such as unknown voxels.

[0122] Therefore, by the spatial statistics calculation unit 1332B calculating the spatial unknown ratio, the outlier determination unit 1332D at the subsequent stage can determine whether or not an occupied voxel is an obstacle.

[0123] As with the number of elements, when the criteria for determining whether a false obstacle exists are changed depending on the height of the occupied voxel from the floor, the spatial statistics calculation unit 1332B calculates the height of the occupied voxel from the floor for which the spatial unknown ratio is calculated.

[0124] The spatial statistics calculation unit 1332B outputs the calculated spatial unknown ratio and height information relating to the height from the floor to the outlier determination unit 1332D.

[0125] Although the number of surrounding voxels is set to 27 in this example, the number is not limited to 27. The number of surrounding voxels may be less than 27 or may be 28 or more.

[0126] (Time statistics calculation unit 1332C) The time statistic calculation unit 1332C calculates the ratio of time change in the state of the Occupancy Map as the time unknown ratio.

[0127] For example, it is assumed that the information processing device 100 includes a buffer (not shown) that stores the state of a target voxel for which the temporal unknown ratio is to be calculated for the past 10 frames. The buffer is included in the storage unit 120, for example.

[0128] The temporal statistic calculation unit 1332C acquires the state of the target voxel for the past 10 frames from the buffer, and calculates the proportion of the 10 frames in which the state is unknown as the temporal unknown proportion.

[0129] For example, suppose the state of a target voxel transitions from Unknown to Occupied or Free, but then transitions back to Unknown within a few frames, i.e., the majority of the 10 frames are Unknown. In this case, the transition of the target voxel to Occupied or Free is likely due to depth noise.

[0130] Therefore, by the time statistic calculation unit 1332C calculating the time unknown ratio, the outlier determination unit 1332D at the subsequent stage can determine whether or not the target voxel is an obstacle.

[0131] When saving the state of the target voxel in the buffer, the temporal statistic calculation unit 1332C switches whether to save the state depending on whether the target voxel is within the distance measurement range of the distance measurement device.

[0132] As described above, the state of the occupancy map is updated over time. At this time, if the target voxel is outside the range of the distance measurement device, the reliability of the state of the target voxel becomes low, and the state transitions to Unknown.

[0133] Therefore, even if the target voxel is outside the distance measurement range, if the time statistics calculation unit 1332C saves the state of the voxel in a buffer, the state of the target voxel will often be saved as Unknown, even if an object actually exists in the target voxel.

[0134] Therefore, the temporal statistic calculation unit 1332C according to an embodiment of the present disclosure stores the state of a target voxel in a buffer when the target voxel is within the ranging range. In other words, the temporal statistic calculation unit 1332C determines whether the state of the target voxel has changed based on the ranging range of the ranging device. The temporal statistic calculation unit 1332C calculates a temporal unknown ratio according to the change in the state of the target voxel over time when the target voxel is within the ranging range of the ranging device.

[0135] This point will be described with reference to Figures 13 to 15. Figures 13 and 14 are diagrams for explaining the relationship between a target voxel and the ranging range of a ranging device according to an embodiment of the present disclosure.

[0136] Fig. 13 shows a case where the target voxel B is included within the ranging range R of the ranging device 260. Fig. 14 shows a case where the target voxel B is not included within the ranging range R of the ranging device 260. The ranging device 260 corresponds to, for example, the sensor unit 220 of the terminal device 200 (see Fig. 5).

[0137] 13 and 14, the distance measurement range R changes in accordance with the movement of the distance measurement device 260, so that the target voxel B is sometimes located within the distance measurement range R and sometimes located outside the distance measurement range R. It is also assumed that an object exists in the target voxel shown in FIGS.

[0138] An example of the state transition of the target voxel B in this case will be described with reference to Fig. 15. Fig. 15 is a diagram for explaining an example of the state transition of the target voxel B according to an embodiment of the present disclosure.

[0139] In the example of FIG. 15, the target voxel is included in the ranging range R in periods T1 and T3 (see FIG. 13), and the target voxel is not included in the ranging range R in period T2 (see FIG. 14).

[0140] Figure 15 shows the state at each time, the time unknown ratio, etc. for each frame. Here, the observation starts from the first frame, and the state transition of the target voxel B up to the 16th frame is shown. Also, in Figure 15, the buffer holds the state of the target voxel B for three frames.

[0141] 15, when the ranging device 260 starts observing the target voxel B at the beginning of the period T1, the existence probability calculated by the information processing device 100 gradually increases. Therefore, the state of the target voxel B, which was Unknown (Un) in the first frame, transitions to Occupied (Occ) in the second frame.

[0142] In addition, in the first and second frames, the target voxel B is located within the ranging range R, so when determining whether it is within the observation range, the time statistic calculation unit 1332C determines Yes (within the observation range). In this case, the time statistic calculation unit 1332C saves the state of the target voxel B in a buffer.

[0143] In addition, since the buffer does not store the status of three frames for the first and second frames, the time statistics calculation unit 1332C treats the time unknown ratio as N / A (not applicable) regardless of the status stored in the buffer.

[0144] Because the distance measuring device 260 continues to observe the target voxel B until the sixth frame when the period T1 ends, the state of the target voxel B is Occupied (Occ) from the second frame to the sixth frame. Also, the determination result of whether or not the target voxel B is within the observation range is "Yes" from the first frame to the sixth frame.

[0145] Therefore, the buffer holds the state of the target voxel B for the past three frames. Specifically, in the second frame, the buffer holds "Unknown," "Occupied," and "Occupied." In the third to sixth frames, the buffer holds "Occupied" for all three frames.

[0146] The temporal statistic calculation unit 1332C calculates a temporal unknown ratio of "0.33" for the second frame, and calculates a temporal unknown ratio of "0" for the third to sixth frames.

[0147] Next, in period T2, when target voxel B moves out of measurement range R, target voxel B is no longer observed and its existence probability gradually decreases. Therefore, the state that was "Occupied (Occ)" up until the 7th frame transitions to "Unknown (Un)" from the 8th frame to the 12th frame.

[0148] However, from the 7th frame to the 12th frame, the target voxel B is not within the observation range (the observation range determination result is "No"), so the state of this period is not held in the buffer. Therefore, from the 7th frame to the 12th frame, the buffer continues to hold the state of the past three frames up to the 6th frame. Therefore, the temporal statistics calculation unit 1332C calculates the temporal unknown ratio of "0" for the 7th frame to the 12th frame, the same as for the 6th frame.

[0149] When period T2 ends and period T3 begins, target voxel B again enters the ranging range R. Therefore, when period T3 begins, the existence probability gradually increases, and the state of target voxel B, which was Unknown (Un) in the 13th frame, transitions to Occupied (Occ) in the 14th to 16th frames. In addition, the determination result of the observation range is "Yes" from the 13th to 16th frames.

[0150] Therefore, the buffer holds the state of target voxel B for the past three frames. Specifically, in the 13th frame, the buffer holds "Occupied", "Occupied", and "Unknown". In the 14th frame, the buffer holds "Occupied", "Unknown", and "Occupied". In the 15th frame, the buffer holds "Unknown", "Occupied", and "Occupied". In the 16th frame, the buffer holds "Occupied" for all of the past three frames.

[0151] The temporal statistic calculation unit 1332C calculates a temporal unknown ratio of "0.33" from the thirteenth frame to the fifteenth frame, and calculates a temporal unknown ratio of "0" for the sixteenth frame.

[0152] In this way, the temporal statistic calculation unit 1332C calculates the temporal unknown ratio in accordance with the state change when the target voxel B is included in the ranging range R (observation range). This allows the temporal statistic calculation unit 1332C to calculate the temporal unknown ratio with higher accuracy by excluding cases where the target voxel B transitions to unknown due to not being observed.

[0153] As with the number of elements, when the criteria for determining whether a false obstacle exists are changed depending on the height of the occupied voxel from the floor, the time statistics calculation unit 1332C calculates the height from the floor of the target voxel B for which the time unknown ratio has been calculated.

[0154] The time statistic calculation unit 1332C outputs the calculated time unknown ratio and height information relating to the height from the floor surface to the outlier determination unit 1332D.

[0155] Although the buffer holds 10 or 3 frames here, the number of frames held by the buffer can be selected appropriately depending on, for example, the size of the buffer, the number of target voxels B, etc.

[0156] (Outlier determination unit 1332D) Returning to Fig. 12, the outlier determination unit 1332D calculates the outlier rate of each voxel based on the calculation results of the element number calculation unit 1332A, the spatial statistics calculation unit 1332B, and the temporal statistics calculation unit 1332C.

[0157] As shown in FIG. 12, the outlier determining unit 1332D includes an outlier rate L1 calculating unit 1332E, an outlier rate L2 calculating unit 1332F, an outlier rate L3 calculating unit 1332G, and an outlier rate integrating unit 1332H.

[0158] The outlier rate L1 calculation unit 1332E calculates a first outlier rate L1 using the number of elements calculated by the number of elements calculation unit 1332A. The outlier rate L2 calculation unit 1332F calculates a second outlier rate L2 using the spatial unknown ratio calculated by the spatial statistics calculation unit 1332B. The outlier rate L3 calculation unit 1332G calculates a third outlier rate L3 using the temporal unknown ratio calculated by the temporal statistics calculation unit 1332C. The outlier rate integration unit 1332H calculates an outlier rate L for each voxel from the first to third outlier rates L1 to L3.

[0159] (Outlier rate L1 calculation unit 1332E) The outlier rate L1 calculation unit 1332E determines the first outlier rate L1 according to the number of elements of the obstacle that includes the calculation target voxel that is the calculation target of the first outlier rate L1 among the occupied voxels. The outlier rate L1 calculation unit 1332E calculates the first outlier rate L1 based on the following formula (1).

[0160]

number

[0161] Here, n is the number of elements, and n0 and n1 are thresholds (parameters) whose values ​​are determined according to, for example, the height of the obstacle from the floor surface.

[0162] In this way, the outlier rate L1 calculation unit 1332E can determine the first outlier rate L1 according to the height from the floor surface by changing the values ​​of n0 and n1 according to the height from the floor surface of the obstacle.

[0163] (Outlier rate L2 calculation unit 1332F) The outlier rate L2 calculation unit 1332F determines the second outlier rate L2 according to the spatial unknown ratio among the occupied voxels. The outlier rate L2 calculation unit 1332F calculates the second outlier rate L2 based on the following formula (2).

[0164]

number

[0165] Here, k is the spatial known ratio, which is calculated as k = 1 - spatial unknown ratio. Furthermore, k0 and k1 are thresholds (parameters) whose values ​​are determined depending on the height from the floor of the calculation target voxel that is the calculation target of the second outlier rate L2, for example.

[0166] In this way, the outlier rate L2 calculation unit 1332F can determine the second outlier rate L2 according to the height from the floor by changing the values ​​of k0 and k1 according to the height from the floor of the voxel to be calculated.

[0167] (Outlier rate L3 calculation unit 1332G) The outlier rate L3 calculation unit 1332G determines the third outlier rate L3 according to the time unknown ratio among the occupied voxels. The outlier rate L3 calculation unit 1332G calculates the third outlier rate L3 based on the following formula (3).

[0168]

number

[0169] Here, h is the time known ratio, which is calculated as h = 1 - time unknown ratio. Also, h0 and h1 are thresholds (parameters) whose values ​​are determined depending on the height from the floor of the calculation target voxel that is the calculation target of the third outlier rate L3, for example.

[0170] In this way, the outlier rate L3 calculation unit 1332G can determine the third outlier rate L3 according to the height from the floor by changing the values ​​of h0 and h1 according to the height from the floor of the voxel to be calculated.

[0171] (Outlier rate integration unit 1332H) The outlier rate integrating unit 1332H integrates the first to third outlier rates L1 to L3 to determine the outlier rate L of each voxel. The outlier rate integrating unit 1332H calculates the weighted average of the first to third outlier rates L1 to L3 as the outlier rate L, for example, as shown in equation (4).

[0172]

number

[0173] Alternatively, the outlier rate integrating unit 1332H may calculate the minimum value of the first to third outlier rates L1 to L3 as the outlier rate L, as shown in equation (5).

[0174]

number

[0175] The outlier rate integrating unit 1332H outputs the determined outlier rate L to the outlier information adding unit 1332K.

[0176] (Outlier information assignment part 1332K) The outlier information assigning unit 1332K assigns an outlier rate L to mesh data corresponding to a calculation target voxel in the display labeled mesh information. The outlier information assigning unit 1332K generates mesh information with outliers by assigning the outlier rate L to the mesh data. Note that the mesh data corresponding to the calculation target voxel is mesh data calculated using TSDF information held in the voxel that is the target of calculation of the outlier rate L.

[0177] Also, although it has been described here that the outlier information assigning unit 1332K assigns the outlier rate L to the mesh data calculated from the TSDF information, this is not limiting. The outlier information assigning unit 1332K may assign the outlier rate L to the TSDF information. In this case, the outlier information assigning unit 1332K generates mesh information with outliers by assigning the outlier rate L to the TSDF information held in the voxel for which the outlier rate L is to be calculated.

[0178] The outlier rate integration unit 1332H outputs the generated mesh information with outliers to the display change unit 1333.

[0179] (Display change unit 1333) Returning to Fig. 6, the display change unit 1333 generates a two-dimensional display image to be presented to the user U based on the mesh information with outliers.

[0180] As described above, the mesh information with outliers includes at least one of a display label and an outlier rate L. The display change unit 1333 highlights the mesh data to which the display label has been assigned. Furthermore, the display change unit 1333 suppresses the display of mesh data to which the outlier rate L has been assigned in accordance with the outlier rate L.

[0181] The display change unit 1333 highlights the mesh data to which a display label has been assigned by displaying the edge lines of the mesh data to which a display label has been assigned in a different shade from that of the mesh data to which a display label has not been assigned.

[0182] Alternatively, the display change unit 1333 may highlight the mesh data to which a display label has been assigned by displaying the face of the mesh data to which a display label has been assigned in a color or hatching that is different from that of the mesh data to which a display label has not been assigned.

[0183] Furthermore, the display change unit 1333 suppresses the display of the mesh data to which the outlier rate L has been assigned by changing the transparency of the mesh data to which the outlier rate L has been assigned in accordance with the value of the outlier rate L. For example, the display change unit 1333 suppresses the display of the mesh data by setting the outlier rate L of the mesh data to the transparency of the mesh data.

[0184] The display change unit 1333 may generate a display image in which mesh data to which a display label has been assigned is highlighted, and the highlighting method is not limited to the above-described example. For example, the display change unit 1333 may highlight mesh data to which a display label has been assigned by blinking the data.

[0185] Furthermore, the display change unit 1333 may make the mesh data less visible to the user U in accordance with the outlier rate L, and the method of making the mesh data less visible, i.e., suppressing the display, is not limited to the above-described example. For example, the display change unit 1333 may set the transmittance of mesh data whose outlier rate L is equal to or greater than a predetermined value to 100% or change the color to the background color.

[0186] The display change unit 1333 transmits the generated display image to the terminal device 200 via the communication unit 110.

[0187] <<3. Information Processing Example>> <3.1. Image generation processing> Fig. 16 is a flowchart showing an example of the flow of image generation processing according to an embodiment of the present disclosure. The image generation processing shown in Fig. 16 is executed at a predetermined cycle by, for example, the information processing device 100. Note that the predetermined cycle may be the same as the distance measurement cycle (frame cycle) of the distance measuring device.

[0188] 16, the information processing device 100 executes three-dimensional information generation processing (step S101). As the three-dimensional information generation processing, the information processing device 100 estimates a camera pose and a direction of gravity from IMU information and a camera image acquired from the terminal device 200, for example. The information processing device 100 generates an occupancy map and mesh information using the camera pose, the direction of gravity, and distance information acquired from the terminal device 200.

[0189] The information processing device 100 executes obstacle division processing using the occupancy map and mesh information (step S102) to generate mesh information with display labels. The obstacle division processing will be described later.

[0190] The information processing device 100 executes a false obstacle determination process using the occupancy map and the mesh information (step S103) to generate mesh information with outliers. The false obstacle determination process will be described later.

[0191] The information processing device 100 executes a display image generation process using the mesh information with outliers (step S104) to generate a display image. The display image generation process will be described later.

[0192] <3.2. Obstacle segmentation processing> 17 is a flowchart showing an example of the flow of the obstacle segmentation process according to an embodiment of the present disclosure. The obstacle segmentation process shown in FIG. 17 is executed in step S102 of the image generation process in FIG.

[0193] The information processing device 100 extracts voxels to be divided using the occupancy map and floor information (step S201). The information processing device 100 selects occupied voxels whose state is occupied from among the voxels in the occupancy map. The information processing device 100 uses the floor information to extract, from among the selected occupied voxels, occupied voxels excluding occupied voxels that are floors, as voxels to be divided.

[0194] The information processing device 100 clusters the extracted voxels to be divided (step S202). The information processing device 100 assigns display labels to the clustered voxels (step S203) and generates mesh information with display labels (step S204).

[0195] <3.3. Fake Obstacle Detection Processing> 18 is a flowchart showing an example of the flow of the fake obstacle determination process according to an embodiment of the present disclosure. The fake obstacle determination process shown in FIG. 18 is executed in step S103 of the image generation process in FIG.

[0196] The information processing device 100 counts the number of elements of occupied voxels using connectivity information of occupied voxels among the voxels in the occupancy map (step S301). The information processing device 100 calculates a first outlier rate L1 according to the counted number of elements (step S302).

[0197] The information processing device 100 calculates a spatial unknown ratio using voxels surrounding the occupied voxels among the voxels in the occupancy map (step S303). The information processing device 100 calculates a second outlier ratio L2 according to the spatial unknown ratio (step S304).

[0198] The information processing device 100 calculates the time unknown ratio according to the time change in the state of the occupancy map (step S305).The information processing device 100 calculates a third outlier rate L3 according to the time unknown ratio (step S306).

[0199] The information processing device 100 calculates the outlier rate L based on the first to third outlier rates L1 to L3 (step S307), and generates mesh information with outliers (step S308).

[0200] <3.4. Display image generation process> 19 is a flowchart showing an example of the flow of the display image generation process according to an embodiment of the present disclosure. The display image generation process shown in FIG. 19 is executed in step S104 of the image generation process in FIG.

[0201] 19, the information processing device 100 highlights the obstacle Ob based on the display label of the mesh information with outliers (step S401). The information processing device 100 highlights the obstacle Ob, for example, by highlighting the mesh data to which the display label has been assigned.

[0202] The information processing device 100 suppresses and displays the fake obstacles based on the outlier rate L of the mesh information with outliers (step S402). The information processing device 100 sets the outlier rate L as a transparency, for example. The information processing device 100 changes the transparency of the mesh of the fake obstacle when it is displayed so that the mesh in the voxel is not displayed as the transparency approaches 1.

[0203] The information processing device 100 generates a display image by highlighting the obstacles and suppressing the display of the fake obstacles (step S403).

[0204] As described above, the information processing device 100 according to an embodiment of the present disclosure can divide mesh data of mesh information into obstacles by clustering voxels in an occupancy map. In this case, the information processing device 100 does not need to use a large-scale classifier, and can detect obstacles at high speed while suppressing an increase in resources.

[0205] Furthermore, the information processing device 100 can suppress the display of meshes that are low in reliability, in other words, that are likely to be false obstacles, by using voxels in an unobserved state (Unknown) in the occupancy map. In particular, the information processing device 100 suppresses the display of meshes by using the size of the clustered voxels, spatial statistics (spatial Unknown ratio), and temporal statistics (temporal Unknown ratio). This allows the information processing device 100 to generate a display image that is more reliable, in other words, that has fewer false obstacles.

[0206] For example, there are objects that are prone to errors (noise) depending on the distance measurement method of the distance measurement device 260. For example, in the case of a distance measurement device that measures distance using a stereo camera, noise is likely to occur on textureless surfaces. In this way, even when noise occurs on a specific object, the information processing device 100 can generate mesh information with higher accuracy.

[0207] As described above, the information processing device 100 can detect obstacles and determine whether they are fake obstacles at high speed and with low resources. Therefore, even in an HMD system that presents a display image to the user U in real time, the information processing device 100 can present a display image that highlights obstacles to the user U while suppressing the display of fake obstacles.

[0208] In this way, the information processing device 100 can present the user U with a display image that highlights obstacles while suppressing the display of fake obstacles, with low resources and at high speed, allowing the user U to enjoy the content more safely.

[0209] <<4. Modifications>> In the above-described embodiment, the information processing device 100 calculates the spatial Unknown ratio using 3x3x3 surrounding voxels of an occupied voxel, in other words, a cubic shape, as a predetermined condition, but this is not limiting. The information processing device 100 may also calculate the spatial Unknown ratio using 1x1xm surrounding voxels (m is an integer equal to or greater than 3), in other words, a rectangular parallelepiped shape.

[0210] For example, the information processing apparatus 100 determines voxels in the direction of gravity among those around the occupied voxel as surrounding voxels, and calculates the spatial unknown ratio according to the state of the surrounding voxels.

[0211] 20 is a diagram illustrating an example of a real space according to a modified example of the embodiment of the present disclosure. As shown in FIG. 20, it is assumed that the real space is indoors, that is, a space surrounded by a floor and walls.

[0212] In this case, when the real space is expressed as an occupancy map, voxels (spaces) where walls, floors, and objects (obstacles) are located become Occupied voxels. Also, voxels (spaces) within the ranging range of the ranging device 260 excluding Occupied voxels become Free voxels. Also, areas outside the ranging range of the ranging device 260 become Unknown voxels. Furthermore, voxels (spaces) that are in the shadow of objects and cannot be measured by the ranging device 260 become Unknown voxels.

[0213] 20, the area behind the wall or below the floor is blocked by the wall or floor and cannot be measured by the distance measuring device 260, so the voxels in that area become unknown voxels. However, due to noise or the like, an occupied voxel B may be observed behind the wall.

[0214] In this case, if the information processing device 100 calculates the spatial unknown ratio using a 3x3x3 cube as surrounding voxels, the surrounding voxels will include walls, i.e., occupied voxels, which will result in a decrease in the spatial unknown ratio. As a result, the information processing device 100 may not be able to suppress the display of occupied voxel B, and it may end up being displayed in the display image.

[0215] On the other hand, because there are no actual obstacles on the other side of the wall, all voxels other than noise-caused occupied voxels or free voxels are unknown voxels. Therefore, voxels in the z-axis direction (gravity direction) on the other side of the wall (voxels in region R1 in FIG. 20) are highly likely to be unknown voxels. In other words, if the spatial unknown ratio of the surrounding voxel R1 in the z-axis direction is high, occupied voxel B is likely to be a false obstacle located behind the wall.

[0216] Therefore, in this modification, the information processing device 100 calculates the spatial unknown ratio in the voxels R1 surrounding the occupied voxel B in the z-axis direction. The information processing device 100 determines a second outlier rate L2 according to the spatial unknown ratio, and suppresses the display of the occupied voxel B.

[0217] In this way, in this modified example, the information processing device 100 calculates the spatial Unknown ratio of the surrounding voxel R1 in the z-axis direction, thereby making it possible to more accurately determine fake obstacles on the other side of the wall (behind the wall), and to more accurately suppress the display of fake obstacles.

[0218] In this way, the number and shape of the surrounding voxels are not limited to a cube or the direction of gravity, but can be set arbitrarily depending on the position of the occupied voxel B in real space, etc.

[0219] <<5. Other embodiments>> The above-described embodiment and each modified example are merely examples, and various modifications and applications are possible.

[0220] For example, some of the functions of the information processing device 100 of this embodiment may be implemented by the terminal device 200. For example, the terminal device 200 may generate mesh information with outliers.

[0221] In the above-described embodiment, the information processing device 100 highlights the obstacles Ob that exist within the play area PA, but this is not limiting. The information processing device 100 may also divide (classify) and highlight the obstacles Ob that exist outside the play area PA.

[0222] In the above-described embodiment, the information processing device 100 or the user U sets the play area PA of the user U, but this is not limited to this. For example, the information processing device 100 may set a range in which a moving object such as a vehicle or a drone can move safely as the play area. Alternatively, the information processing device 100 may set a range in which a partially fixed object such as a robot arm can be driven safely as the play area. In this way, the object for which the information processing device 100 sets the play area is not limited to the user U.

[0223] For example, a communication program for executing the above-described operations is stored in a computer-readable recording medium such as an optical disk, a semiconductor memory, a magnetic tape, or a flexible disk and distributed. Then, for example, the program is installed in a computer and the above-described processing is executed to configure a control device. In this case, the control device may be a device external to the information processing device 100 or the terminal device 200 (for example, a personal computer). Alternatively, the control device may be a device internal to the information processing device 100 or the terminal device 200 (for example, the control units 130 and 250).

[0224] The communication program may also be stored in a disk device provided in a server device on a network such as the Internet, and may be downloaded to a computer. The above-mentioned functions may also be realized by cooperation between an OS (Operating System) and application software. In this case, the parts other than the OS may be stored on a medium and distributed, or may be stored in a server device and downloaded to a computer.

[0225] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0226] Furthermore, the components of each device shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads and usage conditions. This distribution and integration configuration may also be performed dynamically.

[0227] The above-described embodiments can be combined as appropriate within the scope of the processing content without causing inconsistency. The order of the steps shown in the sequence diagrams of the above-described embodiments can be changed as appropriate.

[0228] Furthermore, for example, this embodiment can also be implemented as any configuration that constitutes an apparatus or system, such as a processor as a system LSI (Large Scale Integration), a module using multiple processors, a unit using multiple modules, a set in which other functions are added to a unit, etc. (i.e., a configuration of a part of an apparatus).

[0229] In this embodiment, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

[0230] Furthermore, for example, this embodiment can be configured as a cloud computing system in which one function is shared and processed jointly by a plurality of devices via a network.

[0231] <<6. Hardware Configuration>> An information processing device such as the information processing device 100 according to each of the above-described embodiments is realized by, for example, a computer 1000 configured as shown in FIG. 21 . The information processing device 100 according to an embodiment of the present disclosure will be described below as an example. FIG. 21 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the information processing device 100 according to an embodiment of the present disclosure. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, an HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0232] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0233] The ROM 1300 stores boot programs such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0234] The HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the HDD 1400 is a recording medium that records a program for a medical arm control method according to the present disclosure, which is an example of program data 1450.

[0235] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0236] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined computer-readable recording medium. Examples of the medium include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disk), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0237] For example, when the computer 1000 functions as the information processing device 100 according to an embodiment of the present disclosure, the CPU 1100 of the computer 1000 executes a program loaded onto the RAM 1200 to realize functions of the control unit 130, etc. Note that the CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, the CPU 1100 may obtain the information processing program from another device via an external network 1550.

[0238] Furthermore, the information processing device 100 according to the present embodiment may be applied to a system consisting of a plurality of devices that is premised on connection to a network (or communication between devices), such as cloud computing, etc. In other words, the information processing device 100 according to the present embodiment described above can also be realized as the information processing system 1 according to the present embodiment by a plurality of devices, for example.

[0239] The above shows an example of the hardware configuration of the information processing device 100. Each of the above components may be configured using general-purpose components, or may be configured using hardware specialized for the function of each component. Such a configuration may be changed as appropriate depending on the technical level at the time of implementation.

[0240] <<7. Conclusion>> Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of different embodiments and modifications may be combined as appropriate.

[0241] Furthermore, the effects of each embodiment described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0242] The present technology can also be configured as follows. (1) acquiring first three-dimensional information relating to an occupancy probability of an object in real space and second three-dimensional information relating to an estimation result of a surface shape of the object; classifying the object based on the first three-dimensional information and floor surface information relating to a floor surface of the real space; a control unit that highlights the surface of the object classified based on the second three-dimensional information. An information processing device comprising: (2) The control unit determining an outlier rate of the object in accordance with a predetermined condition based on the first three-dimensional information and the floor surface information; changing the display of the object in response to the outlier rate; An information processing device according to (1). (3) The information processing device according to (2), wherein the control unit determines the outlier rate according to a size of the object as the predetermined condition. (4) The information processing device according to (2) or (3), wherein the control unit determines the outlier rate according to a ratio of voxels that are in an unobserved state among voxels surrounding the object, as the predetermined condition. (5) The information processing device according to (4), wherein the control unit determines the outlier rate according to a state of the voxels in the gravity direction of the real space within the periphery of the object. (6) The information processing device according to any one of (2) to (5), wherein the control unit determines the outlier rate according to a time change in a state of the first three-dimensional information as the predetermined condition. (7) the first three-dimensional information is updated based on distance information measured by a distance measuring device; the control unit determines whether the state has changed based on the distance measurement range of the distance measurement device. (6) An information processing device according to the present invention. (8) The information processing device according to any one of (2) to (7), wherein the control unit determines the outlier rate according to a height of the object from the floor surface. (9) The information processing device according to any one of (1) to (8), wherein the first three-dimensional information is an occupancy grid map. (10) The information processing device according to any one of (1) to (9), wherein the second three-dimensional information includes mesh data that defines a surface by a plurality of vertices and edges connecting the plurality of vertices. (11) The information processing device according to any one of (1) to (10), wherein the control unit highlights the surface of the object that exists within a movement range of an object that moves in the real space. (12) acquiring first three-dimensional information relating to an occupancy probability of an object in real space and second three-dimensional information relating to an estimation result of a surface shape of the object; classifying the object based on the first three-dimensional information and floor surface information relating to a floor surface of the real space; highlighting a surface of the object classified based on the second three-dimensional information; An information processing method including: (13) On the computer, acquiring first three-dimensional information relating to an occupancy probability of an object in real space and second three-dimensional information relating to an estimation result of a surface shape of the object; classifying the object based on the first three-dimensional information and floor surface information relating to a floor surface of the real space; a control unit that highlights the surface of the object classified based on the second three-dimensional information. A program that functions as a [Explanation of symbols]

[0243] 1. Information Processing Systems 100 Information processing device 110,210 Communications Department 120 Storage section 130,250 Control unit 131 Estimation Department 132 Integrated Processing Unit 133 Display control unit 200 Terminal Device 220 Sensor unit 230 Display section 230 The display unit 240 Input section 260 Rangefinder

Claims

1. acquiring first three-dimensional information relating to an occupancy probability of an object in real space and second three-dimensional information relating to an estimation result of a surface shape of the object; classifying the object based on the first three-dimensional information and floor surface information relating to a floor surface of the real space; a control unit that highlights the surface of the object classified based on the second three-dimensional information. An information processing device comprising:

2. The control unit determining an outlier rate of the object in accordance with a predetermined condition based on the first three-dimensional information and the floor surface information; changing the display of the object in response to the outlier rate; The information processing device according to claim 1 .

3. The information processing apparatus according to claim 2 , wherein the control unit determines the outlier rate according to a size of the object as the predetermined condition.

4. The information processing device according to claim 2 , wherein the control unit determines the outlier rate according to a ratio of voxels that are in an unobserved state among voxels around the object, as the predetermined condition.

5. The information processing device according to claim 4 , wherein the control unit determines the outlier rate according to a state of the voxels in the gravity direction of the real space within the periphery of the object.

6. The information processing device according to claim 2 , wherein the control unit determines the outlier rate according to a time change in a state of the first three-dimensional information as the predetermined condition.

7. the first three-dimensional information is updated based on distance information measured by a distance measuring device; the control unit determines whether the state has changed based on the distance measurement range of the distance measurement device. The information processing device according to claim 6 .

8. The information processing device according to claim 2 , wherein the control unit determines the outlier rate according to a height of the object from the floor surface.

9. The information processing apparatus according to claim 1 , wherein the first three-dimensional information is an occupancy grid map.

10. The information processing apparatus according to claim 1 , wherein the second three-dimensional information includes mesh data that defines a surface by a plurality of vertices and edges connecting the plurality of vertices.

11. The information processing device according to claim 1 , wherein the control unit highlights the surface of the object that exists within a range of movement of a target object that moves in the real space.

12. acquiring first three-dimensional information relating to an occupancy probability of an object in real space and second three-dimensional information relating to an estimation result of a surface shape of the object; classifying the object based on the first three-dimensional information and floor surface information relating to a floor surface of the real space; highlighting a surface of the object classified based on the second three-dimensional information; An information processing method including:

13. On the computer, acquiring first three-dimensional information relating to an occupancy probability of an object in real space and second three-dimensional information relating to an estimation result of a surface shape of the object; classifying the object based on the first three-dimensional information and floor surface information relating to a floor surface of the real space; a control unit that highlights the surface of the object classified based on the second three-dimensional information. A program that functions as a

Images