Information processing apparatus and information processing method
By combining time-of-flight data and image data to detect and track the ground, the problem of SLAM method tracking loss in the lack of feature environment is solved, and positioning accuracy and stability are improved.
Patent Information
- Application Number
- CN202380024801.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-07
- Filing Date
- 2023-02-27
- Publication Date
- 2024-11-08
AI Technical Summary
The existing simultaneous positioning and map construction (SLAM) methods are prone to loss of camera tracking in the absence of sufficient visible features, resulting in reduced position accuracy and troublesome and unreliable return tracking process, which cannot effectively solve the tracking loss problem.
Using a method of combining time of flight data and image data, the positioning of the information processing device is realized by detecting and tracking the ground, and using the time of flight device to provide direct measurement of height and vertical direction, the camera provides image data to improve tracking stability and accuracy.
It effectively reduces the situation of camera tracking loss, improves positioning accuracy and stability, and ensures that the information processing device can be accurately positioned in a lack of characteristic environment.
Smart Images

Figure CN118922863A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure generally relates to an information processing device and a corresponding information processing method for simultaneous positioning and mapping. Background Art
[0002] Generally, various methods for simultaneous localization and mapping ("SLAM") are known. Applications such as, for example, 3D room scanning with a smartphone typically rely on data from an accelerometer and a camera to track the position of the camera as it scans the room. However, this approach can have limitations because tracking can be lost, for example, when the camera does not have enough visible features in front of the scene (e.g., a wall with a uniform color), which prevents the camera from compensating for drift of the accelerometer.
[0003] The typical solution is to detect that tracking has been lost, notify the user and have them return to a place with enough known features visible to restart tracking.
[0004] However, in some cases, such back tracking can be cumbersome and unreliable, and may not resolve the problem of tracking loss resulting in still needing to work around.
[0005] While technologies exist for simultaneous positioning and tracking, it is often desirable to improve upon existing technologies. Summary of the invention
[0006] According to a first aspect, the present disclosure provides an information processing device for simultaneous positioning and map construction, the information processing device comprising a circuit, the circuit being configured to:
[0007] obtaining time-of-flight data of an environment of the information processing device;
[0008] detecting a ground surface in the environment based on the time-of-flight data;
[0009] obtaining image data representing an image of the environment;
[0010] Based on the image data, tracking the ground surface; and
[0011] Based on the detected and tracked ground surface, the information processing device is positioned in the environment.
[0012] According to a second aspect, the present disclosure provides an information processing method for simultaneous positioning and map construction, comprising:
[0013] obtaining time-of-flight data of an environment of the information processing device;
[0014] detecting a ground surface in the environment based on the time-of-flight data;
[0015] obtaining image data representing an image of the environment;
[0016] Based on the image data, tracking the ground surface; and
[0017] Based on the detected and tracked ground surface, the information processing device is positioned in the environment.
[0018] Further aspects are set out in the dependent claims, the following description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Embodiments are described by way of example with reference to the accompanying drawings, in which:
[0020] Figure 1 A block diagram schematically illustrating four embodiments of an information processing device;
[0021] Figure 2 A block diagram schematically illustrating one embodiment of an information processing device;
[0022] Figure 3 An embodiment of the information processing method is schematically shown in a flowchart; and
[0023] Figure 4 An embodiment of the information processing method is schematically shown in a flow chart. DETAILED DESCRIPTION
[0024] In giving reference Figure 1 Before a detailed description of the embodiments of the present invention, a general explanation is given.
[0025] As mentioned at the beginning, generally, various methods for simultaneous localization and mapping ("SLAM") are known. Applications such as, for example, 3D room scanning with a smartphone generally rely on data from an accelerometer and a camera to track the position of the camera as it scans the room. However, this approach can have limitations, as tracking can be lost, for example, when the camera does not have enough visible features in front of the scene (e.g., a wall with a uniform color), which prevents the camera from compensating for drift of the accelerometer.
[0026] The typical solution is to detect that tracking has been lost, notify the user and have them return to a place with enough known features visible to restart tracking.
[0027] However, in some cases, such back tracking can be cumbersome and unreliable, and may not resolve the problem of tracking loss resulting in still needing to work around.
[0028] Therefore, it has been recognized that the number of situations where the camera loses tracking should be reduced while improving position accuracy.
[0029] It is recognized that at least one downward-facing camera and optional additional sensors (e.g., an inertial measurement unit (“IMU”)) may be used, and in some embodiments, their inputs may be incorporated into a position tracking SLAM algorithm as follows:
[0030] • In some embodiments, a downward-facing depth sensing device (also referred to herein as a time-of-flight device) is used to provide a direct, robust measurement of the camera's altitude.
[0031] In some embodiments, a downward-facing depth sensing device is used to calculate a robust acceleration independent of the vertical direction by identifying and measuring the position of the horizontal portion of the ground.
[0032] • In some embodiments, a downward facing camera is used to capture horizontal movement by feeding images of the ground into the SLAM algorithm.
[0033] Therefore, some embodiments relate to an information processing device for simultaneous localization and mapping, wherein the information processing device comprises a circuit configured to:
[0034] obtaining time-of-flight data of an environment of the information processing device;
[0035] detecting a ground surface in the environment based on the time-of-flight data;
[0036] obtaining image data representing an image of the environment;
[0037] Tracking the ground surface based on the image data; and
[0038] Based on the detected and tracked ground surface, the information processing device is positioned in the environment.
[0039] The information processing device can be a smart phone, a tablet computer, an image or video acquisition device, etc.
[0040] The information processing device can be used in, for example, room scanning applications (e.g., scanning the interior of a building) or can be installed in a drone to scan terrain or street layout, etc.
[0041] The circuit may be based on or may include or may be implemented by typical electronic components configured to implement the functions described herein.
[0042] The circuit may be based on or may include or may be implemented as integrated circuit logic, and the functions may be implemented by software executed by a processor or the like.
[0043] The circuit may be based on or may include a CPU (central processing unit), a microcontroller, an FPGA (field programmable gate array), an ASIC (application-specific integrated circuit), a GPU (graphics processing unit), a DSP (digital signal processor), etc., or may be implemented by a CPU (central processing unit), a microcontroller, an FPGA (field programmable gate array), an ASIC (application-specific integrated circuit), a GPU (graphics processing unit), a DSP (digital signal processor), etc.
[0044] The circuit may be based on or may include or may be implemented partly by typical electronic components and integrated circuit logic and partly by software.
[0045] The circuitry may include storage capabilities such as magnetic memory, semiconductor memory, or the like.
[0046] The circuit may include a data bus for sending and receiving data, and may implement a corresponding communication protocol.
[0047] Time of flight ("ToF") data may be direct time of flight data or indirect time of flight data.
[0048] Direct time-of-flight data can be obtained from a direct time-of-flight device, also known as a LiDAR (“Light Detection and Ranging”) device, which in some embodiments includes, as is generally known, a light source configured to emit light pulses, an image sensor having multiple time-of-flight pixels (e.g., SPAD (“Single Photon Avalanche Diode”) pixels), which is configured to detect light and generate electrical signals in response to the detected light, and a control unit configured to control the overall operation of the direct time-of-flight device and generate histogram data based on the electrical signals of the generated time-of-flight pixels, the histogram data indicating multiple electrical signals (which may also be referred to as events) generated in a certain time interval since the emission of the light pulse (“arrival time”), and thereby indicating the distance to an object that reflected at least a portion of the emitted light pulse.
[0049] Indirect time-of-flight data can be obtained from an indirect time-of-flight device, which in some embodiments includes a light source as generally known, which is configured to emit light that is temporally modulated according to a modulation signal having a modulation frequency, an image sensor having a plurality of time-of-flight pixels (e.g., CAPD (“current assisted photon demodulator”) pixels), which is configured to detect light and generate electrical signals in response to the detected light according to a demodulation signal applied to the corresponding time-of-flight pixels, and a control unit, which is configured to control the overall operation of the indirect time-of-flight device and perform at least two correlation measurements - typically four (phase shifts of 0°, 90°, 180° and 270° between the modulation signal and the demodulation signal), and generate phase data (representing a phase image) and amplitude data (representing an amplitude image) based on the electrical signals of the time-of-flight pixels generated within the at least two correlation measurements.
[0050] A direct or indirect time-of-flight device may comprise a light source which is or comprises a full-field illuminator or a point illuminator.
[0051] In some embodiments, the full-field illuminator provides full-field illumination to the environment, such that the environment is illuminated with a continuous spatial light distribution, for example, a light beam having a high intensity region at the center of the beam and a continuously decreasing light intensity away from the center of the beam.
[0052] In some embodiments, the point illuminator provides point illumination to the environment, such that the environment is illuminated with a plurality of light points. In other words, the environment is illuminated by a light pattern (such as, for example, a pattern of light points) of (separate) high-intensity and low-intensity (or substantially zero-intensity) areas. In some embodiments, the point illuminator provides a spatially modulated illumination light field (light pattern) with vertical or horizontal stripes or with a checkerboard pattern. Generally, in some embodiments, the point illuminator provides a spatially modulated illumination field (light pattern) to the environment where the light intensity is low (or substantially zero) in a portion of the light pattern.
[0053] In some embodiments, the information processing device includes a time-of-flight device configured to acquire time-of-flight data and image data.
[0054] In some embodiments, the time-of-flight data is indirect time-of-flight data. In such embodiments, the image data corresponds to amplitude data obtained based on the indirect time-of-flight data. Therefore, depth information and image information of the environment are obtained by a single device.
[0055] Furthermore, in some embodiments, the light source of the indirect time-of-flight device (which is used to acquire indirect time-of-flight data) is or includes a full-field illuminator.
[0056] In some embodiments, the light source of the indirect time-of-flight device (which is used to acquire indirect time-of-flight data) is or includes a point illuminator. In such embodiments, the indirect time-of-flight device operates in a normal mode that is dependent on ambient light.
[0057] In some embodiments, the time-of-flight device includes an image sensor including a plurality of time-of-flight pixels for acquiring time-of-flight data and a plurality of image pixels for acquiring image data. Therefore, depth information and image information of the environment are obtained by a single device, wherein the depth information is acquired using the time-of-flight pixels and the image information is acquired using the image pixels.
[0058] For example, half of the image sensor may include time-of-flight pixels while the other half of the image sensor may include image pixels, which may also be mixed on the image sensor.
[0059] In some embodiments, the information processing device includes a time-of-flight device configured to acquire time-of-flight data and a camera configured to acquire image data. Therefore, the depth information and image information of the environment are obtained using two different devices.
[0060] For example, when the time-of-flight device is a direct time-of-flight device (LiDAR) for acquiring time-of-flight data, the camera is used to acquire image data.
[0061] In some embodiments, the information processing device further comprises a camera configured to acquire second image data representing a second image of the environment, wherein the circuit is configured to also track the ground based on the second image data and also locate the information processing device based on the second image data. In such embodiments, the information processing device comprises a time-of-flight device configured to acquire time-of-flight data and image data and a second device, namely a camera, configured to acquire second image data to improve the stability and accuracy of tracking and positioning.
[0062] In some embodiments, the circuit is configured to obtain an altitude of the information processing device and a vertical direction relative to a detected ground surface in the environment based on the time-of-flight data.
[0063] In some implementations, the circuitry is configured to track the ground based on altitude and vertical orientation.
[0064] In some embodiments, the circuit is configured to obtain a horizontal position and a horizontal orientation of the information processing device based on tracking of the ground.
[0065] In some embodiments, the circuit is configured to locate the information processing device based on the height, vertical direction, horizontal position, and horizontal orientation.
[0066] In some implementations, the circuit is configured to obtain sensor data, and further detect and track ground based on the sensor data.
[0067] In some embodiments, the information processing device includes an inertial measurement unit configured to acquire sensor data.
[0068] In some embodiments, the circuit is configured to also locate the information processing device based on the sensor data.
[0069] Some embodiments relate to a (corresponding) information processing method for simultaneous localization and mapping, wherein the information processing method comprises:
[0070] obtaining time-of-flight data of an environment of the information processing device;
[0071] detecting a ground surface in the environment based on the time-of-flight data;
[0072] obtaining image data representing an image of the environment;
[0073] Based on the image data, tracking the ground surface; and
[0074] Based on the detected and tracked ground surface, the information processing device is positioned in the environment.
[0075] The information processing method may be performed by an information processing device as described herein.
[0076] In some embodiments, when executed on a computer and / or processor, the method described herein is also implemented as a computer program that causes the computer and / or processor to perform the method. In some embodiments, a non-transitory computer-readable recording medium is also provided in which a computer program product is stored, which, when executed by a processor (such as the above-mentioned processor), performs the method described herein.
[0077] return Figure 1 , schematically illustrating a block diagram of four embodiments of information processing devices 1 - 1 , 1 - 2 , 1 - 3 , and 1 - 4 , which are discussed below.
[0078] The information processing devices 1 - 1 , 1 - 2 , 1 - 3 , and 1 - 4 may be smartphones, tablet computers, or the like.
[0079] The information processing device 1 - 1 includes a time-of-flight device 2 - 1 , wherein the time-of-flight device is an indirect time-of-flight device.
[0080] The information processing device 1 - 1 also includes an inertial measurement unit 3 (“IMU”), a processor 4 (e.g., an application processor), a data memory 5 and a data bus 6 (e.g., a data bus according to the MIPI (“Mobile Industry Processor Interface Alliance”) specification).
[0081] The processor 4 may load a computer program ("application") from the data memory 5 for simultaneous localization and mapping, eg for room scanning, and may temporarily store data in the data memory 5 during execution of the application.
[0082] When the application is started, the flight time device 2-1 acquires indirect flight time data and outputs depth data representing depth information of the environment of the information processing device 1-1 and amplitude data (as image data) representing an amplitude image (as an image) of the environment of the information processing device 1-1 to the processor 4 via the data bus 6.
[0083] The IMU 3 acquires sensor data (acceleration and rotation of the information processing device 1 - 1 ), and outputs the sensor data to the processor 4 via the data bus 6 .
[0084] The processor 4 performs room scanning based on the acquired depth data, amplitude data and sensor data, as will be referred to in Figure 2 discussed in more detail.
[0085] The information processing device 1 - 2 includes a time-of-flight device 2 - 2 , wherein the time-of-flight device includes an image sensor including a plurality of time-of-flight pixels for acquiring time-of-flight data and a plurality of image pixels for acquiring image data.
[0086] The information processing device 1 - 2 also includes an IMU 3 , a processor 4 , a data storage 5 and a data bus 6 .
[0087] As described above, the processor 4 may load a computer program ("application") from the data storage 5 for simultaneous localization and mapping, eg, for room scanning, and may temporarily store data in the data storage 5 during execution of the application.
[0088] When the application is started, the time-of-flight device 2 - 2 acquires time-of-flight data and image data, and outputs depth data representing depth information of the environment of the information processing device 1 - 2 and image data representing an image of the environment of the information processing device 1 - 2 to the processor 4 via the data bus 6 .
[0089] The IMU 3 acquires sensor data (acceleration and rotation of the information processing device 1 - 2 ), and outputs the sensor data to the processor 4 via the data bus 6 .
[0090] The processor 4 performs room scanning based on the acquired depth data, image data and sensor data, as will be referred to in Figure 2 discussed in more detail.
[0091] The information processing device 1 - 3 includes a time-of-flight device 2 - 1 .
[0092] The information processing device 1 - 3 also includes an IMU 3 , a processor 4 , a data memory 5 , a data bus 6 , and a camera (eg, a conventional camera such as a grayscale camera or an RGB (“Red-Green-Blue”) camera for acquiring two-dimensional images).
[0093] As described above, the processor 4 may load a computer program ("application") from the data storage 5 for simultaneous localization and mapping, eg, for room scanning, and may temporarily store data in the data storage 5 during execution of the application.
[0094] When the application is started, the flight time device 2-1 acquires indirect flight time data and outputs depth data representing depth information of the environment of the information processing device 1-3 and amplitude data (as image data) representing an amplitude image (as an image) of the environment of the information processing device 1-3 to the processor 4 via the data bus 6.
[0095] The IMU 3 acquires sensor data (acceleration and rotation of the information processing device 1 - 3 ), and outputs the sensor data to the processor 4 via the data bus 6 .
[0096] The camera 7 acquires second image data representing a second image of the environment of the information processing device 1 - 3 , and outputs the second image data to the processor 4 via the data bus 6 .
[0097] The processor 4 performs room scanning based on the acquired depth data, amplitude data, sensor data and second image data, as will be referred to in Figure 2 discussed in more detail.
[0098] The information processing device 1 - 4 includes a time-of-flight device 2 - 2 .
[0099] The information processing device 1 - 4 further includes an IMU 3 , a processor 4 , a data storage 5 , a data bus 6 and a camera 7 .
[0100] As described above, the processor 4 may load a computer program ("application") from the data storage 5 for simultaneous localization and mapping, eg, for room scanning, and may temporarily store data in the data storage 5 during execution of the application.
[0101] When the application is started, the time-of-flight device 2 - 2 acquires time-of-flight data and image data, and outputs depth data representing depth information of the environment of the information processing device 1 - 4 and image data representing an image of the environment of the information processing device 1 - 4 to the processor 4 via the data bus 6 .
[0102] The IMU 3 acquires sensor data (acceleration and rotation of the information processing devices 1 - 4 ), and outputs the sensor data to the processor 4 via the data bus 6 .
[0103] The camera 7 acquires second image data representing a second image of the environment of the information processing device 1 - 4 , and outputs the second image data to the processor 4 via the data bus 6 .
[0104] The processor 4 performs room scanning based on the acquired depth data, image data and sensor data, as will be referred to in Figure 2 discussed in more detail.
[0105] Figure 2 A block diagram schematically illustrates an embodiment of an information processing device 10 discussed below.
[0106] The information processing device 10 includes a time-of-flight (“ToF”) device 11 , a ground detection module 12 , a ground tracker module 13 , and a positioning module 14 .
[0107] Optionally, the information processing device 10 also includes an IMU 15 and a camera 16 .
[0108] The information processing device 10 may correspond to Figure 1 One of the information processing devices 1-1, 1-2, 1-3 and 1-4, wherein Figure 1 The IMU3 corresponds to the optional IMU15 and Figure 1 Camera 7 corresponds to the optional camera 16.
[0109] The ToF device 11 corresponds to Figure 1 The time-of-flight device 2-1 or 2-2.
[0110] The ground detection module 12, the ground tracker module 13 and the positioning module 14 are used by Figure 1 The processor 4 executes a computer program for simultaneous positioning and map building.
[0111] The computer program executed by the processor 4 is a room scanning application.
[0112] The ToF device 11 outputs depth data (as time-of-flight data) of the environment of the information processing device 10 to the ground detection module 12 , wherein it is assumed that the ToF device 11 is at least partially facing downwards.
[0113] The ground detection module 12 detects the ground in the environment of the information processing device 10 based on the depth data. The ground is assumed to be a plane in the horizontal direction.
[0114] The floor detection module 12 determines in a three-dimensional (“3D”) point cloud (represented by depth data obtained from the ToF device 11 ) a (larger) block of coplanar points, eg the center of the 3D point cloud, for detecting the floor (of the room / environment).
[0115] The ground detection module 12 may also detect the ground by determining whether the user's feet are in the field of view of the ToF device 11 , which is a feature that indicates a ground area that the ground detection module 12 may lock onto.
[0116] The ground detection module 12 may further detect the ground by determining whether there are edges or right angles between regions of the 3D points of the 3D point cloud, which may indicate vertical walls on the ground.
[0117] The ground detection module 12 calculates a set of 3D points in a 3D point cloud that is part of the ground, and generates and outputs them as detected ground to the ground tracker module 13 .
[0118] In addition, the ground detection module 12 calculates (determines) the height of the information processing device 10 and the vertical direction (normal line of the ground) relative to the detected ground in the environment based on the depth data. In addition, the ground detection module 12 determines the vertical orientation of the information processing device 10 relative to the detected ground.
[0119] The ground detection module 12 outputs the altitude and vertical direction to the ground tracker module 13 and the positioning module 14 .
[0120] The ToF device 11 outputs amplitude data (as image data) representing an amplitude image (as an image) of the environment to the ground tracker module 13 .
[0121] The ground tracker module 13 tracks amplitude data (as image data), altitude and vertical direction, the ground surface based on the detected ground surface.
[0122] Specifically, the ground tracker module 13 uses the knowledge of the detected ground, altitude and vertical direction to compare the continuous images from the ToF device 11 by projecting amplitude data in the ground plane and calculate the changes in the horizontal position and horizontal orientation (azimuth) of the information processing device 10.
[0123] Thus, the ground surface is tracked, and the ground tracker module 13 determines (or updates) the horizontal position and horizontal orientation of the information processing device 10 .
[0124] The ground tracker module 13 outputs the horizontal position and the horizontal orientation to the positioning module 14 .
[0125] Typically, the positioning module 14 stores the vertical direction, all previous altitudes, horizontal positions, and horizontal orientations (and optionally all previous vertical orientations) used to position the information handling device 10 .
[0126] The positioning module 14 fuses the information on the vertical direction, height, horizontal position and orientation to output the positioning (position and orientation) of the information processing device 10 .
[0127] In other words, the positioning module 14 positions the information processing device 10 in the environment based on the detected and tracked ground by positioning the information processing device 10 based on the altitude, vertical direction, horizontal position and horizontal orientation determined according to the detected and tracked ground.
[0128] The positioning module 14 implements a SLAM algorithm, which is generally known to the skilled person and will not be discussed in further detail.
[0129] Optionally, the IMU 15 provides acceleration and rotation data (sensor data) of the information processing device 10, which is used by the ground detection module 12, the ground tracker module 13 and the positioning module 14 to improve the accuracy and reduce the complexity of ground detection, ground tracking and positioning.
[0130] Optionally, the camera 16 provides second image data representing a second image of the environment (eg, having a higher resolution) which is used by the floor tracker module 13 to increase its accuracy.
[0131] Implementation Overview:
[0132] The ToF device 11 can provide height measurement, vertical direction and feature tracking simultaneously, and the SLAM algorithm can track the position of the information processing device 10 based on the height measurement, vertical direction and feature tracking.
[0133] For example, at least one foot of the user may be in the field of view of the ToF device 11, thereby providing a minimum set of features for lock-on.
[0134] Direct measurements of altitude and vertical direction are made to avoid drift in the vertical direction caused by accelerometer (eg, IMU 15) errors and to separate acceleration from the earth's gravity.
[0135] A downward facing ToF device 10 has predictable conditions, for example, the range is limited to the height of the person carrying the camera.
[0136] The downward facing ToF device 10 can provide direct measurements for critical components of SLAM algorithms that are otherwise measured indirectly, and in some cases, as is generally known, such indirect measurements may be unreliable.
[0137] The downward facing TOF device 10 can be used in conjunction with a (SLAM) camera 17 (and thus does not drift even when a conventional camera does not have a feature to lock onto (eg when it is occluded)).
[0138] Figure 3 An embodiment of the information processing method 100 discussed below is schematically shown in a flow chart.
[0139] The information processing method 100 may be performed by an information processing device as discussed herein.
[0140] At 101 , time-of-flight data of an environment of an information processing device is obtained, as discussed herein.
[0141] At 102 , a ground surface in an environment is detected based on the time-of-flight data, as discussed herein.
[0142] At 103, image data representing an image of an environment is obtained, as discussed herein.
[0143] At 104 , based on the image data, the ground surface is tracked, as discussed herein.
[0144] At 105 , an information processing device is located within the environment based on the detected and tracked ground surface, as discussed herein.
[0145] At 106 , an altitude of the information processing device and a vertical direction relative to a detected ground surface in the environment are determined based on the time-of-flight data, as discussed herein.
[0146] At 107 , the ground is tracked in the image data based on altitude and vertical direction, as discussed herein.
[0147] At 108 , a horizontal position and a horizontal orientation of the information handling device are determined based on tracking of the ground surface, as discussed herein.
[0148] At 109 , the information processing device is positioned based on the height, vertical direction, horizontal position, and horizontal orientation, as discussed herein.
[0149] Figure 4 An implementation of the information processing method 200 discussed below is schematically shown in a flow chart.
[0150] The information processing method 200 may be performed by an information processing device as discussed herein.
[0151] At 201 , time-of-flight data of an environment of an information processing device is obtained, as discussed herein.
[0152] At 202 , a ground surface in an environment is detected based on the time-of-flight data, as discussed herein.
[0153] At 203, image data representing an image of an environment is obtained, as discussed herein.
[0154] At 204 , based on the image data, the ground surface is tracked, as discussed herein.
[0155] At 205 , the information processing device is located in the environment based on the detected and tracked ground surface, as discussed herein.
[0156] At 206 , sensor data is obtained, and the ground surface is also detected and tracked based on the sensor data, and the information processing device is also positioned based on the sensor data, as discussed herein.
[0157] It should be appreciated that the embodiments describe methods with an exemplary order of method steps. However, the specific order of method steps is given only for illustrative purposes and should not be construed as limiting.
[0158] If not stated otherwise, all units and entities described in this specification and claimed in the appended claims may be implemented as integrated circuit logic, for example on a chip, and if not stated otherwise, the functions provided by these units and entities may be implemented by software.
[0159] To the extent that the embodiments of the present disclosure described above are at least partially implemented using software-controlled data processing equipment, it should be understood that providing such software-controlled computer programs and the transmission, storage or other media through which such computer programs are provided are contemplated as aspects of the present disclosure.
[0160] It should be noted that the present technology may also be configured as follows.
[0161] (1) An information processing device for simultaneous positioning and mapping, wherein the information processing device comprises a circuit configured to:
[0162] obtaining time-of-flight data of an environment of the information processing device;
[0163] detecting a ground surface in the environment based on the time-of-flight data;
[0164] obtaining image data representing an image of the environment;
[0165] Based on the image data, tracking the ground surface; and
[0166] Based on the detected and tracked ground surface, the information processing device is positioned in the environment.
[0167] (2) The information processing device according to (1), wherein the circuit is configured to: determine an altitude of the information processing device and a vertical direction relative to a detected ground surface in the environment based on the time-of-flight data.
[0168] (3) The information processing device according to (2), wherein the circuit is configured to: track the ground based on the height and the vertical direction.
[0169] (4) The information processing device according to (3), wherein the circuit is configured to: determine a horizontal position and a horizontal orientation of the information processing device based on tracking the ground surface.
[0170] (5) The information processing device according to (4), wherein the circuit is configured to: position the information processing device based on the height, vertical direction, horizontal position, and horizontal orientation.
[0171] (6) The information processing device according to any one of (1) to (5), wherein the circuit is configured to: obtain sensor data, and also detect and track the ground surface based on the sensor data.
[0172] (7) The information processing device according to (6), wherein the circuit is configured to: locate the information processing device further based on the sensor data.
[0173] (8) The information processing device according to any one of (1) to (7), wherein the information processing device includes a time-of-flight device configured to acquire time-of-flight data and image data.
[0174] (9) The information processing device according to (8), wherein the time-of-flight data is indirect time-of-flight data.
[0175] (10) The information processing device according to (9), wherein the image data corresponds to amplitude data obtained based on indirect time-of-flight data.
[0176] (11) An information processing device according to (8) or (9), wherein the time-of-flight device includes: an image sensor including a plurality of time-of-flight pixels for acquiring time-of-flight data and a plurality of image pixels for acquiring image data.
[0177] (12) The information processing device according to any one of (8) to (11) further includes: a camera configured to acquire second image data representing a second image of the environment, wherein the circuit is configured to track the ground based on the second image data and to position the information processing device based on the second image data.
[0178] (13) The information processing device according to any one of (1) to (7), wherein the information processing device includes: a time-of-flight device configured to acquire time-of-flight data and a camera configured to acquire image data.
[0179] (14) The information processing device according to any one of (6) to (13), wherein the information processing device includes: an inertial measurement unit configured to acquire sensor data.
[0180] (15) An information processing method for simultaneous positioning and mapping, wherein the information processing method comprises:
[0181] obtaining time-of-flight data of an environment of the information processing device;
[0182] detecting a ground surface in the environment based on the time-of-flight data;
[0183] obtaining image data representing an image of the environment;
[0184] Based on the image data, tracking the ground surface; and
[0185] Based on the detected and tracked ground surface, the information processing device is positioned in the environment.
[0186] (16) The information processing method according to (15) further includes: determining the height of the information processing device and the vertical direction relative to the detected ground in the environment based on the flight time data.
[0187] (17) The information processing method according to (16) further includes: tracking the ground based on the height and the vertical direction.
[0188] (18) The information processing method according to (17) further includes: determining the horizontal position and horizontal orientation of the information processing device based on tracking of the ground.
[0189] (19) The information processing method according to (18) further includes: positioning the information processing device based on the height, vertical direction, horizontal position and horizontal orientation.
[0190] (20) The information processing method according to any one of (15) to (19) further includes: obtaining sensor data, and also detecting and tracking the ground based on the sensor data, and also positioning the information processing device based on the sensor data.
[0191] (21) A computer program comprising a program code, which, when executed on a computer, causes the computer to execute the method according to any one of (15) to (20).
[0192] (22) A non-transitory computer-readable recording medium storing a computer program product which, when executed by a processor, causes the method according to any one of (15) to (20) to be performed.
Claims
1. An information processing device for simultaneous positioning and mapping, comprising a circuit, wherein the circuit is configured to: obtaining time-of-flight data of an environment of the information processing device; detecting a ground surface in the environment based on the time-of-flight data; obtaining image data representing an image of the environment; Based on the image data, tracking the ground surface; and Based on the detected and tracked ground surface, the information processing device is positioned in the environment.
2. The information processing device according to claim 1, in, The circuit is configured to determine, based on the time-of-flight data, an altitude of the information processing device and a vertical direction relative to the detected ground surface in the environment.
3. The information processing device according to claim 2, in, The circuit is configured to track the ground surface based on the altitude and the vertical direction.
4. The information processing device according to claim 3, in, The circuit is configured to determine a horizontal position and a horizontal orientation of the information processing device based on tracking the ground surface.
5. The information processing device according to claim 4, in, The circuit is configured to locate the information processing device based on the height, the vertical direction, the horizontal position, and the horizontal orientation.
6. The information processing device according to claim 1, in, The circuit is configured to obtain sensor data and, further based on the sensor data, detect and track the ground surface.
7. The information processing device according to claim 6, in, The circuit is configured to: locate the information processing device further based on the sensor data. 8 . The information processing apparatus according to claim 1 , comprising a time-of-flight device configured to acquire the time-of-flight data and the image data.
9. The information processing device according to claim 8, in, The time-of-flight data is indirect time-of-flight data.
10. The information processing device according to claim 9, in, The image data corresponds to amplitude data obtained based on the indirect time-of-flight data.
11. The information processing device according to claim 8, in, The time-of-flight device includes an image sensor including a plurality of time-of-flight pixels for acquiring the time-of-flight data and a plurality of image pixels for acquiring the image data.
12. The information processing device according to claim 8, further comprising: include: A camera is configured to acquire second image data representing a second image of the environment, wherein the circuit is configured to track the ground surface based on the second image data and to locate the information processing device based on the second image data.
13. The information processing device according to claim 1, include: a time-of-flight device configured to acquire the time-of-flight data; and a camera configured to acquire the image data.
14. The information processing device according to claim 6, include: An inertial measurement unit is configured to acquire the sensor data.
15. An information processing method for simultaneous localization and mapping, include: obtaining time-of-flight data of an environment of the information processing device; detecting a ground surface in the environment based on the time-of-flight data; obtaining image data representing an image of the environment; Based on the image data, tracking the ground surface; as well as Based on the detected and tracked ground surface, the information processing device is positioned in the environment.
16. The information processing method according to claim 15, further comprising: include: Based on the time-of-flight data, the altitude of the information processing device and the vertical direction relative to the detected ground surface in the environment are determined.
17. The information processing method according to claim 16, further comprising: include: Based on the height and the vertical direction, the ground surface is tracked.
18. The information processing method according to claim 17, further comprising: include: Based on the tracking of the ground, the horizontal position and the horizontal orientation of the information processing device are determined.
19. The information processing method according to claim 18, further comprising: include: The information processing device is positioned based on the height, the vertical direction, the horizontal position, and the horizontal orientation.
20. The information processing method according to claim 15, further comprising: include: Sensor data is obtained, and the ground surface is detected and tracked based on the sensor data and the information processing device is positioned based on the sensor data.
