Depth sensor activation for localization based on data from a monocular camera

Abstract

A device is disclosed, which is configured to perform positioning using one or both of a monocular camera (200) and a depth sensor (202) that can be transported with the device. The device includes at least one processor that is operatively connected to the monocular camera and the depth sensor. The device also includes at least one memory that stores program code that is executed by the at least one processor to perform an operation for: receiving image data from the monocular camera. The operation determines, based on the image data, a level of benefit of activating the depth sensor for positioning; and based on determining that the level of benefit of activating the depth sensor satisfies an activation rule, activates the depth sensor for positioning. Related methods and computer program products are also disclosed.

Description

Technical Field

[0001] The present disclosure relates to a device for performing positioning using one or both of a monocular camera and a depth sensor that can be transported together with the device, a method for performing positioning by the device using one or both of a monocular camera and a depth sensor that can be transported together with the device, and a corresponding computer program product. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) is a fundamental technology that allows a device to localize itself in an environment while relying on onboard sensors such as cameras, range sensors, and inertial sensors. This is essential for robots such as drones and autonomous vehicles to navigate and understand their environment or perform tasks, as well as being able to display realistic and persistent content in mixed reality (MR) devices.

[0003] For example, current MR headsets and state-of-the-art smartphones contain RGB cameras, depth / 3D cameras (e.g., passive or active stereo, LIDAR, etc.), and inertial sensors (as part of an inertial measurement unit (IMU), and the same is true for indoor and outdoor robots (e.g., drones and autonomous vehicles). Several SLAM algorithms have been proposed that rely on RGB and IMU sensors, depth sensors, or a combination of all of these sensors. The reason for performing a combination of sensors is to both exploit their strengths and improve their limitations.

[0004] For example, RGB cameras perform poorly in dark or overly bright environments, while depth cameras (such as LIDAR or active stereo cameras) will perform well in such scenarios. In addition, by measuring depth directly, localization and mapping can be performed with higher accuracy, and a larger amount of environmental information can be captured (e.g., building dense maps instead of sparse maps), among other benefits. However, depth cameras typically have greater energy consumption and processing requirements, and may perform poorly under certain conditions. For example, depth cameras have a limited measurement range and may perform poorly in low-texture environments (passive stereo cameras) and areas with direct sunlight or IR interference (active stereo cameras and LIDAR), rain conditions (LIDAR), and other limitations. Summary of the invention

[0005] Currently available devices that use sensors for positioning (e.g., Microsoft Hololens, Magic Leap, ARCore, and ARKit) assume that all such sensors are always active (e.g., powered on), and are unaware of the need and ability to perform selective activation and deactivation of individual sensors.

[0006] Some embodiments of the present disclosure relate to a device configured to perform positioning using one or both of a monocular camera and a depth sensor that can be transported with the device. The device includes at least one processor that is operatively connected to the monocular camera and the depth sensor. The device also includes at least one memory that stores program code that is executed by the at least one processor to perform operations for: receiving image data from the monocular camera. The operations determine, based on the image data, a level of benefit of activating the depth sensor for positioning; and based on determining that the level of benefit of activating the depth sensor satisfies an activation rule, activating the depth sensor for positioning.

[0007] Some other related embodiments are directed to a method for performing positioning by a device using one or both of a monocular camera and a depth sensor that can be transported with the device. The method includes: receiving image data from the monocular camera; and based on the image data, determining a benefit level of activating the depth sensor for positioning. The method activates the depth sensor for positioning based on determining that the benefit level of activating the depth sensor satisfies an activation rule.

[0008] Some other related embodiments relate to a computer program product for performing localization using one or both of a monocular camera and a depth sensor that can be transported with a device. The computer program product includes a non-transitory computer-readable medium storing instructions that can be executed by at least one processor of the device to configure the device to: receive image data from the monocular camera; determine a benefit level of activating the depth sensor for localization based on the image data; and activate the depth sensor for localization based on determining that the benefit level of activating the depth sensor satisfies an activation rule.

[0009] Potential advantages of one or more of these embodiments can include: the device can use image data from the monocular camera to determine when the depth sensor can provide a sufficient level of benefit so that the depth sensor should be activated from a deactivated state for subsequent use in positioning. In this way, the depth sensor does not have to be activated in order to programmatically determine whether it will provide a sufficient level of benefit to be used in positioning to justify its activation and use. These operations can reduce the energy consumption and computing resource utilization of the device when performing positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Various aspects of the present disclosure are illustrated by way of example and not limitation in the accompanying drawings. In the drawings:

[0011] Figure 1Operations are shown according to some embodiments for converting a two-dimensional (2D) image obtained from a monocular camera into three-dimensional (3D) data, and then processing the three-dimensional (3D) data to determine a level of benefit that may be obtained by activating a depth sensor for use in positioning;

[0012] Figure 2 A system diagram showing a device for positioning according to some embodiments of the present disclosure;

[0013] Figure 3 A mixed reality (MR) system is shown that includes a MR head mounted device holding a mobile electronic device that may include or be operatively connected to a set of sensors and is configured to operate according to some embodiments of the present disclosure;

[0014] Figure 4 shows a top view of a device with a monocular camera and a depth sensor moving through an environment along a predicted motion trajectory;

[0015] Figure 5 A block diagram illustrating components of a device configured according to some embodiments of the present disclosure; and

[0016] Figures 6 to 12 A flowchart of operations for controlling activation of a monocular camera and deactivation of a depth sensor by a device for positioning according to some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0017] The concept of the present invention will now be described more fully below with reference to the accompanying drawings, in which examples of embodiments of the concept of the present invention are shown. However, the concept of the present invention can be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided so that this disclosure will be thorough and complete, and the scope of various current inventive concepts will be fully communicated to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. Components from one embodiment may be assumed by default to be present in / used in another embodiment.

[0018] Methods, devices, and computer program products are disclosed for determining when a depth sensor should be activated assuming that only a monocular camera is currently active. In this way, when a monocular camera is already being used for positioning, it is not necessary to activate the depth sensor to understand whether the depth sensor is beneficial for positioning. Some other embodiments involve determining when a monocular camera should be deactivated after the depth sensor has become active. These embodiments can reduce energy consumption and computing resource utilization of a device when performing positioning.

[0019] As will be explained below, based on image data from a monocular camera, for example, based on structural information of physical objects identified in the image data, a level of benefit of activating a depth sensor for positioning is determined. Based on determining that the level of benefit of activating the depth sensor satisfies an activation rule, the depth sensor is activated for positioning. Although various embodiments are described in the context of performing positioning, these and other embodiments may be used to perform combined positioning and mapping operations, such as SLAM. Therefore, the term "positioning" as used herein may interchangeably refer to operations that are configured only to perform positioning functions and operations that are configured to perform a combination of positioning and mapping functions, such as SLAM.

[0020] Various embodiments of the present disclosure are described in the context of a device that includes both a monocular camera (eg, an RGB camera) and a depth sensor. The device may also include an inertial measurement unit (IMU). Figure 1 The following operations are shown: 2D image data 100 obtained from a monocular camera is converted to 3D data 102, and then the 3D data 102 is processed to determine the level of benefit that can be obtained by activating a depth sensor for use in positioning. Typical visual-inertial SLAM algorithms applied to sensors that capture RGB and IMU information can reconstruct the depth of the scene, but the 3D data 102 is typically sparse (e.g., a sparse point cloud), where dense depth reconstruction can be performed at a higher computational cost. The 2D image 100 is shown in the form of a point cloud of image data received from a monocular camera used for SLAM. The points on the 2D image data 100 are visual features extracted from the 2D image data 100. The 3D data 102 is shown as a sparse reconstruction of the environment.

[0021] Although various embodiments are described herein in the context of using 3D depth data from a depth sensor and using 2D image data from a monocular camera, it will be understood that any dimensional (e.g., 1D, 2D, 3D) data may be used. For example, the term "3D depth data" refers to depth data from a depth sensor that provides a three-dimensional indication of the sensed depth of an environmental object. In contrast, the term "depth data" refers to depth data from a depth sensor that provides any dimensional indication of the sensed depth of an environmental object.

[0022] Machine learning based methods can also be used to extract depth information from a single RGB 2D image data 100, or to combine the RGB 2D image 100 with 3D data 102 from 3D information collected by a depth sensor or SLAM algorithm. On the other hand, using a depth sensor generally provides better estimates and denser depth information, but consumes more energy and utilizes more computing resources because the depth sensor contains high-power circuits and high-bandwidth data. Depth sensors also have a limited depth measurement range.

[0023] Potential advantages of one or more embodiments disclosed herein can include: the device can use image data from a monocular camera to determine when a depth sensor can provide a sufficient level of benefit so that the depth sensor should be activated from a deactivated state for subsequent use in positioning. In this way, the depth sensor does not have to be activated in order to programmatically determine whether it will provide a sufficient level of benefit to be used in positioning to justify its activation and use. These operations can reduce the energy consumption and computing resource utilization of the device when performing positioning.

[0024] In some embodiments, the following operations may be performed to determine whether the level of benefit from activating the depth sensor is sufficient to trigger its activation, and may also determine whether the monocular camera should be deactivated:

[0025] 1. Receive image data from monocular data;

[0026] 2. Determine the level of benefit of activating the depth sensor based on the image data from the monocular data;

[0027] 3. Based on the determined benefit level, if activating the depth sensor is beneficial, activating the depth sensor for positioning; and

[0028] 4. (Optional) After activation of the depth sensor has been performed, determine a level of benefit from continuing to use the monocular camera for positioning based on image data from the monocular camera and based on depth data from the depth sensor, and deactivate the monocular camera based on determining that the level of benefit from continuing to use the monocular camera for positioning satisfies a deactivation rule.

[0029] Figure 5 A block diagram of components of an example device 500 configured in accordance with some embodiments of the present disclosure is shown. Figure 6 A flow diagram illustrating operations that may be performed by device 500 for controlling activation of depth sensor 202 for positioning based on determining that a benefit level of activating depth sensor 202 for positioning satisfies an activation rule, according to some embodiments of the present disclosure.

[0030] refer to Figure 5 and Figure 6In some embodiments, the device 500 includes: at least one processor 510 (hereinafter referred to as "processor"), which is operatively connected to the monocular camera 200, the depth sensor 202; and at least one memory 520 (hereinafter referred to as "memory"), which stores program code, which is executed by the processor 510 to perform operations to receive 600 image data from the monocular camera 200. The operations determine 602 a benefit level of activating the depth sensor 202 for positioning based on the image data, and activate 604 the depth sensor 202 for positioning based on determining that the benefit level of activating the depth sensor 202 satisfies the activation rule. The memory 520 may include maps and programs 522 (e.g., a positioning map repository), which can be used as described below. The device 500 may include a wireless transceiver 530, which is configured to communicate via a wireless interface.

[0031] Depth sensor 202 and monocular camera 200 can be shipped with device 500, but are not necessarily part of device 500. For example, although Figure 5 The device 500 is shown to include a depth sensor 202, a monocular camera 200, a processor 510, and a memory 520, but in some embodiments, one or more of these components may be separate from the device 500 and communicatively connected to the device 500 via a wireless transceiver 530 and / or a wired interface. The device 500 may be, but is not limited to, a component of any of a smart phone, a wearable computer, an augmented reality head-mounted device, a virtual reality head-mounted device, a mixed reality head-mounted device, a semi-autonomous or autonomous vehicle, a drone, an aircraft, a robot, and the like.

[0032] Although various embodiments are described in the context of activating and deactivating individual sensors (e.g., a monocular camera and a depth sensor), these embodiments can be used to activate and deactivate groups of sensors. Thus, for example, "monocular camera" can correspond to a group of monocular cameras, and "depth sensor" can correspond to a group of depth sensors. A group of sensors can include sensors of homogeneous or non-isomorphic types.

[0033] Figure 3A mixed reality (MR) system is shown that includes a MR head mounted device 300 holding a mobile electronic device 320, which may be operatively connected to at least one monocular camera 200 and at least one depth sensor 202, for example, via a wired and / or wireless communication interface. The mobile electronic device 320 may include or be operatively connected to a processor 510 and a memory storing program code that configures the processor 510 to activate and deactivate individual ones of the monocular cameras 200 and the depth sensors 202, while performing positioning using the active ones of the monocular cameras 200 and the depth sensors 202.

[0034] The MR head mounted device 300 includes a lens 310 through which a user wearing the MR head mounted device can view real world features. The MR head mounted device 300 also includes a support 321 configured to releasably hold the mobile electronic device 320 in a defined orientation relative to the lens 310 so that an image displayed on a display device of the mobile electronic device 320 is reflected directly or indirectly toward the user's eyes by the lens 310. Although not shown, the MR head mounted device 300 may include an intervening mirror located between the lens 310 and the user's eyes, and thus light may be reflected directly or indirectly toward the user's eyes and / or the camera 202.

[0035] Mobile electronic device 320 may include, but is not limited to, a smart phone, a PDA, a tablet computer, a gaming device, or other computing device. For the sake of brevity, a "mobile electronic device" is also referred to herein as a "mobile device" and a "device".

[0036] Figure 2 A system diagram of a device 500 is shown, according to some embodiments of the present disclosure, the device 500 being configured to perform localization operations and / or combined localization and mapping operations (eg, SLAM) using a depth sensor 202 and a monocular sensor 200. Figure 2, the device 500 includes a sensor activation and deactivation logic 204 configured to: determine a level of benefit of activating the depth sensor 202 for positioning operations based on image data from the monocular camera 200; and selectively activate the depth sensor 202 based on the determination. The sensor activation and deactivation logic 204 can also be configured to: deactivate the monocular camera 200 based on determining that the level of benefit of continuing to use the monocular camera 200 for positioning satisfies a deactivation rule (e.g., when the depth sensor 202 has become activated for positioning and when continuing to use the monocular camera 200 does not provide sufficient continued benefit). When the monocular camera 200 is active, image data is provided from the monocular camera 200 to a positioning algorithm, such as the positioning and mapping algorithm 212. Similarly, when the depth sensor 202 is active, depth data is provided from the depth sensor 202 to a positioning algorithm, such as the positioning and mapping algorithm 212.

[0037] The sensor activation and deactivation logic 204 controls the switching logic 208, which performs deactivation and activation of selected sensors of the sensors 200 and 202. The switching logic 208 can perform activation of the sensor (i.e., the depth sensor 202 or the monocular camera 200) by triggering at least one of the following: transitioning the sensor to a high power state, powering on the sensor, powering on an active component of the sensor that senses the environment (e.g., a LIDAR laser component, an infrared emitter, etc.), increasing the data sampling rate of the sensor or its component to a level used for positioning, increasing the resolution of the sensor to a level used for positioning, changing optical parameters (e.g., focal length, field of view, etc.) to parameters used for positioning, and adapting the positioning algorithm to use the parameters of the sensor (e.g., optical parameters). Conversely, the switching logic 208 may perform deactivation of the sensor by triggering at least one of: transitioning the sensor to a low power state, powering off the sensor, powering off an active component of the sensor that senses the environment (e.g., a LIDAR laser component, an infrared emitter, etc.), reducing the data sampling rate of the sensor or its active component to a level lower than that used for positioning, reducing the resolution of the sensor to a level lower than that used for positioning, changing optical parameters (e.g., focal length, field of view, etc.) to parameters that are not used for positioning, and adapting the positioning algorithm to stop using parameters of the sensor (e.g., optical parameters). Thus, the term "switching" is not limited to off-on switching, but may alternatively or additionally include control logic that performs one or more of the more complex above-mentioned activities for activating and deactivating sensors.

[0038] In one embodiment, the operation of adapting the positioning algorithm to use the optical parameters of the sensor for positioning may include obtaining algorithm parameters corresponding to the optical parameters of the sensor. The algorithm parameters may be predetermined based on offline tuning of the positioning algorithm for different optical parameter sets. Then, based on the optical parameter set defined for the sensor, the corresponding predetermined algorithm parameters are selected for use.

[0039] As will be explained in more detail below, the sensor activation and deactivation logic 204 may be operable to use information provided by the energy budget 206 and / or by the positioning map 210 (which may be located at Figure 5 The information provided by the map 522 in the image may be used to determine when to activate the depth sensor 202 and / or when to deactivate the monocular camera 200.

[0040] In order for the depth sensor 202 to provide depth data that can be used for positioning operations and / or combined positioning and mapping operations (e.g., SLAM), the depth sensor 202 must be able to sense relevant environmental features. Various alternative embodiments of operations that can determine the level of benefit of activating a depth sensor based on the performance of a positioning algorithm (e.g., positioning and mapping algorithm 212) will now be explained.

[0041] Methods for quantifying localization and mapping performance using the depth sensor 202 are discussed below.

[0042] The performance of localization and mapping using depth data from the depth sensor 202 will rely on the ability of the depth sensor 202 to sense structural information from the environment in a robust manner. The information sensed by the depth sensor 202 (e.g., in successive frames of depth data) is used to determine the motion properties of the device, while the information can also be stored as a map and subsequently used for localization, in which matching between the map and the online depth information is performed.

[0043] The more physical structure a scene has, the more information from the depth sensor 202 will likely be sensed and used in the localization and mapping algorithms. Some localization and mapping algorithms rely on identifying different 3D features or structural shapes (e.g., planes), but the 3D shapes of structures used for localization and mapping may vary and be trained using machine learning methods. The degree to which a depth-based localization and mapping algorithm relies on the depth sensor 202 and its robustness may be directly related to the presence or absence of detected 3D features or the number of detected 3D features.

[0044] Depth sensors generally have certain limitations that will degrade the associated localization and mapping performance. Some of the limitations of depth sensors are:

[0045] a. Difficulty sensing structural elements in the presence of reflective surfaces (e.g. mirrors);

[0046] b. Depending on the resolution of the depth sensor, structural elements smaller than this resolution may not be detected; and

[0047] c. Limited measurement range (in terms of both minimum and maximum distance).

[0048] Various methods are discussed below for determining whether the depth sensor 202 is able to capture sufficient relevant object features in the environment to justify activating the depth sensor 2024 for localization based on analyzing the 2D image data captured by the monocular camera 200. Various other related methods for determining the level of benefit of using the depth sensor 202 for localization are discussed below.

[0049] In some embodiments, based on the conversion to 3D data (e.g. Figure 1 ) and then sample and analyze the image data from the monocular camera 200 to determine 602 ( Figure 6 ) to activate the depth sensor 202 for positioning. Fig. 9 The corresponding operations are shown in FIG. 1 , which are configured according to some embodiments of the present disclosure. Fig. 9 , based on processing 900 the image data from the monocular camera 200 by a positioning algorithm to obtain depth points within the environment sensed by the monocular camera 200, determining 602 ( Figure 6 ) A benefit level of activating the depth sensor 202 for positioning. Determining 602 the benefit level is also based on estimating 902 a density of depth points within a range of the depth sensor 202. Determining 602 the benefit level based on the estimate of the density of the depth points.

[0050] In another embodiment, estimating 902 the density of depth points within the range of the depth sensor includes: identifying objects of a determined physical size within the environment within image data from a monocular camera; and determining the range of the depth points based on a comparison of the size of the objects within the image data and the physical size of the objects.

[0051] From the localization and mapping algorithms running in the device 500 using only images from the monocular camera 200, these operations can be configured to extract sparse depth points of the environment (see Figure 1). However, the scale of the depth of the image data from the monocular camera 200 can only be extracted if the size of the objects in the scene is known (this is the typical approach used in visual localization and mapping systems), or if the device 500 has an inertial measurement unit (IMU) used to perform localization and mapping (which can be advantageous, and most mobile devices have an IMU), then the range of the depth points can be estimated or directly inferred. For example, an IMU can measure the distance traveled between consecutive images, which can then be used to determine the scale of the depth points estimated by those images. Thus, given the depth sensor range, the operation is able to determine how much information would likely be collected using the depth sensor 202 if it were to be activated for localization, i.e., what the level of benefit of activating the depth sensor 202 for localization would be.

[0052] In one illustrative embodiment, if the number of depth points (voxels) contained in the data (e.g., point cloud) within the minimum and maximum ranges and within the field of view of the depth sensor 202 (i.e., "point density") is above a threshold X, then activating the depth sensor 202 has a benefit. The threshold X can be defined using an offline method and adapted at runtime. For example, a training session is performed with the monocular camera 200 and the depth sensor 202 active, so a point cloud (including depth data and image data) can be collected and can be combined with the IMU sensor information, and the performance of the localization and mapping algorithm is determined based on the density of the point cloud captured by the depth sensor 202 (e.g., a minimum density d_min_depth is required for reasonable performance), which will correspond to a specific depth density d_monocular of the monocular camera 200, for example, finding the minimum value of d_monocular, which ensures that the depth density d_depth of the depth sensor 202>d_min_depth (given a training set). The threshold can also be adapted at runtime using the same training sequence.

[0053] In another illustrative embodiment, the value of d_min_depth may be 20000, while d_monocular>500 may be found during training to achieve d_depth>20000. For example, commercially available depth sensors may provide depth resolutions from 320x240 to 640x480, which yields point cloud counts between 76,800 and 307,200 points / voxel.

[0054] In some embodiments, the determination 602 ( Figure 6 ) activates the depth sensor 202 for positioning benefit level. Fig.10, determining 602 ( Figure 6 ) The benefit level of activating the depth sensor 202 for localization may include processing 1000 image data from the monocular camera 200 through a localization algorithm to obtain depth points within the environment sensed by the monocular camera 200. Determining 602 ( Figure 6 ) The level of benefit of activating the depth sensor 202 for positioning may also include determining 1002 a number of 3D features within the depth reconstruction data for a portion of the environment based on the frame sequence of image data and the depth points. Based on the number of 3D features, determining 602 ( Figure 6 ) level of benefit.

[0055] In another embodiment, the benefit level of activating the depth sensor is determined to satisfy the activation rule based on the number of 3D features satisfying a minimum threshold. The minimum threshold can be determined based on determining a minimum number of 3D features required for the localization algorithm to perform localization with at least a threshold level of accuracy.

[0056] The previous options can be combined with a 3D depth reconstruction algorithm that reconstructs the depth portion of the environment based on the image data from the monocular camera 200 by extracting sparse points using the localization and mapping algorithm and the pose of the device (the pose of the device can also be calculated by the localization and mapping algorithm). For example, using these operations, a multi-view stereo method or a monocular-depth reconstruction method based on pure machine learning is provided. In this way, the operation can directly infer whether the structural properties of the environment provide sufficient information to the localization and mapping algorithm to obtain the desired performance and robustness using the depth sensor 202 and / or the monocular camera 200. This inference can be performed in the following manner: applying a depth-based localization and mapping algorithm to the 3D reconstructed data, and obtaining an indicator of successful localization and mapping based on such data (for example, sufficient 3D features (such as planes) are detected for a sequence of steps, etc., which is an indicator of good performance of the localization and mapping algorithm). For example, the benefit level may be defined as the number of 3D features detected based on analyzing a point cloud created using a 3D reconstruction algorithm, for which a minimum number of 3D features X should be detected so that the localization and mapping algorithms have the desired performance when using the depth sensor 202. The 3D features may be detected as described in "SegMap: 3D Segment Mapping Using Data-Driven Descriptors" (R. Dubé, A Cramariuc, D. Dugas, J. Nieto, R. Siegwart, and C. Cadena, arXiv: 1804.09557, DOI: 10.15607 / RSS.2018.XIV.003, 2018). The threshold minimum number of 3D features X (threshold X) may be determined based on offline experiments to determine the localization performance (e.g., accuracy) given different values ​​of threshold X.

[0057] In another example, the above method can be directly applied to a point cloud created using a 3D reconstruction algorithm and identifying how many voxels are within the range of the depth sensor 202, where the number of voxels defines the benefit level and if the number of voxels is above a threshold X, then activating the depth sensor 202 would be beneficial.

[0058] In some embodiments, based on sampling and analyzing image data from the monocular camera 200 after conversion to 3D data, a level of benefit of activating the depth sensor 202 for positioning is determined 602. Fig.11 The corresponding operations are shown in FIG. 1 , which are configured according to some embodiments of the present disclosure. Fig.11, based on processing 1100 the image data from the monocular camera 200 by an object recognition algorithm and a localization algorithm to obtain a physical object viewed by the monocular camera 200 (including the size of the physical object and the position of the physical object relative to the device), determining 602 ( Figure 6 ) The benefit level of activating the depth sensor 202 for positioning. Determining 602 ( Figure 6 ) level of benefit.

[0059] The device 500 may apply conventional object recognition algorithms and infer the physical structures of the scene (e.g., tables, chairs, walls, desks, closets, etc.) as well as the dimensions of these structures and their positions relative to the device. The level of benefit of this information may then be proportional to the type and / or size of the detected objects and / or their distance relative to the device 500. For example, assuming that there are class A objects (e.g., desks) in the environment that are within the range of the depth sensor 202, the operation may evaluate the performance of the depth-based positioning and mapping algorithm in an offline manner (e.g., pose uncertainty, positioning error relative to known ground truth measurements, etc.), and thus a table may be created indicating the correspondence between the number of objects of a given type and their level of benefit. Thus, if an object of a defined type is found within the range and field of view of the depth sensor 202, the depth-based positioning and mapping 10 performs well using the depth sensor 202. This option may then be viewed as a combination of object detection, where instead of checking the number of voxels within the range of the depth sensor, the operation checks whether a particular object is within the range of the depth sensor.

[0060] As another example, the benefit level may be defined as the size of the detected object (e.g., a 3D bounding box around the object as proposed herein) or the number of objects of a particular type detected (e.g., all furniture objects), and if the size of the object and / or the number of detected objects is above a minimum amount X, then it is determined that activating the depth sensor 202 is beneficial. Again, the threshold X may be defined by performing offline experiments, where the localization and mapping performance (e.g., accuracy) is evaluated as a function of the size and / or type of the object.

[0061] In some embodiments, based on sampling and analyzing image data from the monocular camera 200 after conversion to 3D data, a level of benefit of activating the depth sensor 202 for positioning is determined 602. Fig.12 The corresponding operations are shown in FIG. 1 , which are configured according to some embodiments of the present disclosure. Fig.12 , based on determining 1200 the position of the depth sensor 202 according to the image data from the monocular camera 200, determining 602 ( Figure 6) The benefit level of activating the depth sensor 202 for positioning. The benefit level is also based on accessing 1202 a historical positioning map repository (e.g., Figure 2 and generating 1204 an approximation of depth information that may be obtained from a depth sensor (if activated) based on the historical image data. Determining 602 a benefit level based on the approximation of the depth information.

[0062] Figure 4 4. A top view of a device 500 with a monocular camera 200 and a depth sensor 202 is shown moving through an environment along a predicted motion trajectory 402. The predicted motion trajectory 402 can be used to retrieve information from a historical positioning map repository (e.g., Figure 2 210 in the map 210) to obtain historical image data, and generate 1204 an approximation of depth information that can be obtained from the depth sensor 202 (if activated) based on the historical image data.

[0063] Thus, these operations can use historical image data by positioning relative to a historical positioning map (built for the current environment using the monocular camera 200 and / or the depth sensor 202) to determine the amount of information that would be captured if the depth sensor 202 was activated at the current location for positioning or at another location along the predicted motion trajectory 402 of the device 500. The historical positioning map can be a sparse point cloud or can be a denser point cloud that is built based on image data from the monocular camera 202 and / or depth data from the depth sensor 202. The device 500 can use the historical positioning map in an online manner to perform positioning.

[0064] Various operations for activating the depth sensor 202 for positioning based on the determined activation benefit level are explained below.In some embodiments, operations for activating 604 the depth sensor 202 for positioning when the benefit level satisfies the activation rule include determining that a value of the benefit level satisfies a threshold.

[0065] Example operational determinations that the benefit level satisfies the activation rules for activating the depth sensor 202 may include any one or more of the following:

[0066] a. The level of benefit is above the defined threshold;

[0067] b. a function of the level of benefit obtained by a set of measurements (e.g. the average of the last N measurements using a monocular camera) being above a defined threshold; and

[0068] c. A function of the level of benefit obtained by a set of measurements performed on data from the positioning map, given both the current measurement of the current pose of the device 500 and the predicted motion trajectory 402 of the device 500 , is above a defined threshold.

[0069] After the depth sensor 202 is activated, the device 500 can determine whether the monocular camera 200 should remain activated or be deactivated.

[0070] The above-described operational embodiments may also be combined with an energy budget for the device. For example, if the above-described conditions are met, a further determination may be made to confirm that sufficient energy budget remains for the monocular camera 200 to remain active after the depth sensor 202 becomes active. As described above, depth sensors typically consume more energy than monocular cameras. If sufficient energy budget remains after activating the depth sensor 202, the monocular camera 200 may remain, otherwise, when the remaining energy budget is insufficient, the monocular camera 200 is deactivated after the depth sensor 202 is activated.

[0071] The performance of the localization algorithm can be improved by using data from both the monocular camera 200 and the depth sensor 202. The visual information of the scene captured by the monocular camera 200 can be processed by the localization algorithm in combination with the depth data from the depth sensor 202. For example, where the image data frames are used to determine specific objects in the scene, this can help characterize the spatial ordering and / or visual characteristics of objects in the environment (e.g., you are in front of store X, or person Y is in front of you, or it is sunny today). In this case, the monocular camera 200 can be used for SLAM processing and provide semantic understanding of the environment.

[0072] Another example reason why the monocular camera 200 can continue to be used for positioning after the depth sensor 202 is activated is that the image data frame can be a preferred method for identifying a specific location in operation and optimizing a map given that location, which is an integral part of the SLAM framework. In this way, the monocular camera 200 can be used to perform a complete SLAM and calculate the pose estimate of the device. When the depth sensor 202 is also activated, the monocular camera 200 can still be used for position recognition and loop closure, while the depth sensor 202 performs a complete SLAM in addition to position recognition and loop closure. The depth sensor 202 can be executed to build an environmental map and calculate the movement of the device based on the map. This is an approach considered in another popular framework, RTAB-MAP, as described in “RTAB-Map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation” (M. Labbé and F. Michaud, Journal of Field Robotics, Vol. 36, No. 2, pp. 416–446, https: / / doi.org / 10.1002 / rob.21831, Wiley, 2018).

[0073] refer to Figure 7 In an embodiment, the operations performed by the device 500 further include: after activating 604 the depth sensor 202 has been performed, determining 700 a level of benefit of using the monocular camera 200 for positioning based on image data from the monocular camera 202 and based on depth data from the depth sensor 202. Figure 8 In another embodiment, the operations then deactivate 800 the monocular camera 200 when the determined 700 benefit level of using the monocular camera for positioning satisfies the deactivation rule.

[0074] In some embodiments, the operation for determining 700 that the level of benefit of using the monocular camera 200 for positioning satisfies the deactivation rule includes: determining the number of feature descriptors in the image data from the monocular camera 200, and determining that the number of feature descriptors in the image data within a common field of view of both the depth sensor 202 and the monocular camera 200 meets a threshold number of feature descriptors required to perform positioning.

[0075] In another embodiment, the number of feature descriptors in the image data from the monocular camera 200 is limited to include only feature descriptors that meet a feature quality threshold.

[0076] Various operations are now described that can determine the level of benefit of using the monocular camera 200 based on analyzing the performance of the localization and mapping algorithm processing image data from the monocular camera 200, and that can be performed based on determining the number of features and / or the number and quality of features detected in the image data. When the level of benefit satisfies the deactivation rule, the monocular camera 200 can be deactivated (e.g., turned off). Otherwise, the monocular camera 200 can remain active and be used to obtain superior performance / robustness of the localization algorithm.

[0077] In order for the monocular camera 200 to provide image data that can be used for localization operations and / or combined localization and mapping operations (e.g., SLAM), the monocular camera 200 must be able to capture relevant environmental features. Various alternative embodiments of operations that can determine the level of benefit of activating the depth sensor 202 based on the performance of a localization algorithm (e.g., the localization and mapping algorithm 212) using image data from the monocular camera 200 will now be described.

[0078] First, the performance of the localization algorithm using image data from the monocular camera 202 is explained. The performance may directly depend on the detection of visual features by the monocular camera 202. If the visual features required for localization cannot be detected in a robust manner, the localization algorithm (e.g., the localization and mapping algorithm 212) will fail in operation. Feature descriptors (e.g., SIFT, BRISK, ORB, based on machine learning) typically describe high contrast areas of an image, such as edges, corners, etc. Not all measured features in the localization and mapping algorithm (e.g., SLAM) contribute to accurate localization in the estimation process, so the operations herein may only utilize features that contribute to accurate localization. The ultimate goal is to be able to detect features that can provide geometric information related to the motion of the monocular camera 202 in a robust manner based on a sequence of image data frames of a scene. Therefore, the performance of the localization algorithm (e.g., the localization and mapping algorithm 212) using the monocular camera 202 may be determined based on an evaluation of the number and quality of the detected features. Various embodiments disclosed herein relate to determining the level of benefit of activating the depth sensor 202 for localization based on predicting the performance of the localization algorithm based on an analysis of image data from the binocular camera 200. The depth sensor 202 is then selectively activated for localization and / or combined localization and mapping (eg, SLAM) based on whether the determined benefit level satisfies the activation rules.

[0079] The above embodiments may be combined with using the energy budget 206 of the device 500 to determine whether the level of benefit of activating the depth sensor 202 satisfies the activation rule and / or whether continuing to use the monocular camera 200 for positioning after activating the depth sensor 202 satisfies the deactivation rule. The use of the energy budget 206 enables the device 500 to avoid situations where activating the depth sensor 202 while the monocular camera 200 is active will result in power consumption exceeding the energy budget 206. The device 500 may activate the depth sensor 202, but then deactivate the monocular camera 200 in order to avoid prolonged power consumption exceeding the energy budget 206. In a corresponding embodiment, determining 700 that the level of benefit of using the monocular camera 200 for positioning satisfies the deactivation rule includes determining that the energy consumed by using both the depth sensor 202 and the monocular camera 200 for positioning is at a level greater than the energy budget 206 of the device 500.

[0080] In another embodiment, determining 700 that the level of benefit of using the monocular camera 200 for positioning satisfies the deactivation rule includes determining that energy consumed by using both the depth sensor 202 and the monocular camera 200 for positioning is at a level greater than the energy budget 206 of the device 500 .

[0081] Deactivation of a sensor (e.g., the monocular camera 200 or the depth sensor 202) in various embodiments of the present invention may be performed by triggering at least one of: transitioning the sensor to a low power state, powering off the sensor, powering off an active component of the sensor that senses the environment (e.g., a LIDAR laser component, an infrared emitter, etc.), reducing the data sampling rate of the sensor or its active component to a level lower than that used for positioning, reducing the resolution of the sensor to a level lower than that used for positioning, changing optical parameters (e.g., focal length, field of view, etc.) to parameters that are not used for positioning, and adapting the positioning algorithm to stop using parameters of the sensor (e.g., optical parameters).

[0082] As described above, activation of a sensor (i.e., the depth sensor 202 or the monocular camera 200) may be performed by triggering at least one of: transitioning the depth sensor to a high power state, powering on the sensor, powering on an active component of the sensor that senses the environment (e.g., a LIDAR laser component, an infrared emitter, etc.), increasing the data sampling rate of the sensor or its active component to a level used for positioning, increasing the resolution of the sensor to a level used for positioning, changing optical parameters (e.g., focal length, field of view, etc.) to parameters used for positioning, and adapting a positioning algorithm to use parameters of the sensor (e.g., optical parameters).

[0083] Thus, in some embodiments, activation 604 of the depth sensor 202 includes triggering at least one of: transitioning the depth sensor 202 to a high power state, powering on the depth sensor 202, increasing the data sampling rate of the depth sensor 202 to a level used for positioning, increasing the resolution of the depth sensor 202 to a level used for positioning, and adapting the positioning algorithm to use depth sensing parameters of the depth sensor 202.

[0084] Some other related embodiments relate to a corresponding method for performing positioning by a device using one or both of a monocular camera and a depth sensor that can be transported with the device. The method includes: receiving 600 image data from the monocular camera; determining 602 a benefit level of activating the depth sensor for positioning based on the image data; and activating 604 the depth sensor for positioning based on determining that the benefit level of activating the depth sensor satisfies an activation rule. In various other embodiments, the method also includes performing the above Figure 1-12 Any operation described in the context of .

[0085] Some other related embodiments relate to a computer program product for performing positioning using one or both of a monocular camera 200 and a depth sensor 202 that can be transported with a device 500. The computer program product includes a non-transitory computer-readable medium 520 storing instructions that can be executed by at least one processor 510 of the device to configure the device 500 to: receive image data from the monocular camera 200; determine, based on the image data, a level of benefit of activating the depth sensor 202 for positioning; and activate the depth sensor 202 for positioning based on determining that the level of benefit of activating the depth sensor 202 satisfies an activation rule. In various other embodiments, the instructions also configure the at least one processor 510 of the device 500 to: also perform the above in Figure 1-12 Any operation described in the context of .

[0086] Other definitions and embodiments are explained below.

[0087] In the above description of various embodiments of the present invention, it should be understood that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those commonly understood by those of ordinary skill in the art to which the present invention belongs. It will also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of this specification and related fields, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0088] When an element is referred to as "connected", "coupled" to another element or "responsive to" (or its variant) another element, it can be directly connected, coupled to another element or responsive to another element, or there can be an intermediate element. In contrast, when an element is referred to as "directly connected", "directly coupled" to another element or "directly responsive to" (or its variant) another element, there is no intermediate element. The same number always refers to the same element. In addition, "coupling", "connection", "responsive to" or its variants used herein may include wireless coupling, connection or response. As used herein, the singular "one", "an" and "the" are intended to also include plural forms, unless the context clearly states otherwise. For brevity and / or clarity, known functions or structures may not be described in detail. The term "and / or" includes any and all combinations of one or more associated listed items.

[0089] It should be understood that although the terms first, second, third, etc. can be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Therefore, without departing from the teaching of the present invention, the first element / operation in some embodiments may be referred to as the second element / operation in other embodiments. Throughout the specification, the same reference numerals or the same figure indicators represent the same or similar elements.

[0090] As used herein, the terms "comprises," "including," "having," or variations thereof, are open ended and include one or more stated features, integers, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integers, elements, steps, components, functions, or combinations thereof. In addition, as used herein, the general abbreviation "eg," derived from the Latin phrase "exempli gratia," may be used to introduce or specify one or more general examples of previously mentioned items, and is not intended to limit such items. The general abbreviation "ie," derived from the Latin phrase "id est," may be used to specify a specific item from a more general narrative.

[0091] Example embodiments are described herein with reference to block diagrams and / or flowchart illustrations of computer-implemented methods, devices (systems and / or equipment), and / or computer program products. It should be understood that a block diagram and / or flowchart illustration of a frame and a combination of frames in the block diagram and / or flowchart illustration can be implemented by computer program instructions executed by one or more computer circuits. These computer program instructions can be provided to a processor circuit of a general-purpose computer circuit, a special-purpose computer circuit, and / or other programmable data processing circuit to produce a machine so that instructions executed by a processor of a computer and / or other programmable data processing device transform and control transistors, values ​​stored in storage cells, and other hardware components within such circuits to implement the functions / actions specified in the block diagram and / or one or more flowchart frames, thereby creating a device (function) and / or structure for implementing the functions / actions specified in the block diagram and / or one or more flowchart frames.

[0092] These computer program instructions may also be stored in a tangible computer-readable medium that can direct a computer or other programmable data processing device to work in a specific manner so that the instructions stored in the computer-readable medium produce an article of manufacture including instructions for implementing the functions / actions specified in the block diagram and / or one or more flowchart blocks. Therefore, embodiments of the present inventive concept may be embodied in hardware and / or software (including firmware, resident software, microcode, etc.) running on a processor such as a digital signal processor, which may be collectively referred to as "circuits", "modules" or variations thereof.

[0093] It should also be noted that in some alternative implementations, the function / action mentioned in the frame may not occur in the order mentioned in the flow chart. For example, two frames shown in succession can actually be performed substantially simultaneously, or these frames can sometimes be performed in reverse order, depending on the function / action involved. In addition, the function of a given frame of a flow chart and / or block diagram can be divided into a plurality of frames and / or the function of two or more frames of a flow chart and / or block diagram can be integrated at least in part. Finally, other frames can be added / inserted between the illustrated frames, and / or frames / operations can be omitted without departing from the scope of the inventive concept. In addition, although some figures include arrows on the communication path to illustrate the main direction of communication, it should be understood that communication can occur in the direction opposite to the arrows shown.

[0094] Without departing substantially from the principles of the inventive concept, many changes and modifications may be made to the embodiments. All of these changes and modifications are intended to be included within the scope of the inventive concept. Therefore, the subject matter disclosed above should be considered to be illustrative rather than restrictive, and the attached examples of the embodiments are intended to cover all such modifications, enhancements, and other embodiments that fall within the spirit and scope of the inventive concept. Therefore, to the maximum extent permitted by law, the scope of the inventive concept will be determined by the broadest permitted interpretation of the present disclosure including the following examples of the embodiments and their equivalents, and should not be bound or limited by the aforementioned detailed description.

Claims

1. A device (500) configured to perform positioning using one or both of a monocular camera (200) and a depth sensor (202) capable of being transported together with the device, the device (500) comprising: at least one processor (510) operatively connected to the monocular camera (200) and the depth sensor (202); at least one memory (520) storing program code, the program code being executed by the at least one processor (510) to perform operations for: Receiving (600) image data from the monocular camera; determining (602) a level of benefit of activating the depth sensor for positioning based on sampling and analyzing the image data from the monocular camera after conversion to 3D data; as well as activating (604) the depth sensor for positioning based on determining that the level of benefit from activating the depth sensor satisfies an activation rule; Wherein determining (602) the level of benefit of activating the depth sensor (202) for positioning comprises: processing (900) the image data from the monocular camera by a localization algorithm to obtain depth points within an environment sensed by the monocular camera; and estimating (902) a density of the depth points within a range of the depth sensor, wherein the benefit level is determined (602) based on the estimate of the density of the depth points; Wherein, estimating the density of the depth points within the range of the depth sensor comprises: identifying within the image data from the monocular camera an object having a determined physical size within the environment; and A range of the depth points is determined based on a comparison of the size of the object within the image data and the physical size of the object.

2. The device (500) according to claim 1, wherein The operations further configure the at least one processor to: determine ( 700 ) a level of benefit of using the monocular camera for localization based on the image data from the monocular camera and based on depth data from the depth sensor after activating ( 604 ) the depth sensor has been performed; wherein the operations further configure the at least one processor to: deactivate (800) the monocular camera based on determining that the level of benefit of using the monocular camera for positioning satisfies a deactivation rule; Wherein, determining (700) that the level of benefit of using the monocular camera for positioning satisfies the deactivation rule comprises: determining a number of feature descriptors in the image data from the monocular camera, and It is determined whether a number of feature descriptors in the image data within a common field of view of both the depth sensor and the monocular camera satisfies a threshold number of feature descriptors required to perform localization.

3. The device (500) according to claim 2, wherein: The number of feature descriptors in the image data from the monocular camera is limited to include only feature descriptors that meet a feature quality threshold.

4. The device (500) according to claim 2, wherein: Determining (700) that the level of benefit of using the monocular camera for positioning satisfies the deactivation rule includes determining that energy consumed using both the depth sensor and the monocular camera for positioning is at a level greater than an energy budget of the device.

5. The device (500) according to claim 2, wherein: Based on hardware resource utilization obtained for the device that uses the monocular camera to perform positioning, determining (700) that the benefit level of using the monocular camera for positioning satisfies the deactivation rule is performed, wherein the hardware resource utilization includes at least one of processor utilization, memory utilization, and network utilization.

6. The device (500) according to claim 1, wherein: Activating (604) the depth sensor includes triggering at least one of: transitioning the depth sensor to a high power state, powering on the depth sensor, increasing a data sampling rate of the depth sensor to a level used for positioning, increasing a resolution of the depth sensor to a level used for positioning, and adapting a positioning algorithm to use depth sensing parameters of the depth sensor.

7. The device (500) according to claim 1, wherein The benefit level of activating the depth sensor is determined to satisfy the activation rule based on the density of the depth points within the range of the depth sensor satisfying a minimum threshold.

8. The device (500) according to claim 7, wherein The minimum threshold is determined based on determining a minimum density of the depth points required for the positioning algorithm to perform positioning with at least a threshold level of accuracy.

9. The device (500) according to claim 1, wherein: Determining (602) the level of benefit of activating the depth sensor for positioning includes: processing (1000) the image data from the monocular camera by a localization algorithm to obtain depth points within an environment sensed by the monocular camera; and determining (1002) a number of three-dimensional (3D) features within depth reconstruction data for a portion of the environment based on the sequence of frames of the image data and the depth points, Therein, the benefit level is determined (602) based on the number of the 3D features.

10. The device (500) according to claim 9, wherein: The level of benefit of activating the depth sensor is determined to satisfy the activation rule based on the number of the 3D features satisfying a minimum threshold.

11. The device (500) according to claim 10, wherein The minimum threshold is determined based on determining a minimum number of the 3D features required for the localization algorithm to perform localization with at least a threshold level of accuracy.

12. The device (500) according to claim 1, wherein Determining (602) the level of benefit of activating the depth sensor (202) for positioning includes: processing (1100) the image data from the monocular camera by an object recognition algorithm and a localization algorithm to obtain a physical object viewed by the monocular camera, including obtaining a size of the physical object and a position of the physical object relative to the device, Therein, the benefit level is determined (602) based on at least one of a type and a size of the physical object and based on a distance between the physical object and the device.

13. The device (500) according to claim 1, wherein Determining (602) the level of benefit of activating the depth sensor for positioning is based on: determining (1200) a position of the depth sensor based on the image data; using the position of the depth sensor to access (1202) a historical location map repository to obtain historical image data; as well as generating (1204) an approximation of depth information that would be available from the depth sensor if the depth sensor were activated, based on the historical image data, Therein, based on the approximation of depth information, the benefit level is determined (602).

14. The device (500) according to claim 1, wherein Based on determining that the benefit level of activating the depth sensor satisfies the activation rule, activating (604) the depth sensor for positioning includes determining that a value of the benefit level satisfies a threshold value.

15. A method of performing positioning by a device using one or both of a monocular camera and a depth sensor capable of being transported with the device, the method comprising: Receiving (600) image data from the monocular camera; determining (602) a level of benefit of activating the depth sensor for positioning based on sampling and analyzing the image data from the monocular camera after conversion to 3D data; as well as activating (604) the depth sensor for positioning based on determining that the level of benefit from activating the depth sensor satisfies an activation rule; Wherein determining (602) the benefit level of activating the depth sensor for positioning comprises: processing (900) the image data from the monocular camera by a localization algorithm to obtain depth points within an environment sensed by the monocular camera; and estimating (902) a density of the depth points within a range of the depth sensor, wherein the benefit level is determined (602) based on the estimate of the density of the depth points; Wherein, estimating the density of the depth points within the range of the depth sensor comprises: identifying within the image data from the monocular camera an object having a determined physical size within the environment; and A range of the depth points is determined based on a comparison of the size of the object within the image data and the physical size of the object.

16. The method of claim 15, further comprising executing the operations performed by at least one processor of any one of claims 2 to 14.

17. A computer program product for performing positioning using one or both of a monocular camera (200) and a depth sensor (202) transportable with a device, the computer program product comprising a non-transitory computer-readable medium storing instructions executable by at least one processor of the device to configure the device to: Receiving (600) image data from the monocular camera; determining (602) a level of benefit of activating the depth sensor for positioning based on sampling and analyzing the image data from the monocular camera after conversion to 3D data; as well as activating (604) the depth sensor for positioning based on determining that the level of benefit from activating the depth sensor satisfies an activation rule; Wherein determining (602) the level of benefit of activating the depth sensor (202) for positioning comprises: processing (900) the image data from the monocular camera by a localization algorithm to obtain depth points within an environment sensed by the monocular camera; and estimating (902) a density of the depth points within a range of the depth sensor, wherein the benefit level is determined (602) based on the estimate of the density of the depth points; Wherein, estimating the density of the depth points within the range of the depth sensor comprises: identifying within the image data from the monocular camera an object having a determined physical size within the environment; and A range of the depth points is determined based on a comparison of the size of the object within the image data and the physical size of the object.

18. The computer program product of claim 17, wherein: The instructions also configure the at least one processor of the device to perform the operations performed by at least one processor according to any one of claims 2 to 14.

Images