Monocular camera activation for localization based on data from depth sensor

Abstract

A device is disclosed, the device being configured to perform positioning using one or both of a depth sensor (200) and a monocular camera (202) that can be transported with the device. The device is adapted to: receive depth data from the depth sensor, determine a benefit level for activating the monocular camera for positioning based on the depth data, and activate the monocular camera for positioning based on a determination that the benefit level for activating the monocular camera satisfies an activation rule. Related methods and computer program products are also disclosed.

Description

Technical Field

[0001] The present disclosure relates to a device that uses one or both of a depth sensor and a monocular camera that can be transported together with the device to perform positioning, a method for a device to perform positioning using one or both of a depth sensor and a monocular camera that can be transported together with the device, and a corresponding computer program product. Background Art

[0002] Localization and mapping algorithms, such as simultaneous localization and mapping (SLAM) algorithms, are technologies that allow a device to localize itself in an environment by relying on onboard sensors such as cameras, range sensors, and inertial sensors. This is critical for robots to navigate and understand the environment to perform tasks, as well as for mixed reality devices to be able to display realistic and persistent content.

[0003] For example, current mixed reality (MR) headsets and state-of-the-art smartphones contain RGB cameras, depth / 3D cameras (e.g., passive or active stereo cameras, LIDAR, etc.), and inertial sensors (IMUs), and the same is true for indoor and outdoor robots (e.g., drones and self-driving cars). For example, Intel and Microvision have recently launched small "consumer" LIDAR products. However, other depth sensors or 3D sensors are very common in headsets and other devices such as robots.

[0004] Several SLAM algorithms have been proposed that rely on RGB and IMU sensors, depth sensors, or a combination of all of these sensors. One reason for using a combination of different types of sensors is to exploit the unique operating advantages of different sensors and improve their individual limitations. For example, RGB cameras do not perform well in dim or too bright environments because it is difficult or impossible to acquire visual features in such environments, while depth cameras such as LIDAR or active stereo cameras will perform well in such scenarios. In addition, by measuring depth directly, localization and mapping can be performed with higher accuracy, and a larger amount of information of the environment can be captured (e.g., building dense maps instead of sparse maps), etc.

[0005] However, depth cameras typically have high energy consumption and processing requirements, and may perform poorly under certain conditions. For example, depth cameras have limited measurement range, and may perform poorly in low-texture environments (passive stereo cameras), in areas with direct sunlight or IR interference (active stereo cameras and LIDAR), in rainy conditions (LIDAR), and under other constraints. Therefore, it is desirable to schedule the use of all available sensors to achieve accurate and robust localization and mapping, while also reducing the energy resources of the device.

[0006] Localization and mapping algorithms (e.g., SLAM algorithms) can be used with mobile devices carrying multiple sensors to simultaneously build a consistent geometric map and determine the position of the mobile device within that map. Some localization and mapping algorithms utilize data from cameras, and sometimes other types of sensors for increased robustness, but still fail to produce consistent maps in some environments. In visually degraded environments (e.g., dimly lit rooms or tunnels), it is difficult to extract useful information from cameras because the probability of features being determined or recognized is low or not possible at all. Summary of the invention

[0007] Currently available devices that use sensors for positioning (e.g., Microsoft Hololens, Magic Leap, ARCore, and ARKit) assume that all such sensors are always active (e.g., powered on) and are unaware of the need for and do not have the ability to perform selective activation and deactivation of individual sensors.

[0008] Some embodiments of the present disclosure relate to a device configured to perform positioning using one or both of a depth sensor and a monocular camera that can be transported with the device. The device includes at least one processor that is operably connected to the depth sensor and the monocular camera. The device also includes at least one memory storing program code, which is executed by at least one processor to perform operations, the operations including receiving depth data from the depth sensor. The operations include determining a benefit level for activating the monocular camera for positioning based on the depth data, and activating the monocular camera for positioning based on a determination that the benefit level for activating the monocular camera satisfies an activation rule.

[0009] Some other related embodiments relate to a method for performing positioning by a device using one or both of a depth sensor and a monocular camera that can be transported with the device. The method includes receiving depth data from the depth sensor, and determining a benefit level for activating the monocular camera for positioning based on the depth data. The method activates the monocular camera for positioning based on a determination that the benefit level for activating the monocular camera satisfies an activation rule.

[0010] Some other related embodiments relate to a computer program product for performing positioning using one or both of a depth sensor and a monocular camera that can be transported with a device. The computer program product includes a non-transitory computer-readable medium storing instructions that can be executed by at least one processor of the device to configure the device to: receive depth data from the depth sensor, determine a benefit level for activating the monocular camera for positioning based on the depth data, and activate the monocular camera for positioning based on a determination that the benefit level for activating the monocular camera satisfies an activation rule.

[0011] Potential advantages of one or more of these embodiments may include: the device is able to use the depth data from the depth sensor to determine when the monocular camera can provide a sufficient level of benefit so that it should be activated from a deactivated state for subsequent use in positioning. In this way, the monocular camera does not have to be activated in order to programmatically determine whether it will provide a sufficient level of benefit for use in positioning, thereby rationalizing its activation and use. These operations can reduce the energy consumption and computing resource utilization of the device when performing positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The various aspects of the present disclosure are illustrated by way of example and not limitation in the accompanying drawings. In the drawings:

[0013] Figure 1 illustrates the conversion of three-dimensional (3D) depth data from a depth sensor to a two-dimensional (2D) depth image for use, wherein the two-dimensional (2D) depth image is processed to determine a level of benefit that may be obtained by activating a monocular camera for use in localization, according to some embodiments;

[0014] Figure 2 A system diagram showing a device for positioning according to some embodiments of the present disclosure;

[0015] Figure 3 A mixed reality MR system is shown that includes a MR headset that holds a mobile electronic device that may include or be operably connected to a set of sensors and is configured to operate according to some embodiments of the present disclosure;

[0016] Figure 4 showing a top view of a device moving through an environment along a predicted trajectory;

[0017] Figure 5 A block diagram illustrating components of a device configured in accordance with some embodiments of the present disclosure;

[0018] Figures 6 to 11A flowchart of operations performed by a device for controlling activation of a monocular camera and deactivation of a depth sensor for positioning according to some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0019] Hereinafter, the inventive concept will be described more fully with reference to the accompanying drawings, in which examples of embodiments of the inventive concept are shown. However, the inventive concept can be embodied in a variety of different forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided so that the disclosure will be comprehensive and complete, and the scope of each inventive concept is fully conveyed to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. The components of an embodiment can be assumed by default to be present in / used in another embodiment.

[0020] Disclosed are methods, devices, and computer program products for determining when a currently deactivated monocular camera should be activated for performing a positioning operation based on depth data received from a depth sensor. As will be described below, the benefit level of activating the monocular camera for positioning is determined based on the depth data from the depth sensor (e.g., based on structural information of a physical object identified in the depth data). The monocular camera is activated for positioning based on a determination that the benefit level of activating the monocular camera satisfies an activation rule. Although various embodiments are described in the context of performing positioning, these and other embodiments may be used to perform combined positioning and mapping operations, such as SLAM. Therefore, the term "positioning" as used herein may interchangeably refer to operations that are configured only to perform positioning functions and operations that are configured to perform a combination of positioning and mapping functions, such as SLAM.

[0021] Various embodiments of the present disclosure are described in the context of a device that includes both a monocular camera and a depth sensor. Figure 1An operation of converting three-dimensional (3D) depth data 100 from a depth sensor (e.g., LIDAR) into a two-dimensional (2D) depth image 102 is shown. According to some embodiments, the 2D depth image 102 may then be processed to determine the level of benefit that may be obtained by activating a monocular camera for use in positioning. An estimate of an image that may be captured by a monocular camera, such as an RGB camera, may be extracted from the 3D depth data 100 by converting the 3D depth data 100 into a 2D depth image 102 using intrinsic parameters of the depth camera. The intrinsic parameters may be obtained or determined using, for example, a software development kit (SDK) for a depth sensor (e.g., see Intel RealSense D400 Camera Series SDK). Although various embodiments are described herein in the context of using 3D depth data, it should be understood that depth data of any dimension (e.g., 1D, 2D, 3D) may be used. Thus, the term "3D depth data" refers to such depth data from a sensor that provides a three-dimensional indication of the sensed depth of an environmental object. Conversely, the term "depth data" refers to such depth data from a sensor that provides any dimensional indication of the sensed depth of an environmental object.

[0022] Potential advantages of one or more of these embodiments may include: the device is able to determine when the monocular camera can provide a sufficient level of benefit so that it should be activated from a deactivated state for use in positioning based on the 3D depth data 100 received from the depth sensor. In this way, the monocular camera does not have to be activated in order to programmatically determine whether it will provide a sufficient level of benefit to be used in positioning. These operations can reduce the energy consumption and computing resource utilization of the device when performing positioning.

[0023] In some embodiments, the following operations may be performed to determine whether the benefit level of activating the monocular camera is sufficient to trigger its activation, and may further determine whether the depth sensor should be deactivated:

[0024] 1. Receive depth data from the depth sensor;

[0025] 2. Determine the benefit level of activating the monocular camera based on the depth data;

[0026] 3. If the benefit level of activating the monocular camera for positioning satisfies the activation rule, then the monocular camera is activated; and

[0027] 4. (Optional) After activation of the monocular camera has been performed, determine a benefit level of using the depth sensor for positioning based on the depth data from the depth sensor and based on the image data from the monocular camera, and deactivate the depth sensor based on a determination that the benefit level of using the depth sensor for positioning satisfies the deactivation rule.

[0028] Figure 5 A block diagram of components of an example device 500 configured in accordance with some embodiments of the present disclosure is shown. Figure 6 A flow chart illustrating operations that may be performed by the device 500 to control activation of the monocular camera 202 for positioning based on a determination that a benefit level for activating the monocular camera 202 for positioning satisfies an activation rule, in accordance with some embodiments of the present disclosure.

[0029] refer to Figure 5 and Figure 6 In some embodiments, the device 500 includes at least one processor 510 (hereinafter referred to as "processor"), which is operably connected to the depth sensor 200, the monocular camera 202, and at least one memory 520 (hereinafter referred to as "memory"), wherein the memory 520 stores program code executed by the processor 510 to perform the following operations: receiving 600 depth data from the depth sensor 200, determining 602 a profit level for activating the monocular camera 202 for positioning based on the depth data, and activating 604 the monocular camera 202 for positioning based on a determination that the profit level for activating the monocular camera 202 satisfies an activation rule. The memory 520 may include a positioning map library 522 (e.g., a historical positioning map), which can be used as described below. The device 500 may include a wireless transceiver 530 configured to communicate via a wireless interface.

[0030] Depth sensor 200 and monocular camera 202 may be shipped with device 500, but are not necessarily part of device 500. For example, although Figure 5 The device 500 is shown to include a depth sensor 200, a monocular camera 202, a processor 510, and a memory 520, but in some embodiments, one or more of these components may be separate from the device 500 and communicatively connected to the device 500 via a wireless transceiver 530 and / or a wired interface. The device 500 may be, but is not limited to, a component of any of a smartphone, a wearable computer, an augmented reality headset, a virtual reality headset, a mixed reality headset, a semi-autonomous or autonomous vehicle, a drone, an aircraft, a robot, and the like.

[0031] Figure 3A mixed reality (MR) system is shown that includes a MR headset 300 that holds a mobile electronic device 320 that can be operably connected to at least one monocular camera 202 and at least one depth sensor 200, for example, via a wired and / or wireless communication interface. The mobile electronic device 320 can include or be operably connected to a processor 510 and a memory storing program code that configures the processor 510 to activate and deactivate individuals of the monocular camera 202 and the depth sensor 200 when performing positioning using activated individuals of the monocular camera 202 and the depth sensor 200.

[0032] Although various embodiments are described in the context of activating and deactivating individual sensors (e.g., a monocular camera and a depth sensor), these embodiments can be used to activate and deactivate groups of sensors. Thus, for example, "monocular camera" can correspond to a group of monocular cameras, and "depth sensor" can correspond to a group of depth sensors. A group of sensors can include sensors of the same type or different types.

[0033] The MR headset 300 includes a lens 310 through which a user wearing the MR headset can view real-world features. The MR headset 300 also includes a bracket 321 configured to releasably hold a mobile electronic device 320 in a specified orientation relative to the lens 310, such that an image displayed on a display device of the mobile electronic device 320 is reflected directly or indirectly toward the user's eyes by the lens 310. Although not shown, the MR headset 300 may include an intermediate mirror located between the lens 310 and the user's eyes, so that light may be reflected directly or indirectly toward the user's eyes and / or the camera 202.

[0034] Mobile electronic device 320 may include, but is not limited to, a smart phone, a handheld computer, a tablet computer, a gaming device, or other computing device. For simplicity, a "mobile electronic device" is also referred to as a "mobile device" and a "device" in this article.

[0035] Figure 2 A system diagram of a device configured to perform a localization operation and / or a combined localization and mapping operation (eg, SLAM) using a depth sensor 200 and a monocular sensor 202 according to some embodiments of the present disclosure is shown. Figure 2, the device includes sensor activation and deactivation logic 204, the logic 204 configured to determine a benefit level of activating the monocular camera 202 for positioning operations based on depth data from the depth sensor 200, and selectively activate the monocular camera 202 based on the determination. The sensor activation and deactivation logic 204 can also be configured to deactivate the depth sensor 200 based on a determination that the benefit level of using the depth sensor 200 for positioning satisfies a deactivation rule, such as when the monocular camera 202 has been activated for positioning and continued use of the depth sensor 200 does not provide sufficient continued benefit. When the monocular camera 202 is active, image data is provided from the monocular camera 202 to a positioning algorithm (e.g., a positioning and mapping algorithm 212). Similarly, when the depth sensor 200 is active, depth data is provided from the depth sensor 200 to a positioning algorithm (e.g., a positioning and mapping algorithm 212).

[0036] The sensor activation and deactivation logic 204 controls the switching logic 208, which performs deactivation and activation of selected sensors of the sensors 200 and 202. The switching logic 208 can perform activation of the sensor (i.e., the depth sensor 200 or the monocular camera 202) by triggering at least one of the following: switching the sensor to a higher power state, powering on the sensor, increasing the data sampling rate of the sensor to a level used for positioning, increasing the resolution of the sensor to a level used for positioning, changing the optical parameters (e.g., focal length, field of view, etc.) to parameters used for positioning, and adjusting the positioning algorithm to use the parameters of the sensor (e.g., optical parameters). Conversely, the switching logic 208 can perform deactivation of the sensor by triggering at least one of the following: switching the sensor to a lower power state, powering off the sensor, reducing the data sampling rate of the sensor to a level lower than that used for positioning, reducing the resolution of the sensor to a level lower than that used for positioning, changing the optical parameters (e.g., focal length, field of view, etc.) to parameters not used for positioning, and adjusting the positioning algorithm to stop using the parameters of the sensor (e.g., optical parameters). Thus, the term "switching" is not limited to on-off switching, but may alternatively or additionally include control logic that performs one or more of the more complex above-mentioned activities for activating and deactivating the sensor.

[0037] In one embodiment, the operation of adjusting the positioning algorithm to use the optical parameters of the sensor for positioning may include: obtaining algorithm parameters corresponding to the optical parameters of the sensor. The algorithm parameters may be predetermined based on adjusting the positioning algorithm offline for different optical parameter sets. Then, the corresponding predetermined algorithm parameters are selected for use based on the prescribed optical parameter set of the sensor.

[0038] As will be described in further detail below, sensor activation and deactivation logic 204 may be operable to use information provided by energy budget 206 and / or by positioning map 210 (which may reside in Figure 5 The information provided by the positioning map library 522 in the positioning map library 522 determines when to activate the monocular camera 202 and / or when to deactivate the depth sensor 200.

[0039] In order for the monocular camera 202 to provide image data that can be used for localization operations and / or combined localization and mapping operations (e.g., SLAM), the monocular camera 202 must be able to capture relevant environmental features. Various alternative embodiments of operations that can determine the benefit level of activating a monocular camera based on the performance of a localization algorithm (e.g., localization and mapping algorithm 212) will now be described.

[0040] The performance of a localization algorithm (e.g., localization and mapping algorithm 212) using a monocular camera 202 is first described. The performance may directly depend on the detection of visual features by the monocular camera 202. If the desired visual features cannot be detected in a robust manner, the localization algorithm (e.g., localization and mapping algorithm 212) will fail in operation. Feature descriptors (e.g., SIFT, BRISK, ORB, machine learning based) typically describe high contrast areas of an image, such as edges, corners, etc. Not all measured features in a localization and mapping algorithm (e.g., SLAM) contribute to accurate localization during the estimation process, so the operations herein may only use those features that contribute. The ultimate goal is to be able to detect these features in a robust manner, which can provide geometric information about the motion of the monocular camera 202 based on a sequence of image data frames of the scene. Therefore, the performance of a localization algorithm (e.g., localization and mapping algorithm 212) using a monocular camera 202 can be determined based on an evaluation of the number and quality of detected features. Various embodiments disclosed herein relate to determining a benefit level of activating the monocular camera 202 for localization based on predicting the performance of a localization algorithm based on analysis of depth data from the depth sensor 200. Then, based on whether the determined benefit level satisfies an activation rule, the monocular camera 202 is selectively activated for localization and / or combined localization and mapping, such as SLAM.

[0041] Various methods for determining whether the monocular camera 202 will be able to capture sufficient relevant object features in the environment based on analyzing the 3D depth data 100 captured by the depth sensor 200 are discussed below. Various other related methods for determining the profitability level of the monocular camera 202 are discussed below.

[0042] In some embodiments, based on converting the 3D depth data 100 into the 2D depth image 102 (eg Figure 1200 for activating the monocular camera 202 for positioning. In one embodiment, the benefit level of activating the monocular camera 202 for positioning is determined 602 based on converting the depth data representing the 3D positioning map 210 into 2D depth data representing the 2D depth image 102, and generating an approximation of the depth information that can be obtained from the monocular camera 202 (if the monocular camera 202 is activated) based on the 2D depth image 102. Then, the benefit level of activating the monocular camera 200 for positioning is determined based on the approximation of the depth information.

[0043] In one specific example, the benefit level is determined based on converting the 3D depth data 100 into a 2D depth image 102 using depth camera intrinsic parameters, where the depth sensor intrinsic parameters may be provided by the manufacturer or may be obtained via calibration of the depth sensor 200. For example, given a depth sensor intrinsic parameter matrix M, a 3D voxel with a pose P_3D may be converted to a 2D pixel with a pose P_2D according to P_2D=M*P_3D. For example, for an Intel RealSense D400 depth camera, this process is implemented by a function rs2_project_point_to_pixel() that is part of the device software, where, given a 3D voxel as input, the function computes the corresponding 2D pixel. Reference Figure 1 In an example of this process shown in , a 2D depth image 102 is generated as an approximation of the information required by the monocular camera 202 (if active). The approximation can be performed more accurately by repeatedly converting a sequence of 3D depth data 100 from different perspectives into a sequence of 2D depth images 102. This can correspond to snapshots of the 3D depth data 100 (which are described as 3D depth data) taken from different angles.

[0044] Fig. 9 The corresponding operations configured according to some embodiments of the present disclosure are shown in FIG. Fig. 9 Operation 900 repeats the conversion of the depth data representing the view in the 3D positioning map 210 to 2D depth data representing the 2D depth image 102 for each of the plurality of different view angles in the 3D positioning map 210. The generation of the approximation of the depth information is performed based on the 2D depth data for the plurality of different view angles in the 3D positioning map 210.

[0045] The determination 602 of the benefit level of activating the monocular camera for localization can be based on the determination of the number of feature descriptors detected by processing the 2D depth data by a feature descriptor detection algorithm. Given the 2D depth image 102, it is possible to understand how many and how good 2D visual features the monocular camera 202 will acquire, for example by feeding the 2D depth image 102 to a feature detector algorithm such as ORB, SIFT, etc. and acquiring the number M of detected features.

[0046] Therefore, in one embodiment, determining 602 the benefit level of activating the monocular camera 202 for localization can also be based on the number of feature descriptors that meet a minimum feature number threshold. For example, the quality Qi of the detected features can also be obtained. Then, the benefit level can be equal to M, or a function of the feature quality and the number of features (e.g., the number of features Mi out of M features, where Qi(Mi)>Qmin), where Qmin is the minimum quality level required for a feature to be considered a good feature. The quality of a feature can be inferred by conventional methods.

[0047] For example, the following public document defines the quality of each feature in Section 2.2 as the feature's τ-time observability score over τ frames, which essentially represents the consistency of the feature observed in all consecutive image frames as the device moves: Zhang, G., & Vela, P. A. (2015), "Good features to track for visual SLAM", 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), doi:10.1109 / cvpr.2015.7298743, 2015. The higher the score, the higher the quality of the feature used for localization and mapping. Other quality metrics can be used. For example, the RANSAC algorithm can be applied to features of several image sequences, and the inliers and outliers can be calculated for the image sequences. Features that are always set in the inliers are considered to have higher quality than features included in the outliers.

[0048] In some embodiments, a benefit level of activating the monocular camera 202 for localization may be determined 602 based on sampling and analyzing the 3D depth data 100 contained in the localization map 210 currently used by the depth camera 200 to perform localization. In one embodiment, the determination of the benefit level of activating the monocular camera for localization may include determining the position of the monocular camera based on the depth data. The position of the monocular camera 202 is used to access a library of historical 3D localization maps (which may be used in conjunction with Figure 5The historical depth data is converted into historical 2D depth data representing the historical 2D depth image. An approximation of the depth information that can be obtained from the monocular camera 202 (if activated) is generated based on the historical 2D depth image. The profit level is then determined based on the approximation of the depth information.

[0049] In other embodiments, using (e.g., Figure 5 The benefit level of activating the monocular camera 202 is determined based on the historical 3D positioning map in the library 522 in FIG. 5 and the presented trajectory of the position of the monocular camera 202 relative to the historical 3D positioning map. Figure 4 and Fig.10 Example operations are described in the context of . Figure 4 A top view of a device 400 having a depth sensor 200 and a monocular camera 202 is shown moving through an environment along a predicted trajectory 402. In some embodiments, for each of a plurality of different views along the predicted motion trajectory 402 of the monocular camera 202 relative to a location in the historical 3D localization map, operation 1000 is repeated to convert historical depth data representing a view in the historical 3D localization map to historical 2D depth data representing the historical 2D depth image 102. The generation of the approximation of the depth information is performed based on the historical 2D depth data along the plurality of different views of the predicted motion trajectory 402 of the monocular camera 202 relative to the historical 3D localization map.

[0050] For example, the device 500 determines its current pose relative to the historical 3D positioning map used for positioning, as well as knowledge of the direction of motion of the device 500. Given this information, these operations obtain 2D depth images 102 of different perspectives corresponding to the estimate of the motion trajectory 402 within the historical 3D positioning map, and perform similar operations as detailed in the previous embodiments. For each selected perspective, the operation can determine the level of benefit of activating the monocular camera 202 for positioning. Therefore, the operation can additionally rely on depth data captured a priori and stored in the historical 3D positioning map, rather than using only the online data currently captured from the depth sensor 200.

[0051] Other operations are now described that may be performed to determine a benefit level of activating the monocular camera 202 for positioning, which may then be used to determine whether the monocular camera 202 should be activated.

[0052] In some embodiments, activating 604 the monocular camera 202 based on whether the revenue level of activating the monocular camera 202 satisfies the activation rule may include determining whether the value of the revenue level satisfies a threshold. Fig.11 The operation shown is used to determine the threshold. Fig.11The operation is based on determining the number of feature descriptors in the depth data from the depth sensor 200 that are estimated to be detectable by the monocular camera 202, and determining that the number of feature descriptors meets the threshold number of feature descriptors required to perform localization, to determine 1100 the threshold. In other embodiments, the number of feature descriptors in the depth data from the depth sensor that are estimated to be detectable by the monocular camera 202 is limited to include only feature descriptors that meet the feature quality threshold.

[0053] In one particular example, the benefit level is equal to the number M of detected feature descriptors being above a desired threshold, i.e., M>Mmin, where Mmin is defined as the minimum number of feature descriptors that must be detected for a localization algorithm (e.g., localization and mapping algorithm 212) to perform well enough for its intended purpose. This can be obtained based on offline experiments using, for example, a SLAM algorithm 212, and then adjusted online, the SLAM algorithm 212 being, for example, as proposed in the following literature: Schneider, T., Dymczyk, M., Fehr, M., Egger, K., Lynen, S., Gilitschenski, I., and Siegwart, R., "Maplab: An open framework for research in visual-inertial mapping and localization", IEEE Robotics and Automation Letters, Vol. 3, No. 3, pp. 1418-1425, 2018. Furthermore, the operation may combine the quantity of feature descriptors with the quality of the feature descriptors and define that the quality Qi of the feature descriptors should be above a defined threshold (ie Qi>Qmin), for example 50% higher, where Qmin may again be defined based on offline experiments and then adjusted online.

[0054] In order to perform the calculation of the device pose given an available map with 3D pose, the minimum number of feature descriptors that need to be extracted from the image of the monocular camera 202 can depend on the localization algorithm used and the quality of the inferred feature descriptors. For example, the operation can be configured to require at least 6 feature descriptors that meet a quality threshold. However, in order to deal with noise when detecting and calculating the pose of the features, a larger number of feature descriptors should be detected.

[0055] In some embodiments, if a function of a profit level obtained over a set of measurements (eg, an average of the most recent N measurements of the depth sensor 200 ) is above an activation threshold, the monocular camera 202 should be activated.

[0056] Reference again Figure 4In some embodiments, when using the measurements of the current pose of the device 400 and the predicted pose of the device 400 along the future motion trajectory 402 in the pair (eg, Figure 5 The monocular camera 202 should be activated when the benefit level determined on the set of measurements performed on the localization map (within the localization map library 522 in the example embodiment) is higher than the activation threshold.

[0057] The determination of whether the benefit level of activating the monocular camera 202 satisfies the activation rule may be based on the device's energy budget 206. For example, if the benefit level of activating the monocular camera 202 satisfies the above conditions, and the energy consumption of the device also needs to be reduced based on the energy budget 206, the monocular camera 202 with lower power consumption may be activated, and thereafter, the depth sensor 200 with higher power consumption may be deactivated to reduce the power consumption of the device.

[0058] In one specific example, the values ​​of Mmin and Qmin are trained based on experiments in which both the monocular camera 202 and the depth sensor 200 are activated and the number M of feature descriptors and their quality Qi are detected for both sensors. For example, in order for the monocular camera 202 to perform at the desired performance, the Mmin and Qmin of the monocular camera 202 image should be at the desired values, and the corresponding M and Qi values ​​for a given depth image are recorded. A model is then established of the relationship between M and Qi corresponding to the 3D depth data 100 and Mmin and Qmin of the monocular image, so that when the calculated benefit level given the 3D depth data 100 is above these values, the monocular camera 202 should be activated.

[0059] Some other embodiments involve determining a benefit level of continuing to use the depth sensor 200 for positioning after the monocular camera 202 has been activated for positioning, which can then trigger deactivation of the depth sensor. The benefit level of continuing to use the depth sensor 200 for positioning can be determined based on the 3D depth data 100 from the depth sensor 200 and the image data received from the currently activated monocular camera 202. Figure 7 A corresponding flow chart of operations for determining a level of benefit of using the depth sensor 200 for positioning is shown. The operations determine 700 a level of benefit of using the depth sensor 200 for positioning based on depth data from the depth sensor 200 and based on image data from the monocular camera 202 after activation 604 of the monocular camera 202 has been performed.

[0060] After activating the monocular camera 202, the device can determine whether the depth sensor 200 should remain activated or be deactivated. Figure 8 In an embodiment of the present invention, the device is operative to deactivate 800 the depth sensor based on a determination that a level of benefit from using the depth sensor for positioning satisfies a deactivation rule.

[0061] In one embodiment, determining 700 that the level of benefit of using the depth sensor 200 for localization satisfies the deactivation rule includes determining the number of feature descriptors in the image data from the monocular camera 202, and determining that the number of feature descriptors in the image data located within a common field of view of both the depth sensor 200 and the monocular camera 202 satisfies a threshold number of feature descriptors required to perform localization. Because the field of view of the monocular camera 202 may be different from the field of view of the depth camera 200, the feature descriptors may be determined, for example, in a 2D depth image space (from the 3D depth data of the depth sensor) corresponding to the field of view of the monocular camera 202.

[0062] In other embodiments, the number of feature descriptors in the image data from the monocular camera 202 is limited to include only feature descriptors that meet a feature quality threshold.

[0063] Given the monocular camera measurements, the decision whether to deactivate the depth sensor 200 or to keep it activated on its behalf can include analyzing the performance of the localization algorithm (e.g., the localization and mapping algorithm 212). If the performance and / or robustness meet the deactivation role, the depth camera 200 is turned off accordingly. The performance and / or robustness can be analyzed based on, for example, whether the number M of features detected in the image data from the monocular camera 202 and the quality Qi of the features Mi are above a defined threshold. The threshold can be obtained and defined offline and then further adjusted online. This can be combined with the properties of the pose obtained from the localization algorithm (e.g., the localization and mapping algorithm 212), such as the error and variance of the output values ​​of the determined localization algorithm, and if those values ​​are below a certain threshold, it can be interpreted that the algorithm performance and / or robustness is acceptable. Alternatively or additionally, the depth camera 200 can remain activated and used to achieve excellent algorithm performance.

[0064] The above embodiments may be combined with the use of the energy budget 206 of the device 500 to determine whether the benefit level of activating the monocular camera meets the activation rule. The use of the energy budget 206 enables the device to react to a situation where there is too much power consumption because the depth sensor 200 and the monocular camera 202 are both activated at the same time, which in response may trigger the deactivation of the depth camera 200. In a corresponding embodiment, the determination 700 that the benefit level of using the depth sensor 200 for positioning meets the deactivation rule includes: determining that the level of energy consumed by using both the depth sensor 200 and the monocular camera 202 for positioning is greater than the energy budget 206 of how much energy the device 500 should provide.

[0065] In various embodiments of the present invention, deactivation of a sensor (e.g., the monocular camera 202 or the depth sensor 200) can be performed by triggering at least one of the following: transitioning the sensor to a lower power state, powering off the sensor, reducing the data sampling rate of the sensor to a level below that used for positioning, reducing the resolution of the sensor to a level below that used for positioning, changing optical parameters (e.g., focal length, field of view, etc.) to parameters not used for positioning, and adjusting the positioning algorithm to stop using parameters of the sensor (e.g., optical parameters).

[0066] In another embodiment, the determination 700 that the benefit level of positioning using the depth sensor 200 satisfies the deactivation rule is performed based on determining the hardware resource utilization obtained by the device using the depth sensor 200 to perform positioning, wherein the hardware resource utilization includes at least one of processor utilization, memory utilization, and network communication utilization.

[0067] As described above, activation of the sensor (i.e., the depth sensor 200 or the monocular camera 202) may be performed by triggering at least one of: transitioning the sensor to a higher power state, powering on the sensor, increasing the data sampling rate of the sensor to a level used for positioning, increasing the resolution of the sensor to a level used for positioning, changing optical parameters (e.g., focal length, field of view, etc.) to parameters used for positioning, and adjusting the positioning algorithm to use the parameters of the sensor (e.g., optical parameters). Thus, in one embodiment, activation 604 of the monocular camera 202 includes at least one of: triggering transitioning the monocular camera 202 to a higher power state, powering on the monocular camera 202, increasing the data sampling rate of the monocular camera 202 to a level used for positioning, increasing the resolution of the monocular camera 202 to a level used for positioning, and adjusting the positioning algorithm 212 to use the optical parameters of the monocular camera 202.

[0068] Some other related embodiments relate to a corresponding method for performing positioning by a device using one or both of a depth sensor and a monocular camera that can be transported with the device. The method includes: receiving 600 depth data from the depth sensor, determining 602 a benefit level for activating the monocular camera for positioning based on the depth data, and activating 604 the monocular camera for positioning based on a determination that the benefit level for activating the monocular camera satisfies an activation rule. In various other embodiments, the method also performs the above Figures 1 to 11 Any operation described in the context of .

[0069] Some other related embodiments relate to a computer program product for performing positioning using one or both of a depth sensor and a monocular camera that can be transported with a device. The computer program product includes a non-transitory computer-readable medium 520 storing instructions, the instructions being executable on at least one processor of the device to configure the device to: receive depth data from the depth sensor, determine a benefit level for activating the monocular camera for positioning based on the depth data, and activate the monocular camera for positioning based on a determination that the benefit level for activating the monocular camera satisfies an activation rule. In various other embodiments, the instructions further configure the device to also perform the above in Figures 1 to 11 Any operation described in the context of .

[0070] Other definitions and embodiments are described below.

[0071] In the above description of various embodiments of the inventive concept, it is to be understood that the terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the inventive concept. Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as those generally understood by a person of ordinary skill in the art to which the inventive concept belongs. It will be understood that terms such as those defined in general dictionaries should be interpreted as having a meaning consistent with their meaning in the context of this specification and the relevant art, and not as an ideal or overly formal meaning, unless such a clear definition is provided herein.

[0072] When an element is referred to as being "connected," "coupled," "responsive," or variations thereof relative to another element, it may be directly connected, coupled to, or responsive to the other element, or there may be an intermediate element. In contrast, when an element is referred to as being "directly connected," "directly coupled," "directly responsive," or variations thereof relative to another element, there is no intermediate element. Throughout the text, similar reference numerals represent similar elements. In addition, "coupling," "connection," "responsiveness," or variations thereof used herein may include wireless coupling, connection, or response. As used herein, the singular forms "one," "an," and "said" are intended to also include plural forms, unless the context clearly indicates otherwise. For the sake of brevity and / or clarity, well-known functions or structures may not be described in detail. The term "and / or" includes any and all combinations of one or more items listed in association.

[0073] It will be understood that although the terms first, second, third, etc. can be used herein to describe each element / operation, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Therefore, the first element / operation in some embodiments can be referred to as the second element / operation in other embodiments without departing from the teaching of the inventive concept. Throughout the specification, the same reference numerals or the same reference signs represent the same or similar elements.

[0074] As used herein, the terms "includes," "comprising," "containing," "covering," "consisting of," "included," "having," "having," "having," or variations thereof are open ended and include one or more recited features, integers, elements, steps, components, or functions, but do not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions, or combinations thereof. In addition, as used herein, the commonly used abbreviation "eg," derived from the Latin phrase "exempligratia," may be used to introduce or specify one or more general examples of a previously mentioned item without being intended as a limitation of that item. The commonly used abbreviation "ie," derived from the Latin phrase "idest," may be used to specify a specific item of a more general reference.

[0075] This article describes example embodiments with reference to block diagrams and / or flowchart illustrations of computer-implemented methods, devices (systems and / or equipment) and / or computer program products. It should be understood that the blocks shown in the block diagrams and / or flowcharts and the combination of blocks shown in the block diagrams and / or flowcharts can be implemented by computer program instructions executed by one or more computer circuits. These computer program instructions can be provided to processor circuits of general-purpose computer circuits, special-purpose computer circuits and / or other programmable data processing circuits to produce a machine, so that the instructions executed by the processor of the computer and / or other programmable data processing device convert and control transistors, values ​​stored in memory locations, and other hardware components within such circuits to implement the functions / actions specified in the block diagrams and / or flowcharts, and thereby create a device (functional body) and / or structure for implementing the functions / actions specified in the block diagrams and / or flowcharts.

[0076] These computer program instructions may also be stored in a tangible computer-readable medium that can direct a computer or other programmable data processing device to act in a specific manner so that the instructions stored in the computer-readable medium produce an article of manufacture, the article of manufacture including instructions for implementing the functions / actions specified in the blocks of the block diagrams and / or flow charts. Thus, embodiments of the inventive concept may be implemented in hardware and / or in software (including firmware, stored software, microcode, etc.) running on a processor such as a digital signal processor, which may be collectively referred to as a "circuit," "module," or variations thereof.

[0077] It should also be noted that in some alternative implementations, the function / action marked in the frame may not occur in the order marked in the flow chart. For example, depending on the function / action involved, the two frames shown in succession can actually be performed substantially simultaneously, or the frame can sometimes be performed in the opposite order. In addition, the function of a given frame of a flow chart and / or block diagram can be divided into multiple frames and / or the function of two or more frames of a flow chart and / or block diagram can be integrated at least in part. Finally, without departing from the scope of the inventive concept, other frames can be added / inserted between the frames shown, and / or frames / operations can be omitted. In addition, although some frames include arrows about the communication path for indicating the main direction of communication, it should be understood that communication can occur in the direction opposite to the arrow represented.

[0078] Under the premise of not departing from the principle of the inventive concept basically, many changes and modifications can be made to the embodiment. All these changes and modifications are intended to be included in the scope of the inventive concept herein. Therefore, the above-mentioned subject matter should be understood as exemplary and not restrictive, and the examples of the attached embodiments are intended to cover all these modifications, improvements and other embodiments that fall within the spirit and scope of the inventive concept. Therefore, to the maximum extent permitted by law, the scope of the inventive concept should be determined by the widest allowed interpretation of the examples of the attached embodiments and their equivalents, and should not be limited to or restricted to the previous specific embodiments.

Claims

1. A device (500) configured to perform positioning using one or both of a depth sensor (200) and a monocular camera (202) capable of being transported together with the device, the device (500) comprising: at least one processor (510) operably connected to the depth sensor (200) and the monocular camera (202); At least one memory (520) storing program code executed by the at least one processor (510) to perform operations: receiving (600) depth data from the depth sensor; Determining (602) a benefit level of activating the monocular camera for positioning based on the depth data; and Based on a determination that a benefit level of activating the monocular camera satisfies an activation rule, activating (604) the monocular camera for positioning, wherein The benefit level of activating the monocular camera for localization is determined (602) based on: Converting the depth data representing the 3D localization map into 2D depth data representing the 2D depth image; generating an approximation of depth information based on the 2D depth image, the depth information being obtainable from the monocular camera when the monocular camera is activated; as well as A benefit level of activating the monocular camera for positioning is determined based on the approximation of the depth information.

2. The device (500) according to claim 1, wherein The operations also configure the at least one processor to determine ( 700 ) a benefit level of using the depth sensor for positioning based on depth data from the depth sensor and based on image data from the monocular camera after activation of the monocular camera has been performed ( 604 ).

3. The device (500) according to claim 2, wherein: The operations also configure the at least one processor to: deactivate ( 800 ) the depth sensor based on a determination that a revenue level for positioning using the depth sensor satisfies a deactivation rule.

4. The device (500) according to claim 3, wherein The determination (700) that the benefit level of using the depth sensor for positioning satisfies the deactivation rule includes: determining a number of feature descriptors in the image data from the monocular camera, and It is determined whether a number of feature descriptors in the image data located within a common field of view of both the depth sensor and the monocular camera satisfies a threshold number of feature descriptors required to perform localization.

5. The device (500) according to claim 4, wherein The number of feature descriptors in the image data from the monocular camera is limited to include only feature descriptors that meet a feature quality threshold.

6. The device (500) according to any one of claims 3 to 5, wherein: The determination (700) that the benefit level of using the depth sensor for positioning satisfies the deactivation rule includes determining that the level of energy consumed by using both the depth sensor and the monocular camera for positioning is greater than an energy budget of how much energy the device should provide.

7. The device (500) according to any one of claims 3 to 5, wherein: The determination (700) that a benefit level of using the depth sensor for positioning satisfies the deactivation rule is performed based on hardware resource utilization obtained by the device using the depth sensor to perform positioning, wherein the hardware resource utilization includes at least one of processor utilization, memory utilization, and network communication utilization.

8. The device (500) according to any one of claims 1 to 5, wherein: Activating (604) the monocular camera includes triggering at least one of: transitioning the monocular camera to a higher power state, powering on the monocular camera, increasing a data sampling rate of the monocular camera to a level used for positioning, increasing a resolution of the monocular camera to a level used for positioning, changing optical parameters to parameters used for positioning, and adjusting a positioning algorithm to use the optical parameters of the monocular camera.

9. The device (500) according to claim 1, further comprising: Repeating (900) converting depth data representing a view in the 3D positioning map to 2D depth data representing the 2D depth image for each of a plurality of different views in the 3D positioning map, The generation of the approximate value of the depth information is performed based on 2D depth data of multiple different viewing angles in the 3D positioning map.

10. The device (500) according to any one of claims 1 to 5, wherein: The determination of a benefit level of activating the monocular camera for localization ( 602 ) is based on a determination of a number of feature descriptors detected by processing the 2D depth data via a feature descriptor detection algorithm.

11. The device (500) according to claim 10, wherein The determination of a benefit level for activating the monocular camera for localization ( 602 ) is also based on the number of feature descriptors that meet a minimum feature number threshold.

12. The device (500) according to any one of claims 1 to 5, wherein: The determination of the benefit level of activating the monocular camera for localization (602) is based on the following: Determining a position of the monocular camera based on the depth data; Using the position of the monocular camera to access a historical 3D positioning library to obtain historical depth data; converting historical depth data representing the historical 3D positioning map into historical 2D depth data representing the historical 2D depth image; generating an approximation of depth information based on the historical 2D depth image, the depth information being obtainable from the monocular camera when the monocular camera is activated; as well as A benefit level of activating the monocular camera for positioning is determined based on the approximation of the depth information.

13. The device (500) according to claim 12, further comprising: Repeating (1000) converting historical depth data representing a view in the historical 3D localization map into historical 2D depth data representing the historical 2D depth image for each of a plurality of different viewpoints along the motion trajectory of the monocular camera relative to the historical 3D localization map, The generation of the approximate value of the depth information is performed based on historical 2D depth data of multiple different perspectives along the motion trajectory of the monocular camera relative to the historical 3D positioning map.

14. The device (500) according to any one of claims 1 to 5, wherein: Activating (604) the monocular camera for positioning based on a determination that a benefit level for activating the monocular camera satisfies the activation rule includes determining that a value of the benefit level satisfies a threshold.

15. The device (500) according to claim 14, further comprising: The threshold is determined ( 1100 ) based on determining a number of feature descriptors in the depth data from the depth sensor that are estimated to be detectable by the monocular camera and determining that the number of feature descriptors meets a threshold number of feature descriptors required to perform localization.

16. The device (500) according to claim 15, wherein The number of feature descriptors in the depth data from the depth sensor that are estimated to be detectable by the monocular camera is limited to include only feature descriptors that meet a feature quality threshold.

17. A method for performing positioning by a device (500) using one or both of a depth sensor (200) and a monocular camera (202) that can be transported together with the device, the method comprising: receiving (600) depth data from the depth sensor; Determining (602) a benefit level of activating the monocular camera for positioning based on the depth data; and Based on a determination that a benefit level of activating the monocular camera satisfies an activation rule, activating (604) the monocular camera for positioning, wherein The benefit level of activating the monocular camera for localization is determined (602) based on: Converting the depth data representing the 3D localization map into 2D depth data representing the 2D depth image; generating an approximation of depth information based on the 2D depth image, the depth information being obtainable from the monocular camera when the monocular camera is activated; as well as A benefit level of activating the monocular camera for positioning is determined based on the approximation of the depth information.

18. The method of claim 17, further comprising performing the operations of any one of claims 2 to 16.

19. A computer program product for performing positioning using one or both of a depth sensor (200) and a monocular camera (202) transportable with a device (500), the computer program product comprising a non-transitory computer-readable medium storing instructions executable by at least one processor of the device to configure the device to: receiving (600) depth data from the depth sensor; Determining (602) a benefit level of activating the monocular camera for positioning based on the depth data; and Based on a determination that a benefit level of activating the monocular camera satisfies an activation rule, activating (604) the monocular camera for positioning, wherein The benefit level of activating the monocular camera for localization is determined (602) based on: Converting the depth data representing the 3D localization map into 2D depth data representing the 2D depth image; generating an approximation of depth information based on the 2D depth image, the depth information being obtainable from the monocular camera when the monocular camera is activated; as well as A benefit level of activating the monocular camera for positioning is determined based on the approximation of the depth information.

20. The computer program product of claim 19, wherein: The instructions further configure the device to perform according to any one of claims 2 to 16.

Images