Method and apparatus for performing loop closure detection

Abstract

A method of performing loop closure detection is described. The method includes: detecting movement of a device having a camera; and adaptively enabling or disabling loop closure detection of the device using a processor of the device based on the detected movement of the device.

Description

Technical Field

[0001] The present disclosure relates to an electronic device for capturing an image, and in particular, to a method and apparatus for performing loop closure detection in the electronic device. Background Art

[0002] Object detection using images captured by an electronic device can have many practical applications. However, object detection can require a lot of computing resources and can take a considerable amount of time. Smart eyewear, such as smart glasses and head mounted display (HMD) wearable devices, have many emerging applications. Useful features that enable such applications for smart eyewear or other tracking devices with cameras (e.g., robotic tracking devices) include object detection, recognition and tracking, and displaying the results on a display. Object detection is also important for simultaneous localization and mapping (SLAM).

[0003] Loop closure detection (LCD) is an important component in SLAM applications. When the SLAM engine runs on an electronic device, the LCD identifies previously visited locations so that global bundle adjustment (BA) can be called in this case. Global BA can be used to optimize and realign camera pose information (of key frames) and 3D map point positions, which accumulates a certain amount of drift during cruising. In traditional implementations of SLAM, the LCD runs at a fixed periodic rate, thus causing unnecessary false alarms when the user is not moving at all or when the user is not interacting with an augmented reality (AR) application. Running the LCD unnecessarily also leads to increased power consumption.

[0004] Therefore, improved LCD operation is needed to improve the efficiency of smart eyewear or other devices with cameras for tracking objects. Summary of the invention

[0005] Technical issues

[0006] Embodiments of the present invention provide a method and apparatus for implementing an LCD to improve the efficiency of smart eyewear or other devices with cameras for tracking objects.

[0007] Solution to the problem

[0008] According to an embodiment of the present disclosure, a method for performing loop detection is disclosed, which includes: identifying movement of a device having a camera; determining whether to perform loop detection based on the identified movement of the device, and enabling loop detection when it is determined to perform loop detection.

[0009] According to an embodiment of the present disclosure, a device for performing loop detection is disclosed. The device includes at least one sensor; and at least one processor coupled to receive data from the at least one sensor, wherein the at least one processor is configured to identify movement of a device having a camera, determine whether to perform loop detection based on the identified movement of the device, and enable loop detection when it is determined to perform loop detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The present disclosure is illustrated in the accompanying drawings, and the same reference numerals indicate corresponding parts in the various drawings throughout the drawings. The embodiments of the present invention will be better understood from the following description with reference to the accompanying drawings, in which:

[0011] Figure 1 is an exemplary block diagram of an electronic device having a camera according to an embodiment of the present disclosure;

[0012] Figure 2 is an exemplary diagram illustrating blocks associated with tracking an object using a camera and loop closure detection according to an embodiment of the present disclosure;

[0013] Figure 3 is an exemplary diagram of a circuit for implementing loop detection according to an embodiment of the present disclosure;

[0014] Figure 4 is a timing diagram illustrating the operation of an adaptive loopback detection circuit according to an embodiment of the present disclosure as compared to a conventional loopback detection circuit;

[0015] Figure 5 is a flow chart illustrating an example of adaptive loop detection according to an embodiment of the present disclosure;

[0016] Figure 6 is an exemplary diagram of a circuit including a pre-filter for implementing adaptive loopback detection according to an embodiment of the present disclosure;

[0017] Figure 7 is another timing diagram illustrating the operation of adaptive loop detection compared to conventional loop detection according to an embodiment of the present disclosure;

[0018] Figure 8 is an exemplary diagram of a circuit for implementing adaptive loopback detection and including a post-filter according to an embodiment of the present disclosure;

[0019] Fig. 9 is a chart showing a comparison of conventional high-rate and low-rate periodic loop detection and adaptive loop detection according to an embodiment of the present disclosure; and

[0020] Fig.10 is a flow chart showing a method for implementing adaptive loop detection according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The embodiments of this article and its various features and advantageous details are explained more fully with reference to the non-limiting embodiments shown in the drawings and described in detail in the following description. The description of known components and processing techniques is omitted so as not to unnecessarily obscure the embodiments of this article. In addition, the various embodiments described herein are not necessarily mutually exclusive, because some embodiments can be combined with one or more other embodiments to form new embodiments. Unless otherwise stated, the term "or" as used herein refers to a non-exclusive or. The examples used herein are intended only to facilitate understanding of the manner in which the embodiments of this article can be practiced, and further enable those skilled in the art to practice the embodiments of this article. Therefore, the examples should not be interpreted as limiting the scope of the embodiments of this article.

[0022] Convention in this field is to describe and illustrate the embodiment with the block that performs one or more functions described.These blocks that can be referred to as unit or module etc. in this article are physically implemented by analog or digital circuit (such as logic gate, integrated circuit, microprocessor, microcontroller, memory circuit, passive electronic component, active electronic component, optical component, hard-wired circuit etc.), and can be driven by firmware and software as needed.For example, the circuit can be implemented in one or more semiconductor chips, or implemented on substrate supports such as printed circuit boards etc.The circuit constituting the block can be implemented by dedicated hardware, or implemented by processor (for example, one or more programmed microprocessors and associated circuits), or by the combination of dedicated hardware of some functions of execution block and processor of other functions of execution block.Without departing from the scope of the present disclosure, each block of the embodiment can be physically divided into two or more interacting and discrete blocks.Similarly, without departing from the scope of the present disclosure, the block of the embodiment can be physically combined into more complex blocks.

[0023] The accompanying drawings are used to help easily understand various technical features, and it should be understood that the embodiments presented herein are not limited by the accompanying drawings. Therefore, the present disclosure should be interpreted as extending to any changes, equivalents and substitutions other than those specifically set forth in the accompanying drawings. Although the terms first, second, etc. can be used herein to describe various elements, these elements should not be limited by these terms. These terms are usually only used to distinguish one element from another element.

[0024] As used herein, terms such as "first", "second", etc. may refer to various elements of various embodiments of the present disclosure, but are not limited to these elements. For example, "first user device" and "second user device" may indicate different user devices, regardless of their order or priority. For example, "first user device" and "second user device" indicate different user devices. For example, without departing from the scope of the present disclosure, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.

[0025] As used herein, the expression "configured to" may be used, for example, as an expression "suitable for", "capable of", "designed to", "suitable for", "manufactured to", or "capable of", depending on the circumstances. The term "configured to" does not necessarily mean only "specially designed to" in hardware. On the contrary, the expression "a device configured to..." may indicate that the device is "capable of" operating with another device or other component. For example, "a processor configured (or set) to perform A, B, and C" may represent a dedicated processor (e.g., an embedded processor) for performing the corresponding operations or a general-purpose processor (e.g., a central processing unit (CPU) or an application processor (AP)) that performs the corresponding operations by executing one or more software programs stored in a memory device.

[0026] The LCD determines when a current scene has been previously seen during the process of a device with a camera traversing a particular area. If the scene has been previously seen, a loop is considered to have been detected, and certain processing can be performed, such as a global bundle adjustment to improve pose estimation based on what is currently detected. The method and apparatus can implement an adaptive LCD, which can be, for example, an event-driven LCD, and solves the problem of when the LCD is triggered based on, for example, user motion, user interaction with an AR application, and a particular AR application being executed. Some embodiments include monitoring user motion when using an AR application to determine when to enable / disable the LCD, monitoring user interaction when using an AR application to determine when to enable / disable the LCD, and monitoring the use of AR application-specific decisions to determine when to enable / disable the LCD.

[0027] Referring now to the drawings and more particularly to Figures 1 to 10 , wherein like reference numerals indicate corresponding features throughout the drawings, there is shown a preferred embodiment.

[0028] First go to Figure 1, a block diagram of an electronic device having multiple cameras is shown. The exemplary electronic device 100 can be any type of device having one or more cameras. The electronic device 100 can include a processor 102 coupled to multiple cameras 104 and 105. Although cameras 104 and 105 are shown, it should be understood that the cameras include image recording devices (such as image sensors) and that the cameras can be independent of each other or can share circuitry. The cameras can be part of an HMD, where one camera can be used to provide a view of a scene and another camera can be used to perform eye tracking (i.e., movement and viewing direction) of the eyes of a user of the HMD. The electronic device 100 can be any type of device for performing position tracking including SLAM, such as a wearable device or other robotic device. Therefore, the processor 102 may include blocks for performing position tracking including SLAM, and reference will be made to Figure 2 Describes the configuration of the block in detail.

[0029] Processor 102 is an integrated electronic circuit, such as, for example, an ARM processor, an X86 processor, a MIPS processor, a graphics processing unit (GPU), a general-purpose GPU, or any other processor configured to execute instructions stored in a memory. Processor 102 may be implemented in one or more processing devices, wherein the processors may be different. For example, an electronic device may include, for example, a central processing unit (CPU) and a GPU.

[0030] The processor 102 may be coupled to a display 106 for displaying information to a user. The processor 102 may also be coupled to a memory 108 that allows for the storage of information related to data or information related to achieving a goal. The memory 108 may be implemented as part of the processor 102 or may be implemented outside of any cache memory of the processor, as is well known. The memory 108 may include any type of memory, such as a solid state drive (SSD), flash memory, read-only memory (ROM), or any other memory element that provides long-term storage, wherein the memory may be any type of internal memory of the electronic device or an external memory accessible to the electronic device.

[0031] A user interface 110 is also provided to enable a user to input and receive data. Some activity tracking may require manual input by the user. The user interface may include a touch screen user interface commonly used on a portable communication device, and other input / output (I / O) elements such as a speaker and a microphone. The user interface may also include a device for inputting or outputting data, which may be attached to the mobile device via an electrical connector or via a wireless connection such as a Bluetooth or near field communication (NFC) connection.

[0032] The processor 102 may also be coupled to other elements that receive input data or provide data, including various sensors 111, an inertial measurement unit (IMU) 112, and a global positioning system (GPS) device 113 for activity tracking. For example, the inertial measurement unit (IMU) 112 may provide various information related to the motion or orientation of the device, while the GPS 113 provides location information associated with the device. As examples, sensors that may be part of or coupled to the mobile device may include a light intensity (e.g., ambient light or UV light) sensor, a proximity sensor, an ambient temperature sensor, a humidity sensor, a heart rate detection sensor, a galvanic skin response sensor, a skin temperature sensor, a barometer, a speedometer, an altimeter, a magnetometer, a Hall sensor, a gyroscope, a WiFi transceiver, or any other sensor that may provide information to the mobile device. The processor 102 may receive input data through an input / output (I / O) port 114 or a transceiver 116 coupled to an antenna 118.

[0033] Now go to Figure 2 , an exemplary diagram illustrates blocks associated with a processor 200 for performing position tracking including SLAM. That is, blocks associated with the processor 200 perform capturing images and implementing loop detection associated with tracking operations. The processor 200 includes an initialization block 202, the output of which is coupled to a tracking block 204. The initialization block 202 initializes the device to perform operations such as mapping operations performed by a mapping block 206 by extracting, for example, features tracked during a tracking phase. After a certain number of frames are written to memory and the extracted features can be tracked, tracking operations such as SLAM tracking processing can be performed by the tracking block 204.

[0034] As part of the mapping operation, repositioning including the LCD can be performed. Repositioning is usually performed when the drift of the device performing tracking exceeds a certain value or the feature is lost (such as when the lights are off or there is an occlusion). The LCD enables the camera to identify objects or positions when crossing the path, and can be used as part of the repositioning by the repositioning block 208. The LCD block 210 is capable of adaptive loop detection, wherein loop detection can be enabled or disabled according to certain criteria associated with the user of a specific AR application, as described in more detail below. The LCD determines when the current scene has been seen before in the process of crossing the specific area. If the scene has been seen before, it is considered that a loop has been detected, and certain processing (such as global bundle adjustment) can be performed to improve the pose estimation. According to some embodiments, adaptively disabling or enabling loop detection of the device may include disabling loop detection based on the mode of the device. For example, a mode can be a specific AR application, wherein loop detection can be based on input from the application layer of the AR application, such as environmental history and / or location settings associated with the operation of the AR application.

[0035] Now go to Figure 3 , shows an exemplary diagram of a circuit 300 for implementing loop detection, which can be implemented in the loop detection block 208. The circuit 300 receives an LCD enable signal, which can be responsive to a specific application running on the eyewear that requires LCD operation to adaptively enable LCD operation. Image data from a camera of a device (e.g., an eyewear or other device for tracking images) is provided to a processing circuit 302. The processing circuit 302 includes a feature extraction block 306, where the extracted features are matched in a feature matching block 308. According to one embodiment, the extracted features are compared with values ​​generated by a memory 310 having a vocabulary 312 and a database 314. As will be described in more detail below, the memory 310 may include pre-stored images with certain features, or images with features added as the device traverses different paths. The match determination block 316 will generate an output signal based on the results of the feature matching process. As will be described in more detail below, each feature stored in the memory is compared with a feature of the image data to determine a match.

[0036] If the features compared in the feature matching block 308 are determined to be matched by the match determination block 316, a match indication is provided to a geometric verification block 318, which performs a geometric verification of the compared features. More specifically, while the match comparison may be performed with little spatial context, another verification may be performed from another perspective (such as from a different angle) to determine whether the objects match. If it is determined that both a feature match and a geometric verification exist, a detected loop signal is generated.

[0037] However, if the feature match or geometric verification fails, as indicated by the outputs generated by the match determination block 316 and the geometric verification determination block 320, respectively, the control circuit 322 enables writing to the memory so that the new features can be stored. According to one embodiment, the data stored in the memory is sequentially accessed and matched with the image data to determine whether the camera generating the image data is in the loop (i.e., recognizes the previously viewed features), in which case an LCD can be implemented. According to another embodiment, the image in the memory (or the portion of the image including the features) can be compared with the input data image. For example, assume that there are N images, each of which is encoded as M feature vectors in a vocabulary. When the camera captures the input image frame, the image features (usually M) are extracted from it and matched with the M features of each of the N images. Therefore, for each of the N images, there will be N memory read enable signals, which are generated to produce M feature vectors for each image. These feature vectors are then compared with the M feature vectors of the input image in the feature matching stage.

[0038] More specifically, the output (e.g., logic 1) of the match determination block 316 and the geometry verification determination block 320 is provided to an OR gate 324, which indicates that there is no match during feature point matching, or that the geometry verification of the matched feature point has failed. The output (e.g., logic 1) of the OR gate 324 may be generated to indicate that a write should be performed, wherein a memory write enable (MemWrEnable) and a keyframe (or a relevant portion of a keyframe including features of the keyframe) are provided to the memory. According to one embodiment, the memory may include a vocabulary 312 and a database 314, the vocabulary 312 may be an offline portion including predetermined features, and the database 314 may compare new features of the image data added to the memory. It should be noted that Figure 3 One example of a circuit for implementing an LCD is provided, but an apparatus and method for implementing an LCD may be implemented with any type of circuit for implementing an LCD.

[0039] Now go to Figure 4 The timing diagram shows the operation of the adaptive loop detection compared to the conventional loop detection circuit. Figure 4 As shown, the conventional device that periodically operates the LCD not only operates the LCD when no event occurs, but also may miss the event. Figure 4When the periodic timing shown is run, event B and event D will be missed, and the LCD will run unnecessarily between event A and event B, and run twice between event C and event D. That is, the LCD in the conventional device runs periodically and is unaware of user events that may not be periodic. As a result, events are missed and unnecessary power is required. In contrast, the LCD can be used Figure 3 The circuit implements an adaptive LCD by adaptively controlling the LCD enable signal, for example, adaptively operating the LCD based on the detection of certain events. For example, the LCD may be operated for event A, event B, and event D based on the detection of certain events, but may not be operated for event C and event E. That is, although event C and event E are detected, the LCD is intentionally not operated for these events. The drift is shown by the dotted line, where the repositioning process occurs after the accumulated drift reaches a certain point.

[0040] Therefore, if Figure 4 As shown, the adaptive LCD will not perform LCD when no event is detected, and may not perform LCD even when an event is detected. For example, LCD may or may not be performed in response to detecting a certain event, depending on the type of motion, other actions of the user, or the application being run (e.g., a certain type of AR application), as will be described in more detail below. Figure 5 Additional details related to the adaptive LCD are provided.

[0041] Now go to Figure 5 , flowchart 500 shows the adaptive operation of loop detection. The AR application can set the mode to implement asynchronous or synchronous LCD. During synchronous mode, the device will adaptively decide whether to run the LCD periodically, such as based on a predetermined count of the clock. During asynchronous mode, the LCD can be executed whenever an event that meets certain predetermined criteria is detected.

[0042] The user motion filter can be used to detect whether the user is moving when using the AR application. If valid user motion is detected, the loop enable signal is turned on and LCD is performed. The user action filter can also be used to detect whether the user has created an anchor point in the AR scene. The loop enable signal can be turned on based on the determination of the anchor points in the scene. According to one embodiment, if all anchor points are deleted, the loop enable signal is turned off. That is, since there is no augmented object to be anchored to the real scene, LCD does not need to be performed.

[0043] Specific reference Figure 5, after running the AR application, the mode of the AR application being used is determined at box 502. If the adaptive synchronization mode is being used, a counter can be run at box 504 to determine when the LCD is running, and the count rate is determined at box 506. If the count is equal to the count rate, it is determined at box 508 whether there has been a user action, in which case the user action is identified at box 510 (where the user action can be associated with creating or deleting an anchor point). If an anchor point has been inserted at box 512, loop enable is asserted (i.e., loop detection is performed), and processing returns to the start of AR application monitoring at box 502.

[0044] If no anchor point has been inserted, then a determination is made at block 514 as to whether there is an anchor point deletion. If not, then loop disabled is asserted, and processing returns to the beginning of block 502. If there is an anchor point deletion, then a determination is made at block 516 as to whether the number of anchor points is equal to zero. If not, then loop enabled is asserted, and processing returns to the beginning of block 502. If yes, then loop disabled is asserted, and processing returns to the beginning of block 502.

[0045] If there is no user action at block 508, user motion is also checked at block 518, where user motion can be used to determine whether the user is traversing a path. If it is determined that there is no user motion, the loop is disabled and processing returns to the beginning of block 502. If there is user motion, user motion is identified at block 520, and a determination is made at block 522 whether there is translation. If not, the loop is disabled and processing returns to the beginning of block 502. If translation is detected at block 522, a determination is made at block 524 whether the translation exceeds a threshold. If not, the loop is disabled and processing returns to the beginning of block 502. If yes, the loop is enabled and processing returns to the beginning of block 502. Reference will be made to the following. Figure 6 Specific examples of user actions and user motions are described in more detail. Figure 5 The flowchart of discloses AR application, but it should be understood that Figure 5 The method described in can be used with any type of application that uses a camera to provide loop detection.

[0046] Now go to Figure 6 , showing a method for implementing loop detection (such as described above in Figure 5 The pre-filter 602 is adapted to generate an LCD enable signal to enable or disable the loop detection circuit, such as Figure 3The prefilter 602 receives various inputs to generate an LCD enable signal that enables or disables LCD operation for use as an adaptive LCD. More specifically, the prefilter 602 receives a filter enable signal that enables prefilter operation and various inputs to adapt the LCD operation, wherein the various inputs are processed to determine whether the LCD enable signal enables or disables LCD operation.

[0047] The application filter block 604, illustrated here with the AR application filter as an example, receives input from the AR application. The output of the AR application filter block is provided to an adaptive LCD filter 610 that generates an LCD enable signal. That is, the need to enable / disable the LCD may be different based on the AR application. For example, for an office AR application, loop detection may need to be enabled, but for a gym AR application, loop detection may not be expected to be enabled. The adaptive LCD allows the application layer of the AR application to determine whether to enable or disable the LCD, and also determines whether the event-driven LCD should run synchronously or asynchronously.

[0048] The user action filter block 606 receives input from the user (such as the user's head movement) to identify objects in the room. The output of the user action filter block is provided to the adaptive LCD filter 610 that generates the LCD enable signal. As an example, an AR application can be used to learn new cooking recipes. If a user opens a recipe application in front of a kitchen stove and turns left, an anchor point in the form of an enhanced recipe book can be created. The adaptive LCD can detect a loop after the user has created an anchor point, even if the user may be stationary. That is, a loop is detected in response to a user action even if there is no user motion.

[0049] For example, a user motion filter block 608, which may be used for an AR application, may receive input from the AR application. The output of the user motion filter block is provided to an adaptive LCD filter 610 which generates an LCD enable signal. As an example, an AR application may track a user walking through a grocery store and stopping at a particular aisle to check out some items. Because the user is stationary, the LCD does not need to be running as there are no potential loops to detect. Conversely, in the case where a user is running an AR application for recipes, the LCD may be running based on the user's head motion even when the user is stationary. The LCD enable signal generated by the pre-filter 602 is used to enable the LCD, which may be used as described above. Figure 3 circuit or some other circuit to achieve this.

[0050] It should also be noted that if a loop is enabled and detected, a global bundle adjustment can be triggered even if LCD is not performed. That is, in some cases, a global bundle adjustment can be performed based on the criteria used to determine whether to enable LCD, as will be referenced below. Figure 8 Described in more detail.

[0051] Now go to Figure 7 , another timing diagram illustrates the operation of adaptive loopback detection. In a high rate periodic LCD of a conventional device, all events are captured, but the LCD often operates when no events occur (i.e., between events A and B and between events B and C), thereby resulting in power loss. In contrast, in a low rate periodic LCD of another conventional device, the LCD operates at a reduced frequency, resulting in reduced power, but event D is missed. However, as shown using an adaptive LCD, the LCD only operates when events occur, thus using power only when necessary without missing events. The adaptive LCD can determine to detect event A-event E based on the determination of the pre-filter 602 as described above or other determinations. For example, as described above with reference to Figure 4 As described above, even when an event is detected, the LCD may not operate.

[0052] Now go to Figure 8 , shows an exemplary diagram of a circuit including a post filter for implementing loop detection. Figure 3 In addition to the loop detection signal generated by circuit 300 as described above, post filter 802 may be similar to Figure 6 The pre-filter 602 receives various inputs to determine whether to perform other functions during tracking operations (e.g., SLAM). More specifically, the post-filter 802 receives a loop detection signal that enables the post-filter and various inputs, wherein the various inputs are processed to determine whether to enable the operation of the functions of the circuit that performs tracking (such as performing global BA).

[0053] The application filter block 804 of the post filter 802 (which may be used, for example, for AR applications) receives input from the AR application. The output of the AR application filter block 804 is provided to an adaptive LCD post filter 810, which generates a detected loop signal that can be used to determine whether to perform a certain function, such as global BA. The need to enable / disable LCD may vary based on the AR application. As described above with respect to Figure 6As described, for an office AR application, it may be desirable to enable loop detection, whereas for a gym AR application, it may not be desirable to enable loop detection. The adaptive LCD allows the application layer of the AR application to make this determination. The user action filter block 806 receives input from the user, wherein the output of the user action filter block is provided to the adaptive LCD filter 810 that generates a detected loop signal, as described above with reference to FIG. Figure 6 Finally, the user motion filter block 808, which may be used, for example, in an AR application, receives input from the AR application, as described above with reference to Figure 6 As described, and can also be used to generate a detected loop signal.

[0054] Thus, a loop detection circuit generated by a loop detection circuit such as circuit 300 can be used to generate a function enable signal that can be used for functions such as global BA. Depending on the SLAM engine, the global BA optimizer may or may not be part of the loop detection. If a loop is detected by loop detection, global bundle adjustment can be enabled / disabled based on user events (such as by post filter 802). Figure 8 The implementation allows enabling / disabling of the LCD as well as controlling the global BA blocks within the SLAM. While the pre-filter 602 will be in a static "always on" state, the post-filter 802 will typically be dynamically "woken up" when a loop is detected.

[0055] Now go to Fig. 9 , the graph shows the comparison between traditional loop detection and adaptive loop detection. Fig. 9 As shown, the adaptive LCD according to the present disclosure not only responds faster than a low-rate periodic LCD, but also requires less power than a high-rate periodic LCD. Use case analysis also determined that the adaptive LCD reduces power consumption and improves operation. Considering an AR application in which a user walks through a grocery store and spends 10 minutes in the store, the user may be stationary for about 2 minutes while staying in different aisles. Assuming that the traditional LCD solution runs at a fixed rate of 3Hz (3 times per second), and the adaptive LCD runs at 3Hz when the user is moving, and runs at the minimum necessary rate of 1Hz (i.e., triggering repositioning) when the user is stationary, the number of detected false loops can be significantly reduced based on user motion, user actions, and AR applications. In addition, power consumption can be significantly reduced because loop detection can run adaptively instead of at a fixed periodic rate.

[0056] Table 1 shows the results of the Performance breakdown of the SLAM engine (ORB-SLAM2) when running on a Galaxy S7 phone. As shown in the table, the last row gives the frames processed by loop detection (LC), the cumulative time consumed by loop detection, and the processing time per frame. The ratio of key frames to total frames is 375 / 2965=0.13. All 375 key frames are used in the loop (LC) thread. When SLAM runs for about 600 seconds, about 10 seconds of which are used by the loop thread, then 10 / 600=0.0167 is the time fraction occupied by LC. The total power consumption of SLAM is 4.5W-4.3W=200mW. Therefore, the information in Table 1 is used to calculate power savings, where the ratio of key frames to total frames is 0.1, and where user motion is considered to adaptively change the rate of LC. If it is assumed that the conventional LCD operates at 3Hz (e.g., 3 times per second) and the adaptive LCD operates at 3Hz if the user is moving, but operates at the minimum necessary rate of 1Hz (to trigger repositioning) if the user is stationary, false alarms are reduced from 360 with the conventional LCD to 120 with the adaptive LCD, and the number of LCD operations is reduced from 1800 with the conventional LCD to 1560 with the adaptive LCD. Computing power can also be reduced by 43 microwatts (μW), and memory power requirements are reduced by 30μW.

[0057]

[0058]

[0059] Table 1

[0060] It should also be noted that adaptive LCD can be implemented in the image-to-image matching framework of loop closure detection as well as other frameworks such as map-to-map matching and image-to-map matching, since the local map will always be available even if the user is stationary or the anchors are removed.

[0061] Now go to Fig.10 , the flowchart shows a method for implementing adaptive loop detection. More specifically, in box 1002, the movement of a device having a camera is identified. The device may be, for example, a device for implementing SL AM. In box 1004, the type of AR application being used is determined. As described above, certain actions or movements of the user may affect whether the LCD is disabled. In box 1006, the user action is analyzed, and in box 1008, the user movement is analyzed. For example, user actions associated with the movement of the head are used to identify actions associated with a particular activity. User movements can also be analyzed to determine the path or other changes in the user's position. In box 1010, the LCD can be adaptively enabled or disabled using the processor of the device based on at least one of the type of AR application, user action, or user movement.

[0062] That is, the device determines whether to perform loop detection based on the identified movement, and when it is determined to perform the loop detection, the device enables the loop detection.

[0063] Fig.10 The various elements of the method can be used as described Figure 1-Figure 8 Although specific elements of the method are described, it should be understood that the method can be implemented according to Figure 1-Figure 8 The disclosed content can be used to implement other elements of the method or other details related to the elements.

[0064] Thus, it will be appreciated that a novel circuit and method for implementing an adaptive LCD has been described. Those skilled in the art will appreciate that there will be many alternatives and equivalents that incorporate the disclosed invention. Therefore, the present invention is not limited by the foregoing embodiments, but only by the appended claims.

Claims

1. A method for performing loop closure detection, the method comprising: Identifying movement of a device having a camera, wherein the step of identifying movement of the device having a camera comprises: identifying an action of a user of the device and / or identifying motion of a user of the device; analyzing user actions and / or user movements; determining whether the identified movement of the device is associated with the action of the user of the device and / or the movement of the user of the device based on the analyzed action of the user and / or the movement of the user of the device; and Based on whether the identified movement of the device is associated with the user's action and / or the user's motion, loop closure detection of the device is adaptively enabled or disabled.

2. The method according to claim 1, wherein: The step of identifying the actions of a user of the device includes tracking the user's actions using the device.

3. The method according to claim 2, wherein: The steps to adaptively enable or disable loop detection include: In response to user interaction with the scene, loop closure detection of the device is disabled based on the detection of the anchor point.

4. The method according to claim 1, wherein: The step of identifying movement of a user of the device includes tracking movement of a user of the head mounted device.

5. The method according to claim 1, wherein: The steps to adaptively enable or disable loop detection include: Disabling loop closure detection of the head mounted device based on at least one of translation or rotation of the head mounted device.

6. The method according to claim 1, wherein: The steps to adaptively enable or disable loop detection include: Disable loopback detection based on the device mode.

7. The method according to claim 1, wherein: The step of identifying movement of the device having the camera includes synchronously detecting movement of the device.

8. The method according to claim 1, wherein: The step of identifying movement of the device having the camera includes asynchronously detecting movement of the device.

9. The method according to claim 1, further comprising: A pre-filter is implemented wherein the step of adaptively enabling or disabling loop closure detection is based on an augmented reality (AR) application.

10. The method according to claim 1, further comprising: Implements a post filter where a global bundle adjustment is performed when a loop is detected.

11. A device for performing loop detection, the device comprising: at least one sensor; as well as at least one processor coupled to receive data from the at least one sensor, Wherein, the at least one processor is configured to: Identifying movement of a device having a camera, wherein the operation of identifying movement of the device having a camera includes: identifying an action of a user of the device and / or identifying a motion of a user of the device; analyzing the user's actions and / or the user's movements, determining whether the identified movement of the device is associated with the action of the user of the device and / or the movement of the user of the device based on the analyzed action of the user and / or the movement of the user of the device, and Based on whether the identified movement of the device is associated with the user's action and / or the user's motion, loop closure detection of the device is adaptively enabled or disabled.

12. The device according to claim 11, wherein The operation of identifying the actions of a user of the device includes tracking the user's actions using the device.

13. The device according to claim 11, wherein: The at least one processor is further configured to disable loop closure detection of the device based on detection of the anchor point in response to user interaction with the scene.

14. The device according to claim 11, wherein: The at least one processor is also configured to disable loop closure detection of the head mounted device based on at least one of translation or rotation of the head mounted device.

15. The apparatus according to claim 11, wherein: The at least one processor identifies movement of the device synchronously or asynchronously.

Images