Loop closure detection method, apparatus and system

Abstract

The present disclosure provides a method, an apparatus and a system for detecting a loop closure in an environment, the method executed by a processor, and comprising: identifying a text entity comprising a text content in the environment from an image frame associated with an image sensor; estimating a pose of the text entity in a current local coordinate frame associated with a depth sensor based on an estimated pose of the text entity in the image frame and an extrinsic parameter between the depth and image sensors; retrieving from a database using the text content a candidate pose of the text entity in a previous local coordinate frame; calculating a relative pose constraint relating to a transformation between the current and previous local coordinate frames based on the pose and the candidate pose of the text entity; and detecting the loop closure based on the relative pose constraint.

Description

LOOP CLOSURE DETECTION METHOD, APPARATUS AND SYSTEMCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of priority of Singapore patent application no.10202402114V, filed on 17 July 2024, the contents of which being hereby incorporated by reference in its entirety for all purposes.TECHNICAL FIELD

[0002] The present disclosure generally relates to a method, an apparatus and a system for detecting a loop closure in an environment.BACKGROUND

[0003] In recent years, depth sensor-based odometry, such as Light Detection and Ranging (LiDAR)-inertial odometry (LIO), has become a pillar stone in the field of mobile robotics. Notably, the latest LiDAR technology offers a wider field of view, longer range and improved accuracy. In terms of continuous localization, LiDAR-based methods have decisively demonstrated better accuracy and robustness over conventional visual Simultaneous Localization and Mapping (SLAM), eliminating the need for visual SLAM in most applications, as reflected in the Hilti SLAM challenge. However, LiDAR-based loop closure detection (LCD) methods, such as Stable Triangle Descriptor (STD), Scan Context (SC), and Intensity’ Scan Context (TSC), using local coordinate frames such as LiDAR frames, often struggle to find accurate matches in degenerative and repetitive environments.

[0004] While visual-based LCD, using camera or image frames, offers larger feature descriptor dimensions, it requires more computational power to generate and match dense temporal or semantic descriptors. Additionally, despite dense feature representation, visual- based LCD failures still occur. The LCD challenge is further exacerbated when dealing with featureless, degenerative, and repetitive (FDR) environments with computational constraints. There exists a gap for an efficient, simple, and intuitive LCD solution for LIO in FDR environments, mirroring human-like processes.[1)005] There is thus an object for an improved loop detection method, apparatus and system to address the challenges in mobile robotics in detecting a loop closure in an environment, especially in an FDR environment with improved accuracy and computational efficiency.

[0006] Furthermore, other desirable features and characteristics will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and this background of the disclosure.SUMMARY

[0007] The technical solution seeks to provide a method, an apparatus and a system for detecting a loop closure in an environment that leverages scene text cues, i.e.. detected text entity, in FDR scenes to achieve human-cognition-inspired loop closure detection. In particular, depth sensor scans are projected on the text cue regions in image frames to obtain an accurate relative pose of text entities. The 3D text cues may be stored in a database to support efficient and accurate retrieval of historical observations and candidate poses by using a graph- theoretic scheme. Upon each successful association between observations of the same text entity or content in the current and previous local coordinate frames, a loop closure factor of relative pose priors between poses in the current and previous local coordinate frames that observe the same text entities is then introduced to a pose graph optimizer to detect a loop closure and generate refined poses and map. The pose graph optimizer is subsequently integrated with a front-end LIO to enhance the robustness and accuracy of state estimation within typical FDR environments.

[0008] Stated differently, text entity representation, interpretation, estimation, and management methods are introduced by fusing depth sensor and image data to support efficient loop closure retrieval and alignment. Additionally, by employing a graph-theoretic method to identify the inlier set among ambiguous associations of non-unique text entities, an association scheme to associate current observations and previous observations of the same text entity or contents is also introduced to create loop closures and improve state estimation accuracy.

[0009] In a first aspect, the present disclosure provides a method for detecting a loop closure in an environment comprising: identifying, by a processor of an apparatus, a text entity comprising a text content in the environment from an image frame associated with an image sensor: estimating a pose of the text entity comprising the text content in a local coordinate frame associated with a depth sensor based on an estimated pose of the text entity in the imageframe and an extrinsic parameter between the depth sensor and the image sensor; retrieving, by the processor of the apparatus from a database accessible by the apparatus using the text content, a candidate pose of the text entity in a previous local coordinate frame associated with the depth sensor; calculating, by the processor of the apparatus, a relative pose constraint relating to a transformation between the local coordinate frame and the previous local coordinate frame based on the pose and the candidate pose of the text entity; and delecting, by the processor of the apparatus, the loop closure in the environment based on the relative pose constraint.

[0010] In a second aspect, the present disclosure provides an apparatus for detecting a loop closure in an environment, comprising: at least one processor configured to: identify a text entity comprising a text content in the environment from an image frame associated with an image sensor; estimate a pose of the text entity compri sing the text content in a local coordinate frame associated with a depth sensor based on an estimated pose of the text entity in the image frame and an extrinsic parameter between the depth sensor and the image sensor; retrieve, from a database using the text content, a candidate pose of the text entity in a previous local coordinate frame associated with the depth sensor; calculate a relative pose constraint relating to a transformation between the local coordinate frame and the previous local coordinate frame based on the pose and the candidate pose of the text entity; and detect the loop closure in the environment based on the relative pose constraint.

[0011] In a third aspect, the present disclosure provides a system for detecting a loop closure in an environment, comprising: the apparatus according to the second aspect; an image sensor configured to generate the image frame: and a depth sensor configured to generate the local coordinate frame associated with the depth sensor.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In the drawings, like reference characters generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments are described with reference to the following drawings, in which:FIGS. 1 A-1D show four exemplary pictures illustrating featureless, degenerative, and repetitive (FDR) scenes;FIG. 2 shows a block diagram illustrating a system comprising an apparatus for detecting a loop closure in an environment according to various embodiments of the present disclosure;FIG. 3 shows a flow' chart illustrating a method for detecting a loop closure in an environment according to various embodiments of the present disclosure;FIG. 4 shows a flow diagram illustrating an overview of a process executed by various components of an apparatus for detecting a loop closure in an environment according an embodiment of the present disclosure;FIG. 5 shows a picture illustrating a text entity representation of “Exit" signs in an image frame and three axes of the text entities identified according to an embodiment of the present disclosure;FIG. 6 shows a map illustrating putative associations between two local text entity maps (LTEMs) in an environment according to an embodiment of the present disclosure:FIG. 7 shows a graph network G representing the putative associations of FIG. 6 and a subgraph G* representing putative associations that are found to be geometrically consistent according to an embodiment:FIG. 8 shows graphs illustrating the apparatus’s trajectories in different sequences corresponding to different environments and pictures illustrating scenes of corridors in some sequences according to an embodiment of the present disclosure;FIG. 9 shows two graphs illustrating loop closure detection results of an existing method and a proposed method in a degenerative environment according to various embodiments of the present disclosure, respectively;FIG. 10 shows a map illustrating trajectories generated using existing methods and a proposed method in a multi-floor building environment according to an embodiment of the present disclosure; andFIG. 11 shows three graphs illustrating translation error distribution of various methods for detecting a loop closure in an indoor long corridor environment, a semi -outdoor corridor environment, and a cross floor building environment, respectively.DETAILED DESCRIPTION

[60013] Embodiments described below in context of the apparatus are analogously valid for the respective methods, and vice versa. Furthermore, it will be understood that theembodiments described below may be combined, for example, a part of one embodiment may be combined with a part of another embodiment.

[0014] It should be understood that the terms “on”, “over”, “top”, “bottom”, “down”, “side”, “back”, “left”, “right”, “front”, “lateral”, “side”, “up”, “down” etc., when used in the following description are used for convenience and to aid understanding of relative positions or directions, and not intended to limit the orientation of any device, or structure or any part of any device or structure, in addition, the singular terms “a”, “an”, and “the” include plural references unless tiie context indicates otherwise. Similarly, the word “or” is intended to include “and” unless the context clearly indicates otherwise.

[0015] As described, depth sensor-based odometry has become a pillar stone in the field of mobile robotics. However, loop closure detection (LCD) methods using local coordinate frames generated by depth sensors such as Light Detection and Ranging (LiDAR) frames often struggle to find accurate matches in degenerative and repetitive environments. While visualbased LCD using image frames offers larger feature descriptor dimensions, it requires more computational power to generate and match dense temporal or semantic descriptors. Additionally, despite dense feature representation, visual-based LCD failures still occur The I, CD challenge is further exacerbated when dealing with featureless, degenerative, and repetitive (FDR) environments with computational constraints.

[0016] Visual and LiDAR fusion using both local coordinate frames and image frames for global localization has a common problem in perception tasks. Traditionally, global localization can be achieved by visual odometry or the Simultaneous Localization and Mapping (SLAM) approach. However, visual approaches often lack robustness when dealing with featureless areas, lighting changes, and distant objects. Often, visual factors have to be complemented by other factors, such as Inertial Measurement Unit (JMU) or Ultra-Wideband (UWB), to increase efficiency and robustness. Recently, LiDAR approaches have become the mainstream for front-end odometry estimation as Light Detection and Ranging (LiDAR)- ineitial odometry (LIO) consistently yields relatively better results when compared to any realtime method that fuses with vision. With the emergence of new low-cost LiDAR-based solutions, the visual-based approaches have seen a significant decline in popularity for mobile robotics applications. Another way to achieve global localization is by LCD. Traditionally, the visual-based method was mainstream with handcrafted features such as scale-invariant feature transform (SIFT), binary robust independent elementary features (BRIEF), and Oriented FAST and Rotated BRIEF (ORB), but required significant computational power to match imagesequences. Bag of Words 2 (BoW2) may improve real-time LCD using binary visual word models based on BRIEF. Moreover, learning based methods are commonly used in LCD due to their robustness in handling viewpoint and appearance variations. However, due to limited geometric understanding, FDR environments, and changes in illumination, visual -based LCD is still far from perfect.

[0017] Recently, LiDAR based LCD has been extensively explored in field robotics for precise geometric measurements and illumination invariance. The scan context (SC) series is a commonly used method for LiDAR-based LCD, whose main idea is to employ projection and space partitioning to encode the entire point cloud. The method has also been improved by integrating intensity and semantic information. However, this series of methods cannot estimate the full Special Euclidean group in 3D (SE3) relative pose between candidate frames and rely on odometry poses to discard false loops, making them vulnerable to significant odometry drift.

[0018] A triangle -based descriptor by aggregating local point features has also been proposed, using the length of each side as a key in a hash table for finding loop closure candidates through a voting scheme. An improved technique involving Binary Triangle Combined (BTC) combines Stable Triangle Descriptor (STD) with binary patterns to improve speed and viewpoint invariance. However, such an improved technique involving BTC is currently in early access and not yet available for open-source verification. Despite all the improvements, these methods still struggle with FDR scenes, where similar space shape, intensity, and semantics can cause ambiguity in loop closure.

[0019] Integration of ORB-SLAM with text features has been explored to enhance robot localization in commercial plazas, utilizing large advertisement features for improved accuracy in both day and night settings. This approach is effective in commercial plazas with rich text information but also becomes less useful in FDR environments, like narrow, degenerative corridors where front-end ORB-SLAM tends to drift and fail dramatically. This limitation significantly restricts the operational area of mobile robots. Additionally, it fails to differentiate between unique text and multiple texts with identical content, leaving their association ambiguous.

[0020] All these challenges in the conventional techniques highlight the need for innovative loop detection solutions that facilitate robust and effective global loop closure.

[0029] The following paragraphs describe a method, an apparatus, and a system for detecting a ioop closure in an environment that leverages scene text cues, i.e., detected text entity, in FDR scenes to achieve human-cognition-inspired loop closure detection,

[0022] As described herein, the term “loop closure” refers to the process of recognizing that an apparatus lias returned to a previously visited location in an environment, despite the accumulation of positional drift over time during operation As a robot navigates through an environment, small errors in motion estimation can compound, causing its estimated trajectory and the constructed map to gradually diverge from reality. When loop closure is successfully detected, such detection and recognition may be used to correct accumulated errors by aligning the current observation with earlier observation, thereby improving the overall consistency and accuracy of the estimated map and tlse robot’s trajectory. Loop closure detection typically involves comparing sensor data (e g , LiDAR scans, visual images, or feature descriptors) with previously collected data to identify correspondences indicative of revisiting a known place.

[0023] The term “environment” refers to a physical space or surroundings in which a device moves, operates, navigates, and collects sensor data. The environment may include indoor and / or outdoor settings and can comprise various structural, natural, or man-made elements such as walls, floors, obstacles, signage, furniture, roadways, buildings, vegetation or a combination thereof. In the context of loop closure detection, the environment selves as the spatial domain in which sensor observations, such as LiDAR scans, images, or textual features, are acquired and matched to detect revisited locations. The characteristics of the environment, including its geometry, texture, or semantic content, influence the reliability and distinctiveness of features used for localization and mapping.

[0024] As described herein, the term “text entity” refers to a data structure or object that encapsulates or represents a text content identified and extracted from the environment, such as alphanumeric characters, words, phrases, signage, labels, or other forms of human-readable text. A text entity may include associated metadata, such as spatial location, orientation, timestamp, confidence score, or semantic classification. According to the present disclosure, the text content of a text entity may serve as a textual cue or semantic landmark for loop closure detection by enabling a mobile robot to recognize that it has returned to a previously visited location based on the presence, similarity, or match of textual information. This allows for enhanced localization and map correction in environments where text-bearing objects (e.g., room numbers, street signs, or product labels) provide consistent and distinguishable features.

[0025] The term “depth sensor” refers to a sensor device that measures the distance from the sensor to surfaces or objects in the environment, thereby generating spatial depth information. Examples of depth sensors include, but are not limited to, LiDAR ( Light Detection and Ranging) sensors, structured light sensors, time -of flight (ToF) cameras, and stereo vision systems. A depth sensor may produce a depth map or a point cloud, where each measurement is defined relative to a local coordinate frame associated with the depth sensor. Although various embodiments are illustrated using LiDAR sensors, it is appreciated that similar methods and techniques may be applicable to other apparatus wi th other depth sensors such as ToF cameras and stereo vision systems.[fi(M)26] The term “local coordinate frame” refers to a sensor-centric reference frame in which depth measurements are defined. This frame typically originates at the depth sensors optical or emission center, with axes aligned according to the physical orientation of the depth sensor. In this frame, each depth point or measurement is represented as a 3D coordinate relative to the sensor’s position and orientation at the time of data acquisition. The local coordinate frame facilitates spatial transformations, sensor fusion, map budding, and pose estimation and mapping operations in robotics systems.

[0027] The term “image sensor” refers to a device that captures visual information from the environment in the form of digital images. In some embodiments, the image sensor may comprise a camera, such as a monocular, stereo, or red-green-blue (RGB) camera, and may operate in the visible spectrum or other wavelengths (e.g., infrared). The image data obtained from the image sensor may be used for perception tasks such as object detection, scene understanding, or visual localization, and may be represented in a structured format such as an image frame or pixel array. It may be contemplated that the image sensor may include video recorders or camcorders.

[0028] The term “image frame” as described herein refers to a two-dimensional array of pixel values, i.e., image data, captured by an image sensor at a timestamp, i.e., a given moment in time. Each image frame represents a snapshot of the visual scene and may include color (RGB ), grayscale, or other types of pixel data depending on the sensor modality. In computer vision or robotic applications, an image frame may also be associated with intrinsic and extrinsic camera parameters, enabling the mapping between pixel coordinates and corresponding spatial features in the environment. Where a camera is used as the image sensor, the term “image frame” may be used interchangeably with and refer to the term “camera frame” accordingly.

[0029] As described herein, the term “pose'’ refers to the positiors and orientation of a sensor or robotic platform in a given coordinate frame, typically expressed as a rigid body transformation in three-dimensional space. Stated differently, a pose may represent the spatial configuration of a depth sensor at a specific timestamp and may be defined by a translation component (e.g., x, y, z coordinates) and a rotation component (e.g., roll, pitch, yaw angles or a rotation matrix) A pose may be represented mathematically as a transformation in SE(3), allowing the mapping of points or measurements between different local coordinate frames. Accurate estimation of poses over time enables the construction of a consistent map and supports recognition of previously visited locations during loop closure.

[0030] As described herein, the term “relative pose constraint” refers to a spatial relationship estimated between two different poses, typically corresponding to two separate locations or two different timestamps, based on matching observations, such as depth sensor scans, other environmental features, or according to trie present disclosure, text entity or text content. A relative pose constraint is derived when a mobile robot recognizes that it has revisited a previously observed area, and a transformation is computed that relates the current sensor pose to a prior / candidate pose selected from previous observations or local coordinate frames. This transformation typically includes a relative rotation and translation and is used to impose a constraint within a pose graph or optimization framework. By incorporating such constraints, accumulated localization errors can be corrected and the global consistency of the estimated trajectory and a world map representing the environment can be improved.

[0031] As described herein, a “world frame” refers to a fixed, global reference coordinate system in which the positions and orientations (poses) of a mobile robot, Lil) AR sensor, and environmental features are expressed. The world frame may provide a common basis for representing all sensor measurements and estimated trajectories in an environment, enabling consistent integration of data collected at different times or from different locations in the environment. The world frame ts typically established at the start of mapping or a designated origin point and serves as the coordinate space in which the accumulated map and corrected poses are optimized during loop closure and global trajectory refinement.

[80032] FIGS. I A ID show four exemplary pictures illustrating featureless, degenerative, and repetitive (FDR) scenes. FDR scenes are common and humans naturally navigate using wayfinding signs, numbers, nameplates, and language signage, for example, the numbers, alphabets, or alphanumeric characters “10-262” in FIG. 1A, “9312” in FIG. IB, “AL-0102” in FIG. 1C and “Dr. Xie Lihua”, “Professor”, “S2 62c - 94” in FIG. ID in an environment toidentify their locations and navigate their directions. According to various embodiments below, text entities detected in an environment from image frames associated with an image sensor are utilized, with depth sensor data, to detect loop closure.

[0033] FIG. 2 shows a block diagram illustrating a system 200 comprising an apparatus 202 for detecting a ioop closure in an environment according to various embodiments of the present disclosure. The apparatus 202 may be a robot, a laptop computer, a desktop computer, a tablet computer, an automobile computer, a smartphone, a personal digital assistant, a server, or other electronic devices capable of running computer applications. In some embodiments, the apparatus 202 includes a processor 204. In some embodiments, the apparatus 202 may further include a memory 206. Although FIG. 2 shows one processor and one memory, it is appreciated that the apparatus 202 may include more than one processor and / or more than one memory configured to collectively perform a designated function or individually perform separate functions. Reference to the processor 204 hereinafter refers one or more processor of the apparatus 202 or at least one processor, and reference to the memory 206 hereinafter refers to at least one memory.

[0034] The system 200 may include a depth sensor 208 and an image sensor 210 which are in wired or wireless data communication with the apparatus 202 configured to provide depth data such as local coordinate frames and image data such as image frames to the apparatus 202, respectively. It may be appreciated that the image sensor 210 and the depth sensor 208 may be a part of the apparatus 202 or housed together in a common housing or other assembly, forming the system 200 to perform operations described in FIG. 3 for detecting a loop closure in an environment. One or more servers (not shown) may be configured to provide (e.g., transmit) real-time data to the apparatus 202.

[0035] The processor 204 can execute instructions, for example, tn generate output data based on data inputs such as image data and the depth data provided by the image sensor 210 and the depth sensor 208, respectively. The instructions can include programs, codes, scripts, or other types of data stored in memory (e.g., memory 206). Additionally, or alternatively, the instructions can be encoded as pre -programmed or re -programmable logic circuits, logic gates, or other types of hardware or firmware components or modules. The processor 204 may include, a multicore processor having a plurality of cores, and each such core may have an independent power domain and can be configured to enter and exit different operating or performance states based on workload. Additionally, or alternatively, the processor 204 may be or may include, a general-purpose microprocessor, as a specialized co-processor or anothertype of data processing apparatus. In some cases, the processor 204 performs high-level operation of the apparatus 202. For example, the processor 204 may be configured to execute or interpret software, scripts, programs, functions, executables, or other instructions stored in the memory 206.

[0036] The example memory 206 may include computer-readable storage media, for example, a volatile memory device, a non volatile memory device, or both. The memory 206 may include one or more read-only memory devices, random-access memory devices, buffer memory devices, or a combination of these and other types of memory devices. In some instances, one or more components of the memory can be integrated or otherwise associated with another component of the apparatus 202. The memory 206 may store instructions that are executable by the processor 204. In some embodiments, the memory' 206 may store instructions for an operating system and for application programs.

[0037] In some embodiments, the apparatus 202 may not include the memory 206. Instead the processor 204 may be configured to execute instructions stored in a database or an external source accessible by the processor 204, and cause the apparatus 202 to operate, perform its designated functions and generate output data.

[0038] The apparatus 202 may be configured to operate in a wireless, wired, or cloud network environment (or a combination thereof). In some embodiments, the apparatus 202 can access the network using a network interface(s). The network interface(s) can include one or more adapters, modems, connectors, sockets, terminals, ports, slots, and the like. The wireless network that the apparatus 202 accesses may operate, for example, according to a wireless network standard or another type of wireless communication protocol. For example, the wireless network may be configured to operate as a Wireless Local Area Network (WLAN), a Personal Area Network (PAN), a metropolitan area network (MAN), or another type of wireless network. Examples of WLANs include networks configured to operate according to one or more of the 802.11 f amily of standards developed by IEEE (e.g., Wi-Fi networks), and others. Examples of PANs include networks that operate according to short-range communication standards (e.g., BLUETOOTH®, Near Field Communication (NFC), ZigBee), millimeter wave communications, and others. The wired network that the apparatus 202 accesses may, for example, include Ethernet, SONET, circuit-switched networks (e.g., using components such as SS7, cable, and the like), and others.

[0039] The processor 204 may be used to execute computer program code or instructions, stored locally in the example memory 206, for example, within the computer-readable storagemedia in the example memory 206, or stored in a database or an external source accessible by the processor 204, to perform operations described in FIG. 3 for detecting a loop closure in an environment according to various embodiments of the present disclosure. The processor 204 is configured to receive image data and depth data from the image sensor 210 and the depth sensor 208, respectively. Alternatively, or additionally, the image data and depth data generated and provided by the image sensor 210 and the depth sensor 208 may be stored in the memory' 206 or in a database accessible by the processor 204, and the processor 204 is configured to retrieve the image data and the depth data from the memory; 206 or the database accordingly to perform operations described in FIG. 3 for detecting a loop closure in an environment.

[0040] FfO. 3 shows a flow chart illustrating a method 300 for detecting a loop closure in an environment according to various embodiments of the present disclosure. The steps 302, 304, 306, 308, 310 may be executed by a processor (e.g., processor 204 of apparatus 202 of system 200). The processor (e.g., a text entity identification module of the processor) may be configured to, in step 302, identify a text entity comprising a text content in the environment from an image frame associated with an image sensor (e.g., image sensor 210). The processor (e.g., a pose estimation module of the processor) may be configured to, in step 304, estimating a pose of the text entity comprising the text content in a local coordinate frame associated with a depth sensor (e.g., depth sensor 208) based on an estimated pose of the text entity in the image frame and an extrinsic parameter between the depth sensor and the image sensor. The processor (e.g., a candidate pose retrieval module of the processor) may be configured to, in step 306, retrieving, from a database accessible by the processor using the text content, a candidate pose of the text entity in a previous local coordinate frame associated with the depth sensor. The processor (e.g., a relative pose constraint calculation module) may be configured to, in step 308, calculate a relative pose constraint relating to a transformation between the local coordinate frame and the previous local coordinate frame based on the pose and the candidate pose of the text entity. The processor (e.g., a loop closure detection module of the processor) may be configured to, in step 310, detect the loop closure in the environment based on the relative pose constraint.

[0041] In some embodiments, the processor (e.g., frame aggregation module of the processor) may be configured to combine respective depth measurements in one or more local coordinate frames generated within a time period from a timestamp at which the image frame is generated into the local coordinate frame, the local coordinate frame comprising the respective depth measurement.

[0042] Tn some embodiments, the image frame is generated at a first timestamp, and the local coordinate frame is generated at a second timestamp immediately before the first timestamp, and, in step 304. the processor (e.g., the pose estimation module of the processor) may be configured to include computing an interpolated pose between (i) a first pose of the depth sensor in a world frame at the second timestamp immediately before the first timestamp and (ii) a second pose of the depth sensor in the world frame at a third timestamp immediate after the first timestamp, and estimate the pose of the text entity in the local coordinate frame associated with the depth sensor further based on the interpolated pose. In some embodiments, the second timestamp and the third timestamp may not be immediately before or after the first timestamp. Instead, they may be sufficiently close to the first timestamp, such as within a predefined threshold time window (e.g., + 0.5 seconds), before and after the first timestamp, respectively.

[0043] In some embodiments, in step 302, the processor (e.g., a point coordinate identification module) may be configured io identify a first point coordinate associated with a first edge of the text entity in the image fiame and a second point coordinate associated with a second edge of the text entity opposite to the first edge in the image frame. The processor (e.g., an axis identification module of the processor) may be configured to identify an axis connecting the first point coordinate and the second point coordinate. The processor (e.g., a plane parameter computation module of the processor) may further be configured to compute a plane parameter associated with a surface in the environment on which the text entity is situated based on the axis, the first point coordinate, and the second point coordinate. The processor (e.g., the pose calculation module of the processor) may be configured to compute the estimated pose of the text entity in the image frame based on the plane parameter, the first point coordinate, and the second point coordinate.

[0044] In some embodiments, for example upon identification of the text entity comprising the text content in step 302. the processor (e.g., data storage and database management module of the processor) may be configured to store data such as an index identifying the local coordinate frame, the text content, and the pose of the text entity in the local coordinate frame into the database. In one embodiment, the database may include a text database, and the processor (e.g., a text content and index grouping module of the processor) may be configured to group by each unique text content, a plurality of indices identifying a plurality of local coordinate frames associated with the depth sensor and respective first poses of a first plurality of text entities comprising the each unique text content in the plurality of local coordinateframes in the text database. Additionally, nr alternatively, the database may include a frame database, and the processor (e.g., a text content and index grouping module of the processor) may be configured to group, by each index identifying a specific local coordinate frame, a plurality of text contents and respective second poses of a second plurality of text entities comprising the plurality of text contents in the specific local coordinate frame in the frame database. Such text observation management utilizing a text database and / or a frame database may support and provide efficient loop closure storage, retrieval, and alignment.

[0045] In some embodiments, the processor (e.g., a local text entity map generation module of the processor) may be further configured to generate a first local text entity map comprising a respective first pose of each of a plurality of first text entities observed by the depth sensor within a distance from a current pose of the depth sensor in a current world frame, and, in step 306, the processor (e.g., the candidate pose retrieval module of the processor) may be configured to retrieve, from the database, a plurality of candidate text entity poses with the text content, and the processor (e.g., the local text entity map generation module of the processor) may be configured to, for each of the plurality of candidate text entity poses, generate a second local text entity map comprising a respective second pose of each of a plurality of second text entities observed by the depth sensor within the distance from a previous pose of the depth sensor in the current world frame. The processor (e.g., a graph network generation module of the processor) may be configured to, also for each of the plurality of candidate text entity poses, generate a graph network comprising a plurality of nodes connected to one another, each of the plurality of nodes represents an association between one of a plurality of first poses in the first local text entity map and one of a plurality of second poses in the second local text entity map with a same text content, and in step 308, the processor (e.g., the relative pose constraint calculation module) may be configured to calculate the relative pose constraint based on the one of the plurality of first poses in the first local text entity map and the one of the plurality of second poses in the second local text entity map with the same text content.

[0046] In some embodiments, for example, upon generating the graph network comprising the plurality of nodes connected to one another, the processor (e.g., a graph network generation module of the processor) may be configured to determine whether the plurality of nodes comprises three or more nodes.

[0047] In some embodiments, for example, upon generating the graph network comprising the plurality of nodes connected to one another, the processor (e.g., a geometrical consistent score calculation and determination module of the processor) may be configured to calculate ageometrical consistent score between two connected nodes of the plurality of nodes in the graph network and determine whether the geometrical consistency score is lower than a threshold value, where low geometrical consistency score lower than the threshold value indicates that associations between respective poses in the first local text entity map and the second local text entity map represented by the two connected nodes are geometrically consistent.

[0048] FIG. 4 shows a flow diagram illustrating an overview of a process 400 executed by various components of an apparatus for detecting a loop closure in an environment according to an embodiment of the present disclosure. The process 400 may broadly include front-end odometry 402 and a back-end text cues-based loop closure detection (LCD) 404. During frontend odometry 402, de-skewed scans and initial poses may be obtained from IMU 412 and LiDAR sensor 414 by a LiDAR-inertial odometry (LIO) module 416 to perform LIO and generate a LIO result 418. The de-skewed scans and odometry poses are used for the back-end text cues-based LCD 404. In particular, the back-end text cues-based LCD 404 may include a text entity observation process 422.

[0049] During the text entity observation process 422, text detection and recognition, commonly known as Optical Character Recognition (OCR) may be applied to image data or image frames obtained from a camera 424 using an OCR module 426 to perceive the contents and boundaries or edges (collectively as text entities) of text embedded in the image data or image frames. Text content interpretation and text entity representation may be earned out to perceive or detect the text contents and edges of the text entities. The text entities including respective text contents and boundaries are sent to a pose estimation module 428. The pose of the detected or observed text entity in the image frame may be estimated using the pose estimation module 428. Such estimation may be further based on the de-skewed scans from the LIO module 416. The estimated pose of the text entity may be stored tn the database 406. Such database 406 may be an online or remotely maintained database.

[0050] Candidate loop closures having candidate poses with the same text content as that of the estimated pose estimated by the pose estimation module 428 are retrieved from the database 406 and the authenticity of each candidate loop closure is determined by a graph-theoretic identification scheme in the LCD module 430 hi particular, the LCD module will determine if a new loop is closed, i.e , a loop closure is detected from the candidate loop closures. If a new loop is dosed, a pose graph optimization module 432 will execute a pose graph optimization, and refined poses and a map (e.g., world map) is generated by a refined poses and mapgeneration module 434, effectively mitigating cumulative odometry drift and ensuring the consistency of global poses and maps.

[0051] The following paragraphs describe an exemplary' method executed by the pose estimation module 428 for estimating a pose of a text entity according to an embodiment of the present disclosure.

[0052] Four coordinate frames may be defined: (i) world frame, (ii) LiDAR frame, (iiii camera frame, (iv) and text entity frame. T™is used to represent the Special Euclidean group in 3D (SE3) pose of LiDAR frame in the world frame at timestamp t. As described herein, when referring to a SE3 pose in the world frame, the superscript W may be omitted, for example, T* may be rewritten as TLf. In addition, LiDAR poses, i.e., poses in theLiDAR frame, are adopted as the nodes in the back-end pose graph optimization performed by the pose graph optimization module 432. Similarly,refers to the extrinsic parameters between camera and LiDAR; and T[-extand Ttextrefer text entity poses expressed in camera and LiDAR frame, respectively. 7’ is the measurement of pose T.A. Text Entity Observation

[0053] A text entity may contain two attributes: (i) text content and (ii) spatial pose. Text contents are perceived by text detection and recognition technology, while pose observation is mainly achieved by leveraging the precision of LiDAR measurement.1. Text Content Interpretation

[0054] During the text entity observation process 422, text content interpretation may be carried out using text detection and recognition technology, commonly referred to as OCR. OCR is a mature technique that typically involves identifying and locating regions in an image where these texts may be present and then representing them as rectangles or polygonal vertices, i.e., text entities. Subsequently, text recognition algorithms are employed to convert the region of interest into readable text content. In one example, AttentionOCR is adopted to extract scene texts from image frames.2. Text Entity Representation

[0055] Next, text entity representation may be applied to identify scene text entities, for example, represented by rectangles or polygonal vertices encapsulating the text contents. It isassumed that the scene text entities are situated on flat, surfaces or local planes. Examples include notices on bulletin boards, room numbers on doors, nameplates on fire fighting facilities, and emergency exit signs.

[0056] FIG. 5 shows a picture illustrating a text entity representation of “Exit” signs 502, 504 in an image frame 500 and three axes of the text entities identified according to an embodiment of the present disclosure. The midpoints of a left edge and a right edge of a scene text region is identified, for example, based on OCR technology. The midpoint of the left edge (e.g., in terms of its point coordinate in the frame) may be defined as the origin of the text entity. An in-plane axis, e.g., x-axis, connecting the two midpoints from the left edge towards the right edge of the text entity is defined, and an ont-of-plane axis, e.g., z-axis, aligned with the normal direction of the local plane and points towards the camera may also be defined, and the y-axis is determined by the right-hand rale.3. Pose Estimation

[0057] During pose estimation performed by pose estimation module 428, in order to estimate the SE3 pose of the text entity in the camera frame, scans in the past one second may be accumulated into a local point cloud map, or stated differently, multiple local coordinate frames generated within the past one second from the current timestamp in which the image frame is generated are combined into a single local coordinate frame representing the local point cloud map, and the local point cloud map is cast into the camera frame using an extrinsic parameter between the LiDAR sensor and the camera. Equation (1) shows an exemplary extrinsic parameter.[Equation (1 )1 where pLis a LiDAR point coordinate in the LiDAR frame, Tf is the extrinsic parameter between LiDAR and the camera frame, and pLis the point coordinate in the camera frame.

[0058] Then, the point will be projected into image coordinates, as shown in Equation (2):where K is the intrinsic matrix of the camera, and [u, v]' is the pixel coordinate that the LiDAR point falls in.

[0059] As the scene text is typically affixed to a local plane, the plane parameters in the camera frame can be estimated through Random Sample Consensus (RANSAC) on the set of points detected within the region where the text is present. The plane is represented in the camera frame using Equation (3), as follows:where n is the normal of the plane, p is any point that lies on the plane, and d is the distance from the camera's optical center to the plane.

[0060] Given plane parameters (n, d) and the projection coordinates [it, v] of a point p6, the depth of the point can be recovered using Equation (4), and its position pccould be expressed using Equation (5) as follows:[fiOOtil ] The midpoint pixel of the left boundary or edge of the text bounding box [u;, iy] is used to approximate the projection of the text entity origin on the image, and the midpoint pixel of the bounding box right boundary [ur, i?r] to approximate the projection of the right edge midpoint of the text. Therefore, the position of the text entity origin p and right edge midpoint p in the camera frame could be estimated by Equations (3) to (5), and the pose matrix of the text entity in the camera frame then could be expressed using Equation (6) as follows:[Equation (6)] the normalized vector of p, — p .

[0062] Since the camera and LiDAR are different modal sensors and are triggered at different timestamps, the text entity will be further anchored into the latest LiDAR frame at a timestamp i-t just before the image timestamp and its SE3 pose in the LiDAR frame will be expressed using Equations (7) and (8):fc --[Equation (8)] where t, and tkare the two nearest LiDAR timestamps just before and after the image timestamp, respectively. Althoughand tfcbeing the nearest LiDAR timestamps immediately before and after the image timestamp t,- are preferable, they are not necessarily required to the nearest / closest ones to A. In some embodiments,and ticmay be selected as two other LiDAR timestamps that are sufficiently close to ty, such as within a predefined threshold time window (e.g., ± 0.5 seconds), before or after the image timestamp. interpolate(T,s) is the linear interpolation between the identity transform and T by a factor s G is the SE3 pose of the text entity in its anchored LiDAR frame. For simplicity, henceforth, the text entity's pose relative to the LiDAR frameare dealt with and used for subsequent illustration of the loop closure detection process.B. Text Observation Management

[0063] In addition, to support efficient loop closure storage, retrieval, and alignment, all of the historical text entity observations associated with previous LiDAR frames are stored in the database 406, which consists of a text database 442 representing a text dictionary where frameindices identifying respective LiDAR frames and poses are grouped by each unique text content(e.g., text;, textn, text?) and a frame database 444 representing a frame dictionary where texts detected and poses estimated are grouped by each frame or its frame index, implemented by a hash map as shown in FIG. 4. Stated differently, the text database 442 uses text strings as keys, indices of all LiDAR frames observing the text content, as well as their estimated text entity poses as values, and this enables fast retrieval of candidate frames that observe a certain text content. The frame dictionary 444 utilizes frame indices as keys, contents and estimated poses of all observed text entities in each frame as value, facilitating the creation of local text entities map near the candidate frame.C. Text Entity Based Loop Closure Detection and Alignment

[0064] The following paragraphs describe an exemplary method executed by the LCD module 430 for loop closure detection and alignment based on text entities according to an embodiment of the present disclosure.

[0065] There is a diverse array of scene texts found across various environments, offering insights into the function and location of the entity to which it is affixed. Unlike QR codes or other landmarks, scene text holds the advantage of not requiring specialized deployment and can seamlessly integrate with human navigation. Based on their application characteristics, scene texts can be broadly classified into two categories: (i) unique texts and (ii) repeatable texts. l;..Un.i.que.text.s

[0066] Unique texts, or identifier (ID) texts, refers to texts that follow special conventions designed by humans to identify specific objects within a building or map For instance, S1 - B4c-14 denotes building SI, the fourth basement level (a below-ground floor), section c and room 14, while S2-B3c-AHU3, encodes building S2, the third basement level, section c, and Air Handling Unit 3. Such texts can be picked out according to the predefined pattern of the application environment. ID texts are usually designed to be exclusive, such as door numbers or equipment numbers, thus repeated detection of the same ID text content at different times indicates a high possibility of a loop closure. Regarding unique or ID texts, besides text cuebased loop closure retrievals, the relative pose constraalso known as relative pose prior) between the current loop closure po and a candidate loop closure posea pose in a previous LiDAR frame retrieved from the database 406 can be determined usingEquation (9).[Equation (9)]

[0067] For enhanced constraints between loop closure frames, especially where the text may have multiple instances at different locations, for example, a room may have multiple doors with the same number, the initial relative pose measurements are further refi ned by point cloud registration methods such as iterative closest point (TCP) to rule out the wrong loop candidates, where a rigid body transformation that best aligns the current pose and the candidate pose by minimizing the distance between the two poses is determined. Moreover, 1CP can provide a more precise relative pose prior, which is beneficial to the global pose graph optimization task.In one embodiment, the respective point coordinates in the current pose and the candidate pose are aligned and a world frame may be updated based on the aligned point coordinates.2. Repeatable Texts

[0068] Regarding repeatable texts, or generic texts, in general, a substantial portion of scene texts do not indicate exclusive location information, are repeatable arid can appear multiple times within a scene, such as "EXIT", "No Parking" and "STOP". The association of such text entities can be ambiguous, especially when large odometry pose drift occurs. To tackle this issue, an association scheme based on the local text entity maps (LTEMs) is proposed. An LTEM may encode the spatial arrangements and may be used as a token of the current pose to verify the authenticity of candidate loops with other poses.

[0069] Specifically, an LTEM is a set of ah text entities observed by a set of LiDAR poses. Assuming that at the current pose Tc(the subscript c refers to “current”), a text entity Ecis observed. Mcis defined as the LTEM that contains all of the text entities ( first text entities) observed by the continuous poses Tc= {Tc..w...Tc], wherein Tc-wis the earliest pose within a certain distance d from T... Note that Mcmay contain other text entities whose content, i.e. the text string, is different from the text entity in question Ec. Then, the content of Ecis used to search for all past poses that see a text entity with the same content from the text database 442. The set of these poses is denoted as T.

[0070] For each candidate previous pose Tp€ T (the subscript p refers to “past”), the text entity with the same content with Ecobserved by Tpis denoted as E„ . An LTEM of all text entities (second text entities) observed by the continuous posesTp " {Tp-H,whereTp„wand Tp+Ware the earliest and latest pose within the same distance d from Tp, respectively. This LTEM of text entities (second text entities) is denoted as Mp.

[0071] Given Mcand Mp, an association relationships set A = [a,;, ... } = {(£f,£j'), ... } may be constructed where £f 6 Mc, E? G Mp, and Ef andhave the same text content. The set A is called the set of putative associations.

[0072] FIG. 6 shows a map 600 illustrating putative associations between two local text entity maps (LTEMs) in an environment according to an embodiment of the present disclosure. The historical trajectory 602 and current local trajectory 604 in which the text entities (e.g., text entity Ec606) are observed at timestamp t„, tcare also shown in the map 600 at the lower trajectory and the upper trajectory, respectively. As described, taking the current local trajectory 604 as example, the pose 608 of the text entity 606 as well as other poses of the text entity Ec606 or other text entity observed by LIDAR sensor 610 within tlie distance d 612 as the sensor 610 moves along the trajectory 604 are used to construct the LTEM. As shown in FIG. 6, LTEM Mcand Mpcontain a set of text entities observed by continuous LiDAR poses Tcand Teat the upper trajectory and the lower trajectory, respectively Text entities with identical text contents are connected by lines to indicate putative associations between the two LTEMs. Putative associations are connected by solid lines, while dashed lines indicate false associations. For example, the only “Exit” signs in Mcis associated with three different entities in Mp.

[0073] FIG. 7 shows a graph network G 700 representing the putative associations of FIG. 6 and a subgraph G ’720 representing putative associations that are found to be geometrically consistent according to an embodiment. Such a graph network G may be referred to as a consistency graph. The graph network G representing an association relationship set A may be generated comprising a plurality of nodes connected to one another, each of the plurality of nodes corresponding to a putative association in FIG. 6, i.e., an association between a current pose from Mcand a candidate pose from Mp, and an edge connecting two nodes on the graph network G represents the affinity relationship between two putative associations in A. Stated differently, the connection between arbitrary two nodes afand a, represents theircompatibility, with the darkness of the line signifying a geometrical consistency score calculated using Equation (10), as follows:[Equation (10)1are positions of the two text entities associated by a.,, p, and q, are positions of text entities associated by cij, || ■ ||nieans Euclidean norm of a vector, s: R -» [0,1] is a loss function subject to s(0) = 1 and s(x) = 1 if x > E where E is the threshold geometrical consistency score value. A geometrical consistency score lower than the threshold geometrical consistency score value indicates that the pair of associations a; and cij are geometrically consistent.

[0074] As shown in FIG. 6 and FIG. 7(a), initially, A may contain inappropriate associations due to some repeated text contents. For example, associations a5, az, a3are mutually exclusive to each other, as they try to associate the same text entity from Mcto three different entities from Mp. While associationmay be the exclusively correct one, inappropriate associations such as azand a3may be identified and rejected using the graph theoretic loop closure verification process as described herein

[0075] Next, a fully connected subgraph G* c G, as shown in FIG. 7(b), as well as their nodes subset A* c A is identified such that any pair of associations a, and cq in A* are mutually consistent, for exa mple, by determining that the geometrical consistency scores of the associations pairs are lower than the threshold value. The problem is a variant of the maximum clique problem, and the problem is formulated as finding the densest subgraph G* which may be solved by employing CLIPPER.

[60076] In one implementation, the process may include determining whether the nodebelow illustrates an algorithm summarizing the general process for determining the association for repeatable text entities.

[0077] It is noted that, in the multi -modal LCD and alignment scheme described above, LiD AR inertial odometry LIO output is tightly integrated with the visual detection information, first in the estimation of the text entity’s pose and second in the constructions of the LTEM. The high accuracy for short-term navigation of LIO is crucial for the performance of the loop closure detection process described in the present disclosure, and cannot be achieved with visual inertial odometry' (VIO), due to its poorer depth perception, and larger location drift.[Table 1]

[0078] The following paragraphs describe the effectiveness, accuracy and robustness of the method, device, and system for detecting a loop closure in an environment according to an embodiment of the present disclosure.

[0079] Data from several repetitive and degenerative scenes are collected and comprehensive experiments are conducted. The experiments are implemented on a laptop with an Intel i7-10875H central processing unit (CPU) @ 2.30 GHz and an NVIDIA GeForce RTX 2060 graphic processing unit (GPU).A. Dataset and Experiment Setup

[0080] A customized equipment kit is equipped with a camera with image resolution 1920x1080 , a Livox Mid360 LIDAR, and its embedded IMU. A high-precision prior point map of the scenes is built with Leica MS60 total station and the corresponding ground truth trajectories are obtained by point cloud registration methods. A total of eight data sequences are collected from three different FDR environments: indoor corridor environment, semi- outdoor corridor environment, and cross-floor building environment with distances ranging from 200 to 500 meters.

[0081] Table 2 below shows the exemplary characteristics of the data sequences collected from three different FDR environments.[Table 2]

[0082] The method according to various embodiments of this present disclosure (hereinafter proposed method) is compared to other popular open-source state-of-the-art (SOTA) works (hereinafter existing methods), including Scan Context (SC), Intensity Scan Context (ISC), and Stable Triangle Descriptor (STD). To facilitate fairness in the experiment, Fast LiDAR -Inertial Odometry version 2 (FAST LIO2) is integrated with different loop closure methods to form complete Simultaneous Localization and Mapping (SLAM) systems for evaluation For SC and STD, they have already open sourced integrated solutions. For ISC. it is bootstrapped with Fast LiDAR Odometry and Mapping (FLOAM) as the front-end. Therefore, necessary modifications are made to its code, particularly replacing its front end by FAST-LTO2.

[0083] All of the parameters are kept to the default value, except that the ikdlree map size is set to be 100m x 100m with a resolution 0.2m, and scans are downsampled at a voxel resolution 0.1 tn, two frames are determined to create a true loop closure if their travel distance is larger than 10m and direct distance less than 3m. As the method depends on the existence of text to find loop closure, the frames that have text entities observed are mainly taken into consideration while calculating the recall rate.

[0084] FIG. 8 shows graphs 802, 804, 806, 808, 810, 812, 814, 816 illustrating the apparatus’s trajectories in different sequences corresponding to different environments and pictures 818 illustrating scenes of corridors in some sequences according to an embodiment of the present disclosure. Sequences 1-6 (indoor corridors, semi-outdoor corridors), whose results are shown in graphs 802, 804, 806, 808, 810, 812, are captured on the same floor, whereas sequences 7 and 8 (cross floor buildings), whose results are shown in graphs 814, 816, traverses through different floors and vertical staircases. The overlapped darkened lines in the graphs indicate locations of loop closure incidents. Note only part of the trajectories in Sequences 3 and 6 in graphs 806, 812 had loop closure incidents detected. The pictures (a)-(d) 818 illustrates different scenes at different corridors in Sequences 1 and 4.B. LCD Precision and Recall Analysis1 . True Loop Closure Incidents

[0085] From the ground truth poses, each pose is evaluated to determine if a loop closure detection may happen. Specifically, considering a pose Tfr, the set = (Tp: ||TfeB || <

[0086] For each loop closure method, its recall rate and precision rate are evaluated. The prediction of the method at pose T&can either be a true positive (TP), false positive (FP), true2. Recall

[0087] The recall rates of different methods vary greatly and largely depend on the specific implementation of the algorithm. Table 3 shows the recall rates ([%]) of the existingmethods and the proposed method with different thresholds (1.0 m / 1.7 m), where “MM” denotes “Multi -Modality”, denotes no loop detected, bold text refers to the best results and underlined text refers to the second best results.

[0088] As shown in Table 3, when r = 1,0 in, Semantic Aided Loop closure via Anomaly- aware Detection (SALAD) achieves the best recall performance, exceeding 70% in most cases. Both SC and the proposed method show competitive results, recalling more than 50% of loops in 4 sequences. SC achieves the highest recall rates in Sequences 1 and 2 (indoor corridors). The reasons are twofold. Firstly, SC is robust to translation and yaw angle change. Secondly, SC heavily relies on odometry poses to check the authenticity of candidate loops, so a high recall rate can be easily achieved by setting a low threshold for loop retrieval by SC and odometry poses will help filter out the false loops. It can be expected that as odometer drift becomes more severe in more challenging FDR environments, its recall rate will decrease accordingly. ISC and STD have lower recall rates than the proposed method. This is due to ISC and STD setting strict rules to confirm the authenticity of the loop before adding them to the pose graph. These strict rides lead to their relatively low recall rate but high precision rate, as shown in Tables 3 and 4.

[0089] In the context of loop closure, a higher recall rate may not indicate decisive superiority, since detecting at least one accurate loop for each repeated corridor is sufficient to significantly reduce odometry shift. Precision rate, however, may be more cri tical because false loop closure can corrupt global pose estimation and map building. In pose graph optimization error evaluation described below, the adequacy of recalls can be verified by the effective reduction of absolute translation errror when incorporating the detected loops into the pose graph optimization stage.3. Precision

[0090] FIG. 9 shows two graphs illustrating loop closure detection results 902, 904 of an existing method and a proposed method in a degerative environment according to various embodiments of the present disclosure, respectively. As shown in the loop closure detection results 902 of the existing method in FIG. 9, two trajectories (e.g., historial and current trajectories) do not align well due to odometry drift caused by the degenerative environment; whereas in the loop closure detection results 904 of the proposed method, such drift is reduced through text cue-based loop closure technique as described in various embodiments above, resulting in more aligned trajectories.

[0091] Table 4 shows the precision rates ([%]) of the existing methods and the proposed method with different thresholds (1.0 m / 1.7m) where “MM” denotes “Multi-Modality”, denotes no loop detected, bold text refers to the best results and underlined text refers to the second-best results.[Table 4]

[0092] As shown in Table 4, when T = 1.0 m , SALAD demonstrates competitive performance with 5 second best scores, while ISC stands out as the most effective pure LiDAR- based loop closure methods, achieving precision rates above 80% in 4 sequences. Directed BoW (DBoW) performs effectively in Sequences 1 and 2 (indoor corridors), with a precision rate above 80%, but the performance significantly drops to around 40% in Sequences 4 and 5 (semi-outdoor corridors). This means that DBOW method faces challenges with the FDR scenes like those shown in the pictures 818 of FIG. 8 as it generates many false loop closures. The proposed method achieves the highest precision rate at 100% , owing to the scene text association strategy. In indoor corridor sequences, ISC also performs very well as the intensity of indoor objects is relatively distinguishable, but such scenes can be difficult for SC becausevertical variance is usually limited within an indoor corridor. It should be emphasized that both SC and ISC are not capable of achieving SE3 relative pose estimation between candidate frames and heavily rely on odometry poses to create loop closure constraints, indirectly improving their precision and recall rate. In other words, these existing methods only play the role of loop retrieval in the SLAM system.

[0093] hi the following semi outdoor corridor sequences, STD performs relatively better than the other two methods (SC and ISC). Because there are no sufficient planes and corners present in the semi -outdoor corridor environment, it conservatively creates loops within the elevator halls located at the two ends of the corridor.

[0094] The multi-floor building in Sequences 7 and 8 is a typical repetitive scene for all of the SOT A loop closures, as the layout of different floors is very similar, as shown in picture (c) and picture (d) of FIG 8. It is hard even for humans to figure out which floor they are standing on and differentiate the corridors without the help of the text indicators unless there is level information on the wall. The precision rate of STD seems attractive compared with other methods (SC and ISC), as it selectively identifies loops within structured areas such as elevator halls or stairwells. However, this characteristic also imposes limitations on its recall rate.

[0095] One notable drawback of all existing methods is their susceptibility to generating catastrophic false loop closures, FIG. 10 shows a map 1000 illustrating trajectories generated using exisitng methods and a proposed method in a multi floor building environment (e.g.. Sequence 8) according to an embodiment of the present disclosure. STD and ISC trajectories diverged significantly from the ground truth. Both STD and ISC falsely associate frames from different floors as loop closures, causing their trajectories to deviate from the ground truth.

[0096] Tn contrast, the trajectory of the proposed method remains consistently close to the ground truth. The proposed method leverages textual cues from room numbers, nameplates, and other signs to consistently differentiate between floors and establish genuine associations, avoiding the risk of fanning erroneous loops, which makes our solution closer to human perception of the environment. This capability enables our approach to maintain a high level of trajectory accuracy, effectively avoiding the pitfalls encountered by other methods.

[0097] Moreover, the precision of all existing methods deteriorates significant in partially overlapped sequences compared to the fully overlapped one, even though they are collected from the same environment, i.e., Sequence 3 vs Sequences 1-2, and Sequence 6 vs Sequences4 5. This reveals their propensity to predict false loop closures, which become more pronounced when the trajectory overlap is relatively low.

[0098] Comparatively, the proposed method consistently achieves the best performance and obtains high precision rates of more than 95% on all sequences (environments), benefiting from graph-theoretic loop closure identification scheme that effectively leverages the spatial arrangements of the text entities. The reason for the imperfect performance may be due to a tight threshold T = 1.0 m set to determine the incidence of loop closure. If the threshold is relaxed slightly to 1.7 ro, the proposed method can achieve 100% accuracy while maintaining the same recall rate as shown in Tables 3 and 4.C. Pose Error Evaluation

[0099] FAST-LIO2 is taken as the front- end baseline, and global pose graph optimization will be conducted while loop closure is detected. The absolute pose error is calculated based on the global pose output of the back end pose graph optimizer at the end moment of the data.As the rotation part of the pose error is relatively small and there is no significant difference between different methods, presenting the translation part is the main focus her<[000100] Table 5 shows the mean absolute translation errors of the existing methods and proposed method.[Table 5][000101] As shown in Table 5, the proposed method effectively minimizes the odoroetry drift in the indoor corridor scene. Moreover, the mean absolution translation error of the other three existing methods is even larger than front-end odometry. This is because the layout of the indoor corridor is symmetric and repetitive, which tends to cause false closures between different sides of the closed loop route.[000102] In the semi-outdoor corridor datasets, although forming loop closures is avoided between poses within a travel distance of 10m, SC can retrieve loops between poses whosetravel distance is larger than this value. Consequently, SC can introduce numerous relati ve pose constraints between nonadjacent poses before a loop is closed. This behaviour contributes to a smaller mean translation error for SC compared with FAST-L1O2, ISC, and STD. By leveraging textual cues for loop closure, the approach in the proposed method effectively minimizes odometry drift and achieves the lowest error in such a scenario.[000103] In multi -floor buildings, both ISC and STD tend to generate false loop closures between different floors. 'This erroneous behaviour significantly elevates their mean translation error. Despite encountering erroneous cross-floor loop closures, SC’s final trajectory is gradually corrected by discovering more genuine loops and applying a loss function in its implementation. The result of the proposed method is slightly better than that of SC. The main reason for this may be that stairwells with rich geometric features and obvious height differences are very conducive to finding loops for SC, while there are nearly no scene texts available as references for our method in such places.[000104] FIG. 11 shows three graphs 1100, 1102, 1 104 illustrating translation error distribution of various methods for detecting a loop closure in an indoor long corridor environment (Sequence 2), a semi-outdoor corridor environment (Sequence 5), and a crossfloor building environment (Sequence 8), respectively.[000105] As shown in FIG. 11, the proposed method has the minimum error upper bound as compared with existing methods because all of the loops created by the proposed method are authentic, and no erroneous constraints are introduced into the pose graph. Moreover, the spread of the localization error in the proposed method is also consistent across different sequences, as the distribution of the text cues is similar across the sequences, whereas existing methods rely on the structure of the surroundings.D. Runtime Analysis[000106] Respective time costs of different operation stages of the proposed method are evaluated in an indoor corridor environment, a semi-outdoor corridor environment, and a crossfloor building environment. Table 6 shows the respective mean runtimes of different operation stages in an indoor corridor environment (Sequence 1), a semi -outdoor corridor environment(Sequence 4) and a cross-floor building environment (Sequence 7).[T able 6][000107] As shown in Table 6, text detection and recognition are generally the most timeconsuming stage among all operation stages, indicating that the usage of an improved OCR in the future may be beneficial in further improving the runtimes of loop closure detection.[000108] Various embodiments above illustrate a method, an apparatus, and a system for detecting a loop closure in an environment that addresses the shortcomings of existing methods and fills the gaps in LiDAR or visual loop closure detections in repetitive and degenerative scenes, by utilizing scene text cues inspired by human biological navigation, fusing LiDAR and visual information to observe text entities in the environment and identifying the authenticity of candidate loop closures through a graph -theoretic scheme, with 100% precision rates in loop closure detection.[000109] While the invention has been particularly shown and described with reference to specific embodiments, it should be understood by those skilled in the art that various changes, modification, variation in form and detail may be made therein without departing from the scope of the invention as defined by the appended claims. The scope of the invention is thus indicated by the appended claims and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced.

Claims

CLAIMS1. A method for detecting a loop closure in an environment, the method executed by at least one processor, and comprising: identifying a text entity comprising a text content in the environment from an image frame associated with an image sensor; estimating a pose of the text entity comprising the text content in a local coordinate frame associated with a depth sensor based on an estimated pose of the text entity in the image frame and an extrinsic parameter between the depth sensor and the image sensor; retrieving, from a database accessible by the at least one processor using the text content, a candidate pose of the text entity in a previous local coordinate frame associated with the depth sensor; calculating, a relative pose constraint relating to a transformation between the local coordinate frame and the previous local coordinate frame based on the pose and the candidate pose of the text entity; and detecting, the loop closure in the environment based on the relative pose constraint.The method of claim 1, wherein the image frame is generated at a first timestamp and the local coordinate frame is generated at a second timestamp immediately before the first timestamp, the method further comprising: computing an interpolated pose between (i) a first pose of the depth sensor in a world frame at the second timestamp immediately before the first timestamp and (ii) a second pose of the depth sensor in the world frame at a third timestamp immediately after the first timestamp; wherein the estimating of the pose of the text entity in the local coordinate frame associated with the depth sensor is further based on the interpolated pose.

3. The method of claim ( or 2, further comprising: storing an index identifying the local coordinate frame, the text content, and the pose of the text entity in the local coordinate frame into the database.The method of claim 3, wherein:(i) the database comprises a text database, and the method further comprises grouping, by each unique text content, a plurality of indices identifying a plurality of local coordinate frames associated with the depth sensor and respective first poses of a first plurality of text entities comprising the each unique text content in the plurality of local coordinate frames in the text database; and / or(ii) the database comprises a frame database, and the method further comprises grouping, by each index identifying a specific local coordinate frame, a plurality of text contents and respective second poses of a second plurality of text entities comprising the plurality of text contents in the specific local coordinate frame in the frame database.

5. The method of any one of claims 1 -4, further comprising: generating a first local text entity map comprising a respective first pose of each of a plurality of first text entities observed by the depth sensor within a distance from a current pose of the depth sensor in a current world frame; wherein the retrieving of the candidate pose of the text content comprises: retrieving from the database, a plurality of candidate text entity poses with the text content; and for each of the plurality of candidate text entity poses: generating a second local text entity map comprisi ng a respective second pose of each of a plurality of second text entities observed by the depth sensor within the distance from a previous pose of the depth sensor in the current world frame; generating a graph network comprising a plurality of nodes connected to one another, each of the plurality of nodes represents an association between one of a plurality of first poses in the first local text entity map and one of a plurality of second poses in the second local text entity map with a same text content; and wherein the calculating of the relative pose constraint relating to the transformation between the local coordmate frame and the previous local coordinate frame based on the pose and the candidate pose of the text entity comprises:calculating the relative pose constraint based on the one of the plurality of first poses in the first local text entity map and the one of the plurality of second poses in the second local text entity map with the same text content.

6. The method of claim 5, wherein the generating of the graph network comprising the plurality of nodes connected to one another comprises: calculating a geometrical consistency score between two connected nodes of the plurality of nodes; determining whether the geometrical consistency score is lower than a threshold value, wherein a low geometrical consistency score lower than the threshold value indicates that associations between respective poses in the first local text entity map and the second local text entity map represented by the two connected nodes are geometrically consistent.

7. The method of claim 5 or 6, wherein the generating of the graph network comprising the plurality of nodes connected to one another comprises : determining whether the plurality of nodes comprises three or more nodes.

8. The method of any one of claims 1 - 1, further comprising: aligning respective point coordinates in the local coordinate frame and the previous local coordinate frame; and generating an updated world frame matching the local coordinate frame and the previous local coordinate frame using the aligned point coordinates.

9. The method of any one of claims 1-8, wherein the identifying of the text entity in the environment from the image frame comprises: identifying a first point coordinate associated with a first edge of the text entity in the image frame and a second point coordinate associated with a second edge of the text entity opposite to the first edge in the image frame; identifying an axis associated with the text entity, the axis connecting the first point coordinate and the second point coordinate;computing a plane parameter associated with a surface in the environment on which the text entity is situated based on the axis, the first point coordinate, and the second point coordinate; and computing the estimated pose of the text entity in the image frame based on the plane parameter, the first point coordinate and the second point coordinate.

10. The method of any one of claims 1-9, further comprising: combining respective depth measurements in one or more local coordinate frames generated within a time period from a timestamp at which the image frame is generated into the local coordinate frame, the local coordinate frame comprising the respective depth measurements.

11. An apparatus for detecting a loop closure in an environment, comprising: at least one processor configured to: identify a text entity comprising a text content in the environment from an image frame associated with an image sensor; estimate a pose of the text entity comprising the text content in a local coordinate frame associated with a depth sensor based on an estimated pose of the text entity in the image frame and an extrinsic parameter between the depth sensor and the image sensor: retrieve, from a database using the text content, a candidate pose of the text entity in a previous local coordinate frame associated with the depth sensor: calculate a relative pose constraint relating to a transformation between the local coordinate frame and the previous local coordinate frame based on the pose and the candidate pose of the text entity; and detect the loop closure in the environment based on the relati ve pose constraint12. The apparatus of claim 11, wherein the image frame is generated at a first timestamp and the local coordinate frame is generated at a second timestamp immediately before the first timestamp, and wherein the at least one processor is further configured to: compute an interpolated pose between (i) a first pose of the depth sensor in a world frame at the second timestamp immediately before the first timestamp and (ii) a second poseof the depth sensor in the world frame at a third timestamp immediately after the first time st amp; wherein the estimating of the pose of the text entity in the local coordinate frame associated with the depth sensor is further based on the interpolated pose.13 The apparatus of claim 1 i or 12, wherein the at least one processor is further configured to: store an index identifying the local coordinate frame, the text content, and the pose of the text entity in the local coordinate frame into the database.

14. The apparatus of claim 13, wherein:(i) the database comprises a text database, and the at least one processor is further configured to group, by each unique text content, a plurality of indices identifying a plurality of local coordinate frames associated with the depth sensor and respective first poses of a first plurality of text entities comprising the each unique text content in the plurality of local coordinate frames in the text database; and / or(ii) the database comprises a frame database, and the at least one processor is further configured to group, by each index identifying a specific local coordinate frame, a plurality of text contents and respective second poses of a second plurality of text entities comprising the plurality of text contents in the specific local coordinate frame in tire frame database.

15. The apparatus of any one of claims 11-14, wherein the at least one processor is further configured to: generate a first local text entity map comprising a respective first pose of each of a plurality of first text entities observed by the depth sensor within a distance from a current pose of the depth sensor in a current world frame; retrieve, from the database, a plurality of candidate text entity poses with the text content; and for each of the plurality of candidate text entity poses: generate a second local text entity map comprising a respective second pose of each of a plurality of second text entities observed by the depth sensor within the distance from a previous pose of the depth sensor in the current world frame;generate a graph network comprising a plurality of nodes connected to one another, each of the plurality of nodes represents an association between one of a plurality of first poses in the first local text entity map and one of a plurality of second poses in the second local text entity map with a same text content; and calculate the relative pose constraint based on the one of the plurality of first poses in the first local text entity map and the one of the plurality of second poses in the second local text entity map with the same text content.

16. The apparatus of claim 15, wherein the at least one processor is configured to: calculate a geometrical consistency score between two connected nodes of the plurality of nodes; determine whether the geometrical consistency score is lower than a threshold value, wherein a low geometrical consistency score lower than the threshold value indicates that associations between respective poses in the first local text entity map and the second local text entity rnap represented by the two connected nodes are geometrically consistent.

17. The apparatus of claim 15 or 16, wherein the at least one processor is configured to: determine whether the plurality of nodes comprises three or more nodes.18 The apparatus of any one of claims 11-17, wherein the at least one processor is further configured to: align respective point coordinates in the local coordinate frame and tire previous local coordinate frame; and generate an updated world frame matching the local coordinate frame and the previous local coordinate frame using the aligned point coordinates.19 The apparatus of any one of claims 11-18, wherein the al least one processor is further configured to: identify a first point coordinate associated with a first edge of the text entity in the image frame and a second point coordinate associated with a second edge of the text entity opposite to the first edge;identify an axis associated with the text entity, the axis connecting the first point coordinate and the second point coordinate; compute a plane parameter associated with a surface in the environment on which the text entity is situated based on the axis, the first point coordinate, and the second point coordinate; and compute the estimated pose of the text entity in the image frame based on the plane parameter, the first point coordinate and the second point coordinate.

20. The apparatus of any one of claims 11-19, wherein the at least one processor is further configured to: combine respective depth measurements in one or more local coordinate frames generated within a time period from a timestamp at which the image frame is generated into the local coordinate frame, the local coordinate frame comprising the respective depth measurements.

21. A system for detecting a loop closure in an environment, comprising: the apparatus of any one of claims 11-20: an image sensor configured to generate the image frame; and a depth sensor configured to generate the local coordinate frame associated with the depth sensor.

Images