Device and method for asynchronous factor transfer in real-time visual-inertial slam

Abstract

Disclosed is a device in which asynchronous factor transfer is employed for enabling real-time visual-inertial simultaneous localization and mapping (VI-SLAM) in the device. The device includes visual sensors and inertial sensors to capture image data and inertial data respectively. The device includes a first processor that stores a first subproblem for short-range tracking of the device and a second processor that stores a second subproblem for long-range tracking. The subproblems are represented by factor graphs that may be updated for optimizing the subproblems. The asynchronous factor transfer allows achieving tight coupling between the subproblems such that there is bidirectional information exchange. The exchange allows overcoming inconsistencies between the subproblems and facilitates collaboration between to incrementally determine solutions of a global bundle adjustment problem and open new possibilities for real-time VI-SLAM.

Description

[0001] DEVICE AND METHOD FOR. ASYNCHRONOUS FACTOR TRANSFER IN REAL-TIME VISUAL-INERTIAL SLAM

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to visual-inertial simultaneous localization and mapping (VI-SLAM). The present disclosure also relates to a device and a method for asynchronous factor transfer in real-time VI-SLAM.

[0004] BACKGROUND

[0005] Visual-inertial odometry (VIO) and Simultaneous Localization and Mapping (SLAM) are critical concepts in computer vision and robotics. Monocular VIO or stereo VIO may be used for real-time trajectory and orientation estimation, while SLAM enables autonomous systems to map their environment while simultaneously tracking their movement within the environment. SLAM may be fundamental for various applications such as robotics, autonomous vehicles, drone applications, augmented reality, and so on. For example, a robot or a device navigating in a previously unknown environment may be able to track its own position relative to a starting point using SLAM. Visual-Inertial SLAM (VI-SLAM) integrates inertial measurements with visual data, thereby enhancing robustness and accuracy, especially in visually challenging situations or during rapid movements. VI-SLAM may be modelled mathematically as a problem in which current and previous state variables (such as position, orientation, and so on) of an autonomous system (such as a robot) are unknown. The structure of the problem may be encoded in a factor graph that includes the state variables connected by factors representing data measured by sensors of the robot. Additionally, the robot builds a map of the environment (i.e., its surroundings) that may include landmarks (i.e., 3 Dimensional points in space observed from some, but not all, positions of the robot). The landmarks are variables of the factor graph that are connected to the state variables by factors. As the robot moves, new state variables, new landmarks, and additional factors are added to the problem. These additions increase computational cost of the problem. Eventually, the computational cost reaches to such an extent that real-time computation of the state variables / landmarks becomes infeasible in mobile devices. Thus, existing VI-SLAM methods may have limitations in terms of their inconsistency in maintaining and defining latency bounds. Latency, which refers to a time-delay between capturing data and processing the data, is crucial in applications such as flight controllers, drones, virtual reality headsets, etc., where providing an immediate (or time-bound) response is imperative. In such applications, a minor lag can lead to significant performance deviations, such as a drone veering off-course or a user wearing a virtual reality headset experiencing motion sickness. In order to improve latency, the methods remove older state variables and landmarks from the factor graph or merge data (factors) at various stages. The removal of the older state variables and / or the landmarks is achieved by use of a rolling window that models only recent localization history (i.e., recent state variables) and corresponding map parts (i.e., landmarks associated with the recent state variables). The rolling window may also include prior factors and / or linearization points, associated with older state variables and landmarks, which are obtained based on compression, merging, or summarization of the older state variables, landmarks, and associated factors.

[0006] However, the rolling window approach may lead to cumulative drift error in an estimated trajectory (of the robot), which can be fixed if the entire problem (i.e., all the state variables, landmarks, and associated factors) is used (which is not possible if real-time computation is a requirement). To overcome the issue of cumulative drift error, a dual system with a rolling-window front-end and a SLAM back-end is used. The front-end may perform real-time computations and the SLAM back-end may manage computations such as loop closures and map optimization) that are not latency critical. Although the dual system may significantly simplify the distribution of computational load involved in accurately localizing and mapping robots (or any other autonomous system) in the environment, accuracy of determined state variables or landmarks, used for updating the factor graph, is dependent on synchronization and / or communication between the front-end and the back-end of the dual system. In an absence of a robust transfer mechanism for exchange of information between the front-end and the back end, the distribution of the computational load and accuracy of formulating the VI-SLAM problem is compromised.

[0007] Therefore, considering the foregoing discussion, there exists a need to overcome the aforementioned drawbacks.

[0008] SUMMARY

[0009] The aim of the present disclosure is to provide a device and a method for asynchronous factor transfer in real-time visual-inertial simultaneous localization and mapping (VI-SLAM). The provided device and a method allow incrementally updating an approximate solution to a visual-inertial global bundle adjustment problem in real-time with minimal, bounded latency. The asynchronous factor transfer may ensure a robust coupling between two parallelly or concurrently operating visual-inertial bundle adjustment subproblems using asynchronous communication. The visual- inertial global bundle adjustment subproblem includes a low-latency first subproblem functioning at front-end based on visual-inertial odometry for short-range tracking, and a second subproblem functioning at back-end based on SLAM for long-range tracking. The aim of the present disclosure of ensuring accurate localization of an autonomous system and mapping of surroundings of the autonomous system while guaranteeing bounded latency, which is crucial in real-time applications, is achieved by the provided device and the method for asynchronous factor transfer in realtime VI-SLAM as defined in the appended independent claims to which reference is made to. Advantageous features are set out in the appended dependent claims.

[0010] Throughout the description and claims of this specification, the words "comprise" , "include", "have", and "contain" and variations of these words, for example "comprising" and "comprises" , mean "including but not limited to", and do not exclude other components, items, integers, or steps not explicitly disclosed also to be present. Moreover, the singular encompasses the plural unless the context otherwise requires. In particular, where the indefinite article is used, the specification is to be understood as contemplating plurality as well as singularity, unless the context requires otherwise.

[0011] BRIEF DESCRIPTION OF THE DRAWINGS

[0012] FIG. 1 is a schematic diagram of a device in which real-time visual-inertial simultaneous localization and mapping (VISLAM) is enabled, according to an embodiment of the present disclosure;

[0013] FIG. 2a illustrates an exemplary first factor graph that represents a first sub-problem associated with short-range tracking of the device, in accordance with an embodiment of the present disclosure;

[0014] FIG. 2b illustrates an exemplary second factor graph that represents a second sub-problem associated with long-range tracking of the device, in accordance with an embodiment of the present disclosure;

[0015] FIG. 3a illustrates an exemplary first factor graph, which is a simplified version of the exemplary first factor graph, in accordance with an embodiment of the present disclosure;

[0016] FIG. 3b illustrates an exemplary second factor graph, which is a simplified version of the exemplary second factor graph, in accordance with an embodiment of the present disclosure; FIG. 4 illustrates an exemplary reception of information that enables updating the exemplary first factor graph, i.e., the first sub-problem, in accordance with an embodiment of the present disclosure;

[0017] FIG. 5 illustrates exemplary stages involved in updating the exemplary first factor graph / first sub-problem and the exemplary second factor graph / second sub-problem for solving a VISLAM problem in real-time, in accordance with an embodiment of the present disclosure; and

[0018] FIG. 6 depicts steps of a method for implementing real-time VISLAM in the device, in accordance with an embodiment of the present disclosure.

[0019] DETAILED DESCRIPTION OF EMBODIMENTS

[0020] The following detailed description illustrates embodiments of the present disclosure and ways in which they can be implemented. Although some modes of carrying out the present disclosure have been disclosed, those skilled in the art would recognize that other embodiments for carrying out or practicing the present disclosure are also possible.

[0021] In a first aspect, the present disclosure provides a device comprising: a set of visual sensors that are operable to capture image data of an environment; a set of inertial sensors that are operable to capture inertial data associated with the device in the environment; and a pair of processors including a first processor and a second processor, wherein the pair of processors are operable to:

[0022] - store, by the first processor, a first subproblem represented by a first factor graph, wherein the first factor graph includes a first set of variables, a second set of variables, first information, and a set of first factors that link variables included in the first factor graph,

[0023] - store, by the second processor, a second subproblem represented by a second factor graph, wherein the second factor graph includes the first set of variables, the second set of variables, and a set of second factors that link variables included in the second factor graph,

[0024] - update, by the first processor, the first subproblem by adding a third set of variables and third factors to the first factor graph, wherein the third factors link variables of the third set of variables, wherein one or more factors amongst the third factors link variables of the first set of variables and variables of the third set of variables, and wherein the addition is based on the image data and the inertial data,

[0025] - receive, by the first processor from the second processor, a first message that includes information to enable the first processor to update the first subproblem, wherein the update is associated with the first set of variables, and

[0026] - transmit, by the first processor to the second processor, a second message that includes the third set of variables and the third factors, wherein contents of the second message enable the second processor to update the second subproblem, and wherein the update involves at least one variable of the third set of variables included in the second factor graph.

[0027] In a second aspect, the present disclosure provides a method for realtime visual-inertial simultaneous localization and mapping (VISLAM) in a device, the method comprising: employing a set of visual sensors to capture image data of an environment and a set of inertial sensors to capture inertial data associated with the device in the environment; and employing a pair of processors including a first processor and a second processor, wherein the pair of processors are operable for:

[0028] - storing, by the first processor, a first subproblem represented by a first factor graph, wherein the first factor graph includes a first set of variables, a second set of variables, first information, and a set of first factors that link variables included in the first factor graph,

[0029] - storing, by the second processor, a second subproblem represented by a second factor graph, wherein the second factor graph includes the first set of variables, the second set of variables, and a set of second factors that link state variables included in the second factor graph,

[0030] - updating, by the first processor, the first subproblem by adding a third set of variables and third factors to the first factor graph, wherein the third factors link variables of the third set of variables, wherein one or more factors amongst the third factors link variables of the first set of variables and variables of the third set of variables, and wherein the addition is based on the image data and the inertial data,

[0031] - receiving, by the first processor from the second processor, a first message that includes information to enable the first processor to update the first subproblem, wherein the update is associated with the first set of variables, and

[0032] - transmitting, by the first processor to the second processor, a second message that includes the third set of variables and the third factors, wherein contents of the second message enable the second processor to update the second subproblem, and wherein the update involves at least one variable of the third set of variables included in the second factor graph.

[0033] The present disclosure provides the aforementioned first aspect and the aforementioned second aspect for incrementally updating solutions (state variables and / or landmarks) of a visual-inertial global bundle adjustment problem in real-time. Specifically, approximate solutions are determined for the global bundle adjustment problem while ensuring that the latency involved in the determination is minimal or is bounded, and that memory consumption involved in the determination is optimized. In accordance with an embodiment, the global bundle adjustment problem is split into two subproblems that can be solved using two concurrent processes. A first of the two subproblems or processes is bounded in terms of the number of state variables and landmarks that can be included in the subproblem. This is to ensure that solution determination for the first subproblem or process is computationally efficient and manageable, and execution time involved in determination of the solution adheres to a predefined upper bound, thereby enabling real-time applicability of the first process. The second subproblem or process may be solved to enhance accuracy of the global bundle adjustment problem. The solution determination for the second subproblem / process involves non-linear optimization by use of a larger number of state variables and landmarks (key frames).

[0034] Additionally, a distinctive feature provided by the aforementioned first aspect and the aforementioned second aspect is a mechanism that allows robust coupling between the first and second processes, i.e., the first and second subproblems. The coupling is based on bidirectional information exchange between the first subproblem and the second subproblem. The information exchange mechanism is referred to as Asynchronous Factor Transfer (AFT). AFT may enable coupling a pair of visual-inertial bundle adjustment problems (i.e., the first and second subproblems) based on asynchronous communication. The bidirectional information exchange mechanism or AFT facilitates communicating the following: results of the non-linear optimization, obtained by the second process, to the first process; real-time computations obtained by the first process, based on which the state variables and / or the landmarks are determined, to the second process; information, computed by the second process, such as marginalization priors, linearization points, and gauge fix factors, to the first process; and information, computed by the second process, that enables self-calibration of sensor parameters (such as refining of biases of the inertial sensors) based on which the real-time computations may be performed by the first process, to the first process. The seamless communication ensures that each process benefits from computations performed by the other process, thereby enabling obtaining refined and accurate solutions in real-time. The AFT, thus, makes the first aspect and the second aspect crucial, even critical, in real-time applications such as flight controllers, virtual reality headsets, augmented reality headsets, drones, etc., where immediate response is imperative. Furthermore, the information exchange enables synchronizing the processes and prevents conflicts between the processes, which may arise when both processes perform computations towards the same end (for example, both processes performing computations for determination of marginalization prior). The information exchange enables configuring both processes to cohesively perform computations for obtaining solutions that lead to optimization of the global bundle adjustment problem.

[0035] In an embodiment, the first subproblem may be related to Visual-inertial odometry (VIO) and the second subproblem may be related to VISLAM. VIO may be closely related to inertial navigation (INS), technology that can compute similar outputs as VIO using only an IMU, but without visual sensors. However, INS generally requires very high-end expensive sensors, such as ring laser (R.LG) or fiber-optic (FOG) gyroscopes, which may be large, expensive, and export-controlled, which renders INS infeasible for low-cost and lightweight devices. On the other hand, VIO and VISLAM can operate on very small and inexpensive consumer-grade MEMS IMUs (Micro-Electro-Mechanical Systems).

[0036] The image data of an environment, in which the device is situated, is captured by the set of visual sensors. For example, the device may be a robot, a flight controller, an autonomous vehicle, a virtual reality goggle, a mixed reality goggle, an augmented reality goggle, a drone, and so on. Each visual sensor of the set of visual sensors may be a camera, a stereo camera pair, a depth sensor, a lidar sensor, a laser sensor, thermal camera, or a combination of the above. Throughout the present disclosure, the term "visual sensors" refers to devices that detect light from the real-world environment at its photo-sensitive surface, when the light is incident thereupon. Throughout the present discourse, the term light includes visible light, infrared radiation, and ultraviolet radiation. Each visual sensor comprises a plurality of photo-sensitive elements, which collectively form the photo-sensitive surface of the corresponding visual sensor. Upon such detection of the light from the real-world environment, the plurality of photo-sensitive elements captures a plurality of image signals. The plurality of image signals are electrical signals pertaining to a real-world scene of the real-world environment. The plurality of image signals may be processed (by an image signal processor or the processor of the imaging apparatus) to generate a digital image. The image data corresponds to images of the environment, depth information of the environment, and so on, which may be processed for extraction of visual features such as landmarks and 3-Dimensional locations of the landmarks.

[0037] The inertial data associated with the device (situated in the environment) is captured by the set of inertial sensors. The inertial sensors of the set of inertial sensors include an Inertial Measurement Unit (IMU) that may provide measurements of linear and angular motion of the device. An IMU consists of a triaxial gyroscope (for measuring angular velocity or rotational motion around three axes) and a tri-axial accelerometer (for measuring linear acceleration along three coordinate axes). Throughout the present disclosure, the term "inertial sensors" refers to devices that measure and detect changes in motion, orientation, and acceleration of objects in space, and are based on the principles of inertia. The inertial data may include information associated with movement of the device such as acceleration, and angular velocity. Based on processing of the inertial data, information such as 3-Dimensional (3D) position of the device, 3D orientation of the device, 3D velocity, and biases associated with the inertial sensors may be determined.

[0038] The visual data and the inertial data may be obtained at different time instances, as the device navigates through the environment, and, subsequently, integrated to improve accuracy and / or consistency of navigation results. The accuracy may be improved as the integration allows tracking the device and gauging incremental motion of the device within the environment in real-time from sequential data inputs (collected at the different time instances), which may offer critical insights and / or indications of positional, velocity, and orientation shifts, of the device at the different time instances.

[0039] The device includes the pair of processors that include the first processor and the second processor. The first processor is operable for short-term tracking and the second processor is operable for long-term tracking of the device. The first processor may acquire information from the set of visual sensors and set of inertial sensors and perform computations to suit applications (such as virtual reality application or augmented reality application) that require real-time responses. The second processor may perform operations such as recalibrating parameters measured by the set of visual sensors and set of inertial sensors, updating variables such as state of the device and / or landmark locations) for optimizing the global bundle adjustment problem, providing optimization results to the first processor, and collaborate with the first processor to provide time-bound solutions for the global bundle adjustment problem.

[0040] Each of the first processor or the second processor may be implemented as one of, but not limited to, a microprocessor, a microcontroller, or a controller. For example, the first processor or the second processor may be implemented as an application-specific integrated circuit (AISC) chip, or a reduced instruction set computer (RISC) chip. The first processor stores the first subproblem represented by the first factor graph. The first factor graph includes the first set of variables, the second set of variables, the first information, and the set of first factors that link variables included in the first factor graph. The second processor stores the second subproblem represented by the second factor graph. The second factor graph includes the first set of variables, the second set of variables, and the set of second factors that link variables included in the second factor graph. The variables of the first set of variables or the second set of variables are states of the device or locations of landmarks in the environment where the device may be localized (both of which may be determined based on the factors linking the variables in respective factor graphs). The factors of the set of first factors or the set of second factors may include odometry factors (obtained based on inertial data) or observation factors (obtained based on image data).

[0041] Throughout the present disclosure, the term "factor graph" may refer to a graph-based data structure that functions as a means for describing a structure of a VISLAM optimization problem. For example, the first factor graph may be a means for describing a structure of the first subproblem. Similarly, the second factor graph may be a means for describing a structure of the second subproblem. The factor graph includes two types of nodes, viz., variables and factors. The factors are directly connected to one or more variables (via edges), which they may affect. However, there are no direct connections (edges) between variables or between factors.

[0042] Optionally, each variable (of the first set of state variables or the second set of state variables), included in the first factor graph or the second factor graph, is a state of the device or a location of a landmark in the environment. As, in each of the first factor graph and the second factor graph, the first set of variables are added after the addition of the second set of variables, states of the device that are included in the second set of variables may precede states of the device that are included in the first set of variables. The state of the device includes a position of the device, a velocity of the device, an orientation of the device, and biases associated with the set of inertial sensors. The state of the device or the location of the landmark may be determined by the first processor at a particular time-instance based on the image data and / or the inertial data (by use of the set of visual sensors and / or the set of inertial sensors). The first processor may be instrumental in real-time navigation, and capturing and processing of instantaneous inertial data, for determination of the state of the device. Furthermore, the first processor ensures that visual features extracted from the captured image data are processed in near-real-time for the determination of the location of the landmark.

[0043] Optionally, each factor of the set of first factors is either an odometry factor, an observation factor, or a prior factor. Similarly, each factor of the set of second factors is an odometry factor, an observation factor, or a prior factor. The odometry factor links a pair of consecutive states of the device in the first factor graph or the second factor graph. The odometry factor may be one of a raw odometry factor, a pre-integrated Inertial Measurement Unit (IMU) factor (which is integrated from multiple measurements), a visual-inertial odometry factor, or external-wheel odometry factors. The observation factor links a pair of a state of the device and a location of a landmark in the first factor graph or the second factor graph. The observation factor is a monocular factor, a stereoscopic factor, or a depth-supervised factor. The observation factor is a monocular factor if the set of visual sensors includes a single camera or if the landmark is visible in one camera (out of a pair of cameras of a stereo camera). The observation factor is stereoscopic factor if the set of visual sensors includes a stereo camera. The observation factor is a depth-supervised factor if the set of visual sensors includes a Red-Green- Blue-Depth (RGB-D) camera. The prior factor may be an initial prior that may limit the initial velocity or position. The prior factor may be a gauge fix factor (having four degrees of freedom), a 3D gravity prior, 3D initial velocity prior, or a full initial bias prior. Each factor of the set of first factors and each factor of the set of second factors is determined based on the image data and / or the inertial data.

[0044] Optionally, the odometry factor, linking a pair of consecutive states of the device in either the first factor graph or the second factor graph, may be determined based on a set of sub-factors or a pair of sub-factors. The odometry factor may be the raw odometry factor if the odometry factor is determined based on a composition of a set of sub-factors. The set of sub-factors may include a random walk factor for each bias component associated with each inertial sensor of the set of inertial sensors, an accelerometer factor, a position factor, and a gyroscope factor. On the other hand, the odometry factor may be the pre-integrated IMU factor if the odometry factor is determined based on a marginalization of an intermediate between the pair of sub-factors and a subsequent combining of the pair of sub-factors. The pair of sub-factors may be two target states (for example, two image data samples) that are combined after the marginalization of the two image data samples. The marginalization of the intermediate may be based on a discrete time formulation.

[0045] The first processor initially adds the variables of the second set of variables to the first factor graph. Further, the first processor transfers the second set of variables to the second processor. The second processor adds the second set of variables to the second factor graph. Thus, the second set of variables are stored in the second factor graph based on transfer of the second set of variables by the first processor. The first processor adds the first set of variables to the first factor graph after the addition of the second set of variables.

[0046] During the addition of the second set of variables to the first factor graph, a first subset of first factors linking the second set of variables is added to the first factor graph. Furthermore, during the addition of the first set of variables to the first factor graph, a second subset of first factors linking the second set of variables with the first set of variables, and a third subset of first factors linking the first set of variables, are added to the first factor graph. The first processor may transfer the first subset of first factors to the second processor. The second processor adds the first subset of first factors to the second factor graph during the addition of the second set of variables to the second factor graph. Thus, the set of second factors includes the first subset of first factors.

[0047] At any time-instant, the first processor may transfer the first set of variables to the second processor. The transfer of the first set of variables also includes the second subset of first factors and the third subset of first factors. Optionally, the second processor is operable to receive, from the first processor, the first set of variables, first factors of the set of first factors that link the first set of variables with the second set of variables (i.e., the second subset of first factors), and first factors of the set of first factors that link the first set of variables (i.e., the third subset of first factors). The second processor may add the first set of variables to the second factor graph based on the reception. Thus, the second processor adds the first set of variables to the second factor graph after the addition of the second set of variables. The second processor may further add the second subset of first factors, (i.e., the first factors that link the first set of variables with the second set of variables) and the third subset of first factors (i.e., the first factors that link the first set of variables) to the second factor graph. Thus, the first set of variables, the first factors linking the first set of variables with the second set of variables (i.e., the second subset of first factors), and the first factors linking the first set of variables (i.e., the third subset of first factors), are stored in the second factor graph based on the reception. The first set of state variables and the second set of state variables may enable localization of the device and, simultaneously, generation of a map of the environment. Thus, the set of second factors further includes the second subset of first factors and the third subset of first factors. It may be noted that the second factor graph includes additional variables and additional factors. The set of second factors includes the additional factors. The additional variables and additional factors are included prior to the inclusion of the first set of variables, the second set of variables, the first subset of first factors, the second subset of first factors, and the third subset of first factors. This is because the second processor is operable to perform longterm tracking of the vehicle, and, hence, stores all variables and all factors received from the first processor (unlike the first processor which stores recent variables and factors as the first processor is operable to perform short term tracking and cater to applications that require realtime responses). At this time-instant, each variable of the first set of variables may be a set of shared variables. Both the first processor and the second processor are operable to store the set of shared variables in the first factor graph and the second factor graph respectively. Therefore, the variables of the first set of variables cannot be marginalized from the first factor graph.

[0048] The second processor transfers the first information to the first processor. Thus, storage of the first information in the first factor graph is based on the transfer of the first information by the second processor to the first processor. The first information is determined by the second processor based on the second set of variables. The first information is determined further based on second factors of the set of second factors that link the second set of variables (i.e., the first subset of first factors) and second factors of the set of second factors that link the second set of variables with the first set of variables (i.e., the second subset of first factors). In accordance with an embodiment, on receiving the second set of variables, the second processor optimizes a non-linear loss function (cost function) associated with the second subproblem. The optimization may result in updates in variables of the second set of variables. Thereafter, the first information may be determined based on the second set of variables (or updated second set of variables) and transferred to the first processor.

[0049] It may be noted that, although both the first factor graph and the second factor graph include the second set of variables, the second set of variables are not included in the set of shared variables. This is because, the first processor is operable to marginalize variables of the second set of variables based on the reception of the first information in the first factor graph and inclusion of additional variables (such as the third set of variables) in the first factor graph.

[0050] Optionally, the second processor is operable to compute a first marginalization prior and a first set of linearization points. The first marginalization prior is computed based on factors amongst the second factors that link variables of the second set of variables (i.e., the first subset of first factors). The first set of linearization points are computed based on the second set of variables stored in the second factor graph. For each variable of the second set of variables, a linearization point may be computed. The computation involves freezing or fixing a numeric value (i.e., a linearization point) associated with each variable of the second set of variables. The fixed value associated with each variable may be used as reference for all future computations involving the corresponding variable. Thereafter, the first information, which includes the first marginalization prior and the first set of linearization points, is obtained. The second processor transfers the first information to the first processor.

[0051] Based on the transfer, one or more marginalization priors in the first factor graph, computed by the first processor based on first factors of the set of first factors linking variables of the second set of variables (i.e., the first subset of first factors), are replaced by the first marginalization prior. Thereafter, the first processor may marginalize state variables of the second set of state variables and the first subset of first factors that link variables of the second set of variables from the first factor graph based on the first information. The marginalization of the state variables of the second set of state variables and the first subset of first factors may be initiated or triggered based on addition of new variables (such as the third set of variables) and new factors (such as the third factors) to the first factor graph. It may be noted that the variables of the first set of variables cannot be marginalized from the first factor graph at this stage. However, on reception of second information (which is described later) associated with the first set of variables from the second processor, variables of the first set of variables may be marginalized.

[0052] Optionally, the second processor is operable to compute a first gauge fix factor based on a newest variable of the second set of variables. The newest variable is the last variable of the second set of variables added to the first factor graph and is chosen as a key-frame for gauge fixing. To ensure that the global bundle adjustment problem is invariant to rotations or translations of states (such as position or orientation) of the device or landmark positions (coordinates), gauge fix factors (such as the first gauge fix factor) may be computed. Without the computation of the first gauge fix factor (which is a type of prior factor), rotations (in 2- Dimensional (2D)) and / or translations (in 3D) along a gravity direction may affect accuracy of factors and trajectory solution representatives (i.e., the first set of variables and the second set of variables) of the global bundle adjustment problem.

[0053] In accordance with an embodiment, an ambiguity of an initial 3D position of the device and an ambiguity of an initial heading angle of the device may be eliminated by usage / inclusion of the first gauge fix factor in the first subproblem, i.e., the first factor graph. The first gauge fix factor may be computed by the second processor based on constraints that fix the 3D position of the device and the heading angle of the device. The 3D position may be fixed at a reference position and the heading angle may be fixed at a reference orientation by use of a gravity direction vector in 3D coordinates. For example, the gravity direction vector may be (0, 0, 1). Thereafter, the second processor may transmit the first gauge fix factor to the first processor. The first information may include the first gauge fix factor. It may be noted that gauge fix factors are periodically computed (such as a second gauge fix factor computed at later stages) to improve numerical stability of global bundle adjustment problem.

[0054] The transfer of the first information (which may be obtained based on the computations of the first marginalization prior, the first set of linearization points, and the first gauge fix factor) enables the first processor to use the first information to marginalize the second set of variables from the first factor graph. Furthermore, the first information eliminates necessity of the first processor to compute the first marginalization prior, the first set of linearization points, and the first gauge fix factor. The elimination prevents potential conflicts between the computations of the first processor and that of the second processor. The availability of the first information frees the first processor to cater to applications requiring performance of computations such that responses and feedback related to the first process or the first subproblem is provided in real-time.

[0055] The first processor updates the first subproblem by adding a third set of variables and third factors to the first factor graph. In accordance with an embodiment, the first subproblem is updated based on storage of the third set of variables and the third factors in the first factor graph. The third set of variables may include newer states of the device and new / old locations of landmarks in the environment where the device is situated. The third factors link variables of the third set of variables. Some third factors link variables of the first set of variables and variables of the third set of variables. The addition of the third set of variables and third factors is based on the image data obtained from the set of visual sensors and the inertial data obtained from the set of inertial sensors. After the update, the first factor graph or the first subproblem includes the first set of variables, the second set of variables, the third set of variables, the set of first factors, the third factors, and the first information.

[0056] Optionally, the third set of variables is added to the first factor graph within a period that includes a set of time-instances. For example, the third set of variables may include three variables that are added to the first factor graph in three time-instances (i.e., the set of time-instances may include three time-instances). Each variable of the third set of variables is added to the first factor graph at each time-instance of the set of time-instances. The addition of each variable of the third set of variables is based on at least one variable of the third set of variables that precedes the corresponding variable, at least one variable of the first set of variables, or the first information. In an example, a new variable of the third set of variables may be added to the first factor graph based on one or more older variables of the third set of variables. In an example, a new variable of the third set of variables may be added to the first factor graph based on an older variable of the third set of variables and a variable of the first set of variables. The corresponding variable of the third set of variables may be added to the first factor graph such that a non-linear loss function (cost function) associated with the first subproblem is minimized. The non-linear loss function is minimized based on accurate determination of solutions (i.e., trajectory of movement of the device in the environment) of the global bundle adjustment problem. The accuracy of a determined solution, which may correspond to variables of the third set of variables such as position or orientation (heading angle) of the vehicle, landmark location, and so on, may be improved based on the first information.

[0057] The inclusion of each variable of the third set of variables, i.e., each new variable, in the first factor graph may lead to a concurrent marginalization of an oldest variable of the second set of variables in the marginalization window and factors associated with the oldest variable. Optionally, the first processor is operable to marginalize each variable of the second set of variables from the first factor graph at each time-instance of the set of time-instances. Thus, a variable of the second set of variables is marginalized from the first factor graph when a variable of the third set of variables is added to the first factor graph (as a variable of the third set of variables is added at each time-instance). It may be noted that all variables of the second set of variables are marginalized after elapsing of the period. This is because the period includes the set of time-instances, and each variable of the second set of variables is marginalized at each time-instance the set of time-instances. Consequently, the second set of variables is marginalized after the elapsing of the set of time-instances.

[0058] The first factor graph may include two windows within which the first set of variables, the second set of variables, and the third set of variables, may be included. One of the windows is a rolling window and the other is a marginalization window. Prior to the updating of the first subproblem, i.e., inclusion of the third set of variables, the first set of variables are included in the rolling window and the second set of variables are included in the marginalization window. In accordance with an embodiment, the inclusion of a variable in the marginalization window indicates that the variable may be marginalized based on inclusion of new variables (such as the third set of variables) in the first factor graph. Furthermore, the marginalization of variables, such as the variables of the second set of variables, within the marginalization window may be based on the first information (such as the first marginalization prior obtained based on factors associated with the variables). The variables that are within the marginalization window cease to be included in the set of shared variables (even though those variables are included in the second factor graph) because those variables may be marginalized. Therefore, at this stage only the first set of variables are included in the set of shared variables. Optionally, the first processor may expand the marginalization window at each time-instance to include each variable of the first set of variables within the marginalization window. This is because at each time-instance, each variable of the second set of variables is marginalized and the marginalization window becomes short of one variable. The second set of variables are within the marginalization window prior to initiation of the period and all variables of the first set of variables are included within the marginalization window after elapsing of the period. This is because all variables of the second set of variables are marginalized after the elapsing of the period (due to the inclusion of the third set of variables in the first factor graph at the set of time-instances). The marginalization of all variables of the second set of variables causes the inclusion of all variables of the first set of variables within the marginalization window.

[0059] At this stage, the first set of variables are not included in the set of shared variables. This is because of the inclusion of the first set of variables within the marginalization window (i.e., the first set of variables can be marginalized from the first factor graph). However, the first processor may marginalize the first set of variables based on reception, from the second processor, of the second information associated with the first set of variables. While the second set of variables are marginalized, the first set of variables are included in the marginalization window, and the third set of variables are added to the first factor graph, the second processor may determine the second information and transmit the same to the first processor. The first information is discarded from the first factor graph after elapsing of the period. This is because there is no requirement for the first information after the second set of variables are marginalized.

[0060] The first processor may expand the rolling window at each time-instance to include a newest variable of the third set of variables within the rolling window. This is because at each time-instance, each variable of the first set of variables is included in the marginalization window and the rolling window becomes short of one variable. All variables of the first set of variables are included within the rolling window prior to initiation of the period, and, after elapsing of the period, all variables of the third set of variables are included within the rolling window. This is because all variables of the first set of variables are included in the marginalization window after the elapsing of the period (due to the marginalization of all variables of the second set of variables from the first factor graph during the period). The inclusion of all variables of the first set of variables in the marginalization window causes the inclusion of all variables of the third set of variables within the rolling window.

[0061] The second processor may be operable to update at least one variable of the first set of variables, included in the second factor graph, to minimize the non-linear loss function associated with the second subproblem. The update is triggered by the reception of the first set of variables from the first processor. The update is performed based on at least one variable of the second set of variables and older variables included in the second factor graph. The second processor initiates the update independently of the first processor, and, therefore, the update is not associated with whether the third set of variables are added to the first factor graph. Instead, the update may be based on the storage of the first set of variables in the second factor graph. The second processor, on receiving the first set of variables from the first processor, may determine whether the second subproblem is optimized based on the inclusion of the received first set of variables in the second factor graph or the second subproblem. The determination may be based on variables of the second set of variables and other variables included in the second subproblem (longterm tracking of the state of the device). If it is determined that the second subproblem is not optimized, for non-linear optimization, the at least one variable of the first set of variables is updated such that the non-linear loss function associated with the second subproblem is minimized. Based on the at least one updated variable, an updated first set of variables may be obtained. The second processor may obtain the second information based on the updated first set of variables.

[0062] Optionally, the second processor is operable to compute a second marginalization prior and a second set of linearization points. The computation may be based on the updated first set of variables. Based on the computation, the second information is obtained. The second information includes the second marginalization prior and the second set of linearization points. In accordance with an embodiment, the second marginalization prior is computed based on factors amongst the second factors that link variables of the first set of variables in the second factor graph and factors amongst the second factors that link variables of the first set of variables with variables of the second set of variables in the second factor graph. The second set of linearization points are computed based on the updated first set of variables. For each variable of the updated first set of variables, a linearization point may be computed. The computation involves freezing or fixing a numeric value (a linearization point) associated with each variable of the updated first set of variables. The fixed value associated with each variable may be used as reference for all future computations involving the corresponding variable.

[0063] Thereafter, the second processor may be operable to transfer the second information to the first processor. On reception of the second information by the first processor (i.e., transfer by the second processor), the first processor may be operable to marginalize variables of the first set of variables and one or more marginalization priors in the first factor graph that have been computed by the first processor based on the third subset of first factors linking the first set of variables. The obtaining of the second information and the transfer of the second information by the second processor to the first processor may be independent of updating of the first subproblem by the first processor (whereby the third set of variables and the third factors may be added to the first factor graph). Optionally, the second processor is operable to compute a second gauge fix factor based on a newest variable of the updated first set of variables. The newest variable is the last variable of the first set of variables added to the first factor graph by the first processor. Thereafter, the computed second gauge fix factor may be transmitted to the first processor. The second information may further include the second gauge fix factor (in addition to the second marginalization prior and the second set of linearization points). Thus, the second gauge fix factor is transmitted as part of the second information. The second gauge fix factor is computed to improve numerical stability of global bundle adjustment problem. In accordance with an embodiment, the newest variable of the updated first set of variables may be chosen as the key-frame for gauge fixing based on the marginalization of the second set of variables from the first factor graph. The computation of the second gauge fix factor may enable elimination of ambiguities that may be associated with the 3D position of the device and the heading angle of the device.

[0064] Optionally, for the computation of the second gauge fix factor, the second processor is operable to fix a position of the device to a reference position. The position of the device to be fixed is included in the newest variable of the updated first set of variables. Thereafter, a gravity direction vector is selected. In an embodiment, the gravity direction vector may be in the world coordinate system. For example, a gravity direction vector in the world coordinate system may be selected as (0, 0, 1). Furthermore, a heading angle of the device may be determined based on a reference orientation. The orientation of the device is obtained from the newest variable of the updated first set of variables. The second gauge fix factor may be computed based on the fixing of the position of the device to the reference position, the selected gravity direction vector, and the heading angle determined based on the reference orientation. The transfer of the second information eliminates necessity of the first processor to compute the second marginalization prior, the second set of linearization points, or the second gauge factor. The elimination of the requirement to perform the computations prevents any potential conflict with computations of the first processor. Furthermore, the availability of the second information frees the first processor to cater to applications that require responses and feedback to be provided in real-time.

[0065] The first processor receives, from the second processor, the first message that includes information to enable the first processor to update the first subproblem. In accordance with an embodiment, the information that is included in the first message is the second information transmitted by the second processor and the update is associated with the first set of variables. Optionally, the first processor receives the second information from the second processor as the first message. The update associated with the first set of variables is initiated by the first processor upon reception of the second information. Based on the reception of the second information, the first processor may store the second information in the first factor graph. Thereafter, the second processor extracts the second marginalization prior from the second information. The extracted second marginalization prior may be used to replace one or more marginalization priors in the first factor graph, which are computed by the first processor based on factors amongst the first factors linking variables of the first set of variables (i.e., the third subset of first factors). The first subproblem is updated based on the replacement of the one or more marginalization priors in the first factor graph. It may be noted that the update to the first subproblem, based on the information included in the first message, can be initiated after / during the update in which the third set of variables and the third factors are added to the first factor graph.

[0066] In some embodiments, on reception of the second information, the first processor may marginalize variables of the first set of variables. However, for such marginalization, the first processor may be required to include additional variables (such as a fourth set of variables) and additional factors (such as fourth factors) in the first factor graph. In such scenarios, the third set of variables may be included in the set of shared variables.

[0067] The first processor transmits, to the second processor, the second message that includes the third set of variables and the third factors. Optionally, the second processor is further operable to receive the second message transmitted by the first processor. The contents of the second message enable the second processor to update the second subproblem. The second processor extracts the contents from the second message. The extracted contents include the transmitted third set of variables and the transmitted third factors. The update involves at least one variable of the third set of variables included in the second factor graph. The second processor updates the second subproblem by adding the third set of variables and the third factors to the second factor graph.

[0068] The second processor, on adding the third set of variables to the second factor graph, determines whether the second subproblem is optimized based on the addition. The determination may be based on variables of the first set of variables, variables of the second set of variables, and other variables included in the second subproblem. If it is determined that the second subproblem is not optimized, for non-linear optimization, the at least one variable of the third set of variables may be updated. The at least one variable of the third set of variables, included in the second factor graph, may be updated to minimize the non-linear loss function associated with the second subproblem. The at least one variable of the third set of variables is updated based on one or more variables of the third set of variables that precede the at least one variable, at least one variable of the first set of variables, or at least one variable of the second set of variables. The second processor obtains, based on the at least one updated variable of the third set of variables, an updated third set of variables. The second subproblem may be updated further based on the updated third set of variables.

[0069] The present disclosure also relates to the second aspect as described above. Various embodiments and variants disclosed above, with respect to the aforementioned first aspect, apply mutatis mutandis to the second aspect.

[0070] The aforementioned first aspect and the aforementioned second aspect enable operations / computations for solving the first subproblem within a predefined latency boundary. Moreover, a tight coupling between the first process (for fast or short-term tracking) and the second process (for long-range consistent tracking) enables overcoming inconsistencies that may arise between the parallel / concurrent processes. The tight coupling may allow the first process and the second process to efficiently collaborate for determining a solution of the global bundle adjustment problem and open new possibilities for real-time SLAM operation. The tight coupling is achieved using asynchronous factor transfer (AFT).

[0071] DETAILED DESCRIPTION OF THE DRAWINGS

[0072] Referring to FIG. 1, there is shown a schematic diagram of a device 100 in which real-time visual-inertial simultaneous localization and mapping (VISLAM) is enabled, in accordance with an embodiment of the present disclosure. The device 100 may be navigating in an environment. The VISLAM, enabled in the device 100, may facilitate localizing the device 100 in the environment and, simultaneously, mapping the environment. The device 100 includes a set of visual sensors 102, a set of inertial sensors 104, and a pair of processors 106. The pair of processors 106 includes a first processor 106a and a second processor 106b. The device 100 may be operable to collect image data from the set of visual sensors 102 and inertial data from the set of inertial sensors 104. The first processor 106a stores a first factor graph 108 that is representative of a first subproblem 108a. The second processor 106b stores a second factor graph 110 that is representative of a second subproblem 110a. Based on the collected image and inertial data, the first factor graph 108 and the second factor graph 110 may be generated and updated.

[0073] For enabling real-time VISLAM, the device 100 is operable to determine solutions of a global visual-inertial bundle adjustment problem in realtime. The global visual-inertial bundle adjustment problem may be split into two subproblems, i.e., the first subproblem 108a and the second subproblem 110a, that are solved by the first processor 106a and the second processor 106b, respectively. Each subproblem is a non-linear optimization problem which requires minimizing a non-linear loss function, which is the sum of loss functions corresponding to factors of a factor graph (i.e., the first factor graph 108 or the second factor graph 110). The factors include reprojection errors (corresponding to observation factors), odometry constraints (associated with odometry factors), and loss functions associated with prior factors. The loss function corresponding to each factor may be L2 loss function, in which case the non-linear optimization problem may be referred to as a non-linear least squares problem. In some scenarios, the loss function corresponding to each factor may be a robust loss function, such as Huber loss, in which case the optimization problem may be referred to as robustified nonlinear least squares problem. The minimization involves determinations of optimum solutions of the global visual-inertial bundle adjustment problem that are represent variables of each of the first factor graph 108 and the second factor graph 110.

[0074] The first factor graph 108 (or the first subproblem 108a) is limited by the number of variables that can be included in the first factor graph 108. The first processor 106a is operable to solve the first subproblem 108a for short-range tracking of the device 100. The first processor 106a generates responses and performs computations within a predefined latency due to inclusion of a limited number of variables in the first factor graph 108 and is suitable for applications that require real-time responses. The second processor 106b is operable to solve the second subproblem 110a for long-range tracking of the device 100 based on non-linear optimization of the second subproblem 110a.

[0075] The first subproblem 108a is tightly coupled to the second subproblem 110a. Such coupling is achieved based on a bidirectional information exchange between the first subproblem 108a and the second subproblem 110a. The bidirectional information exchange includes transfer of results of non-linear optimization of the second subproblem 110a to the first processor 106a by the second processor 106b. Based on the received results, the first processor 106a may update the first subproblem 108a. The second processor 106b additionally transfers marginalization priors, linearization points, and gauge fix factors, which are associated with variables stored in the first factor graph 108, to the first processor 106a. Furthermore, the second processor 106b transfers information, to the first processor 106a, that enables self-calibration of sensor parameters (such as refinement of biases of the set of inertial sensors 104) based on which computations are performed by the first processor 106a. The bidirectional information exchange enables configuring both the first processor 106a and the second processor 106b to cohesively perform computations for obtaining solutions that lead to optimization of the global bundle adjustment problem.

[0076] As depicted in FIG. 1, the first processor 106a and the second processor 106b are included in the device 100. Hence, the first subproblem 108a and the second subproblem 110a are executed in the same hardware, i.e., hardware of the device 100. The bidirectional information exchange between the first subproblem 108a and the second subproblem 110a may be intra-device communication. However, the first subproblem 108a and the second subproblem 110a may also be executed in different hardware. For example, the device 100 may include only the first processor 106a and only the first subproblem 108a may be executed in the hardware of the device 100. The second processor 106b may be included in a server and the second subproblem 110a may be executed in the hardware of the server. The bidirectional information exchange between the first subproblem 108a and the second subproblem 110a may be inter-device communication.

[0077] It may be understood by a person skilled in the art that FIG. 1 includes a simplified architecture of the device 100, for sake of clarity, which should not unduly limit the scope of the claims herein. It is to be understood that the specific implementation of the device 100 is provided as an example and is not to be construed as limiting. The person skilled in the art will recognize variations, alternatives, and modifications of embodiments of the present disclosure.

[0078] Referring to FIG. 2a, there is illustrated the exemplary first factor graph 108 that represents the first sub-problem 108a associated with short- range tracking of the device 100, in accordance with an embodiment of the present disclosure. The first factor graph 108 includes a first set of variables 202, a second set of variables 204, and a set of first factors 208a-208n. The first set of variables 202 includes variables 202a-202e and the second set of variables 204 includes variables 204a-204e. The set of first factors 208a-208n link variables included in the first factor graph 108. Each variable of the first set of variables 202 or the second set of variables 204 (included in the first factor graph 108) is a state of the device 100 or a location of a landmark in the environment. For example, amongst the first set of variables 202, the variable 202a, the variable 202b, and the variable 202c represent states of the device 100 at different time-instances. The variable 202d and the variable 202e represent locations of landmarks in the environment. Amongst the second set of variables 204, the variable 204a, the variable 204b, and the variable 204c represent states of the device 100 at different timeinstances. The variable 204d and the variable 204e represent locations of landmarks in the environment.

[0079] Each first factor of the set of first factors 208a-208n is an odometry factor or an observation factor. For example, the first factors 208a, 208b, 208f, 208i, and 208m are odometry factors that link consecutive states of the device 100 in the first factor graph 108. On the other hand, each of the first factors 208c, 208d, 208e, 208g, 208h, 208j, 208k, 2081, and 208n are observation factors that link a pair of a state of the device 100 and a location of a landmark in the first factor graph 108. The set of first factors 208a-208n include a first subset of first factors that link the second set of variables 204, a second subset of first factors that link the second set of variables 204 with the first set of variables 202, and a third subset of first factors that link the first set of variables 202. The first subset of first factors includes the first factors 208a, 208b, 208c, 208d, and 208e. The second subset of first factors includes the first factors 208f, 208g, and 208h. The third subset of first factors includes the first factors 208i, 208j, 208k, 2081, 208m, and 208n.

[0080] FIG. 2a is merely an example, which should not unduly limit the scope of the claims herein. A person skilled in the art will recognize many variations, alternatives, and modifications of embodiments of the present disclosure. For example, a count of variables of the first set of variables 202, a count of variables of the second set of variables 204, or a count of factors of the set of first factors 208a-208n may be different that shown in FIG. 2a. Additionally, any number of factors or any number of variables may be included in the first factor graph 108.

[0081] Referring to FIG. 2b, there is illustrated the exemplary second factor graph 110 that represents the second sub-problem 110a associated with long-range tracking of the device 100, in accordance with an embodiment of the present disclosure. Initially, i.e., at T-l (time- instance-1), the second factor graph 110 includes the second set of variables 204, an older set of variables 206, the first subset of first factors (i.e., the first factors 208a, 208b, 208c, 208d, and 208e), and older factors 208o-208x. The inclusion of the second set of variables 204, the older set of variables 206, the older factors 208a-208x, and the first subset of first factors, in the second factor graph 110 is based on their reception by the second processor 106b from the first processor 106a or transmission by the first processor 106a to the second processor 106b.

[0082] At T-2, the first processor 106a may transfer content 210 to the second processor 106b. The content 210 is a portion of the first factor graph 108. The content 210 may include the first set of variables 202, the second subset of first factors (i.e., the first factors 208f, 208g, and 208h), and the third subset of first factors (i.e., the first factors 208i, 208j, 208k, 2081, 208m, and 208n). Based on the reception, the second processor 106b may add the content 210 to the second factor graph 110. Thus, the first set of variables 202 are added to the second factor graph 110 after the second set of variables 204 and the older set of variables 206. The second factor graph 110 includes a set of second factors 208a-208x that link variables included in the second factor graph 110. The set of second factors 208a-208x comprises the first subset of first factors (i.e., the first factors 208a-208e), the second subset of first factors (i.e., the first factors 208f-208h), the third subset of first factors (i.e., the first factors 208i-208n), and the older factors 208o-208x. It may be noted that each first factor of the set of first factors 208a-208n is included in the second factor graph 110 as a second factor of the set of second factors 208a-208x.

[0083] The older set of variables 206 includes variables 206a-206g. The older factors 208o-208x include second factors that link variables of the older set of variables 206, and second factors that link the second set of variables 204 with the older set of variables 206. Each variable of the older set of variables 206 is a state of the device 100 or a location of a landmark in the environment. For example, amongst the older set of variables 206, the variable 206a, the variable 206b, the variable 206c, and the variable 206d represent states of the device 100 at different time-instances. The variable 206e, the variable 206f, and the variable 202g represent locations of landmarks in the environment.

[0084] Each of the older factors 208o-208x is either an odometry factor or an observation factor. For example, the older factors 208o, 208r, 208t, and 208w are odometry factors that link consecutive states of the device 100 in the second factor graph 108. On the other hand, each of the older factors 208c, 208d, 208e, 208g, 208h, 208j, 208k, 2081, and 208n are observation factors that link a pair of a state of the device 100 and a location of a landmark in the first factor graph 108.

[0085] FIG. 2b is merely an example, which should not unduly limit the scope of the claims herein. A person skilled in the art will recognize many variations, alternatives, and modifications of embodiments of the present disclosure. For example, a count of variables of the older set of variables 202 or a count of factors of the set of second factors 208a-208n may be different that shown in FIG. 2. Additionally, any number of factors or any number of variables may be included in the second factor graph 110.

[0086] Referring to FIG. 3a, there is illustrated an exemplary first factor graph 300A, which is a simplified version of the exemplary first factor graph 108, in accordance with an embodiment of the present disclosure. The exemplary first factor graph 300A includes variables of each of the first set of variables 202 and the second set of variables 204 that represent the states of the device 100 at different time-instances. The variables of the second set of variables 204 are included in a marginalization window 302 and the variables of the first set of variables 202 are included in a rolling window 304. The variables in the marginalization window 302 may be marginalized from the first factor graph 300A based on inclusion of new variables in the first factor graph 108. The variables in the rolling window 304 are included in a set of shared variables, i.e., shared by the first factor graph 108 and the second factor graph 110. The variables in the rolling window cannot be marginalized.

[0087] FIG. 3a is merely an example, which should not unduly limit the scope of the claims herein. A person skilled in the art will recognize many variations, alternatives, and modifications of embodiments of the present disclosure. For example, each of the marginalization window 302 the rolling window 304 may include any number of variables.

[0088] Referring to FIG. 3b, there is illustrated an exemplary second factor graph 300B, which is a simplified version of the exemplary second factor graph 110, in accordance with an embodiment of the present disclosure. The exemplary second factor graph 300B includes variables of each of the first set of variables 202, the second set of variables 204, and the older set of variables 206 that represent the states of the device 100 at different time-instances.

[0089] FIG. 3b is merely an example, which should not unduly limit the scope of the claims herein. A person skilled in the art will recognize many variations, alternatives, and modifications of embodiments of the present disclosure.

[0090] Referring to FIG. 4, there is illustrated an exemplary reception of first information 402 that enables updating the exemplary first factor graph 108, i.e., the first sub-problem 108a, in accordance with an embodiment of the present disclosure. The first processor 106a may receive the first information 402 from the second processor 106b. The first information 402 may be associated with the second set of variables 204. The first information 402 may include a first marginalization prior 404, a first set of linearization points 406a-406c, and a first gauge fix factor 408. The first marginalization prior 404 is computed by the second processor 106b based on the second factors of the set of second factors 208a-208x that link the second set of variables 204 (i.e., the first factors 208a-208e or the first subset of first factors). In some embodiments, computation of the first marginalization prior 404 is further based on the second factors of the set of second factors 208a-208x that link the second set of variables 204 with the first set of variables 202 (i.e., the first factors 208f-208h or the second subset of first factors).

[0091] For each variable of the second set of variables 204, a linearization point may be computed by the second processor 106b. The linearization point 406a is determined based on the variable 204a, the linearization point 406b is determined based on the variable 204b, and the linearization point 406c is determined based on the variable 204c. The first gauge fix factor 408 is computed by the second processor 106b based on the newest variable, i.e., the variable 204a, of the second set of variables 204. The first gauge fix factor 408 is computed by fixing a 3-Dimensional (3D) position of the device 100 and the heading angle of the device 100, as indicated in the variable 204a, to a reference position and a reference heading angle respectively.

[0092] The reception of the first information 402 from the second processor 106b enables the first processor 106a to use the first information 402 to update the first factor graph 108 or the first sub-problem 108a based on marginalization of the second set of variables 204 from the first factor graph 108 and one or more marginalization priors computed by the first processor 106a based on the first factors 208a-208e.

[0093] FIG. 4 is merely an example, which should not unduly limit the scope of the claims herein. A person skilled in the art will recognize many variations, alternatives, and modifications of embodiments of the present disclosure. Referring to FIG. 5, there is illustrated exemplary stages involved in updating the exemplary first factor graph 300A / first sub-problem 108a and the exemplary second factor graph 300B / second sub-problem 110a for solving a VISLAM problem in real-time, in accordance with an embodiment of the present disclosure. Prior to T-l, the first factor graph 300A includes the first set of variables 202, the second set of variables 204, the first information 402, and the set of first factors 208a-208n that link variables included in the first factor graph 300A. The second set of variables 204 are within the marginalization window 302 and the first set of variables 202 are within the rolling window 304. The second factor graph 300B includes the first set of variables 202, the second set of variables 204, and a set of second factors 208a-208x that link variables included in the second factor graph 300B. At this stage, the first set of variables 202 are included in the shared state of variables that cannot be marginalized. The inclusion of the first information 402 in the first factor graph 300A (based on reception of the first information 402 from the second processor 106b) enables the first processor 106a to marginalize the second set of variables 204 from the first factor graph 300A upon inclusion of new variables in the first factor graph 300A.

[0094] The first processor 106a updates the first subproblem 108a by adding a third set of variables 502 and third factors to the first factor graph 300A. The third factors link variables of the third set of variables 502. One or more of the third factors link variables of the first set of variables 202 and variables of the third set of variables 502. The addition of the third set of variables 502 and the third factors by the first processor 106a is based on the image data and the inertial data collected from the set of visual sensors 102 and the set of inertial sensors 104 respectively. The third set of variables 502 is added to first factor graph 300A within a period that includes a set of time-instances. For example, the third set of variables 502 are added in three time-instances, viz., at T-l, T-2, and T- 3. The set of time-instances include T-l, T-2, and T-3. Each variable of the third set of variables 502 is added to the first factor graph 300A at each time-instance of the set of time-instances. The third set of variables 502 includes a variable 502A, a variable 502B, and a variable 502C. The variable 502A is added at T-l, the variable 502B is added at T-2, and the variable 502C is added at T-3.

[0095] The addition of each variable of the third set of variables 502 is based on at least one variable of the third set of variables that precedes the corresponding variable, at least one variable of the first set of variables 202, or the first information 402. For example, the variable 502A may be added based on the at least one variable of the first set of variables 202 or the first information 402. The variable 502B may be added based on the variable 502A, the at least one variable of the first set of variables 202, or the first information 402. Similarly, the variable 502C may be added based on the variable 502A, the variable 502B, the at least one variable of the first set of variables 202, or the first information 402.

[0096] The first processor 106a marginalizes each variable of the second set of variables 204 from the first factor graph 300A at each time-instance of the set of time-instances. All variables of the second set of variables 204 are marginalized after elapsing of the period, i.e., after the elapsing of the set of time-instances. For example, the variable 204c is marginalized at T-l based on inclusion of the variable 502a. The variable 204b is marginalized at T-2 based on inclusion of the variable 502b. The variable 204a is marginalized at T-3 based on inclusion of the variable 502c. The marginalization window 302 expands at each time-instance to include each variable of the first set of variables 202 within the marginalization window 302. At T-l, the variable 202c is included in the marginalization window 302. At T-2, the variable 202b is included in the marginalization window 302. At T-3, the variable 202a is included in the marginalization window 302. All variables of the first set of variables 204 are included within the marginalization window 302 after elapsing of the period, i.e., after T-3. The first information 402 is discarded from the first factor graph 300A after the elapsing of the period, i.e., after T-3.

[0097] The first processor 106a expands the rolling window 304 at each timeinstance to include a newest variable of the third set of variables 502 within the rolling window 304. All variables of the first set of variables 202 are included within the rolling window 304 prior to initiation of the period (i.e., prior to T-l), and all variables of the third set of variables 502 are included within the rolling window 304 after elapsing of the period, i.e., after T-3. At T-l, the variable 502a is included in the rolling window 304. At T-2, the variable 502b is included in the rolling window 304. At T-3, the variable 502c is included in the rolling window 304.

[0098] The second processor 106b, on receiving the content 210 (i.e., the first set of variables 202 and the first factors 208f, 208g, 208h, 208i, 208j, 208k, 2081, 208m, and 208n), determines if the second subproblem 110a is optimized. The second processor 106b may update at least one variable of the first set of variables 202 to minimize the non-linear loss function associated with the second subproblem 110a such that the second subproblem 110a is optimized. The update is performed based on at least one variable of the second set of variables 204 and / or at least one variable of the older set of variables 206. Based on the update, an updated first set of variables 504 is obtained. The second processor 106b computes a second marginalization prior, a second set of linearization points, and a second gauge fix factor based on the updated first set of variables 504.

[0099] The second marginalization prior is computed based on the second factors that link variables of the first set of variables 202 (i.e., the first factors 208i, 208j, 208k, 2081, 208m, and 208n) and second factors that link variables of the first set of variables 202 with variables of the second set of variables 204 (i.e., the first factors 208f, 208g, and 208h). The second set of linearization points include a linearization point that is computed for each variable of the first set of variables 202 by the second processor 106b. The second set of linearization points may include a linearization point determined based on the variable 202a, a linearization point determined based on the variable 202b, and a linearization point determined based on the variable 202c. The second gauge fix factor is computed by the second processor 106b based on the newest variable of the first set of variables 202 (i.e., the variable 202a). The second gauge fix factor is computed by fixing a 3D position of the device 100 and the heading angle of the device 100, as indicated in the variable 202a, to a reference position and a reference heading angle respectively.

[0100] The second processor 106b obtains second information 506 that includes the second marginalization prior, the second set of linearization points, and the second gauge fix factor. Thereafter, the second processor 106b transmits the second information 506 to the first processor 106a. The second information 506 is included in the first message received by the first processor 106a. The reception of the second information 506 allows the first processor 106a to update the first factor graph 300A or the first sub-problem 108a. The update is associated with the first set of variables 202. The update includes marginalization of the first set of variables 202 from the first factor graph 300A (or the first factor graph 108) and one or more marginalization priors that may have been computed by the first processor 106a based on the first factors 208f, 208g, 208h 208i, 208j, 208k, 2081, 208m, and 208n.

[0101] The first processor 106a transmits a second message 508 that includes the third set of variables 502 and the third factors to the second processor 106b. Upon reception of the second message 508, the second processor 106b adds the third set of variables 502 and the third factors to the second factor graph 300B. The second message 508 enables the second processor 106b to update the second subproblem 110a. The update involves at least one variable of the third set of variables 502 included in the second factor graph 300B. The second processor 106b updates the at least one variable of the third set of variables 502 to minimize the non-linear loss function associated with the second subproblem 110a such that the second subproblem 110a is optimized. In accordance with an embodiment, a variable of the third set of variables 502 may be updated based on one or more variables of the third set of variables 502 preceding the variable, at least one variable of the first set of variables 202, at least one variable of the second set of variables 204, and / or at least one variable of the older set of variables 206. Based on the updating of the at least one variable of the third set of variables 502, an updated third set of variables 510 may be obtained.

[0102] FIG. 5 is merely an example, which should not unduly limit the scope of the claims herein. A person skilled in the art will recognize many variations, alternatives, and modifications of embodiments of the present disclosure.

[0103] Referring to FIG. 6, depicted are steps of a method for implementing realtime VISLAM in the device 100, in accordance with an embodiment of the present disclosure. At step 602, a set of visual sensors 102 are employed to capture image data of an environment and a set of inertial sensors 104 are employed to capture inertial data associated with the device 100 in the environment. At step 604, the pair of processors 106 including the first processor 106a and the second processor 106b, are employed. The pair of processors are operable for functions indicated in the steps 602A-602E. At step 602A, the first subproblem 108a, represented by the first factor graph 108, is stored by the first processor 106a. The first factor graph 108 includes the first set of variables 202, the second set of variables 204, the first information 402, and the set of first factors 208a-208n that link variables included in the first factor graph 108. At step 602B, the second subproblem 110a, represented by the second factor graph 110, is stored by the second processor 106b. The second factor graph 110 includes the first set of variables 202, the second set of variables 204, and a set of second factors 208a-208x that link variables included in the second factor graph 110.

[0104] At step 602C, the first subproblem 108a is updated by the first processor 106a by adding the third set of variables 502 and the third factors to the first factor graph 108. The third factors link variables of the third set of variables 502. One or more third factors link variables of the first set of variables 202 and variables of the third set of variables 502. The addition of the third set of variables 502 and the third factors is based on the image data and the inertial data. At step 602D, the first processor 106a receives the first message from the second processor 106b. The first message includes information (the second information 506) for enabling the first processor 106a to update the first subproblem 108a. The update is associated with the first set of variables 202. At step 602E, the first processor 106a transmits the second message 508 to the second processor 106b. The second message 508 includes the third set of variables 502 and the third factors. Contents of the second message 508 enable the second processor 106b to update the second subproblem 110a. The update involves at least one variable of the third set of variables 502 included in the second factor graph 110.

[0105] The aforementioned steps are only illustrative and other alternatives can also be provided where one or more steps are added, one or more steps are removed, or one or more steps are provided in a different sequence without departing from the scope of the claims herein.

Claims

CLAIMS1. A device (100) comprising:- a set of visual sensors (102) that are operable to capture image data of an environment;- a set of inertial sensors (104) that are operable to capture inertial data associated with the device in the environment; and- a pair of processors (106) including a first processor (106a) and a second processor (106b), wherein the pair of processors are operable to:- store, by the first processor, a first subproblem represented by a first factor graph (108), wherein the first factor graph includes a first set of variables (202), a second set of variables (204), first information (402), and a set of first factors (208a-208n) that link variables included in the first factor graph,- store, by the second processor, a second subproblem represented by a second factor graph (110), wherein the second factor graph includes the first set of variables, the second set of variables, and a set of second factors (208a-208x) that link variables included in the second factor graph,- update, by the first processor, the first subproblem by adding a third set of variables (502) and third factors to the first factor graph, wherein the third factors link variables of the third set of variables, wherein one or more factors amongst the third factors link variables of the first set of variables and variables of the third set of variables, and wherein the addition is based on the image data and the inertial data,- receive, by the first processor from the second processor, a first message that includes information to enable the first processor to update the first subproblem, wherein the update is associated with the first setof variables, and- transmit, by the first processor to the second processor, a second message that includes the third set of variables and the third factors, wherein contents of the second message enable the second processor to update the second subproblem, and wherein the update involves at least one variable of the third set of variables included in the second factor graph.

2. The device (100) according to claim 1, wherein each variable, included in the first factor graph (108) or the second factor graph (110), is a state of the device or a location of a landmark in the environment, wherein the state of the device includes at least one of a position of the device, velocity of the device, an orientation of the device, and biases associated with the set of inertial sensors (104), wherein the state of the device or the location of the landmark is determined at a time-instance based on at least one of the image data and the inertial data.

3. The device (100) according to claims 1 and 2, wherein each factor included in each of the set of first factors, the set of second factors, or a factor of the third factors, is one of an odometry factor, an observation factor, or a prior factor, wherein the odometry factor links a pair of consecutive states of the device in the first factor graph (108) or the second factor graph (110), wherein the observation factor links a pair of a state of the device and a location of a landmark in the first factor graph or the second factor graph, and wherein each factor is determined based on at least one of the image data and the inertial data.

4. The device (100) according to claim 3, wherein the odometry factor, linking a pair of consecutive states of the device in the first factor graph (108) or the second factor graph (110), is determined based on a set of sub-factors or a pair of sub-factors, wherein the odometry factor is a raw odometry factor composed of the set of sub-factors that include a randomwalk factor for each bias component associated with each inertial sensor of the set of inertial sensors (104), an accelerometer factor, a position factor, and a gyroscope factor, or a pre-integrated Inertial Measurement Unit (IMU) factor determined based on marginalization of an intermediate between the pair of sub-factors, and a combination of the pair of subfactors, and wherein the marginalization of the intermediate is based on a discrete time formulation.

5. The device (100) according to claim 1, wherein the second processor (106b) is further operable to receive, from the first processor (106a), the first set of variables (202), first factors of the set of first factors that link the first set of variables with the second set of variables, and first factors of the set of first factors that link variables of the first set of variables,- wherein the first set of variables, the first factors linking the first set of variables with the second set of variables, and the first factors linking the first set of variables, are stored in the second factor graph (110) based on the reception, and- wherein states of the device included in the second set of variables precede states of the device included in the first set of variables.

6. The device (100) according to claims 1 and 5, wherein the second processor (106b) is further operable to:- compute a first marginalization prior, and a first set of linearization points, wherein the first marginalization prior is computed based on factors amongst the second factors that link variables of the second set of variables (204), and the first set of linearization points is computed based on the second set of variables;- obtain the first information (402) that includes the first marginalization prior, and the first set of linearization points; and- transfer the first marginalization prior factor, and the first set oflinearization points to the first processor (106a), wherein the first information is stored in the first factor graph (108) based on the transfer, and wherein one or more marginalization priors in the first factor graph, computed by the first processor based on factors amongst the first factors linking variables of the second set of variables, are replaced by the first marginalization prior.

7. The device (100) according to claims 1 and 5, wherein the second processor (106b) is further operable to:- compute a first gauge fix factor based on a newest variable of the second set of variables (204); and- transmit the first gauge fix factor to the first processor (106a), wherein the first information (402) further includes the first gauge fix factor.

8. The device (100) according to claim 1,- wherein the third set of variables (502) is added to first factor graph (300A) within a period that includes a set of time-instances,- wherein each variable of the third set of variables is added to the first factor graph at each time-instance of the set of time-instances,- wherein addition of each variable of the third set of variables is further based on at least one variable of the third set of variables that precedes the corresponding variable, at least one variable of the first set of variables (202), or the first information (402), and- wherein the corresponding variable of the third set of variables is added to first factor graph such that a non-linear loss function associated with the first subproblem is minimized.

9. The device (100) according to claims 1 and 8, wherein the first processor (106a) is further operable to:- marginalize each variable of the second set of variables (204) from thefirst factor graph (300A) at each time-instance of the set of timeinstances, wherein and all variables of the second set of variables are marginalized after elapsing of the period;- expand a marginalization window at each time-instance to include each variable of the first set of variables (202) within the marginalization window, wherein the second set of variables are within the marginalization window prior to initiation of the period, wherein all variables of the first set of variables are included within the marginalization window after elapsing of the period, and wherein the first information (402) is discarded from the first factor graph (300A) after elapsing of the period; and- expand a rolling window at each time-instance to include a newest variable of the third set of variables (502) within the rolling window, wherein all variables of the first set of variables are included within the rolling window prior to initiation of the period, and wherein all variables of the third set of variables are included within the rolling window after elapsing of the period.

10. The device (100) according to claim 1, wherein the second processor (106b) is further operable to:- update at least one variable of the first set of variables (202), included in the second factor graph (300B), to minimize a non-linear loss function associated with the second subproblem, wherein the update is triggered by the reception of the first set of variables from the first processor (106a), and wherein the update is performed based on at least one variable of the second set of variables (204) included in the second factor graph;- obtain, based on the at least one updated variable, an updated first set of variables (504); and- compute a second marginalization prior, and a second set of linearization points, wherein the second marginalization prior is computed based on factors amongst the second factors that link variables of the first set of variables and factors amongst the second factors that link variables of the first set of variables with variables of the second set of variables, and wherein the second set of linearization points is computed based on the updated first set of variables;- obtain second information (506) that includes the second marginalization prior and the second set of linearization points; and- transmit the second information to the first processor (106a), wherein the information included in the first message is the second information.

11. The device (100) according to claims 1 and 10, wherein the second processor (106b) is further operable to:- compute a second gauge fix factor based on a newest variable of the updated first set of variables (504); and- transmit the second gauge fix factor to the first processor (106a), wherein the second information (506) further includes the second gauge fix factor.

12. The device (100) according to claim 11, wherein the second processor (106b) is further operable to:- fix a position of the device to a reference position, wherein the position of the device is included in the newest variable of the updated first set of variables;- select a gravity direction vector in world coordinate system;- determine a heading angle of the device based on a reference orientation, wherein the orientation of the device is obtained from the newest variable of the updated first set of variables (504); and- compute the second gauge fix factor based on the fixing of the position of the device to the reference position, the selected gravity direction vector, and the heading angle determined based on the reference orientation.

13. The device (100) according to claims 1, 10, and 11, wherein the first processor (106a) is further operable to:- receive the second information (506) from the second processor (106b);- store the second information in the first factor graph (300A) based on the reception;- extract the second marginalization prior from the second information; and- replace one or more marginalization priors in the first factor graph (300A), computed by the first processor based on factors amongst the first factors linking variables of the first set of variables, with the second marginalization prior, wherein the first subproblem is updated based on the replacement.

14. The device (100) according to claim 1, wherein the second processor (106b) is further operable to:- receive the second message transmitted by the first processor (106a);- extract the contents from the second message, wherein the extracted contents include the third set of variables (502) and the third factors;- update the second subproblem by adding the third set of variables and the third factors to the second factor graph (300B);- update the at least one variable of the third set of variables, included in the second factor graph, to minimize a non-linear loss function associated with the second subproblem , wherein the update is performed based onone or more variables of the third set of variables that precede the at least one variable, at least one variable of the first set of variables (202) included in the second factor graph, or at least one variable of the second set of variables (204) included in the second factor graph; and- obtain, based on the at least one updated variable, an updated third set of variables (510), wherein the second subproblem is updated further based on the updated third set of variables.

15. A method (600) for enabling asynchronous factor transfer (AFT) for real-time visual-inertial simultaneous localization and mapping (VISLAM) in a device (100), the method comprising:- employing a set of visual sensors (102) to capture image data of an environment and a set of inertial sensors (104) to capture inertial data associated with the device in the environment; and- employing pair of processors (106) including a first processor (106a) and a second processor (106b), wherein the pair of processors are operable for:- storing, by the first processor, a first subproblem represented by a first factor graph (108), wherein the first factor graph includes a first set of variables (202), a second set of variables (204), first information (402), and a set of first factors (208a-208n) that link variables included in the first factor graph,- storing, by the second processor, a second subproblem represented by a second factor graph (110), wherein the second factor graph includes the first set of variables, the second set of variables, and a set of second factors (208a-208x) that link variables included in the second factor graph,- updating, by the first processor, the first subproblem by adding a third set of variables (502) and third factors to the first factor graph,wherein the third factors link variables of the third set of variables, wherein one or more factors amongst the third factors link variables of the first set of variables and variables of the third set of variables, and wherein the addition is based on the image data and the inertial data, - receiving, by the first processor from the second processor, a first message that includes information to enable the first processor to update the first subproblem, wherein the update is associated with the first set of variables, and- transmitting, by the first processor to the second processor, a second message that includes the third set of variables and the third factors, wherein contents of the second message enable the second processor to update the second subproblem, and wherein the update involves at least one variable of the third set of variables included in the second factor graph.

Images