Method and apparatus for optimizing monocular vision-inertial positioning systems

Abstract

The method and apparatus disclosed herein propose a method comprising: capturing multiple optical data at corresponding locations within a portion of an environment using optical sensors mounted on a device moving in an environment (1002); capturing an encoder dataset corresponding to the multiple optical data at the corresponding locations using a wheel encoder (258, 602) mounted on the device (1004); determining a first relative motion based on the multiple optical data (1006); determining a corresponding second relative motion based on the encoder dataset (1008); and incrementing a counter indicating a slip event of the wheel encoder (258, 602) based on the determination that the difference between the first relative motion and the corresponding second relative motion is greater than a first threshold (1010), wherein the slip event corresponds to wheel movement of the device and the corresponding second relative motion is less than a second threshold (1012).

Description

Technical Field

[0001] This disclosure generally relates to Simultaneous Localization and Mapping (SLAM) techniques in an environment, and more particularly to systems and methods for characterizing a physical environment and using image data to localize a mobile robot relative to its environment. Background Technology

[0002] Localization, position recognition, and environmental understanding enable mobile robots to become fully autonomous or semi-autonomous systems within their environment. Simultaneous Localization and Mapping (SLAM) is a method that builds a map of the environment while simultaneously estimating the mobile robot's pose within that environment (e.g., using the estimated pose from the robot's cameras). SLAM algorithms enable mobile robots to build maps of unknown environments and determine their own position within those environments to perform tasks such as path planning and obstacle avoidance. Summary of the Invention

[0003] Monocular camera-based localization techniques extract information, such as features (points and lines) or raw pixel values, from consecutive frames of captured surroundings to solve for the relative pose (e.g., orientation and translation) between those frames using, for example, epipolar geometric constraints or viewpoint n-point solving of a 3D geometry problem. Since a single RGB camera cannot measure the depth of a scene (e.g., the distance to objects captured in a camera frame cannot be directly measured), the distance from the associated features to the camera center in two related frames is unknown when using epipolar geometric constraints for solving. With scaling calibration, the solved translation between two related frames is only valid when it reaches a certain scale (e.g., the solved translation is valid after multiplying by an arbitrary scale). Before scaling calibration, there is a scaling ambiguity problem for monocular camera-based localization techniques. In some embodiments, the scale refers to the physical distance between the poses of two frames. Without accurate scaling estimates, monocular camera-based localization methods may not be able to provide accurate position information to their host devices.

[0004] Therefore, there is a particular need for more efficient methods and systems to provide scale information to visual data collected from monocular cameras. The methods and systems described in this paper do not involve formulating factor graph-based optimization problems, but instead use the relative pose changes between frames measured by an inertial measurement unit to solve the factor graph-based optimization problem. Therefore, the methods and systems presented in this paper are less susceptible to numerical instability and have lower computational costs, resulting in more accurate localization solutions and faster responses from mobile robots.

[0005] As disclosed herein, one solution relies on the use of a monocular camera, MEMS inertial sensors, wheel encoders, and optical flow sensors. Such a solution utilizes a multi-sensor approach to cross-check the pose from different odometry modules. The method and system also detect and reject data collected during wheel slippage events, which can be common in mobile robot applications involving traversing different terrains (e.g., carpet, wooden floors, tiled floors, etc.). By doing so, the method and system effectively improve localization accuracy by rejecting accumulated errors from wheel odometry data accumulated during slippage events. Furthermore, the backend of the algorithm for localization performs proportional calibration and online optimization based on accurately calculated poses from various odometry sources to recover and dynamically adjust the scale of the visual odometry, and asynchronously fuses the poses to obtain robust and accurate pose estimates for the robot.

[0006] The methods and systems described in this paper offer several advantages. First, they are computationally efficient and stable because they do not involve formulating factor graph-based optimization problems, making them better suited for real-time applications. Second, the scale (correlated with images recorded by a monocular camera) can be recovered and dynamically adjusted more accurately. Scale calculations are improved by excluding data measured during wheel slippage events (e.g., when data is recorded showing the robot's wheels rotating but the robot's displacement remains substantially constant), making the entire localization algorithm more adaptable to scenarios with varying scales. Finally, the multi-sensor scheme used in the system described in this paper allows for the replacement and deployment of different types of sensors, making the system and methods flexible and scalable for diverse applications.

[0007] The systems and methods described in this paper provide stable and accurate online scaling calibration (e.g., performing online scaling calibration while a mobile robot is operating or running) and optimization with low computational cost. The systems and methods described in this paper use high-frequency inertial odometry and optical tracking sensor (OTS) odometry information to perform wheel slippage detection and visual odometry (VO) attitude outlier rejection, resulting in more accurate odometry attitude for scaling calibration and optimization. The methods and systems described in this paper are adapted to scenarios with variable scaling and use scaling calibration and optimization strategies to solve the scaling ambiguity problem associated with monocular camera visual odometry. The methods and systems described in this paper employ a loosely coupled sensor fusion framework, allowing for modifications to the SLAM algorithm and sensors to provide greater flexibility.

[0008] According to a first aspect of this application, a method for locating a mobile robot includes: capturing multiple image frames recorded in a first coordinate reference frame at corresponding locations within a portion of the environment during a first time period using a camera mounted on a device moving in an environment; capturing an inertial odometry dataset recorded in a second coordinate reference frame using an inertial measurement unit mounted on the device, the inertial odometry dataset corresponding to the multiple image frames at corresponding locations during the first time period; storing matched pairs of image frames and the inertial odometry dataset satisfying a first criterion in a buffer; determining a rotation transformation matrix corresponding to a relative rotation between the first and second reference frames based on a determined number of stored matched pairs of image frames and inertial odometry data; and determining a scaling factor based on the matched pairs of image frames, wherein the rotation transformation matrix defines the orientation of the device, and the scaling factor and rotation transformation matrix calibrate the multiple image frames captured by the camera.

[0009] According to a second aspect of this application, an electronic device includes one or more processors, a memory, and a plurality of programs stored in the memory. The programs include instructions that, when executed by the one or more processors, cause the electronic device to perform the methods described herein.

[0010] According to a third aspect of this application, a non-transitory computer-readable storage medium stores a plurality of programs, which are executed by an electronic device having one or more processors. The programs include instructions that, when executed by the one or more processors, cause the electronic device to perform the methods described herein.

[0011] In addition to the reduction of computational complexity and the improvement of the speed and accuracy of locating mobile robots in the environment, as described above, various additional advantages of the disclosed technical solution become apparent from the following description. Attached Figure Description

[0012] In the following detailed description of preferred embodiments taken in conjunction with the accompanying drawings, the foregoing features and advantages of the disclosed technology, as well as their additional features and advantages, will be more clearly understood.

[0013] To more clearly describe the embodiments of the technologies disclosed herein or the technical solutions in the prior art, the accompanying drawings used in describing the embodiments or the prior art are briefly introduced below. Obviously, the drawings in the following description only illustrate some embodiments of the technologies disclosed herein, and those skilled in the art can still obtain other drawings based on these drawings without creative effort.

[0014] Figure 1A This is a schematic diagram of the layout of an environment according to some embodiments, in which a mobile robot navigates.

[0015] Figure 1B This is a schematic diagram of a mobile robot including an inertial measurement unit (IMU) and a camera, according to some embodiments.

[0016] Figure 2A This is a schematic diagram of a system for providing location information to a mobile robot according to some embodiments.

[0017] Figure 2B This is a schematic diagram of a system for providing location information to a mobile robot according to some embodiments.

[0018] Figure 3A A flow chart of an exemplary process for calibrating a VO pose to a corresponding IO pose, according to some embodiments, is described.

[0019] Figure 3B This is a flowchart illustrating an exemplary process for calibrating the scaling factor and orientation according to some embodiments.

[0020] Figure 4 Different coordinate systems used by different sensor modules according to some embodiments are depicted.

[0021] Figure 5A This is a flowchart illustrating an exemplary process for optimizing the scaling factor according to some embodiments.

[0022] Figure 5B A flow chart of an exemplary process for optimizing a scaling factor according to some embodiments is described.

[0023] Figure 6 A schematic diagram of a system including a wheel slip detection module according to some embodiments is depicted.

[0024] Figure 7A This is a flowchart illustrating an exemplary process for detecting wheel slippage using an optical tracking sensor module according to some embodiments.

[0025] Figure 7B This is a flowchart illustrating an exemplary process for detecting wheel slippage using a visual odometer module according to some embodiments.

[0026] Figure 7C An exemplary VO outlier detection method according to some embodiments is described.

[0027] Figure 8A An exemplary differential drive robot according to some embodiments is depicted.

[0028] Figure 8B Different motion patterns of a mobile robot according to some embodiments are described.

[0029] Figure 9An exemplary flowchart of a process for positioning a mobile robot in an environment, according to some embodiments, is shown.

[0030] Figure 10 An exemplary flowchart of a process for detecting wheel slippage according to some embodiments is shown.

[0031] Figure 11 This is a block diagram of an electronic device configured to perform the methods described herein, according to some embodiments.

[0032] In several views of the accompanying drawings, the same reference numerals refer to corresponding parts. Detailed Implementation

[0033] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. Numerous specific details are set forth in the following detailed description to provide a thorough understanding of the subject matter presented herein. However, it will be apparent to those skilled in the art that the subject matter can be practiced without these specific details. In other instances, well-known methods, processes, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0034] Figure 1A This is a schematic diagram of the layout of an exemplary environment 100 according to some embodiments, in which a mobile robot navigates. In Figure 1, for example, environment 100 is an indoor environment comprising multiple connected rooms separated by walls. Environment 100 includes structured objects (e.g., windows 130, floors 128, doors 110 and 126, and walls 132) and unstructured objects (e.g., sofa 101, table 106, television 108, bed 112, bedside tables 114 and 116, chair 120, desk 118, toilet 124, and bathtub 122). In some embodiments, environment 100 is an outdoor environment, optionally including structured objects (e.g., buildings, landmarks, streets, etc.) and unstructured objects (e.g., trees, mailboxes, street signs, cars, picnic tables, tents, etc.). In some embodiments, environment 100 (or multiple portions of environment 100) is digitally captured (e.g., via an RGB color camera, a monochrome camera, an infrared camera, a depth camera, etc.), various image processing and object segmentation and / or recognition methods are used to detect structured and unstructured objects, and the structured and unstructured objects are identified as semantic objects from the captured images. For example, semantic annotation of objects can be implemented using algorithms for pixel-level 2D semantic segmentation (e.g., deep neural networks). Each semantic object includes one or more semantic annotations and descriptors, such as category labels, appearance descriptors, shape descriptors, size attributes, material attributes, 3D position and orientation, etc. For example, a semantic object representing a chair 120 captured on an image of a portion of environment 100 may include the following semantic annotations and descriptors:

[0035] Table 1: Semantic annotations and descriptors for chair objects

[0036] Object ID 001 object type Chair Group ID living room type static Functional visibility Portable Function Take your seats property Wooden, padded, with multiple legs Appearance Heavier, medium size Location on the image (8, 2.5, 0), distribution N(0.2, 0.2, 0.2) 3D bounding box (0.8, 0.8, 0.5), oriented (0, 0, 0)

[0037] In some embodiments, the mobile robot 102 (e.g., an autonomous or semi-autonomous cleaning device, delivery device, transport device, monitoring device, etc.) navigates in the environment (e.g., navigating on floor 128 in environment 100) to perform pre-programmed tasks (e.g., vacuuming / mopping floors, performing security checks, delivering food or medicine, and / or traveling to a charging station or a user-selected destination, etc.). In some embodiments, the mobile robot has onboard processing capabilities to process images and self-localize in the environment using object semantic information. In some embodiments, the mobile robot includes communication devices that communicate with a host device (e.g., a control station, home station, remote server, etc.) to transmit image data to and receive localization results from the host device. In some embodiments, the mobile robot 102 is equipped with a forward-facing camera (e.g., facing forward) and a top-facing camera (facing upward) to capture images in environment 100 from different perspectives. In some embodiments, the mobile robot 102 is also equipped with a rear-facing camera and / or a downward-facing camera to capture images in environment 100 from different perspectives. In some embodiments, the mobile robot 102 transmits captured images to an onboard computer (e.g., or via a wireless connection to a remote computer) to extract object semantic information for localization purposes (e.g., calculating the pose of the robot or its camera in environment 100). In some embodiments, the mobile robot obtains the information required for localization from a host device as needed. In some embodiments, some or all of the steps described for the mobile robot may be performed by the host device communicating with the mobile robot.

[0038] Figure 1B This is a schematic diagram of a mobile robot including an inertial measurement unit (IMU) and a camera according to some embodiments. In some embodiments, the mobile robot 102 includes an IMU 256 mounted to a central location of movement and a camera 262 offset from the IMU 256.

[0039] Figure 2A This is a schematic diagram of a system for providing positioning information to a mobile robot according to some embodiments. System 200 includes three different functional components. The front end of the system includes multiple odometry modules, each detecting the pose of the mobile robot. The pose detected by the front end is preprocessed into a preprocessed pose. The back end of system 200 uses the preprocessed pose for pose fusion to obtain positioning information using data collected by the different odometry modules.

[0040] In some embodiments, the odometry module includes an inertial odometry (“IO”) module 202, an optical tracking sensor odometry module 204, and a visual odometry module 206. The inertial odometry module 202 includes an inertial measurement unit (IMU). The IMU uses a combination of one or more of accelerometers, gyroscopes, and magnetometers to measure and report one or more of a specific force, angular rate, and orientation of the mobile robot. The inertial odometry module 202 also includes a wheel encoder. The wheel encoder counts the number of motor rotations. The readings from the wheel encoder can be used to calculate the distance traveled or rotated by the driven robot.

[0041] The Optical Tracking Sensor (“OTS”) odometer module 204 includes an optical flow sensor. The optical flow sensor is a visual sensor capable of measuring optical flow or visual motion and outputting measurements based on the optical flow. Optical flow can also be defined as the distribution of the apparent velocity of a brightness pattern moving within an image. In some embodiments, the optical tracking sensor uses a pair of infrared (IR) emitting LEDs and IR receivers to detect a reflective surface in front of the sensor.

[0042] The visual odometry (“VO”) module 206 includes a monocular camera. The monocular camera captures frames of images of its surroundings at a camera frame rate (e.g., 10 frames per second, 20 frames per second, etc.). For example, the monocular camera is an onboard camera of a mobile robot that captures images from a vantage point of the mobile robot navigating in environment 100.

[0043] In some embodiments, the IO module 202, OTS module 204, and VO module 206 capture odometry data at different rates (e.g., the IMU detects and reports inertial measurements at 50 Hz, the wheel encoder reports encoder readings at 30 Hz, the monocular camera captures image frames at 10 frames per second, and the OTS system 204 detects and reports optical flow information at 30 Hz).

[0044] Effectively extracting and combining odometry data received from each odometry module at the front end helps improve the quality of localization operations (e.g., more precise localization by using multiple sensors that collect different types of measurements). The pose of mobile robot 102 describes the state of the mobile robot (e.g., the robot's position, orientation). IO module 202 includes sensors (e.g., IMU and wheel encoders) that report the mobile robot's pose from an inertial odometry perspective. The output of IO module 202 is the IO pose of mobile robot 102. OTS module 204 includes one or more sensors (e.g., optical flow sensors) that report the mobile robot's pose from an optical tracking perspective. The output of OTS module 204 is the OTS pose of mobile robot 102. Similarly, VO module 206 includes one or more sensors (e.g., a monocular camera) that report the mobile robot's pose from a visual perspective. The output of VO module 206 is the VO pose of mobile robot 102.

[0045] The preprocessing of IO pose, OTS pose, and VO pose includes step 212 of timestamp alignment and delay compensation. Step 212 of timestamp alignment and delay compensation helps align poses that may be acquired at different times and / or at different frequencies or rates on different modules. For example, the IO pose may be reported at a higher rate (e.g., 50 Hz) compared to the VO pose. In some embodiments, step 212 helps match the VO pose with the IO pose that is closest to the VO pose, compensates for any time delay between the VO pose and the IO pose, and aligns the timestamps of the IO pose and the VO pose.

[0046] During attitude preprocessing, step 214 for detecting wheel slippage is also performed. (See reference) Figure 6 The description up to Figure 8 provides further details on wheel slip detection. Wheel slip may occur when the motors of mobile robot 102 rotate, but mobile robot 102 does not move or moves less than a first distance threshold (e.g., the first distance threshold is less than 5 inches, the second distance threshold is less than 2 inches). Wheel slip detection can help identify when the mobile robot is stuck in a specific area or location. Based on the determination that wheel slip has been detected, the IO pose, OTS pose, and VO pose detected or received at the time of wheel slip detection are discarded and / or not passed to the backend so as not to be used for pose fusion.

[0047] During pose preprocessing, step 216 for detecting outliers in the VO pose is performed on the VO pose. (See reference) Figure 7CThe description provides further details on VO pose outlier detection. VO pose outliers can occur when noise, other interference, or instrumentation issues (e.g., hardware or software problems) cause the camera to record image frames that significantly deviate from the actual conditions of the environment (e.g., extreme noise levels or errors in measurement). Random Sample Consensus (RANSAC) is an example of an outlier detection method. Based on the determination that a VO pose outlier has been detected, the anomalous VO pose is discarded and / or not propagated to the backend so as not to be used for pose fusion.

[0048] Once preprocessing aligns the timestamps of the IO, OTS, and VO poses and compensates for delays between these poses, the poses that have withstood wheel slippage and VO pose outlier detection are fed to the backend for pose fusion. Pose fusion includes a scaling and transformation calibration step 218. This is described below and... Figures 3A to 5B More details on scaling and transformation calibration are provided in the description. Once the initial scale is determined from step 218, the scale is optimized in step 220. The process of determining the pose of the mobile robot 102 ends after step 222, where asynchronous pose fusion is performed and the quality of the fused pose is evaluated.

[0049] Figure 2B This is a schematic diagram of a system for providing location information to a mobile robot according to some embodiments. Figure 2B A system 250 for providing positioning information to a mobile robot 102 is illustrated. In system 250, an OTS sensor provides measurements to an OTS odometry module 254, which in turn provides the acquired OTS attitude to an inertial odometry module 260. An IMU, such as a MEMS (microelectromechanical system) IMU 256, provides inertial odometry data, such as angular velocity and acceleration values, to the inertial odometry module 260. A wheel encoder 258 also provides encoder scales including the number of motor rotations of the mobile robot. Information about the encoded scale is provided to the IO module 260. A camera 262 (e.g., at the camera's frame rate) captures images of the area around the mobile robot and sends the information to the visual odometry module 264. A component 266 performing scale recovery and coordinate transformation receives an input VO attitude, excluding scale, from the VO module 264. The output of component 266 is the VO attitude including scale. The VO attitude including scale is provided to a component 270 including an asynchronous Kalman filter for attitude fusion.

[0050] VO module 264 also sends a copy of the VO pose without scaling to component 268. Component 268 receives VO poses from VO module 265 and IO poses from IO module 260. Component 268 calculates the scaling of the VO poses and the transformation matrix T based on the received VO poses and IO poses. vw, the transformation matrix, transforms the coordinate system between the IO pose and the VO pose. As the additional IO pose, VO pose, and OTS pose are recorded by system 250, component 268 continues to optimize the scaling.

[0051] Component 270 receives IO attitude from IO module 260 and VO attitude including scaling information from component 266. Component 270 calculates the fused attitude (or VIO attitude).

[0052] Figure 3A A flow chart depicts an exemplary process for calibrating a VO pose to a corresponding IO pose according to some embodiments. The process 300 for calibration, comprising a VO pose and a corresponding IO pose, begins at step 302, waiting for a pair of poses. In some embodiments, process 300 is performed at component 268. After receiving a pair of poses (e.g., a non-scaled VO pose from VO module 264, and an IO pose from IO module 260), it is determined at step 304 whether the received pose is eligible for calibration. If it is determined that the received pose is not eligible for calibration, process 300 returns to step 302 to wait for the next pair of poses. Eligibility criteria include, for example, a minimum distance and / or a maximum distance between IO poses. Ensuring a minimum distance between IO poses helps prevent the mobile robot 102 from getting stuck in a position (e.g., collecting measurements during wheel slippage). Simultaneously, limiting the distance between IO poses to no more than a maximum distance threshold enables better and more accurate positioning by minimizing jumps between acquired poses. Since the IO pose can accumulate drift over distance traveled, having a nearby pose reduces the chance that the IO pose used for calibration may have significant drift, and thus improves calibration accuracy.

[0053] Based on the determination that the received poses are eligible for calibration, paired poses are buffered at step 306. Based on the determination in step 308 that the number of buffered paired poses reaches a first threshold (e.g., ten pairs of eligible poses, twenty pairs of eligible poses, thirty pairs of eligible poses), the process begins scaling calibration and calculates the transformation matrix at step 310. Based on the determination at step 312 that the calibration result from step 310 is successful, process 300 ends at step 316, and the calibration result is applied to the measured VO data. Based on the determination at step 312 that the calibration result from step 310 is unsuccessful, the calibrator is reset at step 314. The calibrator includes the buffers used in step 306 and other stored data used in step 310 to calculate the scaling and transformation matrices. When a reset occurs, assuming the previously collected data is not suitable for calibration, the entire calibration process will restart. All previously collected poses will be cleared from the buffers and discarded.

[0054] Figure 3B This is a flowchart illustrating an exemplary process for calibrating the scaling factor and orientation according to some embodiments. Figure 3B A flowchart 320 illustrates the steps associated with process 300. After processing each camera frame (e.g., via VO module 264), a VO pose without true scale and transformation is generated. A VIO calibration pipeline (e.g., component 268) receives (322) the VO pose and its associated IO pose.

[0055] If the paired poses are eligible for scaling and coordinate transformation, the paired poses are buffered (324). Eligibility criteria include, for example, minimum and / or maximum distances between IO poses. Scaling and transformation are performed (326) based on the determination that the number of buffered paired poses reaches a predetermined threshold. During the calibration process, orientation is first solved to align the buffered VO poses with respect to the IO poses. Based on the solved orientation, a least-squares problem based on the VO and IO poses is solved (328) to produce scaling and translation vectors. The solved scaling, orientation, and translation vectors are applied to the VO poses to obtain the corrected VO poses. The corrected VO poses are compared with their corresponding IO poses (330). The orientation and translation errors between the IO poses and the corrected VO poses indicate the quality of the calibration process. If the quality of the calibration process meets the required criteria, the corrected VO poses are used in the fusion module (332). Otherwise, the calibrator is reset, and a new set of poses is collected to repeat the calibration process until the calibration process meets the required criteria.

[0056] Figure 4 Different coordinate systems, used by different sensor modules according to some embodiments, are depicted. The mobile robot's IMU (e.g., the IMU in the inertial odometry module 202, MEMSIMU 256) records measurements along three orthogonal axes in the world coordinate reference system 402. The mobile robot's camera (e.g., the monocular camera 262 in the visual odometry module 206) records measurements along three orthogonal axes in the visual coordinate reference system 404. Suppose 406 illustrates a world coordinate reference frame 402 and a visual coordinate reference frame 404. To align these two coordinate reference frames, a rotation is performed (e.g., indicated by the thick arrow, described below). Operations such as scaling (e.g., scaling by scaling parameter s) and translation (e.g.) are also available. The operation is described in detail below.

[0057] Scale and coordinate transformation

[0058] In the following description, general pose transformations in 3D space are represented by 4×4 homogeneous coordinates, for example:

[0059]

[0060] Where R is a 3×3 rotation matrix, P is a 3×1 translation vector, and 0 and 1 are scalars. Subscripts and superscripts are used to represent coordinates and their corresponding references. For example, w represents the world coordinate reference frame, v represents the visual coordinate reference frame, b represents the robot's main frame, and c represents the camera frame. Therefore, This refers to the pose of the robot's main frame relative to the world coordinate system. Subscripts are used with the above symbols to indicate the temporal order of the pose. For example, This means the pose of the robot's main frame relative to the world coordinate system when the k-th image is captured, and so on. Here, the definition of each coordinate system and how the pose is represented when referring to each coordinate system are provided, but it is not specified whether the pose comes from the IO module or the VO module.

[0061] The goal of calibration is to solve for the scaling and coordinate transformation to recover the true scale of the VO pose, and then transform the reference frame of the VO pose from the visual coordinate reference frame to the world coordinate reference frame. Specifically, this is achieved by constructing a least-squares problem using the following equation:

[0062]

[0063] in It is the pose of the robot's main frame when the k-th image is captured, referencing the camera frame. It is essentially not time-varying and does not change the positioning of the monocular camera or IMU on the mobile robot 102. It is the pose of the camera frame relative to the visual coordinate reference system when the k-th image is captured.

[0064]

[0065]

[0066] in It is the pose of the visual coordinate reference system with reference to the world coordinate reference system. It is not time-varying and will not change the positioning of the monocular camera or IMU on the mobile robot 102.

[0067]

[0068] Where s (the unknown term to be solved) is a scaling factor, which is multiplied by the translation vector of the camera frame in the reference visual coordinate system when the k-th image is captured. To resolve the scaling blur associated with the image.

[0069] in This refers to the pose of the robot's main frame when the k-th image is captured, referencing the camera frame. It is essentially not time-varying and does not change the positioning of the onboard monocular camera or IMU on the mobile robot 102. In some embodiments, this can be achieved before operating the mobile robot 102 (e.g., when the monocular camera and IMU are installed on the mobile robot 102, for example, as...). Figure 1B (As shown) Pre-calibration The geometric relationship between the IMU and the camera was revealed mathematically.

[0070] For calibration purposes, s, and This is the unknown term to be solved. Therefore, equation (2) is extended as follows:

[0071]

[0072] Perform matrix multiplication and collect relevant terms:

[0073]

[0074]

[0075] Solving based on equation (7)

[0076]

[0077] In equation (9), the rotation matrix of the robot's main frame referenced to the camera frame is used when the k-th image is captured. The inverse matrix (i.e.) The rotation matrix of the camera frame relative to the visual coordinate reference frame when the k-th image is captured. The inverse matrix (i.e.) Multiply by the matrix and then multiply by the rotation matrix of the robot's main frame in the world coordinate reference frame when the k-th image was captured. A single pairwise dataset (the k-th camera frame in a pair of poses and the IO measurements associated with the k-th camera frame) is used to determine... In contrast, using n datasets to determine s and In some embodiments, n Each dataset is solved using a different formula. In some embodiments, a single pair of datasets works as multiple datasets, and can be implemented more easily.

[0078] Solve using equation (9) Then, s and s are solved by rearranging equation (8) into equation (10).

[0079]

[0080] Where I is an identifier (e.g., 1).

[0081] Rearrange the data from different observations, based on different camera frames k to k+n, according to equation (10). arrive ) and their corresponding associated IMU measurements (e.g. arrive A series of corresponding equations. Further, the solution from equation (9) will be obtained. Substitute into the following equation (11):

[0082]

[0083] Equation (11) can be rewritten as the least squares problem Ax = b, where The unknown term to be solved is the element and Translation vector The ratio s is an (unknown) constant in frames k to k+n, and once a sufficient number of observations have been collected (e.g., observations k to k+n, where n satisfies a first threshold), the translation vector can be solved using the least squares formula. And the ratio s. In some embodiments, n is at least 10, or at least 20 (e.g., n is 20, n is greater than 20, n is 30, n is 40). In equation (11), it is known that the solution obtained in equation (9) is... And for each data point, Same. Assume and It will not change over time because... and It is determined solely by the mounting locations of the IMU and camera.

[0084] The solution s and the corresponding translation vector Multiply by (the k-th camera frame in the reference visual coordinate system) to produce The result is an initial estimate of the true proportion of the locations. yes and Relative translation between (e.g., not) and (relative translation between them).

[0085] Proportion optimization

[0086] Scale calibration (e.g., s obtained by solving equation (11)) provides an initial estimate of the visual odometry scale factor. In some embodiments, calibration errors due to measurement errors, wheel slippage, or other factors may result in inaccurate visual-inertial odometry (e.g., odometry based on data from IMU sensors and images captured from cameras). While the system is running (e.g., after the scale factor has been initially determined based on equation (11) above), an iterative algorithm for on-the-flying optimization of the scale factor (or “online scale optimization”) is described below.

[0087] Figure 5A A flow chart of an exemplary process for optimizing a scaling factor according to some embodiments is described. Once an initial calculation of s (i.e., the scaling factor) has been obtained using equation (11), the process 500 for optimizing the scaling factor can be initiated. The process 500 is activated to monitor potential scaling changes during the operation of the mobile robot. Scaling changes can occur when the mobile robot enters a new part of the environment 100 (e.g., a room or a portion of a room with different ceiling heights, an area with different background textures, or an area in the background with some significant scaling changes).

[0088] The scaling optimization process 500 begins at step 502, where the process waits for paired poses. In some embodiments, process 500 is performed at component 268. After receiving a paired pose (e.g., a non-scaled VO pose from VO module 264 and an IO pose from IO module 260), it is determined at step 504 whether the received pose is eligible for calibration. The pose received at step 504 is an additional pose received after the initial scaling factor s has been determined (e.g., a pose received after the k+nth image frame). If it is determined that the received pose is not eligible for calibration, process 500 returns to step 502 to wait for the next paired pose. Eligibility criteria include, for example, the relative angle or distance between consecutive poses. Angle measurements are obtained from an IMU, and distance measurements are obtained from a wheel encoder. In some embodiments, distance may also be measured from an IMU. In some embodiments, at least one pose is stored in a temporary buffer to determine eligibility for consecutive poses. In some embodiments, eligibility adopted during the scaling optimization process accepts poses with minimal distance between them and rejects poses if the angle variation is too large. Such eligibility criteria help collect poses with greater translation and less rotation. Ensuring minimal angular variation between IO poses helps prevent the mobile robot 102 from getting stuck in a substantially fixed position.

[0089] Based on the determination that the received poses are eligible for scaling optimization, pairs of poses are buffered in a sliding window at step 506. The sliding window has a predetermined length (e.g., the length of the sliding window is used to store twenty pairs of poses, and if the poses are received at 10Hz and each pair of received poses is eligible for optimization, the sliding window will be filled after 2 seconds of data collection). After buffering the pairs of poses at step 506, the optimized scaling factor is calculated as described below.

[0090] If, at step 508, the optimized scaling factor is determined to be greater than a threshold and the sliding window is full, process 500 moves to step 510. In step 510, the scaling factor is updated using the optimized scaling factor determined at step 508, and the optimizer is reset (e.g., clearing the memory associated with the sliding window and clearing other temporary buffers). If the difference is less than the threshold, the current scaling factor is not updated, and the sliding window continues to receive newly entering eligible poses and calculate the optimized scaling factor.

[0091] Figure 5B A flow chart illustrating an exemplary process for optimizing a scaling factor according to some embodiments is described. According to some embodiments, Figure 5B A flowchart 520 illustrates the steps associated with process 500. After processing each camera frame (e.g., via VO module 264), a VO pose with previously calibrated or optimized scale and transformation is generated. A VIO calibration pipeline (e.g., component 268) receives (522) the VO pose and its associated IO pose.

[0092] If a pair of poses is eligible for scaling and coordinate transformation, then the pair of poses is buffered (524). Eligibility criteria include, for example, minimum relative angles and / or distances between IO poses, and / or maximum relative angles and / or distances. A pair of poses refers to an IO pose and its associated (e.g., a) VO pose. In some embodiments, only IO poses are used when evaluating eligibility because IO poses are more stable. The relative angles between IO poses are measured against previous poses stored in a sliding window.

[0093] Eligible pairs of poses are stored in a sliding window, and for each incoming eligible pair of poses, the optimized scale is updated (526). In some embodiments, buffering pairs of poses in step 524 is equivalent to storing eligible poses in the sliding window. When eligible pairs of poses fill the sliding window and the difference between the optimized scale and the current scale is greater than a threshold, the current scale (e.g., the value of the scale factor obtained from equation (11)) is updated (528), and the sliding window and optimizer are reset. If the difference is less than the threshold, the current scale is not updated, the sliding window continues to receive new incoming eligible poses, and the optimized scale is calculated.

[0094] Scaling optimization compensates for residuals that remain uncorrected after scaling calibration in scaling estimation. For example, It is the relative translation vector between two sampled instances (e.g., k-1 and k) calculated by vision-odometry. The relative translation vector is independent of the original reference coordinate system used for measurement. Using the relative pose between frames helps to avoid the influence of the original reference coordinate system. It is the relative translation vector between two identical (or adjacent) sampled instances calculated using inertial-odometer measurements.

[0095] The residual proportion η used to compensate for the residual r(η) between two sets of measurements is:

[0096]

[0097] Equation (12) shows that the residual r(η) is a function of the residual proportion η, and is a corresponding pair. and The sum of squares of the absolute differences between them, where k ranges from 1 to K.

[0098] To minimize the residual function r(η), we need to optimize the value of η such that... Mathematically speaking, the minimum value of the function r(η) occurs when the first derivative of equation (12) is zero.

[0099] The first derivative of equation (12) with respect to η is:

[0100]

[0101] Let equation (13) be zero, and remove factor 2:

[0102]

[0103] Rearrange equation (14) to obtain the optimized residual ratio η opt Represented as:

[0104]

[0105] Where η opt It is the optimal value that minimizes r(η).

[0106] Use the following equation to calculate the scaling factor s i-1 Updated to s i :

[0107] s i =α(η) opt –1)s i-1 +s i-1 (16)

[0108] s i It is the proportion after the i-th iteration (e.g., the latest iteration), s i-1 The ratio before the update is α, and the optimized residual ratio η is the control ratio. opt To what extent should s be modified? i-1 The learning rate is determined by α. In some embodiments, α is heuristic. For example, the value of α depends on various factors, such as the signal-to-noise ratio, the sensor's sampling rate, and how quickly the ratio stabilizes at an optimized value. In some embodiments, α is a very small number that is tied to the characteristics of the application and some tuning (e.g., less than 0.5, less than 0.1, less than 0.01, less than 0.001). In some embodiments, the signal-to-noise ratio (SNR) of the mobile robot 102 is used to set the learning rate.

[0109] In some embodiments, for each new pair of poses received at step 522 (or step 506), an optimized scale s is calculated. i However, only when the difference between the optimized ratio and the current ratio (s i -s i-1 The scaling factor s, initially determined by Equation 11, is updated only when the absolute value of (η-1) is greater than the threshold and the sliding window storing the additional paired poses received at step 506 is full. In some embodiments, the scaling optimization process terminates when the absolute value of (η-1) is determined to be less than the threshold.

[0110] Wheel slip detection

[0111] Wheel slippage can occur when a mobile robot traverses different terrains (e.g., carpet, wood, or tile floors) or uneven surfaces, etc. (e.g., the wheels on the mobile robot are turning, but the mobile robot essentially does not change its position in environment 100). Wheel slippage is also referred to as a "slippage event." Slippage events can introduce errors into displacement measurements (e.g., via wheel encoder 258), and slippage events occur only on the wheels. Wheel slippage primarily affects odometry measurements performed by the wheel encoder. Since proportional calibration relies on reliable odometry measurements from the wheel encoder, once wheel slippage is detected, the associated measurements are invalidated and repeated (e.g., re-performed) for proportional calibration. During robot traversal of terrain, this results in inaccurate proportional calibration and / or optimization, and reduced positioning accuracy over long distances. The wheel slippage detection method described below helps mitigate this degradation of accuracy in the positioning process. The wheel slippage detection method utilizes multimodal sensing inputs to detect whether one or more robot wheels are slipping and effectively removes or suppresses measurements captured when robot wheels are slipping during pose fusion. The results from the wheel slippage detection also inform the planner to perform further corrective actions. The planner is a module used to plan the robot's route and movement.

[0112] Figure 6 A schematic diagram of a system including a wheel slip detection module according to some embodiments is depicted. System 600 includes multiple modules for collecting sensing data: a wheel encoder 602, an optical tracking sensor 604 (e.g., similar to OTS sensor 252), and a visual odometry module 606 (e.g., similar to camera 262 and visual odometry module 264). System 600 also includes a data buffer 608 for storing multimodal sensing data (e.g., for storing multimodal sensing data over specific time windows such as sliding windows). System 600 includes an OTS-based slip detector 610 and a VO-based slip detector 612. System 600 includes an arbitrator 614 and a publisher 616, the arbitrator 614 determining a final detection result related to wheel slip, and the publisher 616 sending the wheel slip event to a planner module.

[0113] Figure 7A This is a flowchart illustrating an exemplary process for detecting wheel slippage using an optical tracking sensor module according to some embodiments. The process 700 for detecting wheel slippage begins at step 702, where OTS and wheel encoder data are obtained from a buffer (e.g., data buffer 608). In some embodiments, process 700 is performed as part of step 214. At step 704, the relative motion ΔP is calculated by integrating the OTS readings. o Integrating the OTS readings over the time interval from t to t+Δt yields the relative motion measurement value ΔP from t to t+Δt. o In some embodiments, an OTS sensor is used to measure robot motion from t to t+Δt, particularly in the relative motion measurement value ΔP. o When the value is significantly large, it provides an important reference for detecting wheel slippage events (e.g., robot motion is used for comparison with wheel encoder odometer measurements). At step 705, process 700 obtains the relative motion ΔP from the wheel encoder based on the mobile robot's kinematic model. m Based on the ΔP determined from the wheel encoder at step 706. m If the value is significantly larger (e.g., greater than a first threshold), process 700 moves to step 708, where the wheel encoder reading is aligned with the OTS data. For example, even when the robot is stuck, ΔP m It continues to increase. In some embodiments, the first threshold is approximately 0.1m (e.g., greater than 0.05m, greater than 0.2m, greater than 0.3m). If the first threshold is exceeded, ΔP is considered to be... o The value is significantly larger. The first threshold is a tuning parameter used to reduce false positive detections caused by sensor noise. When ΔP is not considered at step 706... mIf the value is significantly large, process 700 returns to step 702 to continue obtaining OTS data and wheel encoder data from the buffer. In some embodiments, wheel slip detection according to process 700 is performed only after the OTS reading confirms that the mobile robot has moved a threshold distance. Wheel slip detection helps prevent errors in proportional calibration (e.g., incorrect proportional calibration). The proportional calibration process will not begin when the robot's relative motion is not significant (e.g., proportional calibration is not performed when the mobile robot is in a state of complete slippage). Therefore, OTS-based slip detection can still provide protection for ongoing proportional calibration. ΔP from the wheel encoder is checked at step 706. m Then, process 700 checks ΔP at step 708. o To detect wheel slippage events.

[0114] Aligning the wheel encoder readings with the OTS readings involves matching the wheel encoder readings to check if the wheel encoder is also measuring a proportional amount of movement determined by the OTS readings. The alignment process primarily aligns the time periods recorded by the sensors of both the OTS and the wheel encoder. Based on the determination at step 710 that the previous state of the mobile robot was not a slipping state (e.g., the previous state of the mobile robot was set to a non-slipping state before any slippage was detected), process 700 checks ΔP at step 712. m Is it significantly greater than ΔP? o In some embodiments, when the ratio ΔP o / ΔP m When the threshold range is between 0.001 and 0.1, ΔP m Significantly greater than ΔP o In some embodiments, the ratio ΔP o / ΔP m A value less than 0.01 indicates that the mobile robot is experiencing a significant wheel slippage event. In some embodiments, the first threshold is proportional to the second threshold (e.g., the first threshold is equal to the second threshold, the first threshold is a multiple of the second threshold, and the second threshold is a multiple of the first threshold).

[0115] Based on the determination of ΔP m Not significantly greater than ΔP o (e.g., compared to ΔP) o If the second threshold is exceeded, process 700 returns to step 702 to continue obtaining OTS data and wheel encoder data from the buffer. Based on the determined ΔP... m Ratio ΔP oAt step 716, the slip count counter is incremented by 1 if the second threshold is exceeded. Based on the determination at step 720 that the slip count counter is not greater than the slip count threshold, process 700 returns to step 702 to continue obtaining OTS data and wheel encoder data from the buffer. Based on the determination at step 720 that the slip count counter is greater than the slip count threshold, process 700 sets the current state of the mobile robot to a slip state at step 724 and sends the state (e.g., slip) to an arbitrator (e.g., arbitrator 614) at step 728. In some embodiments, the slip count counter is used by subtracting ΔP... m Significantly greater than ΔP o "One-off" measurements (e.g., those caused by noise or transient malfunctions) are used to improve the accuracy of wheel slippage detection. When several repeated measurements (e.g., at least 2, at least 5, at least 10) indicate that ΔP... m Significantly greater than ΔP o At that time, the mobile robot's state is only set to slipping.

[0116] Based on the determination at step 710 that the previous state of the mobile robot was slipping, process 700 checks ΔP at step 714. m Is it significantly greater than ΔP? o At step 714, if ΔP o / ΔP m A value greater than 0.9 indicates that wheel slippage is not severe. In some embodiments, when ΔP m Ratio ΔP o When ΔP exceeds the second threshold, m Significantly greater than ΔP o In some embodiments, the second threshold is at least 0.2m (e.g., the second threshold is 0.3m, the second threshold is 0.4m). In some embodiments, the first threshold is proportional to the second threshold (e.g., the first threshold is equal to the second threshold, the first threshold is a multiple of the second threshold, and the second threshold is a multiple of the first threshold).

[0117] Based on the determination of ΔP m Not significantly greater than ΔP o (For example, ΔP) m No comparison with ΔP o If the second threshold is not greater than the non-slip count threshold, process 700 increments the counter indicating the number of non-slip events (1) at step 718. Based on the determination at step 722 that the non-slip count counter is not greater than the non-slip count threshold, process 700 returns to step 702 to continue obtaining OTS data and wheel encoder data from the buffer. In some embodiments, the non-slip count counter is used to improve the accuracy of wheel slip detection by reducing the chance of erroneous changes from a "slip" state to a "no slip" state, wherein such erroneous changes are caused by ΔP.m Not significantly greater than ΔP o The "one-off" measurement (e.g., caused by noise or a brief malfunction) results. When several repeated measurements (e.g., at least 2, at least 5, at least 10) indicate that ΔP... m It was never significantly greater than ΔP o At that time, the mobile robot's state is only set to non-slip state.

[0118] Based on the determination at step 720 that the non-slip count counter is greater than the non-slip count threshold, process 700 sets the current state of the mobile robot from the slipping state to the non-slipping state at step 726, and sends the state (e.g., non-slipping) to the arbitrator (e.g., arbitrator 614) at step 728.

[0119] Based on the determination of ΔP m Ratio ΔP o After reaching the second threshold, process 700 returns to step 702 to continue obtaining OTS data and wheel encoder data from the buffer.

[0120] Figure 7B This is a flowchart illustrating an exemplary process for detecting wheel slippage using a visual odometry (VO) module according to some embodiments. In VO-based slippage detection methods (e.g.) Figure 7B In the process 730 shown, detection accuracy is improved when the VO measurement is qualified before being used as input for slip detection. VO measurements can occasionally be invalid; invalid VO measurements contain values ​​that deviate from valid measurements. Such VO attitude anomalies can lead to incorrect slip detection results and ultimately affect system performance. Furthermore, for attitude fusion processes (e.g., Figure 2A In step 222), VO pose outliers should also be excluded to ensure global consistency of the fused pose.

[0121] The process 730 for detecting wheel slippage begins at step 732, where two up-to-date consecutive VO outputs (e.g., two image frames) are obtained from a buffer (e.g., data buffer 608). In some embodiments, process 730 is performed as part of step 214. At step 734, it is determined whether the VO outputs just acquired are eligible for wheel slippage detection. In some embodiments, statistical tools such as RANSAC are used to determine whether the VO outputs contain outliers. Figure 7C An exemplary VO outlier detection method according to some embodiments is described.

[0122] The first step 762 in the process 760 for detecting VO outliers includes obtaining two latest consecutive VO outputs from the buffer. In some embodiments, step 762 is simply step 732. If process 760 is called from step 734 (of process 730), then step 762 has already been completed by step 732 (e.g., no additional VO output is needed from the buffer; the two latest consecutive VO outputs obtained during step 732 are used in step 762).

[0123] At step 764, wheel encoder readings corresponding to the acquired paired VO outputs are obtained. Based on the wheel encoder readings obtained from step 764, process 760 determines the motion pattern of the mobile robot at step 766. Further details regarding motion pattern detection are provided in the description of Figure 8.

[0124] At step 768, the feasible range of motion is calculated based on the motion pattern determined at step 766. It should be understood that the specific order or hierarchy of steps in the disclosed process is illustrative of an exemplary method. It should be understood that the specific order or hierarchy of steps in the process can be rearranged based on design preferences. Some of these steps may be performed simultaneously. Based on the two most recent consecutive VO outputs obtained at step 762, the relative motion ΔP is determined from the VO outputs at step 770. v (For example, the distance the robot moves between two VO image frames). In some embodiments, step 770 is performed in parallel with steps 764 to 768.

[0125] At step 772 of process 760, determine ΔP. v Whether it is within the feasible range (e.g., calculating the feasible range at step 768). When ΔP v When within the feasible range calculated in step 768, the two consecutive VO outputs obtained at step 762 are eligible as valid VO outputs at step 774 (for further use in process 730). When ΔP v If the two consecutive VO outputs obtained at step 762 are outside the feasible range calculated in step 768, they are not eligible at step 776, i.e., they are invalid VO outputs (and are discarded or excluded and not used further in process 730).

[0126] Return to Figure 7BAfter the VO output is qualified at step 734 (e.g., after completing step 762), process 730 proceeds to step 736, during which wheel encoder data corresponding to the VO output (obtained from step 732) is retrieved. In some embodiments, where step 734 includes completing process 760, the information obtained at step 736 has already been retrieved at step 766 (of process 760). At step 738, process 730 begins kinematic-based VO jump detection. Kinematic-based VO jump detection is similar to the steps outlined in steps 766 and 768 of process 760. In step 739, process 730 determines whether a VO jump has occurred. A VO jump occurs when the VO output obtained from step 732 corresponds to a displacement change greater than the displacement change calculated from the wheel encoder readings. Interpreting the wheel encoder readings under the assumption that no wheel slippage has occurred (e.g., assuming that all wheel rotations recorded by the wheel encoder correspond to the displacement of the mobile robot), if the VO output shows a motion significantly greater than the motion provided by the wheel encoder readings, process 730 determines that a VO jump has occurred. In some embodiments, step 734 may be performed using only RANSAC (e.g., process 760 is not performed at step 734), and steps 736, 738, and 739 are performed to determine whether a VO jump has occurred. In some embodiments, process 760 is used to perform step 734. In such embodiments, once process 760 has determined that two latest consecutive VO poses are eligible for wheel slip detection, process 730 may proceed directly to step 740.

[0127] Based on the determination that a VO jump has occurred at step 739, process 730 returns to step 732 to obtain new paired VO outputs. The previous paired VO outputs corresponding to the VO jump are discarded from wheel slip detection or otherwise excluded.

[0128] At step 740, the relative motion (ΔP) of the mobile robot, as determined by the VO output, is determined. v ) and the relative motion (ΔP) of the mobile robot, as determined by a wheel encoder. m In some embodiments, ΔP v This is determined to be part of step 738 or step 770 and stored in local memory so that only ΔP is calculated in step 740. m .

[0129] Based on the determination at step 742 that the previous state of the mobile robot was not a slipping state (e.g., the previous state of the mobile robot was set to a non-slipping state before any slippage was detected), process 730 checks ΔP at step 744. v Is it significantly smaller than ΔP? m In some embodiments, when ΔPv Ratio ΔP m When ΔP exceeds the third threshold, v Significantly smaller than ΔP m In some embodiments, when the ratio ΔP v / ΔP m When the threshold range is between 0.001 and 0.1, ΔP v Significantly smaller than ΔP m In some embodiments, the ratio ΔP v / ΔP m A value less than 0.01 indicates that the mobile robot is experiencing a significant wheel slippage event. In some embodiments, the third threshold is set to 0.15m.

[0130] Based on the determination of ΔP v Uncompared ΔP m If the third threshold is small, process 730 returns to step 732 to continue obtaining new pairs of consecutive outputs from the buffer. This is based on the determined ΔP. v Ratio ΔP m At step 748, a third threshold is reached, and the slip count counter is incremented by 1. Based on the determination at step 750 that the slip count counter is not greater than the slip count threshold, process 730 returns to step 732 to continue obtaining new pairs of consecutive outputs from the buffer. Based on the determination at step 520 that the slip count counter is greater than the slip count threshold, process 730 sets the current state of the mobile robot to a slip state at step 758 and sends the state (e.g., slip) to an arbitrator (e.g., arbitrator 614) at step 760. In some embodiments, the value of the slip count threshold is the same in processes 730 and 700. In some embodiments, the value of the slip count threshold differs between processes 730 and 700 to account for differences between the sensing modules used in these processes (e.g., expected noise from different measurements).

[0131] In some embodiments, a slippage count counter is used, by subtracting ΔP v Significantly smaller than ΔP m "One-off" measurements (e.g., those caused by noise or transient malfunctions) are used to improve the accuracy of wheel slippage detection. When several repeated measurements (e.g., at least 2, at least 5, at least 10) indicate that ΔP... v Significantly smaller than ΔP m At that time, the mobile robot's state is only set to slipping.

[0132] Based on the determination at step 742 that the previous state of the mobile robot was slipping, process 730 checks ΔP at step 746. v Is it significantly smaller than ΔP? m In some embodiments, when ΔPm Ratio ΔP o When ΔP exceeds the third threshold, v Significantly smaller than ΔP m .

[0133] Based on the determination of ΔP v No comparison with ΔP w If the third threshold is small, process 730 increments (1) the counter indicating the number of non-slip counts at step 752. Based on the determination at step 754 that the non-slip count counter is not greater than the non-slip count threshold, process 730 returns to step 732 to continue obtaining new (e.g., latest) paired VO outputs from the buffer. In some embodiments, the non-slip count counter is used to improve the accuracy of wheel slip detection by reducing the chance of an erroneous change from a "slipping" state to a "non-slipping" state, where such erroneous change is determined by ΔP. v Not significantly less than ΔP w The "one-off" measurement (e.g., caused by noise or a brief malfunction) results. When several repeated measurements (e.g., at least 2, at least 5, at least 10) indicate that ΔP... v It was never significantly less than ΔP m At that time, the mobile robot's state is only set to non-slip state.

[0134] Based on the determination at step 754 that the non-slip count counter is greater than the non-slip count threshold, process 730 sets the current state of the mobile robot from the slipping state to (e.g., changes to) the non-slipping state at step 756, and sends the state (e.g., non-slipping) to the arbitrator (e.g., arbitrator 614) at step 760.

[0135] Based on the determination of ΔP v Ratio ΔP m If the third threshold is reached, process 730 returns to step 732 to continue obtaining new (e.g., the latest) paired VO outputs from the buffer.

[0136] Figure 8A An exemplary differential-driven robot according to some embodiments is depicted. The robot 800 includes a right wheel 802 and a left wheel 804, which are mounted on a common shaft 806 having a length l. The right wheel 802 is driven by a motor to a linear velocity v. r The left wheel 804 is driven by a motor at a linear velocity v. l The differential drive robot controls its posture by providing independent speed control to its left wheel 802 and its right wheel 804. R is the signed distance from the instantaneous center of curvature (ICC) 808 to the midpoint between the left wheel 804 and the right wheel 802.

[0137] Figure 8BDifferent motion patterns of a mobile robot according to some embodiments are described. Figure 8B Figure 820 shows four different types of movement modes: forward, clockwise, counterclockwise, and backward, which depend on the speed v of the right wheel. r And the speed v of the revolver l When v r and v l When both are positive, the mobile robot's motion pattern is forward (in the first quadrant). When v r and v l When both are negative, the mobile robot's movement pattern is backward (in the third quadrant). r For positive and v l When v is negative, the mobile robot's movement pattern is counterclockwise (in the second quadrant). r For negative sums and v l When the time is positive, the movement pattern of the mobile robot is clockwise (in the fourth quadrant).

[0138] In some embodiments, the mobile robot moves in a manner that mixes (e.g., different) motion patterns within a single detection cycle. In such embodiments, the system uses a more lenient estimate of the feasible motion region.

[0139] Equations (13) and (14) describe the feasible motion regions in the first and third quadrants.

[0140]

[0141]

[0142] T is the time interval between paired VO outputs. l is the length of the axle connecting the left and right wheels, r is the radius of the (maximum) trajectory covered by the mobile robot during time T, and θ is the angle traversed by the mobile robot during time T. It is the maximum linear velocity detected by the wheel encoder of the left wheel 804. It is the maximum linear velocity detected by the wheel encoder of the right wheel 802. It is the minimum linear velocity detected by the wheel encoder of the left wheel 804. The minimum linear velocity is detected by the wheel encoder of the right wheel 802. When and / or When it is negative, the minimum linear velocity corresponds to the highest negative value among linear velocities.

[0143] Based on equations (13) and (14), the feasible motion regions in the first and third quadrants are sector regions (e.g., the angle of the sector is defined by θ), and the radius of the sector region is determined by r.

[0144] Equations (15) and (16) describe the feasible motion regions in the second and fourth quadrants.

[0145]

[0146] θ∈[0, 2π] (16)

[0147] exist Between and in The maximum value between these two values ​​is selected to estimate the maximum distance the robot can travel in the second and fourth quadrants.

[0148] The feasible motion region of the mobile robot in the second and fourth quadrants is a 2D spherical shape. Equation (16) shows that in some embodiments, if the mobile robot's speed is high enough, the mobile robot can rotate through the sphere (a circular area anywhere within a 360° range).

[0149] Equations (13) through (16) estimate the motion between two wheel encoder readings, or the relative motion of a "step" between two readings. In some embodiments, the wheel encoder readings are recorded at a higher frequency than the camera sampling rate that generates the VO readings. For example, the wheel encoder may have a frequency of 50 Hz and the camera sampling rate is 10 Hz. In this case, a relative motion of 50 / 10 = 5 small steps is considered in the calculation. In the ideal case (no slippage), the 5 wheel encoder readings are integrated to produce the relative motion. However, for cases involving wheel slippage, a smaller range of "possible" motion is integrated to produce a larger range so that the wheel encoder data is comparable to the VO data. Equation (16) provides a very coarse estimate. While a narrow angular range can be calculated, the integration calculation within such a "range" can be complex due to the irregular shape of the trajectory. Furthermore, considering that the wheel speed can be different for each step, the accumulated shape is complex and computationally intensive for tracking. The estimate provided in Equation (16) offers a faster and simpler solution.

[0150] When a hybrid motion pattern exists during a single detection cycle (e.g., the mobile robot transitions from the first quadrant to the second quadrant within time T), equations (16) and (17) describe feasible motions for such a hybrid motion pattern:

[0151]

[0152] For hybrid motion modes, the feasible region for hybrid motion estimation is a more conservative 2D spherical shape. In some embodiments, more conservative means that the theoretical range of motion should be smaller than the calculated sphere (e.g., a circular shape), which is controlled by r from equation (17), providing an envelope that includes all possible motion outcomes (e.g., hybrid motion and single motion, with / without wheel slippage). Equation (17) can be considered a superset of equations (13) through (16). Equations (13) through (16) provide better estimates than equation (17) when one or more conditions are met (e.g., the robot performs single-mode motion within a time window T). Equation (17) handles the remaining cases not covered by equations (13) through (16). Equation (17) produces a larger radius range than the radius range calculated from equation (15) (e.g., comparing the second and fourth quadrants). Furthermore, the angular range associated with the hybrid motion mode is larger than the angular range calculated from equation (14) (e.g., comparing the first and third quadrants). In some embodiments, a trade-off is made for a larger radius and angle range to simplify range estimation for complex hybrid motion mode cases.

[0153] Extended Kalman filter for VO and IO attitude fusion process

[0154] In step 222 (in Figure 2A In some embodiments, system 200 performs asynchronous attitude fusion. In some embodiments, an extended Kalman filter (EKF) is used to perform asynchronous attitude fusion. In some embodiments, the process model includes data recorded by the inertial odometry module and the OTS module. The process model can be represented as:

[0155] x k =f(x) k-1 u k )+w k (18)

[0156]

[0157] x k It represents the pose of a mobile robot in 2D, and it has three components: P x,k P y,k and θ x,k The vector, where P x,k and P y,k Let θ represent the robot's x and y coordinates (position), respectively. x,kThis represents the robot's orientation (e.g., azimuth). Unlike the earlier chapters that computed and processed the entire pose in 3D space, this chapter (Extended Kalman Filter for VO and IO Pose Fusion Process) computes the pose only in 2D space. This is a sufficient approximation for robot applications on flat ground, simplifying the computation by considering only some components of the 3D pose and applying the EKF only in 2D.

[0158] In some embodiments, the measurement model includes data recorded by the visual odometry module and includes scaling correction. The measurement model can be represented as:

[0159]

[0160]

[0161]

[0162] Where F k H is the Jacobian matrix used in process models. k It is the Jacobian matrix used to measure the model. (twenty three)

[0164] Using EKF, predictions can be made as follows:

[0165]

[0166] P k - =F k P k-1 F k T +Q k-1 (25)

[0167] This allows for the following measurements to be provided:

[0168] K k =P k - H k T (H k P k - H k T +R k ) -1 (26)

[0169]

[0170] P k =(IK k H k )P k- (28)

[0171] Where P k It is the revised state covariance matrix, K k It is the Kalman gain, Q k It is a 3×3 covariance matrix used for the process model, which quantizes the variance of the wheel encoder and IMU gyroscope measurements, and R... k It is a 3×3 covariance matrix used to measure the model, which quantifies the variance of position and orientation measured by the VO module. Q k and R k Both are 3×3 matrices in the Extended Kalman Filter (EKF) pipeline.

[0172] Figure 9 An exemplary flowchart of a process 900 for locating a mobile robot in an environment, according to some embodiments, is shown. According to some embodiments, process 900 may optionally be performed by an onboard processing unit of the mobile robot. In some embodiments, process 900 is performed by a server communicating with the mobile robot. In some embodiments, process 900 is jointly performed by the mobile robot and a server communicating with the mobile robot via a network. For ease of description, the steps of process 900 are described in relation to a mobile robot; this is not limiting, but rather is to be understood that, according to various embodiments, the steps of process 900 are performed by a server, or jointly by the mobile robot and the server.

[0173] As a first step, the mobile robot captures (902) multiple image frames recorded in a first coordinate reference system (e.g., the first coordinate reference system is a visual coordinate reference system) at corresponding locations within a portion of the environment during a first time period using a camera.

[0174] Then, the mobile robot, via an inertial measurement unit (IMU) mounted on the device, captures (904) an inertial odometry dataset recorded in a second coordinate reference system (e.g., the second coordinate reference system is a world coordinate reference system), the inertial odometry dataset corresponding to multiple image frames at corresponding locations within a first time period (e.g., multiple image frames collected at a different frequency compared to an inertial odometry dataset collected by the IMU at a higher frequency than that collected by the camera). The mobile robot stores (906) matching pairs of image frames (e.g., corresponding image frames among multiple image frames) and inertial odometry datasets satisfying a first criterion in a buffer; (e.g., pairing the first frame from the camera with the fifth IMU dataset to form a matching pair). Based on determining a threshold number of stored matching pairs of image frames and inertial odometry data (908): the mobile robot determines (910) a rotation transformation matrix corresponding to the relative rotation between the first and second coordinate reference systems; and determines (912) a scaling factor based on the matching pairs, wherein the rotation transformation matrix defines the orientation of the device, and the scaling factor and rotation transformation matrix calibrate the multiple image frames captured by the camera.

[0175] In some embodiments, the mobile robot further determines a translational transformation vector based on a matching pair, wherein the rotational transformation matrix and the translational transformation vector define a transformation posture for transforming a first coordinate reference frame to a second coordinate reference frame.

[0176] In some embodiments, the scaling factor and translation vector are solved as parameters of a least squares problem.

[0177] In some embodiments, the mobile robot further determines the calibration quality of a corresponding image frame by determining the error between the adjusted visual pose obtained by applying a scaling factor, a rotation transformation matrix, and a translation transformation vector to the corresponding image frame in a matched centering process and the inertial odometry dataset corresponding to the corresponding image frame.

[0178] In some embodiments, the mobile robot further captures additional image frame sequences and additional set sequences to determine an updated scaling factor until the calibration quality meets a second threshold. In some embodiments, determining the rotation transformation matrix includes determining the inverse rotation of the relative rotation between the device's main reference frame and the camera's camera reference frame.

[0179] In some embodiments, the first criterion includes the relative distance between two adjacent inertial odometry datasets that is greater than a minimum distance and less than a maximum distance (e.g., the first criterion includes the relative distance between a (matched pair) inertial odometry dataset and an adjacent inertial odometry dataset (e.g., the preceding inertial odometry dataset, such as the immediately preceding inertial odometry dataset)).

[0180] In some embodiments, the mobile robot further determines an optimized scaling factor by determining the residual in the scaling factor (e.g., summing over multiple poses eligible for optimization), wherein the optimized scaling factor includes multiplying the residual by the scaling factor (e.g., further including multiplying the learning rate by the residual and the scaling factor; setting the learning rate based on the noise level of the image frame and the inertial odometry dataset).

[0181] In some embodiments, determining the residual includes summing the product of (i) the relative translation between adjacent image frames and (ii) the relative translation between adjacent inertial odometry datasets.

[0182] In some embodiments, the optimized scaling factor is determined based on the fact that the residual in the scaling factor is greater than a threshold and the number of inertial odometry datasets and optimized paired image frames satisfies a second threshold (e.g., the second threshold corresponds to the length of the sliding window receiving the optimized pair).

[0183] In some embodiments, corresponding image frames in optimized pairs of image frames are obtained by applying a scaling factor, a rotation transformation matrix, and a translation transformation vector to the corresponding image frames captured by the camera; wherein, the optimized pairs of image frames and inertial odometry datasets are stored in a sliding window based on determining that the optimized pairs meet a second criterion (e.g., the second criterion is related to the relative angle and distance of continuous attitudes).

[0184] Figure 10 An exemplary flowchart of a process for detecting wheel slippage according to some embodiments is shown. According to some embodiments, process 1000 may optionally be executed by an onboard processing unit of a mobile robot. In some embodiments, process 1000 is executed by a server communicating with the mobile robot. In some embodiments, process 1000 is jointly executed by the mobile robot and a server communicating with the mobile robot via a network. For ease of description, the steps of process 1000 are described in relation to a mobile robot; this is not limiting, but rather is to be understood that, according to various embodiments, the steps of process 1000 are executed by a server, or by a mobile robot and a server jointly.

[0185] The mobile robot captures (1002) multiple optical data at corresponding locations within a part of the environment by means of optical sensors set up on devices that move in the environment.

[0186] The mobile robot captures (1004) an encoder dataset corresponding to multiple optical data points at corresponding locations using a wheel encoder mounted on the device (e.g., corresponding means that the optical data is measured at the same time point as the encoder data, or at a different time point than the encoder data but within the same time period). Based on the multiple optical data, the mobile robot determines (1006) a first relative motion. Based on the encoder dataset, the mobile robot determines (1008) a corresponding second relative motion.

[0187] Based on the determination that the difference between the first relative motion and the corresponding second relative motion is greater than a first threshold (1010): the mobile robot increments the counter indicating a slip event of the wheel encoder (1012), wherein the slip event corresponds to the wheel of the device moving forward and the second relative motion is less than a second threshold. {The wheel encoder counts the number of rotations of the motor associated with the device; a slip event occurs when the device is stuck in a certain area (e.g., a position, or a limited range of positions), even after the motor has rotated multiple times and the device remains stuck in that area}.

[0188] In some embodiments, when the counter is higher than a second threshold, the device is set to a first state, and when the device is set to the first state, the mobile robot excludes multiple optical data without further processing (e.g., the second threshold specifies a lower limit of slip events that must be detected within a certain period of time before the device is set to a slip state).

[0189] In some embodiments, the optical sensor includes a camera, and the optical data includes image frames captured by the camera. In some embodiments, the mobile robot determines whether the device's position jumps between two adjacent frames in the image frames captured by the camera.

[0190] In some embodiments, a jump occurs when additional image frames are captured by the camera until the position of the device is not shown in two adjacent frames in the captured image frames before the first relative motion is determined.

[0191] In some embodiments, the mobile robot further determines whether two adjacent frames in the image frames captured by the camera qualify as valid measurements before calculating the first relative motion.

[0192] In some embodiments, the mobile robot further obtains wheel encoder readings between two adjacent frames in the captured image frames; determines a motion pattern based on the wheel encoder readings; and determines a feasible motion range based on the motion pattern. If it is determined that a first relative motion between two adjacent frames is outside the feasible motion range, then the two adjacent frames in the image frames are disqualified as valid measurements; and additional image frames are captured by the camera until the two adjacent frames in the captured image frames contain relative motion within the feasible motion range.

[0193] In some embodiments, the motion mode includes one or more of a forward mode, a clockwise mode, a backward mode, and a counterclockwise mode. In some embodiments, the optical sensor includes an optical tracking sensor, and determining the first relative motion includes integrating measurements captured by the optical tracking sensor. In some embodiments, the mobile robot captures a sequence of image frames corresponding to an optical data sequence and a sequence of encoded data recorded at corresponding locations within a portion of the environment via a camera mounted on the device.

[0194] Figure 11 A block diagram of an apparatus 1100 according to some embodiments is shown. In some embodiments, apparatus 1100 includes a mobile robot 102. In some embodiments, apparatus 1100 includes a server communicating with the mobile robot. According to some embodiments, apparatus 1100 represents an execution Figure 10 The process of electronic devices in 1000.

[0195] The device 1100 includes one or more processors 1102, one or more communication interfaces 1104 (e.g., network interfaces), memory 1106, and one or more communication buses 1108 for interconnecting these components (sometimes referred to as chipsets).

[0196] In some embodiments, the device 1100 includes an input interface 1110 that facilitates user input.

[0197] In some embodiments, the device 1100 includes one or more cameras 1118. In some embodiments, the camera 1118 is configured to capture color images. In some embodiments, the camera 1118 is configured to capture black and white images. In some embodiments, the camera 1118 captures images with depth information.

[0198] In some embodiments, the device 1100 includes one or more inertial measurement units 1666. In some embodiments, the inertial measurement unit 1166 is configured to measure and report specific forces, angular rates, and orientation of the robot 102 (sometimes measured and reported) using a combination of one or more of accelerometers, gyroscopes, and magnetometers (sometimes used).

[0199] In some embodiments, device 1100 includes a battery 1112. Device 1100 also includes sensors 1120, such as a light sensor 1122, a pressure sensor 1124, a humidity sensor 1126, an airflow sensor 1128, and / or a temperature sensor 1130, to facilitate tasks and operations of the mobile robot (e.g., cleaning, conveying, etc.). In some embodiments, device 1100 also includes a liquid reservoir 1134, a stirrer 1136, and / or a motor 1138 to perform cleaning tasks (e.g., sweeping, scrubbing, mopping, etc.).

[0200] In some embodiments, device 1100 includes wireless device 1130. Wireless device 1130 enables one or more communication networks and allows device 1100 to communicate with other devices (e.g., docking stations, remote control devices, servers, etc.). In some implementations, wireless device 1130 is capable of performing data communication using any of a variety of custom or standard wireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA 100.5A, WirelessHART, MiWi, Ultra Wideband (UWB), Software Radio (SDR), etc.), custom or standard wired protocols (e.g., Ethernet, HomePlug, etc.), and / or any other suitable communication protocol (including communication protocols not yet developed as of the filing date of this document).

[0201] Memory 1106 includes high-speed random access memory (e.g., DRAM, SRAM, DDR RAM) or other random access solid-state storage devices; optionally, memory 1106 includes non-volatile memory, such as one or more disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid-state storage devices. Optionally, memory 1106 includes one or more storage devices disposed remotely from one or more processors 1102. Memory 1106, or alternatively, the non-volatile memory within memory 1106 includes a non-transitory computer-readable storage medium. In some implementations, memory 1106, or the non-transitory computer-readable storage medium of memory 1106, stores, or stores subsets or supersets of, programs, modules, and data structures:

[0202] • Operational logic 1140 includes procedures for handling various basic system services and for performing hardware-dependent tasks;

[0203] • Communication module 1142 (e.g., wireless communication module) is used to connect to and communicate with other network devices (e.g., local networks, such as routers providing Internet connectivity, networked storage devices, network routing devices, server systems and / or other connected devices, etc.) coupled to one or more communication networks via communication interface 1104 (e.g., wired or wireless).

[0204] • Application 1144, for performing tasks and self-localization, and for controlling one or more components of device 1100 and / or other connected devices according to preset instructions;

[0205] • Device data 1138 of device 1100 includes, but is not limited to:

[0206] ○ Device settings 1156 of device 1100, such as default options for performing tasks;

[0207] ○ User settings 1158 for performing tasks;

[0208] ○ Sensor data 1160 acquired (e.g., measured) from sensor 1120;

[0209] ○ Camera data 1162 acquired from camera 1118; and

[0210] ○ The stored data 1164. For example, in some embodiments, the stored data 1164 includes semantic and feature maps of the environment according to some embodiments, camera pose, and map points of stored keyframes, etc.

[0211] Each of the executable modules, applications, or sets of procedures described above may be stored in one or more of the previously mentioned storage devices and corresponds to a set of instructions for performing the functions described above. The modules or programs (i.e., the instruction sets) described above do not need to be implemented as separate software programs, processes, or modules; therefore, in various implementations, various subsets of these modules may be combined or otherwise rearranged. In some implementations, memory 1106 stores a subset of the modules and data structures described above. Furthermore, memory 1106 may store additional modules or data structures not described above. In some embodiments, a subset of programs, modules, and / or data stored in memory 1106 is stored on and / or executed by a server system, and / or executed by a mobile robot. While some figures in the various drawings show multiple logical stages in a specific order, stages that are not dependent on order may be reordered, and other stages may be combined or separated. While some reorderings or other groupings are specifically mentioned, other orderings and groupings will be apparent to those skilled in the art, and therefore the orderings and groupings presented herein are not an exhaustive list of alternatives. Furthermore, it should be recognized that these stages can be implemented in hardware, firmware, software, or any combination thereof.

[0212] It will also be understood that, although in some instances the terms first, second, etc., are used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of the various implementations described, a first sensor may be referred to as a second sensor, and similarly, a second sensor may be referred to as a first sensor. Both the first sensor and the second sensor are sensors, but the first sensor and the second sensor are not the same type of sensor.

[0213] The terminology used in the description of the various implementations described herein is for the purpose of describing particular implementations only and is not intended to be limiting. As used in the description of the various implementations described and in the appended claims, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It will also be understood that the term “and / or,” as used herein, refers to and covers any and all possible combinations of one or more of the associated listed items. It will be further understood that, when used in this specification, the terms “comprising,” “including,” “containing,” and / or “having” indicate the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0214] As used herein, the term "if" may optionally be interpreted, depending on the context, as meaning "when," "at," "in response to determining," "in response to detecting," or "according to determining." Similarly, the phrases "if determining..." or "if [the stated condition or event] is detected" may optionally be interpreted, depending on the context, as meaning "when determining," "in response to determining," "when [the stated condition or event] is detected," "in response to detecting [the stated condition or event]," or "according to determining that [the stated condition or event] is detected."

[0215] For illustrative purposes, the foregoing description has been described with reference to specific implementations. However, the illustrative discussion above is not intended to be exhaustive or to limit the scope of the claims to the precise forms disclosed. Many modifications and variations are possible in light of the foregoing teachings. The implementations were chosen to best explain the underlying principles of the claims and their practical application, thereby enabling other skilled in the art to best utilize the implementations and various modifications to suit the intended particular use. The technical solutions in the embodiments of this application have been clearly and completely described above with reference to the accompanying drawings. The described embodiments are merely a part of the embodiments of this application, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort fall within the protection scope of this application. The appended method claims present the elements of each step in a sample order, but this does not imply limitation to the specific order or hierarchy presented.

Claims

1. A method comprising: By using optical sensors mounted on a device that moves within the environment, multiple optical data points at corresponding locations within a portion of the environment are captured; An encoder dataset is captured by a wheel encoder installed on the device, which captures the plurality of optical data corresponding to the respective locations. Based on the aforementioned optical data, the first relative motion is determined; Based on the encoder dataset, the corresponding second relative motion is determined; Based on the determination that the difference between the first relative motion and the corresponding second relative motion is greater than a first threshold: Increase the counter indicating a slippage event of the wheel encoder, wherein the slippage event corresponds to the wheel of the device moving forward, and the corresponding second relative motion is less than a second threshold. The optical sensor includes a camera, and the optical data includes image frames captured by the camera; Further includes: determining whether two adjacent frames in the image frames captured by the camera qualify as valid measurements before calculating the first relative motion; Further includes: obtaining wheel encoder readings between two adjacent frames in the image frames captured by the camera; The motion mode is determined based on the readings from the wheel encoder. Based on the described motion pattern, determine the feasible motion range; Based on the determination that the first relative motion between the two adjacent frames is outside the feasible motion range: This makes two adjacent frames in the image frame ineligible for valid measurement; and Additional image frames are captured by the camera until two adjacent frames in the captured image frames contain relative motion within the feasible range of motion.

2. The method according to claim 1, wherein, When the counter is higher than the second threshold, the state of the device is set to the first state, and the method includes: When the device is set to the first state, the plurality of optical data are excluded without further processing.

3. The method according to claim 1, further comprising: Determine whether the device's position jumps between two adjacent frames in the image frames captured by the camera.

4. The method of claim 3, further comprising: The jump occurs when additional image frames are captured by the camera until, prior to determining the first relative motion, two adjacent frames in the captured image frames do not show the position of the device.

5. The method according to claim 1, wherein, The movement modes include one or more of the following: forward mode, clockwise mode, backward mode, and counterclockwise mode.

6. The method according to claim 1, wherein, The optical sensor includes an optical tracking sensor, and determining the first relative motion includes integrating the measurements captured by the optical tracking sensor.

7. The method of claim 1, further comprising: The camera mounted on the device captures multiple image frames corresponding to an optical data sequence and an encoded dataset recorded at the corresponding location within a portion of the environment.

8. An electronic device, comprising: One or more processing units; Memory; as well as A plurality of programs stored in the memory, when executed by the one or more processing units, cause the one or more processing units to perform the method according to any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium storing a plurality of programs, said plurality of programs being executed by an electronic device having one or more processing units, wherein, When the plurality of programs are executed by the one or more processing units, the plurality of programs cause the processing units to perform the method according to any one of claims 1 to 7.

Images