Calibrating the Gyroscope and Optical Flow Sensor Scales

By calibrating gyroscopes using optical flow sensors through rotation-based integration and frequency sensing, the method addresses aging-related errors in consumer robots, enhancing navigation precision and odometry accuracy.

JP7590358B2Active Publication Date: 2024-11-26CEVA TECHNOLOGIES INC
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2021577607
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-06-28
Filing Date
2020-06-25
Publication Date
2024-11-26
Estimated Expiration
2040-06-25

AI Technical Summary

Technical Problem

Consumer robots face challenges in achieving high-precision navigation due to aging-related changes in gyroscope scales and variations in optical flow sensor performance, which affect the accuracy of sensor calibration and odometry measurements.

Method used

A method and system for calibrating gyroscopes using optical flow sensors by rotating the robot on a calibration surface, integrating gyroscope and image data to derive sensor calibration parameters, and employing techniques such as frequency sensing and autocorrelation to correct scale errors.

Benefits of technology

Enhances the precision of robotic navigation by accurately calibrating gyroscopes and optical flow sensors, compensating for aging and environmental changes, thereby improving the robot's ability to perform high-quality odometry and localization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007590358000047
    Figure 0007590358000047
  • Figure 0007590358000048
    Figure 0007590358000048
  • Figure 0007590358000049
    Figure 0007590358000049
Patent Text Reader

Abstract

A method for calculating a scale factor for a gyroscope may include detecting physical movement of the robot with a gyroscope, detecting one or more image signals containing information with an optical flow (OF) sensor (and / or a camera), and deriving predictions of sensor calibration parameters based on the detected physical movement and information.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] Related Applications This application claims priority to U.S. Provisional Patent Application No. 62 / 868,591, entitled “Methods and Apparatus for Mutual Gyroscope Scale and Optical Flow Sensor Scale Calibration Through Optical Flow Image Quality Metadata,” filed June 28, 2019, the disclosure of which is incorporated herein by reference. [Background technology]

[0002] It is becoming increasingly common and widespread for electronic devices, particularly mobile phones, mobile devices, digital cameras, global positioning system (GPS) units, laptop and palmtop computers, automobiles, robot vacuum cleaners, etc., to include sensors (e.g., motion sensors) to enhance their functionality. For example, sensors may include gyroscopes to measure angular velocity, accelerometers to measure linear acceleration, and / or optical flow (OF) sensors to measure the displacement of image features over time.

[0003] Many applications in surface robotics require the successful implementation of high quality odometry measurements. In some examples, a typical robot platform may include one or more, or any combination of the following sensing qualities: 1) inertial measurements (e.g., gyroscopes to measure angular velocity and / or accelerometers to measure linear acceleration), 2) linear motion prediction relative to the ground (e.g., wheel encoders to convert wheel rotations into linear distance and / or optical flow (OF) sensors to report relative motion of visible points on the ground surface), 3) distance prediction to obstacles in the environment (e.g., light detection and ranging (LIDAR), infrared proximity, ultrasonic), and / or 4) contact detection to obstacles in the environment (e.g., bumper switches).

[0004] A robotic system, such as a consumer robot (e.g., a planar robot, a robotic vacuum cleaner or RVC), may include one or more inertial measurement units (IMUs) (or IMU sensors) and one or more OF sensors for navigation and simultaneous localization and mapping (SLAM). The IMU, which may contain one or more accelerometers and gyroscopes (e.g., MEMS gyroscopes), provides linear acceleration and angular velocity measurements that can be used to obtain an orientation (direction) relative to an initial orientation of something. The OF sensor provides a measurement of the linear velocity of the sensor relative to a ground surface. In some examples, the OF sensor may measure the displacement of an image feature over time. For example, the OF sensor may report a two-dimensional (2D) displacement vector that represents the average number of feature pixels moved between successive sample images each time they are sampled. In some cases, the OF sensor may measure the displacement between two sample points in the x-axis and y-axis.

[0005] A consumer robot may obtain an orientation prediction by integrating scaled gyroscope measurements. The gyroscope scale may vary over the life of the part due to, for example, aging effects. OF sensors may provide image quality data (as well as linear motion relative to a surface).

[0006] For a robotic system, it is essential to obtain sensor calibration parameters to calibrate the sensors since accurate calibration is important to achieve high precision operation of the robotic system. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] U.S. Patent No. 8,073,564 Summary of the Invention [Means for solving the problem]

[0008] According to one embodiment, a method for scale calibration includes sensing physical motion of a robot by a gyroscope, sensing one or more image signals containing information by an optical flow (OF) sensor (and / or a camera), and deriving predictions of sensor calibration parameters based on the sensed physical motion and information.

[0009] According to another embodiment, a system for scale calibration includes a gyroscope configured to sense physical motion of a robot, an optical flow (OF) sensor and / or a camera configured to generate one or more image signals including information, and a processor configured to derive predictions of sensor calibration parameters based on the sensed physical motion and the information.

[0010] According to one embodiment, a method for calibrating a gyroscope disposed on a robot includes placing the robot on a calibration surface, rotating the robot on the calibration surface, determining a first orientation of the robot using the gyroscope, determining a second orientation of the robot using an image sensor, and calibrating the gyroscope based on the first orientation and the second orientation.

[0011] According to one embodiment, a system for calibrating a gyroscope disposed on a robot includes a calibration surface, a robot disposed on the calibration surface, a gyroscope disposed on the robot, an image sensor disposed on the robot, and a processor for generating a first orientation of the robot using data generated by the gyroscope and generating a second orientation of the robot using data generated by the image sensor, the processor further configured to generate calibration parameters for the gyroscope using the first orientation and the second orientation.

[0012] According to one embodiment, a method for calibrating a gyroscope disposed on a robot includes the steps of docking the robot to a docking station, where the robot obtains a first orientation based on information from the gyroscope, rotating the robot away from the docking station, redocking the robot to the docking station, where the robot obtains a second orientation based on information from the gyroscope, and calibrating the gyroscope based on a difference between the first orientation and the second orientation.

[0013] A more detailed understanding will be had from the following detailed description, given by way of example in conjunction with the accompanying drawings, in which: Figures in the description are examples; thus, the figures and detailed description should not be considered as limiting, other equally effective examples being possible and probable; furthermore, like reference numerals in the figures refer to like elements. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 illustrates a diagram of the physical placement of sensors in an example robotic system with symbolic references, in accordance with one or more embodiments. [Diagram 2] 1 is a chart illustrating an example of optical flow scale sensitivity, according to one or more embodiments. [Figure 3A] 1 is a diagram illustrating an example of an optical flow sensor operating on a soft surface, according to one or more embodiments. [Figure 3B] 1 is a diagram illustrating an example of an optical flow sensor operating on a hard surface according to one or more embodiments. [Figure 4] 1 is a diagram illustrating an example of a surface periodic calibration pattern, according to one or more embodiments. [Diagram 5] 1 is a graph illustrating an example of different luminance at a central pixel with respect to motion across a typical field of view, according to one or more embodiments. [Figure 6]FIG. 1 includes a graph showing an example of peak signal frequency preserved across field of view averages, in accordance with one or more embodiments. [Figure 7] FIG. 1 includes graphs illustrating an example of the effect of gyroscope scale error on measured signal and peak frequency in accordance with one or more embodiments. [Figure 8] FIG. 1 is a block diagram of an example of gyro scale calibration with a frequency sensing algorithm, according to one or more embodiments. [Figure 9A] 1 is a diagram of an example surface having natural wood grain. [Figure 9B] 9B is a graph illustrating an example of average intensity sensed as a function of angular / orientation position of an optical flow sensor operating on a surface having natural wood grain (e.g., of FIG. 9A ) in accordance with one or more embodiments. [Figure 10] 13 is a graph showing an example of overall autocorrelation results for circular motion on a wood surface (e.g., without scale error) in accordance with one or more embodiments. [Figure 11A] 1 is a diagram illustrating an example of an engineering pattern for a periodic sensing technique, according to one or more embodiments. [Figure 11B] 11B is a graph illustrating an example of average intensity sensed as a function of angular / orientation position of an optical flow sensor operating on a surface having an engineered pattern (e.g., of FIG. 11A ) in accordance with one or more embodiments. [Figure 12] FIG. 1 includes graphs showing an example of autocorrelation and DFT of motion on an engineered surface without peak frequencies being compromised by global alignment bars, in accordance with one or more embodiments. [Figure 13] 13 is a graph showing an example of the CDF of error introduced by non-circularity, including the situation of autocorrelation when motion-induced changes do not accumulate in the underlying signal, in accordance with one or more embodiments. [Figure 14]FIG. 1 is a block diagram of an example of a gyroscope scale calibration algorithm using image quality autocorrelation, according to one or more embodiments. [Figure 15] FIG. 2 is a block diagram illustrating an example physical arrangement for reference mark-based angular distance sensing in accordance with one or more embodiments. [Figure 16A] FIG. 2 is a block diagram illustrating an exemplary mark sensing mechanism, according to one or more embodiments. [Figure 16B] FIG. 2 is a block diagram illustrating an exemplary mark sensing mechanism, according to one or more embodiments. [Figure 17] 1 is a diagram illustrating an example of a relationship between a reference mark orientation and a sensor orientation, according to one or more embodiments. [Figure 18] 1 is a diagram illustrating an example of a circular motion where the center of rotation is not at the center of the robot, in accordance with one or more embodiments. [Figure 19] FIG. 1 is a block diagram representation of a robot that may be used in connection with the described embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] The following detailed description of the invention refers to the accompanying drawings, in which the same reference numbers in different drawings identify the same or similar elements, and the following detailed description is not intended to limit the invention, the scope of the invention being instead defined by the appended claims.

[0016] As previously mentioned, a robotic system such as a robotic vacuum cleaner may have an IMU including one or more accelerometers and / or gyroscopes to provide orientation information for use in controlling the robot. The robot may include or incorporate other sensors, such as OF sensors that measure the displacement of image features over time. A planar robotic system may also have a controller, such as a processor, that has access to data from all sensors and can therefore calculate the motion and position and / or orientation state of the robot. For example, if both the wheel encoders and the camera indicate no movement and all motors are idle, the controller may reasonably assume that the robot is stationary.

[0017] In addition, the robot may include motors for moving the robot in any number of ways, including rotation about any axis and / or translation in any direction, and / or rotation and translation of parts of the robot relative to other parts of the robot (e.g., a mechanical arm that moves relative to the body of the robot). The controller may also be adapted to direct the motion of the robot by controlling the motors.

[0018] In various embodiments, the OF sensor and the IMU may be on the same robot, so their angular velocities should be generally the same, and their motion parameters may be related such that one may be able to be calibrated using the other. For example, the OF sensor and the IMU may be on the same rigid body (e.g., a planar robotic system), and the OF sensor momentum and the IMU momentum (e.g., their velocities) may be related by a rigid body equation. In one example, the robot's controller (or other processing device) may use data from the IMU or its sensors and / or other sensors and components (e.g., cameras, wheel encoders) to improve the calibration of the OF sensor. In another example, the controller (or other processing device) may use data from the OF sensor to calibrate the robotic system's IMU (e.g., the IMU's gyroscopic scale). For example, if the OF sensor accuracy is good enough, the OF sensor may be used to calibrate the robotic system's gyroscopic scale.

[0019] Sensor System Description An example diagram of the physical placement of sensors in a robotic system is shown in Figure 1. The diagram in Figure 1 summarizes the notation used to relate various quantities and frames of reference to each other.

[0020] Conventions

[0021]

number

[0022] is the vector measured in the frame of reference B at point A

[0023]

number

[0024] The robot frame is specified by R. The rotation of the vector from R to the IMU (I) and optical flow (O) frame is given by the direction cosine matrix R R→I and R R→O For example, these can be expressed as follows:

[0025]

number

[0026] We define the Z axis of the robot as perpendicular to the ground surface on which the robot drives. In some cases, the Z axis of the optical flow sensor must also be perpendicular to the ground for accurate sensor operation. The IMU displacement and the optical flow sensor displacement relative to the robot center are respectively

[0027]

number

[0028] ,

[0029]

number

[0030] It is represented by:

[0031] Under the rigid body assumption, the angular velocity in the global frame is the same at every point. For the common case where motion is constrained to the XY plane, the angular velocity

[0032]

number

[0033] The x and / or y components of may be ignored, i.e.

[0034]

number

[0035] and

[0036]

number

[0037] It is noted that this property can be independent of the reference frame if the Z-axes of these frames are all aligned (e.g., to the Z-axis of the Earth frame). For simplicity, ω is the angular velocity vector

[0038]

number

[0039] or angular velocity ω, and which one is represented can be inferred from the context. In some examples, the quantities can be expressed in absolute Earth frame terms. In some examples, the quantities (e.g., for measurements) are not affected by a linear position or initial orientation in the global coordinate frame.

[0040] Gyroscope Error Sources In various embodiments, for planar motion, the true angular rate ω may be related to the gyroscope measurements x by the relationship:

[0041]

number

[0042] ω zro (k) is the zero rate offset (ZRO) at temperature k, which is the angular rate sensed and reported at rest for the device. More specifically, a stationary gyroscope (e.g., a MEMS gyroscope) typically reports a non-zero angular rate (ZRO, which is the error), and ZRO typically varies as a function of sensor temperature.

[0043] In general, 3-axis gyroscope measurements require calibration of scale, orthogonal axis skew, and rotation parameters to accurately relate the sensor output to physical motion. Assuming planar motion, only one axis (Z) has meaningful data, and these calibration terms can be absorbed into a single scale value α. In practice, many consumer surface robots use single-axis gyroscopes for cost-reducing or accuracy-improving reasons.

[0044] In various embodiments, ω zro Although (k) can be measured, characterized, and / or compensated for, there are several other error sources that should be taken into account. For example, one error source is the effect of sensor aging. More specifically, the gyroscope scale can change over time due to the accumulation of mechanical stress from thermal cycling. In some cases, current commercial-grade MEMS gyroscopes have a typical aging effect of approximately 1%.

[0045] Sensor nonlinearity errors are another source of error that can be introduced when the assumption that gyroscope measurements are linearly related to angular rate does not hold. In general, for current commercial MEMS gyroscopes, the effects of nonlinearity are small, so a simple linear scale may be sufficient for practical purposes.

[0046] Another common concern with MEMS gyroscopes is their sensitivity to linear acceleration (e.g., linear acceleration causes a change in the bias of the angular rate output). However, for planar motion, the largest actual contribution to linear acceleration is gravity, and in most planar motion scenarios, orientation relative to gravity does not change under normal circumstances, and the effect of linear acceleration is relatively small for current commercial MEMS gyroscopes. Therefore, sensitivity to linear acceleration is unlikely to cause significant errors in the gyroscope readings.

[0047] For this reason, various embodiments discussed herein focus on suppressing aging-induced scale changes (as opposed to other sources of error) in field-deployed robots. In some cases, for simplicity, the above equations assume that ZRO has been measured and removed from the gyroscope measurements, and that some range of the nominal scale is known (e.g., the manufacturer provides the average, maximum and / or minimum scale values ​​for each part).

[0048] Optical flow scale sensitivity Optical flow sensors operate by detecting the relative motion of surface features projected onto an image detector. The image detector or sensor chip may be connected to a processor that executes optical flow algorithms to output measurements related to the detected visual motion of, for example, a robot to which the image sensor chip is attached. Figure 2 shows diagrammatically how the distance between the image sensor (top) and the sensing surface (bottom) affects the observed motion of a single tracked point for a sensor with a fixed focal length.

[0049] Although the optical flow sensor is typically installed at some nominal height (e.g., to keep the tracked surface in focus of a fixed lens), different effective surface heights may appear when moving between hard and soft surfaces or when the robot suspension changes (e.g., due to a change in load weight). With reference to Figures 3A and 3B, the grey rectangle represents a side-on view of the robot's wheel, and the triangle indicates the nominal field of view of the optical flow sensor. Figures 3A and 3B show how the field of view of the OF sensor can change between soft surfaces (e.g., carpet) and hard surfaces (e.g., tiles), respectively.

[0050] Related Methods Spatially varying signals may be used to perform position measurements or sensor calibration. Robotic systems may use OF image quality data to calibrate IMUs. Localization of devices by correlation with previously measured RF or magnetic "fingerprints" has been well known in the literature for decades. New or improved approaches focus on exploiting known spatial features of signals rather than attempting to localize sensors by identifying signal characteristics similar to previously observed signals. In some instances, there is an advantage to constrained motion (e.g., rotation on a plane) that associates signals with a particular trajectory, but would not be practical for free motion in space.

[0051] Some current implementations use calibration of the gyroscope scale with magnetometer measurements. The basic assumption in this approach is that the horizontal component of the uncalibrated Earth's magnetic field is repeatable when the device is oriented in the same direction multiple times (e.g., rotated 360°). The underlying magnetic fields used to implement this method may not be stable enough in practice to provide good performance for surface-bound robotic motion. The effects of construction materials (e.g., wiring, metallic reinforcements) and the robot itself (e.g., rotating motors) may create a time-varying magnetic field environment or sufficient displacement to limit the effectiveness of this approach.

[0052] Gyroscope scale estimation from surface image quality measurements In some examples, a standard optical flow sensor may output not only position change over time, but also various image quality metrics. These are typically used to adjust illumination sources or to roughly gauge how reliable the position output can be. The reported values ​​may include one or more of the following: minimum pixel brightness, maximum pixel brightness, average pixel brightness, "surface quality" (typically a proxy for the number of trackable points in the field of view), and / or image sensor gain.

[0053] These image quality signals change as the optical flow sensor moves over the surface. If the relationship between position and image quality is known, this can be exploited to estimate the motion of the connected gyroscope. If the true motion of the gyroscope is predicted, it becomes possible to calibrate the gyroscope. In the case of circular motion, the robotic system may estimate the gyroscope scale by sensing either the frequency of a pattern that repeats in the spatial domain, the period of a pattern that repeats once for every rotation of the device, or the position of a discrete reference mark. Estimating the gyroscope scale is described in more detail below.

[0054] An exemplary embodiment of the technique disclosed above captures the image quality output of the OF sensor and the angular rate output of the gyroscope while rotating a wheeled robot in place, and then combines / compares this data to characterize the gyroscope behavior.

[0055] Frequency detection of patterns on engineered surfaces. In one embodiment, if a device of a robotic system can be rotated (or configured to be) on top of an engineering calibration surface (e.g., with a fiducial marker detectable by an OF sensor), the robotic system can use the observed correspondence between the gyroscope-derived orientation (integrated gyroscope output) and the expected signal characteristics associated with the calibration surface for calibration. FIG. 4 shows one such engineering calibration surface, where the brightness of a point seen by the optical flow sensor corresponds to the detected brightness of that point. For example, in FIG. 4, 53 bands radiate from the center. When the OF sensor moves through a circle centered on this pattern, the brightness of the center pixel of the OF sensor is defined by its angular position relative to the pattern. For a robot that rotates its center of rotation (CoR) at the center of the pattern, the observed brightness at any non-zero radius of the OF sensor will show a triangular pattern in the spatial domain with a frequency of 53 cycles per 360°. This holds true because the brightness is constant along each line radiating outward from the center. The average luminance over the field of view of the OF sensor can be a smoothed version of this. Figure 5 shows the ideal luminance vs. position function for this pattern, as well as some common image quality metrics reported by an exemplary OF sensor (e.g., lowest pixel luminance, highest pixel luminance, average pixel luminance over its field of view).

[0056] In various embodiments, the average luminance of the sensing surface is used to generate a signal suitable for various pitch detection algorithms. The approaches described herein are not limited to a particular image quality metric or pitch detection method. In some examples, instead of using "average luminance over the field of view" and / or discrete Fourier transform (DFT) techniques, another pitch detection technique or image quality metric may be used to achieve the same or similar results.

[0057] In one example, FIG. 6 demonstrates the relationship between the original signal at a single pixel, the average pixel value in a 1 cm square window moving on a surface with a radius of 10 cm in the spatial domain (top), and a subset of the DFT output of the same signal with peak frequencies indicated in the frequency domain (bottom). If the gyroscope and OF sensor are attached to a rigid body and the rigid body moves in a circle around the pattern center, integrating the gyroscope output signal should yield the orientation of the OF sensor (e.g., the orientation between the gyroscope and OF sensor) relative to the pattern modulo a constant angular offset and the unknown initial orientation. This analysis does not require that the initial orientation or true displacement between the involved sensors or between the sensor and the center of rotation be known. In one embodiment, the robotic system performs a DFT on the image quality signal to obtain the dominant frequency of the image quality signal versus the gyroscope angular position in the domain of the gyroscope-derived orientation. Note that this calculation is independent of the initial orientation. From there, the robotic system can then calculate a scale value for the gyroscope-derived orientation that corresponds accurately to the (e.g., known) true pattern frequency obtained from the OF sensor. For example, if the DFT results in the dominant frequency of the OF output corresponding to a 360° rotation measured by the gyroscope being 52 instead of the expected 53, then scaling the gyroscope output by 53 / 52 (i.e., 1.10923) will provide an accurate overall orientation that does not accumulate error with the angular distance traveled.

[0058] FIG. 6 shows the OF intensity metadata output in the position and frequency domains as one moves over the pattern shown in FIG. 4. So the angular position sensed using the gyroscope is the true angular position in this case. The peak at 53 cycles / revolution indicates that 53 peaks are counted for every revolution of the OF sensor. FIG. 7 repeats the measurement of the pattern of FIG. 4 and produces an intensity metadata output similar to that of FIG. 6, but adding a series where the angular position is not the ground truth angular position, and the angular position obtained from the gyroscope has a scale error. This results in a peak at 52 cycles / revolution. Since the expected cycles / revolution is known a priori (53), the system can determine that the discrepancy is due to the gyroscope scale error.

[0059] More specifically, the top graph in Figure 7 shows the relationship between the unscaled gyroscope signal (dashed line) and the actual orientation (e.g., measured with scale error vs. true orientation) relative to the surface pattern / OF signal. The bottom graph in Figure 7 shows the DFT output before scale correction (dashed line, measured with scale error) and after scale correction (solid line, true orientation).

[0060] In some cases, this approach may have some associated prerequisites. For example: The optical flow sensor needs to rotate around the center of the surface pattern. The center of rotation and the center of the surface pattern must be at the same point, otherwise the frequency of the surface pattern will change with position. For wheeled robots, they need to start rotating around a known point (with an arbitrary initial orientation), and / or For wheeled robots, the circular motion must be maintained throughout the rotation. The surface pattern must produce known signal characteristics. The range of values ​​does not necessarily have to be known, but the change in frequency in the spatial domain must be known a priori. o The surface needs to be designed so that it produces a meaningful change in a property that the optical flow sensor can measure (e.g. the reflectivity of the surface). If the optical flow sensor uses automatic gain control that affects the output signal, the automatic gain control should be compensated for best results. The gyroscope measurements, when integrated over the course of a test motion, must be able to produce a stable orientation prediction. o The scale must remain constant throughout the test. Any offset in the gyroscope output induced by temperature (e.g. ZRO) can be reliably removed from the signal prior to integration. The choice of pitch detection method may have some inherent limitations. o The frequency resolution of the DFT is scaled, for example, with the inverse of the data duration, and long measurements may be required to obtain results with the required accuracy.

[0061] Scale calibration according to one embodiment is depicted in Figure 8. In step 800, the robot is centered on a calibration surface, such as the surface shown in Figure 4. Then, in step 802, the robot is rotated in place, and while the robot is rotating, the gyroscope and optical flow sensor capture angular velocity and image quality versus time, respectively. Next, in step 804, the nominal scale factor c nominal The angular velocity ω multiplied by is integrated to obtain the initial gyroscope-derived orientation of the robot. The corresponding image quality values ​​are then interpolated over the uncalibrated orientation grid in step 806, and the result is discrete Fourier transformed in step 808 to obtain the dominant frequency f peak This results in c corrected =c nominal *[f peak / fpattern ], where f pattern is known a priori. If the scaled output is within a predetermined tolerance value (step 812), the corrected scale factor may be saved for gyroscope output correction (step 814) and the process may terminate. If the scaled output is not within the tolerance, proceed to block 816 where a decision may be made whether to recalibrate the sensor.

[0062] The technique described above with reference to FIG. 8 is suitable for practical use in the final calibration of wheeled robots, for example in a factory (where a calibration pattern can be applied, the necessary surface texture for good circular motion can be realized, and / or the robot can be placed in a specific position and then its calibration can begin). This technique has several advantages over the practice of individually calibrating sensors before assembly. For example, this technique can measure sensor characteristics after some stress has been created in the manufacturing process, verify that the optical flow sensor reports image quality values ​​within the expected range (e.g., there are no defects or failures of the lens assembly), and / or can be used to verify the ability of the robot to perform circular motion.

[0063] If this process does not result in a reliable scale calibration of the gyroscope, then one of the above items has failed, indicating a manufacturing issue with this robot.

[0064] Optical flow gain compensation For the best possible position tracking, the OF sensor may implement some sort of automatic gain control (e.g. longer exposure time when the image is dark and / or shorter exposure time when the image is bright). If the signal used for pitch sensing is affected by this, care should be taken to pre-adjust the signal based on the gain setting.

[0065] An exemplary algorithm for accomplishing this is shown below as pseudocode.

[0066]

number

[0067] At a high level in the above code, the gain settings and signals of interest may be recorded and used to create a map of changes in gain settings and the corresponding average change in signal value. An offset may be applied iteratively to map a section of the data adjacent to the most commonly observed gain value to the approximate measurement range observed under the most common gain settings. This process may be repeated until all data has the same approximate gain setting. While the underlying signals on either side of a transition are not necessarily identical, this process / procedure may mitigate the effects of large jumps in the signal that would otherwise occur in the presence of gain control.

[0068] Image quality period detection on any surface Frequency detection techniques have relied on detecting signals with known spatial frequencies and moving in near-perfect circles. This concept can be generalized for use on any surface. In particular, if a few loose constraints are met, a repeating pattern with a frequency of one cycle per revolution can be detected if the sensor can rotate multiple times through the same nearly circular orbit. For example, the signal may be required to have spatial variations that change sufficiently over the course of one revolution and / or have sufficient spatial coherence that the detected image quality metric does not change drastically with small deviations from the circular motion.

[0069] In one embodiment, the approach described herein takes into account the brightness of a surface (e.g., a natural wood surface) over a circular orbit. For example, Figure 9A shows an example of a natural wood grain, and Figure 9B shows a graph of the average brightness sensed as a function of angular position when an OF sensor is circularly moved over such a surface. The angular distance over which the pattern repeats is nominally 360° (vertical dashed line), and the ratio between 360° and the angular distance obtained by integrating the gyroscope output should yield an accurate scale factor for the gyroscope.

[0070] In one example, the period of the signal can be efficiently extracted by 1) recording the estimated angular position and corresponding signal over multiple rotations, and 2) performing an autocorrelation of the resulting sequence. The shift corresponding to the maximum of the autocorrelation is the signal period in the angular position domain of the gyroscope-derived orientation (e.g., 360° rotation).

[0071] FIG. 10 includes two graphs showing an example of a single autocorrelation for a nearly complete circular motion over a natural wood surface using true device orientation (e.g., no scale error). Each label at the bottom of the graph indicates the length (in degrees) measured between the peaks, the error for this period, and the average period error across all data for that point. Note that correlation peaks occur every 360°, but smaller peaks are seen between them. These correspond to 180° shifts in the pattern, highlighting the similarity of the texture on both sides of the natural wood grain. Some of the maxima (e.g., maxima that do not correspond to a 360° shift) may be better correlated than the true 360° correlation, so the minimum and maximum scale values ​​may be used to limit the search area for each peak to minimize or eliminate the possibility of aliasing to inaccurate rotations. If the true scale is known to be within x% of the nominal scale, then the period between the true peaks will be within x% of 360° of the gyroscope orientation. In some examples, this information (eg, typical, minimum, and / or maximum scale values) may be provided by the sensor manufacturer.

[0072] This technique can also be applied to engineered surfaces (e.g., surfaces designed with favorable spatial variation in the signal) if they have a single global maximum or if the scale error is sufficiently bounded to avoid aliasing to the alignment of high frequency patterns. The surface shown in FIG. 4 can be enhanced by adding features that provide a means for global alignment away from the basic pattern, as displayed in FIG. 11A and / or FIG. 11B. Note that the pattern is still dominated by the 53 cycles / revolution signal, but the placement of the wider black bars results in the best single alignment. From rotation to rotation, it is only a 360 degree shift in this pattern that causes all five wide black bars to align with the wide bar (i.e., all five wide bars to align with themselves again). From rotation to rotation, no more than two wider black bars align with another wide black bar (resulting in a weak correlation). For example, when rotated 90 degrees clockwise, the wider black bar at the 3 o'clock position in FIG. 11A may align with the wider black bar at the 6 o'clock position in FIG. 11A, but the other four wide black bars do not. This pattern is suitable for carrying out both this method and the method described in the frequency sensing of patterns on engineered surfaces disclosed above. The patterns depicted herein are representative, and other patterns having one or more preferred signal characteristics may be used.

[0073] When the techniques described in this paragraph (and / or the techniques described in the frequency detection of patterns on engineered surfaces disclosed above) yield different results, this may mean that the gyroscope and optical flow sensor are working as expected (e.g., establishing a good rotation period), but the robot's motion is sufficiently non-circular to shift the dominant frequency from the nominal one. In one example, Figure 12 shows the original signal, its autocorrelation (with the detected period), and the corresponding DFT when performing a nearly circular rotation on the engineered surface depicted in Figure 11A.

[0074] The motion of a wheeled robot cannot be perfectly circular. Thus, obtaining the best single correlation over a long data collection will result in non-circularity appearing as a change in the underlying signal. In one example, these problems can be mitigated by performing multiple correlations over overlapping 720° windows, rather than trying to find the best single period that maximizes correlation over a much longer input series. For example, FIG. 13 shows the distribution of recovered period lengths when simulating non-circular motion on an engineering surface with no orientation error. In this case, the motion was obtained by using position measurements of a two-wheeled robot (trying to rotate in place on a smooth surface) to sample from data collected using a nearly circular motion on the engineering surface. For the data shown in solid lines, period predictions were obtained for each overlapping two-revolution window, and the cumulative distribution function (CDF) of the error of these predictions is plotted. The dashed and dotted lines are the same data used to predict the period using overlapping four-revolution and eight-revolution windows, respectively. In this case, the benefit of having more data for correlation in a long window was outweighed by the disadvantage that accumulated non-circular motion would shift the observed pattern. The motion used to simulate this maintained an approximately constant radius (10 cm) about the center of rotation of the optical flow sensor, but the center of rotation moved approximately 3 mm in a random direction for each rotation. The optimal window length used to predict the period will vary depending on how well the robot can maintain circular motion; for example, the more circular the motion, the longer the window may be.

[0075] The mechanism used to aggregate the results of multiple overlapping windows is somewhat arbitrary: one example involves discarding outliers in the results and then taking a weighted average of the remaining results, with higher weights assigned to windows with higher autocorrelation values.

[0076] 14 is a block diagram illustrating an example gyroscope scale calibration algorithm / mechanism using image quality autocorrelation, according to one embodiment. First, in step 1400, input data (uncalibrated orientation and image quality) is collected by a robot rotating in place over some surface, then in step 1402, the angular velocity ω is scaled with a nominal scale factor c nominal The initial gyroscope-derived orientation of the robot is obtained by integrating the multiplied image quality values, and in step 1404, the corresponding image quality values ​​are interpolated against the uncalibrated orientation grid in the same or similar manner as the corresponding steps 802, 804, and 806 described above. Then, in step 1406, this data is divided into overlapping windows as described above (the window length is defined by the expected degree of circularity of the robot motion). Then, in step 1408, an autocorrelation is performed within each window. Block 1410 shows the substeps related to the autocorrelation. Then, in step 1412, the global maximum in the autocorrelation is identified, and in step 1414, two regions adjacent to this peak, which are within the expected scale error limit (e.g., close to + / - 360° from the global peak), are set as the initial search region. Optionally, this process is repeated (e.g., find the maximum autocorrelation in the current search area and expand the search area) until the current window is peak-free, as described by steps 1416, 1418, and 1420. The peak-to-peak periods found in each window (step 1422) are aggregated in step 1424 to obtain an estimated uncalibrated single period of the gyroscope corresponding to a known 360° rotation. This is used in step 1426 to obtain a calibrated gyroscope scale value. The values ​​of variables F1-F5 referenced in the flow chart are described in Table 1 below.

[0077] [Table 1]

[0078] Identifying the appropriate calibration surface The techniques described herein may use appropriate variation in various embodiments to produce a single global fit over one revolution. The techniques described in this paragraph require appropriate variation to produce a single global best fit over one revolution, but modest variations over the small displacements produced by imperfect rotations give vastly different signals.

[0079] A simple approximation for identifying a surface with one or more features is the short-term variance of the image quality signal.

[0080]

number

[0081] and long-term variance

[0082]

number

[0083] The goal is to record the drift introduced by one revolution of the center of rotation over a short distance window (length d s ) and a long-distance window (length d l The appropriate calibration plane is

[0084]

number

[0085] is the threshold t l during the same measurement period.

[0086]

number

[0087] is the threshold t s The actual selection of these thresholds and distances may vary across sensors and robots. Thus, according to this embodiment, the robot can select its own calibration surface by searching for a surface, for example in a home or office, where the surface has enough large-scale variation to provide the best single correlation, but not enough small-scale variation so that small deviations from a circular motion make the rotations overall different. For example, a checkered pattern has enough large-scale variation to provide the best single correlation, but if the robot slips even a little, the pattern seen in one rotation will be very different from the pattern seen in the next rotation (high level of small-scale variation). On the other hand, a generally uniform surface has low levels of small-scale variation (even if the robot slips, the pattern seen in each rotation will be similar to the pattern seen in the last rotation). However, the large-scale variation is also low and there is no overall best correlation. Thus, a surface that provides enough large-scale variation between those two types of surfaces to be above a first predetermined threshold, but low levels of small-scale variation below a second predetermined threshold, can be identified by the robot as a suitable surface to use for calibration.

[0088] Identifying fixed fiducial marks using image quality A third approach / technique replaces cycle detection, using an OF sensor to determine repeating patterns by sensing discrete points having known spatial relationships to one another.

[0089] In this approach, multiple distinct reflective markers may be placed at known locations on the surface. Figure 15 shows a possible arrangement of reflectors. In this example, the reflectors are two parallel reflective strips 1502a and 1502b. The wheeled robot may position itself so that its center of rotation 1504 is located at some point on a line 1506 that is equidistant and parallel to each strip 1502a, 1502b. The robot then rotates back and forth so that the OF sensor 1510 is first located over the center of one of the strips (e.g., 1502b) and then over the center of the other strip (e.g., 1502a) (the two strips are merely illustrative and there may be more strips), while the gyroscope measures the change in orientation between the two positions. The angular distance measured between these marks may be combined with the known angular distance between the two strips to obtain a scale between the gyroscope and the ground truth measured by the OF sensor. The optical flow sensor is placed on the robot at a fixed radius r from the robot's center of rotation 1504.

[0090] In this case, the robotic system determines the angle when the OF sensor is over the center of the first reflective strip by integrating the angular velocity measured by the gyroscope between when the movement begins and each time the OF sensor is centered over each reflective strip.

[0091]

number

[0092] and the angle when it is over the center of the second reflective strip

[0093]

number

[0094] Using the known distance d between the marks and the known radius r around which the optical flow sensor rotates, the robotic system can determine

[0095]

number

[0096] and then use the scale factor

[0097]

number

[0098] which can be used to correct the gyroscope measurements. In the common case where the starting position is meaningless,

[0099]

number

[0100] As such, by starting the integration just leaving the first detected mark, integration error can be reduced.

[0101] In one embodiment, the overall flow of the algorithm used to perform the scale factor calculation described above with reference to FIG. 15 is depicted in FIG. 16A and FIG. 16B. As set forth above and shown in FIG. 16A, the overall algorithm finds a first mark using a set of steps in step 1600, then finds a second mark using the same (or similar) set of steps in step 1602, and then calculates a scale factor in step 1604 based on the angular positions of the two detected marks (identified as F0 and F1 in the table below). The underlying concept of the steps used to find each mark is that when the OF sensor is over the reflective surface, there is a plateau in the image intensity, and the center of the reflective surface should be somewhere on that plateau (but the actual maximum observed is not directly on the center). Thus, the algorithm obtains the angular position of each edge of this plateau and continues to adjust upwards the threshold required to be on the plateau until the desired accuracy limit is reached.

[0102] To make the algorithm practical, two stopping conditions are introduced. The first stopping condition, indicated by φ in the table below, is the point at which the robotic system stops searching and defines the angular width of the detected mark, which is related to the angular width that the peak observations may span. The second stopping condition τ in the table below defines how close the threshold of visible marks may be to the observed peak value, and is related to the granularity at which the robotic system can resolve the difference between peak and non-peak values. The steps associated with each mark detection step 1600 and 1602 are shown in FIG. 16B.

[0103] This process is performed using the plateau threshold x thresh The process begins with a step (not shown in FIG. 16B, but identified as F11 in Table 2 below) of initializing . Then, in step 1606, the robot checks whether the luminance detected on the mark exceeds a threshold x threshThe angular position at which the detected brightness exceeds the threshold (F3) is stored as the first interim marker edge and processing continues to step 1608 where the robot is rotated (either clockwise or counterclockwise) until the brightness level falls below the threshold, i.e. the robot moves away from the other edge of the marker. This angular position (F4) is recorded as the second interim marker edge and processing continues to step 1610 where a first stopping condition is used to determine whether the difference between the two stored angular positions is large enough (as shown by test F5 in the table below) that the marker edge is considered to have been found. If so, processing continues to step 1616 where the final angular position of the mark is output as F6.

[0104] If, in step 1610, it is determined that x is not large enough, then processing moves to step 1612 where another test is performed. Specifically, a second stopping condition is tested to see if x is greater than a threshold x (given the resolution of the photosensor). thresh It is determined whether lowering threshold is not expected to improve the accuracy of the edge detection process. If the test in step 1612 (e.g. using F7 in the table below) determines that no improvement in accuracy is expected, the process terminates in step 1616 by outputting the final angular position of the detected mark. If it is determined that an improvement in accuracy is expected, the threshold may be lowered (e.g. using F8) in step 1614 and the process is repeated until one of the stopping conditions is met.

[0105] [Table 2]

[0106] In one embodiment, the main requirements for this approach are threefold: · The image quality signal needs to be sufficiently different when the reflective mark is visible than when it is not, to get a good estimate of the center of each reflector. The robot must align its center of rotation with the central axis of the reflector array. The robot's turning radius must be constant and there must be no lateral drift.

[0107] This approach is more susceptible to small disturbances than previously disclosed approaches because the calibration is based on sensing two discrete points rather than correlation across a much larger data set. In some examples, this approach is a solution that can perform gyroscope scale calibration outside of a factory setting. A practical deployment scenario could be the addition of two reflective strips to a charging station used by a domestic robot. The robot needs to align and make electrical contact with the charging station in order to charge, and this can be exploited to minimize the amount of uncontrolled movement required from a known pose to the end of the calibration movement.

[0108] Variations This approach can be extended to include any number of fiducial marks for redundancy and detection of non-circular motion without substantially changing the above expressions.

[0109] This technique can also be performed with a single fiducial mark, and the robot rotates completely between sensing, where θ is 360°.

[0110] Optical Flow Scale Estimation Using Gyroscope Scale and Known Geometry Robotic systems can overcome the surface dependence of the optical flow sensor's distance scale by using precise gyroscope scales, a priori knowledge of the robot's rotational behavior, and sensor geometry. The relationship between the distance reported by the OF sensor and the actual distance it travels is determined primarily by the vertical distance from the image sensor to the tracked surface. As the robot moves across different surfaces (e.g., carpet or tile) or as the robot's characteristics change, the vertical distance from the image sensor to the tracked surface can change (e.g., a change in the weight it carries affects the height to which the robot's wheel suspension lifts its chassis off the ground). For these reasons, it would be desirable to enhance approaches to directly and rapidly measure the optical flow scale during operation.

[0111] Once the gyroscope scale is obtained (e.g., by the methods / techniques described in "Related Techniques"), the robot can be rotated in place and the total distance traveled by the optical flow sensor can be estimated using the gyroscope (using the known device geometry) and also measured by the optical flow sensor itself to obtain a scale constant specific to the surface across which the rotation is performed.

[0112] In various embodiments, the robot may rotate around its center and the displacement of the OF sensor relative to this point

[0113]

number

[0114] In one example, for simplicity, we assume that the motion is planar, the Z axes of each frame are aligned, and the rotation is confined to the XY plane (rotation is a scalar about the Z axis). This means that the robotic system can treat the angular velocity measured by the IMU as if it were measured at the OF sensor. Both the IMU and the OF sensor have a sample frequency

[0115]

number

[0116] are assumed to be sampled simultaneously at,Δt being the time between successive samples.

[0117] The relationship between the OF sensor output and the gyroscope output can be interpreted relative to one another in both the rate and / or position domains.

[0118] Velocity domain interpretation Optical Flow Frame In various embodiments, the linear velocity of the optical flow sensor given by the rigid body relationship is:

[0119]

number

[0120] The robot system is a pure rotational

[0121]

number

[0122] and the angular velocity ω measured by the IMU is assumed to vary over time. I Based on this, we predict the linear velocity of the optical flow sensor.

[0123]

number

[0124] You can get R R→O is a rotation around the Z axis, which does not change over time, and ω ISince (t) is assumed to be planar (Z axis only), the prediction of the linear velocity can be further simplified to:

[0125]

number

[0126] At each point in time, the robot system receives the following from the OF sensor:

[0127]

number

[0128] An OF sensor measures the displacement (number of pixels) between two points in time and does not have to generate a velocity output itself. Assuming the OF sensor is polled at a constant rate (or appropriately post-processed), an equivalent velocity will be obtained.

[0129] The robot system then: V I = αV O By applying a suitable optimization (e.g., least squares fitting) to obtain the relationship I and V O is an n × 2 matrix, and the i-th row of each matrix represents the i-th sample obtained by each sensor.

[0130]

number

[0131] and

[0132]

number

[0133] Corresponds to.

[0134] Ground Frame The displacement reported by an optical flow sensor in a circular orbit is the angular distance

[0135]

number

[0136] can be converted to

[0137]

number

[0138] is the displacement reported by the OF sensor in the i-th sample period (at time t = iΔt). This is the angular velocity predicted by the optical flow sensor.

[0139]

number

[0140] The robot system then uses a suitable optimization scheme to Ω I =αΩ O where, similar to the linear velocity expression above, Ω I and Ω O is an n × 1 vector, and the i-th element of each vector is the i-th angular velocity ω I (iΔt) and ω O corresponds to (iΔt).

[0141] Interpretation of location regions Optical Flow Frame The linear distance traveled by the optical flow sensor (in the optical flow sensor's frame) can be obtained by integrating the velocity over time. The robot uses an IMU to obtain this.

[0142]

number

[0143] Here,

[0144]

number

[0145] is the distance measurement in the optical flow sensor's coordinate system, as calculated by the IMU. This is a two-dimensional quantity,

[0146]

number

[0147] Note that is a constant term vector in the robot frame, and should not be confused with the (time-varying) vector from the center of rotation to the optical flow sensor in the global frame.

[0148] Each time an OF sensor is polled, it may report two-dimensional (2D) displacement (since the last time it was polled). OF Sensor Output

[0149]

number

[0150] It would be sufficient to sum up the following equation for each period:

[0151]

number

[0152] This results in a distance measurement calculated by the optical flow sensor in the coordinate system of the optical flow sensor.

[0153] Under the assumption that the optical flow scale is constant throughout this movement, the gyroscopes are pre-calibrated, and there is no orthogonal axis skew in the optical flow sensor output, the robotic system can relate the optical flow output to the actual linear distance traveled by the following equation:

[0154]

number

[0155] Under the simplifying assumption that the scale is identical for each axis of the optical flow sensor, the robotic system can calculate a single scale constant based on the total distance measured by each sensor (ignoring direction).

[0156]

number

[0157] For example, if one of the axes of the optical flow sensor is perpendicular to the motion, the above calculation can be used.

[0158] Ground Frame The method / approach described above in the velocity domain interpretation can be interpreted in the position domain: the sum of the angular distances measured by the gyroscope can be related to the sum of the angular distances traveled, calculated by the optical flow displacement and the radius, by a constant scale factor c.

[0159]

number

[0160] Relationship to previously described methods The same physical motion used to measure the gyroscope scale vs. image quality period of an unknown surface is used to establish the optical flow sensor distance scale, and these two processes can be combined.

[0161] Alternatives and Extensions The general concept of using periodic signals in surface image quality to estimate robot motion may have further domain-specific applications. This paragraph outlines some related methods that may find application in similar situations.

[0162] Image matching for cycle detection The methods described above work by using the concise, high-level image quality metrics reported by optical flow sensors. Some optical flow sensors give access to an image of the actual surface on which motion is sensed. When this is available, some further improvements are possible.

[0163] Pixel scale reference mark detection The method described in "Identifying a Suitable Calibration Plane" may include estimating the center of the area in which the fiducial mark is visible by scanning back and forth over the fiducial mark. If the shape of this area is not an arc (e.g., due to accidental linear motion) or if the sensed mark area changes over time (e.g., due to illumination or tilt changes), the prediction of the mark center will degrade.

[0164] However, if the robotic system can resolve the actual shape of the mark when the OF sensor is approximately centered over the marker, the effect of these problems can be reduced. If the edge of the mark is sufficiently straight and of known orientation, the robotic system can sense the sensor orientation relative to this edge. Figure 17 shows an example of a mark sensed in the sensor's reference frame and how this can be used to obtain the angular position of the sensor relative to any known orientation that corresponds to the mark's direction (labeled "N" in this case).

[0165] If the fiducial marks have a known orientation relative to each other (e.g., parallel), the robotic system can directly measure the difference in orientation at the start and end of the inspection motion and compare this to the integrated gyroscope measurement. As long as the calibration motion can cover the two fiducial marks, the motion between the two fiducial marks is not subject to the circularity and center of rotation constraints described in "Identifying a Suitable Calibration Plane".

[0166] True orientation on engineering surfaces The above technique can be further extended to obtain near-continuous directional measurements on engineering surfaces consisting of multiple parallel lines.

[0167] Intrinsic detection of non-circular motion When the motion is perfectly circular, the angular velocity measured by the gyroscope should be related to the linear velocity sensed at the optical flow sensor by a simple scale factor (defined by the distance from the OF sensor to the center of rotation and the scale of the respective sensor). The method proposed in this document outlines an approach to estimate the gyroscope scale (vs. true angular velocity) and the optical flow scale (vs. angular distance traveled), and relies on the calibration motion being more or less circular.

[0168] The circularity of the motion can be measured even if neither the scale of the sensor nor the true radius of rotation is known, and this basic technique will be valid for practical applications of the above methods. The representations presented so far have generally assumed, for simplicity, that the robot rotates about its center. Figure 18 shows an example where the center of rotation of a circular motion is not at the robot center, and shows the relationships between relevant quantities during such a motion.

[0169] The linear velocity of the robot and the optical flow sensor in the optical flow frame was previously described using the velocity of the robot and the displacement of the optical flow sensor relative to the robot center.

[0170]

number

[0171] A similar rigid body relationship can be used to obtain the velocity of the optical flow sensor at a given time from the velocity of the center of rotation and the displacement of the optical flow sensor relative to the center of rotation. Note that the center of rotation is not necessarily at the robot center.

[0172]

number

[0173] Then, the linear velocity measured by the optical flow sensor is given by:

[0174]

number

[0175] When the robot motion is circular, the linear velocity of the center of rotation is zero and the radius from the center of rotation to the optical flow is constant in the robot frame. In this case, a single scale relates the linear velocity measured by the optical flow sensor to the angular velocity measured in the IMU throughout the motion, as described above (e.g., in the paragraph on estimating optical flow scale using gyroscope scale and known geometry). In the absence of a single scale that consistently relates these quantities throughout the motion, the robot system may assume non-circular motion, where the center of rotation is moving and / or the distance from the optical flow sensor to the center of rotation is changing.

[0176] Adaptation for non-circular motion The optical patterns presented above (e.g., in the paragraph on frequency detection of patterns on optical surfaces) are optimal when the center of rotation is fixed at the center of the pattern. The mechanisms disclosed above (e.g., in the paragraph on image quality period detection on arbitrary surfaces) may be more robust to non-circular motion, but this is difficult to achieve with frequency detection techniques.

[0177] It may be possible to develop novel patterns that can identify non-circularities and can even be used to reconstruct the entire device trajectory. The concept of using a grid with known spacing, where the surface quality between adjacent cells varies, has been described, for example, in reference (2). Here, a similar concept of collecting the positions of possible cell intersections over a longer motion and using them to reconstruct the most likely trajectory on the engineered surface seems applicable.

[0178] It is also conceivable that engineered patterns could be designed that incorporate, for example, reflective surfaces that can be placed slightly outside the intended range of motion and used to detect drift from the pattern center.

[0179] Calibrating the gyroscope scale by docking exercises Some of the mechanisms disclosed above (e.g., in the paragraph regarding identifying fixed reference marks using image quality) may rely on identifying multiple known angular positions. In a typical consumer application, the robot needs to periodically and automatically return to the charging dock to make physical contact at some roughly known orientation. If there is an orientation difference between valid docking positions, the gyroscope scale can be estimated by leaving the docked position, rotating in place, and docking again. If the maximum angular difference between two docking positions is δ°, then after the robot has rotated n times, the gyroscope scale error will be at most

[0180]

number

[0181] There is a possibility that it will become.

[0182] This method has the advantage that it does not require any infrastructure other than that typical for consumer robots. However, charging surfaces are typically designed to allow a wide range of docking positions, so a δ of the order of 20 degrees is not unusual. To achieve a scale error of 0.25% with a possible alignment error of 20 degrees would require approximately 20 rotations. In contrast, a periodicity detection approach (e.g. the mechanism disclosed in the paragraph on image quality periodicity detection on arbitrary surfaces) would achieve such a scale error in more than 8 rotations, ≈80% of the time, under typical conditions on a wooden surface. Practical issues associated with a dock moving over time (in space, pushed by a docking robot or pushed by a person) may introduce further errors.

[0183] summary In various embodiments, the relationship between motion (e.g., as sensed by a gyroscope) and image quality signals (e.g., as sensed by an OF sensor) may be used to derive estimates of sensor calibration parameters. In some cases, image quality information from an OF sensor may be used to calibrate a gyroscope by estimating the motion of the robot. This is in contrast to the common usage of OF sensors, which only measure linear displacement.

[0184] In various embodiments, the methods and / or techniques disclosed herein for calibrating sensors may support calibration of sensors in their final installed configuration (e.g., after assembly into an end product). In one example, sensors may be recalibrated in the field to account for calibration drift and physical aging that occurs over time. In another example, sensors may be recalibrated in the field to capture the effects of environmental changes on the calibration, particularly to adapt the OF sensor scale to changes in the surface.

[0185] In one embodiment, the gyroscope scale may be calibrated by detecting the frequency of a known pattern in the OF image quality signal. For example, the robotic system may be configured to combine several factory calibration processes with one or more final installed configurations of the sensor. In another embodiment, the gyroscope scale may be calibrated by detecting periodicity of an unknown OF image quality signal. For example, the robotic system may be configured to perform or allow for in-situ recalibration of the gyroscope to account for calibration drift.

[0186] In one embodiment, the gyroscope scale may be calibrated by sensing the fiducial marks with optical flow image quality. The robotic system may be configured to perform or allow in situ recalibration of the gyroscope to account for calibration drift. In another embodiment, the optical flow distance scale may be calibrated by circular motion of the device with a calibrated gyroscope and known geometry. For example, the robotic system may be configured to perform or allow in situ recalibration of the OF sensor to account for the effects of environmental changes.

[0187] An exemplary (non-limiting) high-level architecture of a robot 1900 is shown in Fig. 19. In Fig. 19, one or more processors 1902, which may be included in the robot 1900, include a system bus 1904, which is communicatively coupled to (1) one or more communication devices 1906 that enable communication with other devices through a communication channel 1907, (2) one or more sensors / sensors including one or more accelerometers 1908, one or more gyroscopes 1910, one or more optical flow sensors 1912, (3) one or more manipulators 1914, and (4) one or more moving objects 1916. The communication channel 1907 may be compatible with both wired and wireless communication as well as supporting various communication protocols. By way of example and not limitation, communication channel 1907 may be configured as a serial and / or parallel communication channel, such as, for example, USB, IEEE-1394, 802.11, BLE, cellular (e.g., LTE or 5G), and other wired and wireless communication protocols. If a wireless communication channel is used, communication device 1906 will include a wireless transceiver and antenna (not shown in FIG. 19).

[0188] The sensors may include any number of different sensors, such as optical sensors, inertial sensors (e.g., gyroscopes, accelerometers, etc.), thermal sensors, tactile sensors, compasses, distance sensors, sonar, Global Positioning System (GPS), Ground Penetrating Radar (GPR), lasers for object detection and range sensing, imaging devices, magnetometers, etc. Sensors may also be any other existing sensors within the deployment, but mounted on the robot to obtain the same data distributed across the installation, rather than from a single location (e.g., temperature or humidity sensors) that would otherwise be static.

[0189] Further, one skilled in the art will appreciate that many of these sensors may include both a source and a sensor to combine the sensor input into a meaningful and actionable sense. For example, sonar sensors and GPRs generate acoustic or sub-audio frequency waveforms and sense the reflected waves. Similarly, sensors that include lasers may include a sensor configured to sense the reflected waves from the laser to determine obstructions or phase shifts in the laser beam. An imaging device suitable for use as the optical flow sensor 1912 may be any device suitable for capturing images, such as, for example, an infrared imager, a video camera, a still camera, a digital camera, a complementary metal oxide semiconductor (CMOS) imaging device, a charge-coupled device (CCD) imaging device, etc. Additionally, the imaging device may include optical devices to modify the captured image, such as lenses, collimators, filters, and mirrors. The robot 1900 may also include a pan and tilt mechanism coupled to the imaging device to adjust the orientation of the imaging device.

[0190] The manipulators 1914 may include, for example, vacuum devices, magnetic pick-up devices, arm manipulators, scoops, grippers, camera pan and tilt manipulators, individual or combined actuators, etc. The moving object 1916 may include, for example, one or more wheels, tracks, legs, rollers, propellers, etc. The moving object 1916 may be driven by motors, actuators, levers, relays, etc. for movement or steering capabilities. Additionally, sensors may be configured in association with the manipulators 1914 or the moving object 1916, such as, for example, a odometer and a pedometer.

[0191] The discussion of FIG. 19 above and the hardware associated with the robot in general has been adapted from U.S. Patent No. 8,073,564 (hereinafter the '564 patent), the disclosure of which is incorporated herein by reference. However, those skilled in the art will appreciate that such elements are merely exemplary. Some robots may not include all of the elements shown in FIG. 19, while other robots may include hardware elements that do not fall into the categories depicted in FIG. 19. Nevertheless, the robot 1900 may include the previously described embodiments relating to scale calibration.

[0192] conclusion The systems and methods for processing data according to exemplary embodiments of the present invention may be performed by one or more processors executing sequences of instructions contained in a storage device. Such instructions may be read into the storage device from other computer-readable media, such as a secondary data storage device. The processor operates, for example, as described above, by executing sequences of instructions contained in the storage device. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions to implement the present invention. Such software may run on a processor housed in a device, such as, for example, a sensor or a robot or other device containing the software, or on a processor or computer housed in another device, such as, for example, a system controller, a game console, a personal computer, that communicates with the device containing the sensor. In such cases, data may be transferred wired or wirelessly between the device containing the sensor and the device containing the processor executing the software to perform sensor calibration as described above. According to other exemplary embodiments, some of the processes described above with respect to sensor calibration may be performed in the device containing the sensor, with the remaining processes being performed in a second device after receiving partially processed data from the device containing the sensor.

[0193] Although the exemplary embodiments described above relate to sensing signals including one or more rotational and / or OF sensors, the sensor calibration techniques according to these exemplary embodiments are not limited to only these types of sensors. Devices to which the sensor calibration techniques described herein may be applied include, for example, accelerometers, optical sensors and inertial sensors (e.g., rotational sensors, gyroscopes or accelerometers), magnetometers and inertial sensors (e.g., rotational sensors, gyroscopes or accelerometers), magnetometers, and optical sensors (e.g., OF sensors, cameras, one or more photodiodes, one or more phototransistors), or combinations of other sensors. In addition, while the exemplary embodiments described herein relate to sensor calibration in the context of robots and robotic vacuum cleaners and applications, such techniques are not so limited and may be employed in methods and devices related to other applications, such as, for example, mobile phones, medical applications, games, cameras, military applications, robotic devices, etc.

[0194] The foregoing exemplary embodiments are intended to be illustrative rather than restrictive in all respects of the present invention. Thus, the present invention is susceptible of many variations in detailed implementation that may be derived by those skilled in the art from the description contained herein. For example, although some of the foregoing exemplary embodiments describe, among other things, the use of inertial sensors to sense the motion of the device, other types of sensors (e.g., ultrasonic sensors, magnetic sensors, or optical sensors) may be used instead of or in addition to the inertial sensors, along with the previously described signal processing. All such variations and modifications are deemed to be within the scope and spirit of the present invention as defined by the following claims. No element, operation, or instruction used in the description of this application should be construed as critical or essential to the present invention unless expressly described as such. Also, the article "a" as used herein is intended to include one or more items.

[0195] Although features or elements are described above in certain combinations, one skilled in the art will understand that each feature or element may be used alone or in any combination with other features or elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware embodied in a computer-readable medium for execution by a computer or processor. Examples of non-transitory computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in the WTRU 102, UE, terminal, base station, RNC, or any host computer.

[0196] Moreover, the above embodiments refer to processing platforms, computer systems, controllers, and other devices that contain processors. These devices may contain at least one central processing unit ("CPU") and memory. References to acts or symbolic representations of operations or instructions may be performed by various CPUs and memories, in accordance with practices of those skilled in the art of computer programming. Such acts, operations or instructions may be referred to as being "executed," "computer-executed," or "CPU-executed."

[0197] Those skilled in the art will recognize that acts or symbolically represented operations or instructions include the manipulation of electrical signals by a CPU. The electrical system represents data bits that allow for the transformation and reduction of electrical signals, the maintenance of data bits in memory system locations, and other processing of the signals to reconfigure or alter the operation of the CPU. The memory locations where the data bits are maintained are physical locations that have particular electrical, magnetic, optical, or organic properties that correspond to or represent the data bits. It should be understood that representative embodiments are not limited to the aforementioned platforms or CPUs, and that other platforms and CPUs may support the provided methods.

[0198] The data bits may also be maintained on computer readable media, including magnetic disks, optical disks, and any other volatile (e.g., random access memory ("RAM")) or non-volatile (e.g., read only memory ("ROM")) CPU readable mass storage system. The computer readable media may include cooperating or interconnected computer readable media that reside solely on a processing system or that are distributed among multiple interconnected processing systems that may be local or remote to the processing system. It will be appreciated that representative embodiments are not limited to the storage devices mentioned above, and that other platforms and storage devices may support the described methods.

[0199] In an example embodiment, any of the operations, processes, etc. described herein may be implemented as computer-readable instructions stored on a computer-readable medium. The computer-readable instructions may be executed by a processor of a mobile unit, a network element, and / or any other computing device.

[0200] There is little distinction between hardware and software implementations of aspects of the system. The use of hardware or software is a design choice that generally represents a cost vs. efficiency tradeoff (although this is not necessarily the case in that in certain circumstances the choice between hardware and software may be critical). There may be various means (e.g., hardware, software, and / or firmware) that may affect the processes and / or systems and / or other technologies described herein, and the preferred means may vary depending on the context in which the processes and / or systems and / or other technologies are deployed. For example, a developer may choose a primarily hardware and / or firmware means when he or she determines that speed and accuracy are paramount. If versatility is paramount, the developer may choose a primarily software implementation. Alternatively, the developer may choose some combination of hardware, software, and / or firmware.

[0201] The foregoing detailed description describes various embodiments of the apparatus and / or processes by using block diagrams, flow charts, and / or examples. To the extent that such block diagrams, flow charts, and / or examples include one or more functions and / or operations, those skilled in the art will appreciate that each function and / or operation within such block diagrams, flow charts, or examples may be individually and / or collectively implemented by a wide variety of hardware, software, firmware, or substantially any combination thereof. Suitable processors include, by way of example, general purpose processors, special purpose processors, conventional processors, digital signal processors (DSPs), multiple microprocessors, one or more microprocessors associated with a DSP core, controllers, microcontrollers, application specific integrated circuits (ASICs), application specific standard products (ASSPs), field programmable gate array (FPGA) circuits, any other type of integrated circuit (IC), and / or state machines.

[0202] The present disclosure is not limited in the sense of the specific embodiments described in this application, which are intended as illustrations of various aspects. It will be apparent to those skilled in the art that many modifications and variations can be made without departing from the spirit and scope of the present disclosure. No element, operation, or instruction used in the description of this application should be construed as critical or essential to the invention unless expressly so given. In addition to those enumerated herein, functionally equivalent methods and apparatuses within the scope of the present disclosure will be apparent to those skilled in the art from the foregoing description. Such modifications and variations are intended to be within the scope of the appended claims. The present disclosure should be limited only by the terms of the appended claims, along with the full scope of equivalents to which such claims are entitled. It should be understood that the present disclosure is not limited to a particular method or system.

[0203] In certain representative embodiments, portions of the subject matter described herein may be implemented by application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), and / or other integrated configurations. However, those skilled in the art will appreciate that some aspects of the embodiments disclosed herein may equally be implemented, in whole or in part, in an integrated circuit, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as substantially any combination thereof, and that, in view of the present disclosure, designing circuitry and / or writing software and / or firmware code is well within the skill of those skilled in the art. In addition, those skilled in the art will appreciate that the subject matter described herein may be distributed as a program product in various forms, and that the exemplary embodiments of the subject matter described herein apply regardless of the particular type of signal-bearing medium used for the actual distribution. Examples of signal bearing media include, but are not limited to, recordable type media such as floppy disks, hard disk drives, CDs, DVDs, digital tape, computer memory, and transmission type media such as digital and / or analog communications media (e.g., fiber optic cables, wave guides, wired communications links, wireless communications links, etc.).

[0204] The subject matter described herein may depict various components embedded within or connected to various other components. It should be understood that such depicted architectures are merely examples, and that in fact many other architectures that achieve the same functionality may be implemented. In a conceptual sense, any mechanism of components to achieve the same functionality is effectively "associated" such that the desired functionality may be achieved. Hence, any two components herein that are combined to achieve a particular functionality may be considered to be "associated" with each other to achieve the desired functionality, regardless of architecture or intermediate elements. Similarly, any two components so associated may be viewed as being "operably connected" or "operably coupled" with respect to each other to achieve the desired functionality, and any two components that may be so associated may be viewed as being "operably coupled" with respect to each other to achieve the desired functionality. Particular examples of what may be operably coupled include, but are not limited to, components that may be physically paired and / or physically interacting components and / or wirelessly interacting components and / or wirelessly interacting components and / or logically interacting components and / or logically interacting components.

[0205] With respect to the use of substantially any plural and / or singular terminology herein, one of ordinary skill in the art can convert from plural to singular and / or from singular to plural as appropriate for the situation and / or application. Various singular / plural permutations may be set forth herein for specific clarity.

[0206] Those skilled in the art will understand that, in general, the terms used herein, and particularly in the appended claims (e.g., the body of the appended claims), are generally intended as "open" terms (e.g., the term "including" should be interpreted as "including, but not limited to," the term "having" should be interpreted as "having at least," and the term "including" should be interpreted as "including, but not limited to"). Those skilled in the art will further understand that if a specific number is intended in the introduced claim recitation, such intention will be clearly stated in the claim, and that in the absence of such recitation, no such intention exists. For example, where only one item is intended, the term "single" or similar language may be used. As an aid to understanding, the following appended claims and / or description of this specification may include the use of introductory phrases such as "at least one" and "one or more" to introduce claim recitations. However, the use of such phrases should not be construed as meaning that the introduction of a claim recitation with the indefinite article "a" or "an" means that any particular claim is limited to embodiments containing only one such recitation, even when the same claim includes the introductory phrases "one or more" or "at least one" or an indefinite article such as "a" or "an" (e.g., "a" and / or "an" should be construed to mean "at least one" or "one or more"). The same applies to the use of definite articles used to introduce claim recitations. Also, even if a particular number of recitations of introduced claims is expressly recited, one of ordinary skill in the art would understand that such recitation should be construed to mean at least the recited number (e.g., the bare recitation of "two recitations" without other modifiers means at least two recitations, or more than two recitations).Furthermore, in instances where traditional phrasing similar to "at least one of A, B, and C, etc." is used, such syntax is generally intended to mean the meaning of the traditional phrasing as one of ordinary skill in the art would understand (e.g., "a system having at least one of A, B, and C" would include, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B and C together, etc.). In instances where traditional phrasing similar to "at least one of A, B, or C, etc." is used, such syntax is generally intended to mean the meaning of the traditional phrasing as one of ordinary skill in the art would understand (e.g., "a system having at least one of A, B, or C" would include, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B and C together, etc.). Those skilled in the art will further appreciate that substantially any disjunctive word and / or phrase indicating two or more alternative terms, whether in the description, claims, or drawings, should be understood to contemplate the possibility of including one of the terms, either of the terms, or both of the terms. For example, the phrase "A or B" is to be understood to include the possibility of "A" or "B" or "A and B." Additionally, as used herein, the term "any of" following a list of a number of items and / or a number of categories is intended to include "any of," "any combination," "any plurality," and / or "any combination of a plurality" of the items and / or categories of items, either individually or in conjunction with other items and / or items in other categories. Moreover, as used herein, the term "set" or "group" is intended to include any number of items, including zero. Additionally, as used herein, the term "number" is intended to include any number, including zero.

[0207] Additionally, those skilled in the art will understand that when features or aspects of the disclosure are described in terms of a Markush group, the disclosure is also described thereby in terms of any individual element or subgroup of elements of the Markush group.

[0208] As will be understood by those skilled in the art, for any and all purposes, including in terms of providing a written description, the ranges disclosed herein include any and all possible subranges and combinations of those subranges. Any recited range can be readily understood as fully describing and allowing the same range to be subdivided into at least two, three, four, five, ten, etc. As a non-limiting example, each range discussed herein can be readily subdivided into a lower third, a middle third, and an upper third, etc. As will be understood by those skilled in the art, all phrases such as "up to," "at least," "more than," "less than," and the like, are inclusive of the number recited, and refer to ranges that can in turn be subdivided into subranges as discussed above. Finally, as will be understood by those skilled in the art, a range includes each individual element. Thus, for example, a group having 1 to 3 cells refers to a group having 1, 2, or 3 cells. Similarly, a group having 1 to 5 cells refers to groups having 1, 2, 3, 4, or 5 cells.

[0209] Moreover, the claims should not be read as limited to the order or elements given unless otherwise specified. In addition, the use of the phrase "means for" in any claim is intended to refer to 35 U.S.C. 35 U.S.C. 112, sixth paragraph, or means-plus-function claim format, and no such intent is intended in any claim without the phrase "means for." [Explanation of symbols]

[0210] 1502a reflective strip 1502b reflective strip 1504 Center of rotation 1506 Parallel Lines 1510 OF sensor 1900 Robot 1902 Processor 1904 System Bus 1906 Communication Devices 1907 Communication Channel 1908 Accelerometer 1910 Gyroscope 1912 Optical flow sensor 1914 Manipulator 1916 Moving objects

Claims

1. Sensing physical motion of a robot operating on a surface with a gyroscope; generating one or more image signals by detecting one or more reflective reference marks on the surface by an optical flow (OF) sensor and / or a camera, the one or more image signals comprising information; deriving estimates of sensor calibration parameters for each of the gyroscope and the OF sensor based on the sensed physical motion and the information; Including, the information includes a function of a respective image in each frame of the one or more image signals; A method, wherein said function is related to image quality and / or image brightness.

2. The method of claim 1 , further comprising calibrating the gyroscope based on the information and / or the estimate of sensor calibration parameters.

3. calibrating the gyroscope identifying a relationship between the information and the sensed physical movement; calibrating a scale factor of the gyroscope based on the identified relationship; The method of claim 2 , comprising:

4. The method of claim 1 , wherein the physical motion of the robot is a rotation in place and / or a rotation of more than 360 degrees.

5. The method of claim 1 , wherein the physical movement of the robot is sensed on an arbitrary surface or on a surface having an unknown surface pattern.

6. The method of claim 1 , wherein the one or more image signals are periodic signals.

7. estimating angular motion of the robot relative to the surface based on the information; calibrating a scale factor of the gyroscope based on the estimated angular motion; The method of claim 1 further comprising:

8. relating angular velocity to linear velocity at the OF sensors and / or cameras based on known robot geometry; Calculating the displacement scale of the OF sensor and / or camera; The method of claim 1 further comprising:

9. A gyroscope configured to sense physical motion of a robot operating on a surface; an optical flow (OF) sensor and / or camera configured to generate one or more image signals by detecting one or more reflective reference marks on the surface, the one or more image signals including information; a processor configured to derive estimates of sensor calibration parameters for each of the gyroscope and the OF sensor based on the sensed physical motion and the information; Equipped with the information includes a function of a respective image in each frame of the one or more image signals; The system, wherein the function is related to image quality and / or image brightness.

10. The system of claim 9 , wherein the processor is further configured to calibrate the gyroscope based on the information and / or the estimate of sensor calibration parameters.

11. The processor, identifying a relationship between the information and the sensed physical movement; calibrating a scale factor of the gyroscope based on the identified relationship; The system of claim 10 , further configured to calibrate the gyroscope by:

12. The system of claim 9 , wherein the physical motion of the robot is a rotation in place and / or a rotation of more than 360 degrees.

13. The system of claim 9 , wherein the physical movement of the robot is sensed on an arbitrary surface or on a surface having an unknown surface pattern.

14. The system of claim 9 , wherein the one or more image signals are periodic signals.

15. The processor, estimating angular motion of the robot relative to the surface based on the information; Calibrating a scale factor of the gyroscope based on the estimated angular motion. The system of claim 9 further configured to:

16. The processor, relating angular velocity to linear velocity in the OF sensors and / or cameras based on known robot geometry; Calculate the displacement scale of the OF sensor and / or camera The system of claim 9 further configured to:

Citation Information

Patent Citations

  • Proximity sensing on mobile robots

    JP2016076248A

  • Movement vector calculation method and movement vector calculation device

    JP2019020289A

  • Multi-robot control interface

    US8073564B2

  • Sensor fusion using inertial and image sensors

    WO2016187757A1