A multi-sensor timestamp delay estimation method

Abstract

The present invention relates to a multi-sensor timestamp delay estimation method. A visual sensor is used as a visual compass to perform a test corresponding to an inertial measurement unit. The acquired data is processed with the same frequency through interpolation, and then the time delay is estimated. The time delay between multiple sensors can be estimated through simple testing means and image recognition. The method is suitable for detection systems, positioning systems, and recognition systems currently used by a large number of sensors, and improves the detection accuracy of the system.

Description

Technical Field

[0001] The invention relates to a sensor error detection technology, and in particular to a multi-sensor timestamp delay estimation method. Background Art

[0002] Visual sensors (Camera / TOF) and inertial measurement units (IMU) have become indispensable devices in robot positioning and perception algorithms. Vision can capture image data for posture solution and perception of the external environment (object recognition, target detection, etc.). IMU can obtain the acceleration and angular velocity data of the body. Fusion of the data of the two sensors can complement each other's advantages and help improve the accuracy and robustness of positioning and perception, but the premise is that the timestamps of the two need to be synchronized (generally, the sensor synchronization accuracy needs to be within 10ms). However, the camera module products on the market are uneven. If the timestamp of the visual sensor and the timestamp of the IMU are not based on the same time reference, there may be a large difference in the time delay between the two, which seriously affects the subsequent use. Therefore, verifying the timestamp delay of multiple sensors becomes a key issue.

[0003] At present, the timestamp estimation of multiple sensors in the industry can be divided into offline and online methods. Among them, offline methods generally require the use of external equipment (such as a chessboard of fixed size or a high-precision turntable, etc.), which is expensive or has low accuracy. Online methods generally require the use of external algorithms (such as VIO or visual SLAM). This method has great limitations and has strict requirements on scenes and movement modes. It also requires a large amount of computing power. In addition, the accuracy of the estimation is closely related to the accuracy of the algorithm, which introduces additional uncertainty. In addition, considering that timestamp delay estimation is only one of the optimization items, it needs to be optimized in combination with other optimization items. The online method relies on external algorithms and needs to move according to a certain trajectory, which is cumbersome, so this method is not advisable. Therefore, how to design a simple, convenient, and high-precision timestamp estimation method is a difficult problem. Summary of the invention

[0004] Aiming at the time synchronization problem among multiple sensors, a multi-sensor timestamp delay estimation method was proposed.

[0005] The technical solution of the present invention is: a multi-sensor timestamp delay estimation method, which specifically includes the following steps: 1) selecting a reference object that is fixed and includes a complete straight line, and using a device to make the visual sensor rotate left-right symmetrically in situ within 180 degrees along the direction of the complete straight line, and ensuring that the partial line segments of the complete straight line in the reference object are always within the visual angle range of the visual sensor during the rotation of the visual sensor, the visual sensor repeats the left-right symmetrical rotation for several cycles, and collects time series image data of the reference object. Through image processing technology, the rotation cycle angle data of a cycle of the visual sensor is solved as an angle group, and the calculated rotation angles of several cycles of the visual sensor are combined into a first angle sequence;

[0006] 2) The inertial measurement unit rotates synchronously with the visual sensor in step 1) according to the period and angle range of the visual sensor. The rotation is expressed by Euler angles through the gravity acceleration component to solve the movement of the group of roll and pitch, and the angle data of each rotation period combined in time sequence is solved to obtain the corresponding second angle sequence;

[0007] 3) The frequencies of the two sets of angle sequence data obtained in steps 1) and 2) are different. The low-frequency data sequence is aligned to the high-frequency data sequence, and two sets of sequences with the same frequency are obtained by using the cubic spline interpolation method;

[0008] 4) Perform a sliding dot product operation on the two sets of same-frequency sequences obtained in step 3) to calculate the offset between the two sets of sequences, and estimate the time delay between the two sensors based on the offset and the data frequency.

[0009] Further, the step 1) is specifically implemented by:

[0010] 1.1) Use a horizontally placed top-view visual sensor to capture continuous images of the ceiling, detect the longest point set through Hough transform, and then fit a straight line as the initial main direction;

[0011] The specific steps for determining the main direction are as follows:

[0012] 1.1.1) Convert the image data captured by the top-view visual sensor into a grayscale image;

[0013] 1.1.2) Perform edge extraction on the converted grayscale image;

[0014] 1.1.3) Binarize the image to determine whether it is an edge point;

[0015] 1.1.4) Map each pixel in the image to Hough space;

[0016] 1.1.5) Take the local maximum value and filter the interference line through the set threshold;

[0017] 1.1.6) Take the longest detected line segment as the main direction;

[0018] 1.2) The visual sensor is rotated symmetrically on the spot for several cycles not exceeding 180 degrees. The image acquired in each cycle is detected in the main direction according to the image processing technology of 1.1.1) to 1.1.6). A set of main directions will be obtained. By converting adjacent frames, an angle sequence relative to the initial main direction is obtained, which is the first angle sequence.

[0019] The beneficial effects of the present invention are as follows: the multi-sensor timestamp delay estimation method of the present invention can estimate the time delay between multiple sensors through simple testing means and image recognition, and is suitable for detection systems, positioning systems, and identification systems that are currently used by a large number of sensors to improve the system detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of the multi-sensor timestamp delay estimation method of the present invention;

[0021] Figure 2 Schematic diagram of visual compass detection in the multi-sensor timestamp delay estimation method of the present invention. DETAILED DESCRIPTION

[0022] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0023] The present invention provides a multi-sensor timestamp delay estimation method, such as Figure 1 The flowchart of the multi-sensor timestamp delay estimation method is shown in the figure. The time delay of the data detected by the visual sensor (Camera / TOF) and the inertial measurement unit (IMU) is estimated. The specific method is as follows:

[0024] 1. If Figure 2 As shown, a horizontally placed visual sensor (Camera or TOF) is used to shoot an image of the ceiling (reference object) at the top of the room, and the visual sensor is rotated symmetrically on the spot by no more than 180 degrees (through a turntable device or other devices that can control the rotation angle range and rotation rate of the visual sensor, this process ensures that the ceiling is always within its field of view), as shown in FIG. Figure 2During the rotation, the visual angle of the visual sensor remains unchanged. The dotted line is the angle of view of the camera. Only the visual sensor rotates in place and repeats the left-right symmetrical rotation for several cycles. The collected time series image data is detected by Hough transform to detect the straight line of the top ceiling. The longest one is used as the main direction. According to the transformation of the adjacent frame line segments, the change of the visual sensor's rotation angle relative to the initial position (initial definition is 0 degrees, and clockwise rotation is positive, counterclockwise rotation is negative) is calculated. It can act as a visual compass. One rotation cycle angle data is used as an angle group, and the calculated visual sensor rotation angle is combined into an angle sequence 1. Here, there is no need to use the traditional checkerboard method, nor is there a need for VIO or visual SLAM algorithms with high requirements for motion trajectories. The time data of the visual sensor's mobile acquisition data is solved from the image data of the fixed reference object. The specific implementation steps are as follows:

[0025] 1.1. Use a horizontally placed top-view vision module to capture continuous images of the ceiling, detect the longest point set through Hough transform, and then fit a straight line as the initial main direction;

[0026] The specific details are:

[0027] 1.1.1. Convert the acquired image data into a grayscale image;

[0028] 1.1.2. Extract the edge of the converted grayscale image, usually using the Canny operator;

[0029] 1.1.3. Binarize the image to determine whether it is an edge point;

[0030] 1.1.4. Map each pixel in the image to Hough space;

[0031] 1.1.5. Take the local maximum value and filter the interference line through the set threshold;

[0032] 1.1.6. Take the longest detected line segment as the main direction.

[0033] 1.2. The visual module is rotated symmetrically on the spot for several cycles not exceeding 180 degrees (this process ensures that the ceiling is always within its field of view. The camera can be placed on a regular table indoors to meet this requirement, where the camera FOV exceeds 90 degrees and the camera is more than 2 meters above the ceiling). The images acquired in each cycle are detected in the main direction according to the above-mentioned image processing technology, and a set of main directions will be obtained. By converting adjacent frames, an angle sequence relative to the initial main direction is obtained, which is the first angle sequence.

[0034] 2. The inertial measurement unit (IMU) can obtain 3-axis accelerometer and 3-axis gyroscope data. Since the angle information obtained by integrating the angular velocity of the gyroscope cannot be guaranteed to be 0 degrees at the beginning, and it cannot be guaranteed that the initial angle is the same each time, an operation to align with the visual angle is required. This operation is relatively cumbersome. Therefore, the present invention uses the accelerometer data to rotate the inertial measurement unit using the same means according to the cycle and angle range of step 1, and also collects several cycles. The inertial measurement unit collects data through the gravity acceleration component and uses Euler angles to express the rotation. The rotation changes of the two axes can be solved, and the angle data of each rotation cycle can be solved. The corresponding angle sequence 2 can also be obtained.

[0035] 3. If the frequencies of the angle sequence obtained by the visual sensor and the angle sequence calculated by the gravity of the inertial measurement unit are inconsistent, it is necessary to interpolate and align the low-frequency data sequence to the high-frequency data sequence, and obtain two sets of angle sequences with the same frequency through cubic spline interpolation.

[0036] 4. Perform a sliding dot product operation on the two sets of same-frequency sequences obtained above (which can be implemented through python or matlab code) to calculate the offset of the two sets of data. The time delay between the two sensors can be estimated based on the offset and frequency. This method can also be used to estimate the time delay between multiple sensors based on step 1. For example, the TOF sensor commonly used in the field of robotics can be analogous to a visual sensor. Through a similar image processing method as mentioned above, the corresponding angle sequence can also be obtained. Based on the obtained angle sequence, the time delay between the three sensors of Camera, IMU, and TOF can be estimated. And the analogy can be applied to multiple sensors.

[0037] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A multi-sensor timestamp delay estimation method, characterized in that: The specific steps include: 1) Select a reference object that is fixed and includes a complete straight line, and use a device to make the visual sensor rotate left-right symmetrically in situ within 180 degrees along the direction of the complete straight line, and ensure that the partial line segment of the complete straight line in the reference object is always within the visual angle range of the visual sensor during the rotation of the visual sensor. The visual sensor repeats the left-right symmetrical rotation for several cycles, collects time series image data of the reference object, and uses image processing technology to solve the periodic angle data of the visual sensor rotation as an angle group, and the calculated several periodic visual sensor rotation angles form a first angle sequence; 2) The inertial measurement unit rotates synchronously with the visual sensor in step 1) according to the period and angle range of the visual sensor. The rotation is expressed by Euler angles through the gravity acceleration component to solve the movement of the group of roll and pitch, and the angle data of each rotation period combined in time sequence is solved to obtain the corresponding second angle sequence; 3) The frequencies of the two sets of angle sequence data obtained in steps 1) and 2) are different. The low-frequency data sequence is aligned to the high-frequency data sequence, and two sets of sequences with the same frequency are obtained by using the cubic spline interpolation method; 4) Perform a sliding dot product operation on the two sets of same-frequency sequences obtained in step 3) to calculate the offset between the two sets of sequences, and estimate the time delay between the two sensors based on the offset and the data frequency.

2. The multi-sensor timestamp delay estimation method according to claim 1, characterized in that: The specific implementation steps of step 1) are as follows: 1.1) Use a horizontally placed top-view visual sensor to capture continuous images of the ceiling, detect the longest point set through Hough transform, and then fit a straight line as the initial main direction; The specific steps for determining the main direction are as follows: 1.1.1) Convert the image data captured by the top-view visual sensor into a grayscale image; 1.1.2) Perform edge extraction on the converted grayscale image; 1.1.3) Binarize the image to determine whether it is an edge point; 1.1.4) Map each pixel in the image to Hough space; 1.1.5) Take the local maximum value and filter the interference line through the set threshold; 1.1.6) Take the longest detected line segment as the main direction; 1.2) The visual sensor is rotated symmetrically on the spot for several cycles not exceeding 180 degrees. The image acquired in each cycle is detected in the main direction according to the image processing technology of 1.1.1) to 1.1.6). A set of main directions will be obtained. By converting adjacent frames, an angle sequence relative to the initial main direction is obtained, which is the first angle sequence.

Images