Timestamp delay calibration method and device, equipment and storage medium

Abstract

The invention provides a timestamp delay calibration method, a timestamp delay calibration device, terminal equipment and a computer readable storage medium, and relates to the technical field of computer vision. The method comprises the following steps: acquiring pose data corresponding to a first data acquisition device, and determining angular velocities corresponding to a plurality of preset moments of the first data acquisition device according to the pose data so as to generate a first sequence; acquiring angular velocities corresponding to a plurality of preset moments of a second data acquisition device to generate a second sequence; and determining timestamp delays of the first data acquisition device and the second data acquisition device according to the first sequence and the second sequence. The embodiment of the invention aims to accurately determine the timestamp delay between the data, so that the precision of time migration is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a timestamp delay calibration method, a timestamp delay calibration device, a terminal device, and a computer-readable storage medium. Background Art

[0002] In computer vision systems, visual-inertial odometry (VIO) using cameras and inertial measurement units (IMUs) has achieved excellent results in recent years. This is because the camera data acquired by the camera and the inertial data acquired by the IMU are complementary. These methods that fuse visual and inertial information for pose estimation generally require strict timestamp alignment of the data. Failure to do so not only affects system accuracy but can also easily cause the system to crash.

[0003] Existing methods for timestamp delay calibration generally include offline calibration and online calibration. Offline calibration generally uses an external device or sensor as a trigger to simultaneously trigger data acquisition from the camera and IMU. By detecting the arrival time of the trigger signal in the data, the time offset between the visual and IMU data can be determined. Online calibration, on the other hand, uses synchronized visual and IMU datasets to estimate the time offset through optimization algorithms. However, both methods have drawbacks such as limited application scenarios, high cost of delay timestamp calibration, and low time offset accuracy. Summary of the Invention

[0004] The present application provides a timestamp delay calibration method, a timestamp delay calibration device, a terminal device and a computer-readable storage medium, which aim to accurately determine the timestamp delay between data, thereby improving the accuracy of time offset.

[0005] To achieve the above objectives, the present application provides a timestamp delay calibration method, which includes:

[0006] Acquiring posture data corresponding to a first data acquisition device, and determining angular velocities corresponding to the first data acquisition device at a plurality of preset moments based on the posture data to generate a first sequence;

[0007] acquiring the angular velocities corresponding to the second data acquisition device at the plurality of preset moments to generate a second sequence;

[0008] A timestamp delay of the first data acquisition device and the second data acquisition device is determined according to the first sequence and the second sequence.

[0009] In addition, to achieve the above-mentioned purpose, the present application also provides a timestamp delay calibration device, which includes:

[0010] a first sequence generating module, configured to obtain posture data corresponding to a first data acquisition device, and determine angular velocities corresponding to the first data acquisition device at a plurality of preset moments based on the posture data, so as to generate a first sequence;

[0011] A second sequence generating module is configured to obtain the angular velocities corresponding to the second data acquisition device at a plurality of preset moments to generate a second sequence;

[0012] A timestamp delay determining module is used to determine the timestamp delay of the first data acquisition device and the second data acquisition device according to the first sequence and the second sequence.

[0013] In addition, to achieve the above-mentioned purpose, the present application also provides a terminal device, which includes: a memory and a processor; wherein the memory is connected to the processor for storing programs, and the processor is used to implement any one of the timestamp delay calibration methods provided in the embodiments of the present application by running the programs stored in the memory.

[0014] In addition, to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements any one of the timestamp delay calibration methods provided in the embodiments of the present application.

[0015] The embodiments of the present application disclose a timestamp delay calibration method, a timestamp delay calibration device, a terminal device, and a computer-readable storage medium. By acquiring the posture data corresponding to the first data acquisition device and fitting the control points corresponding to the target curve according to the posture data, the angular velocity corresponding to the first data acquisition device at each moment is obtained; a first sequence is generated according to the angular velocity corresponding to the first data acquisition device at each moment; the angular velocity corresponding to the second data acquisition device at each moment is acquired to generate a second sequence; and the timestamp delay of the first data acquisition device and the second data acquisition device is determined according to the first sequence and the second sequence. In this way, an angular velocity with more precise time accuracy can be obtained by fitting the posture data of the first data acquisition device; and the angular velocity is then correlated with the angular velocity of the first data acquisition device to obtain a more accurate calculation result; thereby accurately determining the timestamp delay between the data, thereby improving the accuracy of the time offset, reducing the cost of delay timestamp calibration, and being applicable to most application scenarios, that is, suitable for scenarios with multiple cameras or multiple imu. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] Figure 1 This is a flowchart of a timestamp delay calibration method provided in an embodiment of the present application;

[0018] Figure 2 This is a schematic diagram of a step of obtaining the angular velocity corresponding to the first data acquisition device at each moment, provided by an embodiment of the present application;

[0019] Figure 3 This is a schematic diagram of the steps of the workflow of a timestamp delay calibration method provided in an embodiment of the present application;

[0020] Figure 4 This is a schematic block diagram of a timestamp delay calibration device provided in an embodiment of the present application;

[0021] Figure 5 This is a schematic block diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] The flowcharts shown in the accompanying drawings are illustrative only and do not necessarily include all content and operations / steps, nor do they necessarily need to be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual order of execution may vary depending on the actual situation. In addition, although the functional modules are divided in the device schematics, in some cases, the module division may be different from that shown in the device schematics.

[0024] The term "and / or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0025] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.

[0026] To facilitate understanding of the embodiments of the present application, some background information involved in the embodiments of the present application is briefly described below.

[0027] Existing methods for timestamp delay calibration generally include offline and online calibration. Offline calibration requires specialized data acquisition methods, resulting in high costs and complex positioning procedures. It is also difficult to use online or in scenes that change frequently, limiting its application scenarios and making it less versatile. Online calibration, through the use of the EKF algorithm or nonlinear optimization, requires a good initial value or a low delay between camera and inertial data. While this allows the VIO algorithm to run, positioning accuracy is also limited.

[0028] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a timestamp delay calibration method provided in an embodiment of the present application. In this embodiment, the timestamp delay calibration method can be applied to a terminal device.

[0029] The terminal device may be a camera, sensor, smart glasses, mobile phone, computer, robot, or other terminal device, and is not specifically limited here. The first data acquisition device and the second data acquisition device are communicatively connected to the terminal device. The first data acquisition device is configured to send posture data corresponding to the device to the terminal device; the second data acquisition device is configured to send the angular velocity corresponding to the device to the terminal device.

[0030] Exemplarily, the first data acquisition device and the second data acquisition device may be arranged on the same terminal device, or may be arranged on different terminal devices, which is not specifically limited here.

[0031] For example, the following describes the timestamp delay calibration method provided by the embodiment of the present application using a terminal device as smart glasses as an example. The smart glasses include a first data acquisition device and a second data acquisition device. The first data acquisition device can be a camera or other device, and the second data acquisition device can be an IMU or other device.

[0032] The smart glasses may be AR (Augmented Reality) glasses, VR (Virtual Reality) glasses, or any other type of glasses including a first data acquisition device and a second data acquisition device, and are not specifically limited here.

[0033] like Figure 1 As shown, the timestamp delay calibration method includes steps S101 to S104.

[0034] S101. Acquire posture data corresponding to a first data acquisition device, and determine the angular velocity corresponding to the first data acquisition device at multiple preset moments based on the posture data to generate a first sequence.

[0035] The first data acquisition device may be a device with an image or video function, such as a camera or sensor. The posture data may include the position information and posture information of the posture corresponding to each frame of the image. Taking a camera as an example, the posture data may include the position information and posture information of the posture corresponding to each frame of the image in the video in the camera data. The position information may indicate the position of the camera in a preset reference system when each frame of the image is captured, and the posture information may describe the orientation of the camera. The preset reference system may be set by the user and is not specifically limited here. The preset time may be a time set in advance by the user, for example, the preset time may be every second, every 3 seconds, or every nanosecond, and is not specifically limited here.

[0036] For example, taking the first data acquisition device as a camera, the camera's pose can include the camera's attitude (which can be represented by the camera's rotation angle). The camera's rotation angle can be determined based on a user-selected reference frame and is not specifically limited here. The camera's pose can be expressed in a variety of ways, such as quaternions, Euler angles, rotation matrices, and rotation vectors.

[0037] In some embodiments, the control points corresponding to the target curve are fitted based on the pose data to obtain the angular velocity corresponding to each of the preset moments of the first data acquisition device; and a first sequence is generated based on the angular velocity corresponding to each of the preset moments of the first data acquisition device. This allows for obtaining angular velocities with greater temporal precision by fitting the pose data of the first data acquisition device (e.g., the values ​​of integer pixels of a two-dimensional image can be fitted to obtain the values ​​of sub-pixel points).

[0038] The target curve may be a B-spline curve or other curves. The B-spline curve is a curve whose shape is locally controlled by control points.

[0039] Specifically, by connecting the control points corresponding to each posture in the posture data, a target curve is generated, and the corresponding target curve expression is generated based on the target curve, so that the angular velocity corresponding to the first data acquisition device at each preset moment can be calculated; finally, the first sequence is generated based on the angular velocity corresponding to the first data acquisition device at each preset moment.

[0040] In some embodiments, based on the posture corresponding to each frame of image, multiple control points corresponding to the target curve are fitted to obtain a target curve expression, wherein the target curve expression is used to represent the relationship between the posture corresponding to each frame of image and the posture of each control point; based on the target curve expression, the angular velocity corresponding to each preset moment is calculated based on the posture corresponding to each frame of image, to obtain the angular velocity corresponding to each preset moment of the first data acquisition device. In this way, by performing B-spline fitting on the posture data (for example, the values ​​of the points of the integer pixels of a two-dimensional image can be obtained by fitting the values ​​of the sub-pixel points), the angular velocity below the video timestamp interval can be accurately obtained.

[0041] Among them, the target curve expression can be a B-spline expression, which is used to represent the relationship between the posture corresponding to each frame image and the posture of each control point. It is used to represent the relationship between the posture corresponding to each frame image and the posture of each control point. In order to facilitate optimization calculations, it can be expressed by Lie groups and Lie algebras.

[0042] like Figure 2 As shown, exemplarily, fitting the control points corresponding to the target curve according to the posture data to obtain the angular velocity corresponding to the first data acquisition device at each moment includes steps S1011 to S1012.

[0043] S1011. Fitting multiple control points corresponding to a target curve according to the posture corresponding to each frame of image to obtain a target curve expression.

[0044] S1012. Based on the target curve expression, the angular velocity corresponding to each preset moment is calculated according to the posture corresponding to each frame of image, to obtain the angular velocity corresponding to each preset moment of the first data acquisition device.

[0045] Specifically, according to the posture corresponding to each frame of image, multiple control points corresponding to the target curve are fitted to obtain the target curve and generate the corresponding target curve expression; by derivatizing the target curve expression and substituting the posture corresponding to each frame of image into the derived target curve expression, the angular velocity corresponding to each posture is calculated, thereby obtaining the angular velocity corresponding to the first data acquisition device at each moment.

[0046] For example, the target curve expression is represented by Lie group and Lie algebra as an example. For a k-order cumulative B-spline curve, the target curve expression corresponding to the B-spline curve on SO3 can be written as follows:

[0047]

[0048] Among them, SO3 is the Lie group and Lie algebra representation of the posture, R(u) is the position information and posture information of the posture corresponding to each frame image, and the position information and posture information of the posture can be expressed as a rotation matrix on SO3, i represents the i-th control point, k represents the order of the b-spline curve, j represents the j-th control point, and u is the normalized time. At time t, this time is located at the control point [t i ,t i+1 ) time range, R i is the position information and posture information corresponding to the i-th control point on the target curve expression, can be expressed as a rotation vector, is the relative rotation angle between adjacent control points, λ j (u) is the weight corresponding to the relative rotation angle between adjacent control points.

[0049] In some embodiments, for a posture corresponding to any frame image, multiple control points corresponding to the posture are obtained, and the posture is normalized based on the multiple control points to obtain a rotation matrix; the postures corresponding to the multiple control points are obtained, and a rotation vector is determined based on the postures corresponding to the multiple control points; a posture mapping relationship is constructed based on the rotation matrix and the rotation vector, and multiple control points on the target curve are fitted based on the posture mapping relationship to obtain a target curve expression. In this way, the target curve expression can be accurately constructed, thereby obtaining the angular velocity corresponding to the first data acquisition device at each moment.

[0050] Among them, the posture corresponding to each frame image can correspond to multiple control points.

[0051] Specifically, the posture mapping relationship can be expressed by Lie groups and Lie algebras, and the following relationship is obtained:

[0052] R=Exp(φ^)

[0053] Where R is the rotation matrix, φ is the rotation vector, and ^ is the antisymmetric symbol used to represent the transformation from the rotation vector to the antisymmetric matrix.

[0054] The normalized time can be expressed as:

[0055]

[0056] Among them, u is the normalized time, i represents the i-th control point, and t represents the time corresponding to the control point.

[0057] Therefore, for the posture corresponding to any frame, multiple control points corresponding to the target curve can be obtained, and the normalized time can be obtained according to the multiple control points, so as to normalize the posture and obtain the rotation matrix.

[0058] In some embodiments, the relative rotation angles between adjacent control points are determined based on the corresponding poses of the adjacent control points; the coefficient and normalization coefficient corresponding to the order of the target curve are obtained, and the weight of each relative rotation angle is determined based on the coefficient and normalization coefficient corresponding to the order of the target curve; and the rotation vector is determined based on the relative rotation angles between adjacent control points and their corresponding weights. In this way, the rotation vector can be accurately determined.

[0059] Specifically, the relative rotation angle between adjacent control points can be expressed as:

[0060]

[0061] Among them, this formula can be obtained by reversing the formula corresponding to the posture mapping relationship. is the relative rotation angle between adjacent control points, Indicates the pose corresponding to the i+j-1th position, R i+j represents the pose corresponding to the i+jth pixel, and v represents the transformation from the antisymmetric matrix to the rotation vector.

[0062] Therefore, the relative rotation angles between adjacent control points can be determined according to the corresponding positions of the adjacent control points.

[0063] Specifically, the weight corresponding to the relative rotation angle between adjacent control points can be expressed as:

[0064] λ(u)=M k [1,u,u 2 ,...,u k-1 ] T

[0065] Among them, λ(u) is the weight corresponding to the relative rotation angle between adjacent control points, M k is the coefficient related to the order of the B-spline curve, u is the normalized time, [1,u,u 2 ,...,u k-1 ] T represents the normalization coefficient, that is, the coefficient related to u, and T represents the column vector transformation of the matrix.

[0066] Therefore, the weight of each relative rotation angle can be determined according to the coefficient corresponding to the order of the target curve and the normalization coefficient, thereby determining the rotation vector according to the relative rotation angles between adjacent control points and their corresponding weights.

[0067] For example, the following is the derivation process of the above formulas. For the accumulated B-spline curve, the control points are evenly distributed, and the B-spline expression on the scalar is as follows:

[0068]

[0069] Among them, p(u) is the corresponding parameter value in the posture corresponding to time u, and p(u) can be x u ,y u or z u Any one of x u ,y u and z u The combination obtains the corresponding pose at time u, x u is the position on the x-axis in the spatial coordinate system, y u is the position on the y-axis in the spatial coordinate system, z u is the position on the z-axis in the spatial coordinate system. i is the parameter value corresponding to the pose of the i-th control point, such as x i ,y i or z i U is the normalization coefficient, that is, U=[1,u,u 2 ,...,u k-1 ] T .

[0070] Therefore, p(u) can be expressed as (p i , p i+1 ,...p i+k-1 )These k control points are calculated.

[0071]

[0072] p i represents the value of the i-th control point, represents the control point p i+j To control point p i+j-1 The difference between this section.

[0073] M k is a coefficient related to the B-spline order. If the order is determined, this item is a definite quantity.

[0074] For example, for a 5th order B-spline,

[0075] In some embodiments, for the pose corresponding to any frame image, the target curve expression is recursively transformed to obtain a recursive expression corresponding to the target curve expression; the recursive expression corresponding to the target curve expression is differentiated to obtain a derivative expression corresponding to the target curve expression; and the angular velocity corresponding to each moment of the first data acquisition device is calculated based on the derivative expression corresponding to the target curve expression. In this way, the angular velocity corresponding to each moment of the first data acquisition device can be accurately determined.

[0076] Specifically, the recursive expression corresponding to the target curve expression can be expressed as:( j ) (u) = R ( j －1) (u)A j-1 (u) and R ( 1 ) (u) = R i .

[0077] Among them, in the formula In the A j (u) is a parameter proposed to facilitate subsequent calculations.

[0078] but It can be expressed as

[0079] Obviously, the above formula is the product of k rotation quantities. Under this definition:

[0080] R=R ( k ) (u) represents the product of k rotation quantities, which can be expressed recursively as follows:

[0081] R ( j ) (u) = R ( j －1) (u)A j-1 (u) and R ( 1 ) (u) = R i .

[0082] Specifically, by respectively ( 1 ) (u) = R i and formula R ( j ) (u) = R ( j －1) (u)A j-1 (u) is differentiated to obtain the derivative expression corresponding to the target curve expression.

[0083] Since the formula R ( 1 ) (u) = R i There is nothing related to u in the formula R ( 1 ) (u) = R i Taking partial derivatives we can get w ( 1 ) =0.

[0084] The following is the derivation process of the formula. ( j) (u) = R ( j －1) (u)A j-1 (u) can be expressed by taking partial derivatives

[0085] Shown as:

[0086]

[0087] From the rotation Poisson formula we get: Therefore, substituting into Poisson's formula we can get:

[0088]

[0089] but

[0090] in, Indicates the time derivative value of the weight corresponding to the relative rotation angle between adjacent control points.

[0091] Finally Substitution In, you can get

[0092] Therefore, the recursive expression of angular velocity is as follows:

[0093] w ( 1 ) =0

[0094]

[0095] Among them, w ( 1 ) is the angular velocity of the first data acquisition device at the first moment, w ( j ) is the angular velocity of the first data acquisition device at the jth moment, w represents the angular velocity of the first data acquisition device at the corresponding moment, j is the jth control point, i is the ith control point, and k is the order of the B-spline curve.

[0096] Therefore, the angular velocity corresponding to the first data acquisition device at each preset moment is calculated by the above formula.

[0097] Specifically, a first sequence is generated according to the angular velocities corresponding to the first data acquisition device at multiple preset moments.

[0098] The first sequence is a sequence composed of the Euclidean norms of the angular velocities corresponding to the first data acquisition device at multiple preset moments.

[0099] In some embodiments, the Euclidean norm of the angular velocity corresponding to each moment of the first data acquisition device is calculated, and the first sequence is generated using the Euclidean norm corresponding to each of the angular velocities.

[0100] The Euclidean norm refers to the distance norm. For example, in Euclidean space, it represents the distance between two points (i.e., the modulus of a vector). In the embodiment of the present application, it can be represented as the modulus of the angular velocity corresponding to the first data acquisition device at multiple preset moments.

[0101] Specifically, for the angular velocity corresponding to each preset moment of the first data acquisition device, the Euclidean norm of each angular velocity is calculated respectively, so as to construct a first sequence through the Euclidean norm of each angular velocity.

[0102] S102: Obtain the angular velocity corresponding to the second data acquisition device at the plurality of preset moments to generate a second sequence.

[0103] The second data acquisition device may be an IMU, a camera, a lidar or other device, which is not specifically limited here.

[0104] In some embodiments, the Euclidean norm of the angular velocity corresponding to the second data acquisition device at multiple preset moments is calculated, and the second sequence is generated using the Euclidean norm corresponding to each of the angular velocities.

[0105] The Euclidean norm refers to the distance norm. For example, in Euclidean space, it represents the distance between two points (i.e., the modulus of a vector). In the embodiment of the present application, it can be represented as the modulus of the angular velocity of the second data acquisition device at multiple preset moments.

[0106] Specifically, for the angular velocity corresponding to each moment of the second data acquisition device, the Euclidean norm of each angular velocity is calculated respectively, so as to construct a second sequence through the Euclidean norm of each angular velocity.

[0107] S104: Determine the timestamp delay of the first data acquisition device and the second data acquisition device according to the first sequence and the second sequence.

[0108] The timestamp delay is used to indicate the delay between the data collected by the first data collection device and the data collected by the second data collection device.

[0109] In some embodiments, a cross-correlation analysis is performed on the first and second sequences to generate a correlation coefficient sequence; the sequence length of the second sequence and the time interval between adjacent parameters in the second sequence are obtained, and the timestamp delay between the first and second data acquisition devices is determined based on the correlation coefficient sequence, the sequence length of the second sequence, and the time interval. Thus, a cross-correlation analysis can be performed by calculating the second norm of the angular velocity between the two data sets to determine the timestamp delay between the data sets.

[0110] The correlation coefficient sequence is a sequence consisting of the correlation coefficients of the first sequence and the second sequence. The first sequence includes the Euclidean norm of the angular velocity corresponding to each moment of the first data acquisition device, and the second sequence includes the Euclidean norm of the angular velocity corresponding to each moment of the second data acquisition device. The adjacent parameters in the second sequence are the adjacent Euclidean norms in the second sequence. The second norm refers to the straight-line distance between two vector matrices in space.

[0111] Since the present application performs cross-correlation analysis on angular velocities, it can more accurately obtain the timestamp delay between data compared to the implementation method of cross-correlation analysis on rotation angles.

[0112] Specifically, the correlation coefficient sequence can be expressed as:

[0113]

[0114] Where c(k) is the correlation coefficient sequence, L1 is the first sequence, L2 is the second sequence, l2 is the sequence length of the second sequence, and k is the kth item in the correlation coefficient sequence.

[0115] In some embodiments, a maximum correlation coefficient is determined from the correlation coefficient sequence, and a timestamp delay between the first and second data acquisition devices is determined based on the maximum correlation coefficient, the length of the second sequence, and the time interval. This allows accurate determination of the timestamp delay between the first and second data acquisition devices.

[0116] The maximum correlation coefficient is the maximum value of all correlation coefficients in the correlation coefficient sequence.

[0117] Specifically, all correlation coefficients in the correlation coefficient sequence are traversed, and the correlation coefficient corresponding to the maximum value is found as the maximum correlation coefficient; the delay is calculated according to the maximum correlation coefficient, the sequence length of the second sequence and the time interval, so as to determine the timestamp delay of the first data acquisition device and the second data acquisition device.

[0118] Exemplarily, the time stamp delay between the first data acquisition device and the second data acquisition device can be expressed as follows:

[0119] t=(k_max-l2)*Δt

[0120] Wherein, t is the time stamp delay between the first data acquisition device and the second data acquisition device, k_max is the maximum correlation coefficient, l2 is the sequence length of the second sequence, and Δt is the time interval between adjacent elements of the second sequence.

[0121] Generally, the time intervals between adjacent elements in the second sequence are the same. If the time intervals between adjacent elements in the second sequence are different, the average value of the time intervals between adjacent elements is taken as Δt.

[0122] According to the above formula, the time delay between the two data can be determined by calculating the second norm of the angular velocity between the two data and performing cross-correlation analysis.

[0123] In some embodiments, as Figure 3 As shown, taking the first data acquisition device as a camera and the second data acquisition device as an IMU as an example, the timestamp delay calibration method proposed in an embodiment of the present application is introduced from the overall process of timestamp delay calibration, including S201-S207.

[0124] S201: Obtain camera data, where the camera data includes the posture corresponding to each frame of image.

[0125] S202 : Fitting a plurality of control points corresponding to a B-spline curve according to the posture corresponding to each frame of image in the camera data to obtain an expression corresponding to the B-spline curve.

[0126] S203 , based on the expression corresponding to the B-spline curve, calculate the angular velocity corresponding to each frame of image according to the posture corresponding to the frame, and obtain the angular velocity corresponding to the camera at each moment.

[0127] S204: Calculate the Euclidean norm of the angular velocity corresponding to the camera at each moment and generate a first sequence.

[0128] S205 , obtaining the angular velocity corresponding to the IMU at each moment, calculating the Euclidean norm of the angular velocity corresponding to the IMU at each moment, and generating a second sequence.

[0129] S206: Calculate the mutual correlation coefficient between the first sequence and the second sequence.

[0130] S207. Determine the timestamp delay between the camera and the imu camera using the maximum value of the mutual correlation coefficient.

[0131] The embodiment of the present application discloses a timestamp delay calibration method. By acquiring the posture data corresponding to the first data acquisition device and fitting the control points corresponding to the target curve according to the posture data, the angular velocity corresponding to the first data acquisition device at each moment is obtained; according to the angular velocity corresponding to the first data acquisition device at each moment, a first sequence is generated; the angular velocity corresponding to the second data acquisition device at each moment is acquired to generate a second sequence; the timestamp delay of the first data acquisition device and the second data acquisition device is determined according to the first sequence and the second sequence. In this way, an angular velocity with more precise time accuracy can be obtained by fitting the posture data of the first data acquisition device; and the angular velocity is then correlated with the angular velocity of the first data acquisition device to obtain a more accurate calculation result; thereby accurately determining the timestamp delay between the data, thereby improving the accuracy of the time offset, reducing the cost of delay timestamp calibration, and being applicable to most application scenarios, that is, suitable for scenarios with multiple cameras or multiple imu.

[0132] See also Figure 4 , Figure 4 This is a schematic block diagram of a timestamp delay calibration device provided in an embodiment of the present application. Figure 4 As shown, the timestamp delay calibration device 300 includes: a first sequence generating module 301 , a second sequence generating module 302 and a timestamp delay determining module 303 .

[0133] A first sequence generating module 301 is configured to obtain posture data corresponding to a first data acquisition device, and determine angular velocities corresponding to the first data acquisition device at a plurality of preset moments based on the posture data to generate a first sequence;

[0134] A second sequence generating module 302 is configured to obtain the angular velocities corresponding to the second data acquisition device at a plurality of preset moments to generate a second sequence;

[0135] The timestamp delay determining module 303 is configured to determine the timestamp delays of the first data acquisition device and the second data acquisition device according to the first sequence and the second sequence.

[0136] In some embodiments, the posture data includes the posture corresponding to each frame of the image. The first sequence generation module 301 is also used to fit multiple control points corresponding to the target curve according to the posture corresponding to each frame of the image to obtain a target curve expression. The target curve expression is used to represent the relationship between the posture corresponding to each frame of the image and the posture of each control point; based on the target curve expression, the angular velocity corresponding to each preset moment is calculated according to the posture corresponding to each frame of the image to obtain the angular velocity corresponding to the first data acquisition device at each preset moment.

[0137] In some embodiments, the first sequence generation module 301 is also used to obtain multiple control points corresponding to the target curve for the posture corresponding to any frame image, and normalize the posture according to the multiple control points to obtain a rotation matrix; obtain the postures corresponding to the multiple control points, and determine the rotation vector according to the postures corresponding to the multiple control points; construct a posture mapping relationship according to the rotation matrix and the rotation vector, and fit the multiple control points on the target curve based on the posture mapping relationship to obtain a target curve expression.

[0138] In some embodiments, the first sequence generating module 301 is further configured to calculate the Euclidean norm of the angular velocity corresponding to each moment of the first data acquisition device, and generate the first sequence using the Euclidean norm corresponding to each angular velocity.

[0139] In some embodiments, the timestamp delay determination module 303 is also used to perform cross-correlation analysis on the first sequence and the second sequence to generate a correlation coefficient sequence; obtain the sequence length of the second sequence and the time interval between adjacent parameters in the second sequence, and determine the timestamp delay of the first data acquisition device and the second data acquisition device based on the correlation coefficient sequence, the sequence length of the second sequence and the time interval.

[0140] In some embodiments, the timestamp delay determination module 303 is further used to determine the maximum correlation coefficient from the correlation coefficient sequence; and determine the timestamp delay of the first data acquisition device and the second data acquisition device based on the maximum correlation coefficient, the sequence length of the second sequence and the time interval.

[0141] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0142] The methods and apparatus of the present application can be used in a wide variety of general-purpose or specialized computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above.

[0143] See also Figure 5 , Figure 5 A schematic block diagram of the structure of a terminal device 400 provided in an embodiment of the present application.

[0144] like Figure 5As shown, the terminal device 400 may include a processor 401 and a memory 402 , and the processor 401 and the memory 402 are connected via a bus 403 , such as an I 2 C (Inter-integrated Circuit) bus.

[0145] In an exemplary embodiment, processor 401 can be used to provide computing and control capabilities to support the operation of the entire terminal device. Processor 401 can be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0146] Specifically, the memory 402 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0147] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the embodiment of the present application, and does not constitute a limitation on the terminal device to which the embodiment of the present application is applied. The specific server may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0148] The processor 401 is configured to run a computer program stored in the memory, and implement any one of the timestamp delay calibration methods provided in the embodiments of the present application when executing the computer program.

[0149] In one embodiment, the processor 401 is used to run a computer program stored in a memory, and implement the following steps when executing the computer program: obtain posture data corresponding to a first data acquisition device, determine the angular velocity corresponding to the first data acquisition device at multiple preset moments based on the posture data to generate a first sequence; obtain the angular velocity corresponding to the second data acquisition device at multiple preset moments to generate a second sequence; determine the timestamp delay of the first data acquisition device and the second data acquisition device based on the first sequence and the second sequence.

[0150] In one embodiment, the processor 401 is also used to implement: fitting the control points corresponding to the target curve according to the posture data to obtain the angular velocity corresponding to the first data acquisition device at each preset moment; and generating a first sequence according to the angular velocity corresponding to the first data acquisition device at each preset moment.

[0151] In one embodiment, the posture data includes the posture corresponding to each frame of the image, and the processor 401 is further used to implement: according to the posture corresponding to each frame of the image, multiple control points corresponding to the target curve are fitted to obtain a target curve expression, and the target curve expression is used to express the relationship between the posture corresponding to each frame of the image and the posture of each control point; based on the target curve expression, the angular velocity corresponding to each preset moment is calculated according to the posture corresponding to each frame of the image, and the angular velocity corresponding to the first data acquisition device at each preset moment is obtained.

[0152] In one embodiment, the processor 401 is also used to implement: for the posture corresponding to any frame image, obtain multiple control points corresponding to the target curve, and normalize the posture according to the multiple control points to obtain a rotation matrix; obtain the postures corresponding to the multiple control points, and determine the rotation vector according to the postures corresponding to the multiple control points; construct a posture mapping relationship according to the rotation matrix and the rotation vector, and fit the multiple control points on the target curve based on the posture mapping relationship to obtain a target curve expression.

[0153] In one embodiment, the processor 401 is further configured to calculate the Euclidean norm of the angular velocity corresponding to each moment of the first data acquisition device, and generate the first sequence using the Euclidean norm corresponding to each angular velocity.

[0154] In one embodiment, the processor 401 is also used to implement: performing cross-correlation analysis on the first sequence and the second sequence to generate a correlation coefficient sequence; obtaining the sequence length of the second sequence and the time interval between adjacent parameters in the second sequence, and determining the timestamp delay of the first data acquisition device and the second data acquisition device based on the correlation coefficient sequence, the sequence length of the second sequence and the time interval.

[0155] In one embodiment, the processor 401 is further used to implement: determining the maximum correlation coefficient from the correlation coefficient sequence; determining the timestamp delay of the first data acquisition device and the second data acquisition device based on the maximum correlation coefficient, the sequence length of the second sequence and the time interval.

[0156] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the terminal device described above can refer to the corresponding process in the aforementioned terminal capability reporting method embodiment, and will not be repeated here.

[0157] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed, any of the timestamp delay calibration methods provided in the embodiments of the present application is implemented.

[0158] The computer-readable storage medium may be an internal storage unit of the smart glasses described in the aforementioned embodiment, such as the memory of the smart glasses. The computer-readable storage medium may also be an external storage device of the smart glasses, such as a plug-in hard drive, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the smart glasses.

[0159] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function, and the like.

[0160] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A time stamp delay calibration method, characterized in that: The method comprises: Acquire posture data corresponding to a first data acquisition device, and determine angular velocities corresponding to the first data acquisition device at a plurality of preset moments according to the posture data to generate a first sequence; Acquire the angular velocity corresponding to the second data acquisition device at a plurality of the preset moments to generate a second sequence; A timestamp delay of the first data acquisition device and the second data acquisition device is determined according to the first sequence and the second sequence.

2. The method according to claim 1, characterized in that Determining the angular velocity corresponding to the first data acquisition device at a plurality of preset moments according to the posture data to generate a first sequence includes: Fitting the control points corresponding to the target curve according to the posture data to obtain the angular velocity corresponding to the first data acquisition device at each preset time; A first sequence is generated according to the angular velocity corresponding to the first data acquisition device at each of the preset moments.

3. The method according to claim 2, characterized in that The posture data includes the posture corresponding to each frame of the image, and the control points corresponding to the target curve are fitted according to the posture data to obtain the angular velocity corresponding to the first data acquisition device at each preset time, including: According to the posture corresponding to each frame of image, multiple control points corresponding to the target curve are fitted to obtain a target curve expression, wherein the target curve expression is used to represent the relationship between the posture corresponding to each frame of image and the posture of each control point; Based on the target curve expression, the angular velocity corresponding to each preset moment is calculated according to the posture corresponding to each frame of image, so as to obtain the angular velocity corresponding to each preset moment of the first data acquisition device.

4. The method according to claim 3, characterized in that: The step of fitting a plurality of control points corresponding to the target curve according to the posture corresponding to each frame of the image to obtain a target curve expression includes: For a posture corresponding to any frame image, a plurality of control points corresponding to the posture are obtained, and the posture is normalized according to the plurality of control points to obtain a rotation matrix; Obtaining positions and postures corresponding to the plurality of control points, and determining a rotation vector according to the positions and postures corresponding to the plurality of control points; A posture mapping relationship is constructed according to the rotation matrix and the rotation vector, and multiple control points on the target curve are fitted based on the posture mapping relationship to obtain a target curve expression.

5. The method according to claim 1, characterized in that The step of generating a first sequence according to the angular velocity corresponding to the first data acquisition device at each moment includes: The Euclidean norm of the angular velocity corresponding to each moment of the first data acquisition device is calculated, and the first sequence is generated by the Euclidean norm corresponding to each of the angular velocities.

6. The method according to claim 1, characterized in that The determining the time stamp delay of the first data acquisition device and the second data acquisition device according to the first sequence and the second sequence includes: Performing cross-correlation analysis on the first sequence and the second sequence to generate a correlation coefficient sequence; The sequence length of the second sequence and the time interval between adjacent parameters in the second sequence are obtained, and the timestamp delay of the first data acquisition device and the second data acquisition device is determined according to the correlation coefficient sequence, the sequence length of the second sequence and the time interval.

7. The method according to claim 6, characterized in that The determining of the timestamp delay of the first data acquisition device and the second data acquisition device according to the correlation coefficient sequence, the sequence length of the second sequence and the time interval comprises: determining a maximum correlation coefficient from the sequence of correlation coefficients; The timestamp delay between the first data acquisition device and the second data acquisition device is determined according to the maximum correlation coefficient, the sequence length of the second sequence and the time interval.

8. A time stamp delay calibration device, characterized in that: The time stamp delay calibration device comprises: A first sequence generating module, used for acquiring posture data corresponding to a first data acquisition device, and determining angular velocities corresponding to the first data acquisition device at a plurality of preset moments according to the posture data, so as to generate a first sequence; A second sequence generating module, used for acquiring the angular velocity corresponding to the second data acquisition device at a plurality of the preset moments to generate a second sequence; A timestamp delay determination module is used to determine the timestamp delay of the first data acquisition device and the second data acquisition device according to the first sequence and the second sequence.

9. A terminal device, characterized in that: The terminal device includes: a memory and a processor; wherein the memory is connected to the processor and is used to store programs, and the processor is used to implement any one of the timestamp delay calibration methods described in claims 1 to 7 by running the programs stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the timestamp delay calibration method according to any one of claims 1 to 7.

Images