Time-space calibration method, device and storage medium
Through the spatiotemporal calibration method of AR devices and external positioning devices, the camera and IMU are used to obtain local position and global positioning, which solves the problem that high-precision global positioning calibration cannot be achieved in the existing technology, and realizes the global position of AR application scenarios Pose sharing and virtual screen support, the calibration process is simple and efficient.
Patent Information
- Application Number
- CN202310012840.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-01-05
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-01-05
AI Technical Summary
The existing technology cannot realize high-precision global pose calibration of AR glasses, cannot support the sharing of global poses and virtual images in outdoor scenes, and cannot convert local coordinate systems and global coordinate systems.
The camera and inertial measurement unit IMU in the AR device obtain the local positioning posture, combine the identification code of the external positioning device to obtain the relative positioning posture and the global positioning posture, and calculate the spatiotemporal calibration parameters using the spatiotemporal calibration method to realize the AR device and the external positioning device. Space-time calibration to obtain high-precision global poses.
It realizes high-precision global pose calibration between AR devices and external positioning devices, supports multi-device and multiple people to share global poses and virtual images, and provides technical support for global pose sharing and three-dimensional virtual object anchoring in AR application scenarios. The calibration process is simple and efficient, and the flexible device startup method does not require hard synchronization.
Smart Images

Figure CN115908586B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a time-space calibration method, a device and a storage medium, and belongs to the technical field of augmented reality. Background Art
[0002] In recent years, with the popularity of augmented reality (AR), positioning and orientation service technology based on AR glasses has attracted widespread attention from researchers.
[0003] In the existing solutions, whether it is the three-dimensional registration method using identification codes or VIO without identification, the postures obtained are all local postures in the global coordinate system, and it is impossible to obtain high-precision global postures, and thus it is impossible to achieve posture sharing in a large range of outdoor scenes. In other words, the current calibration method cannot provide the conversion relationship between the local coordinate system and the global coordinate system of AR glasses, and cannot provide technical support for multiple devices and multiple people to share global postures and virtual images in outdoor scenes. Summary of the invention
[0004] The purpose of the present invention is to provide a time-space calibration method, device and storage medium to solve the problems existing in the prior art.
[0005] In order to achieve the above object, the present invention provides the following technical solutions:
[0006] According to a first aspect, an embodiment of the present invention provides a spatiotemporal calibration method for an AR device, wherein the AR device includes a camera and an inertial measurement unit (IMU). The method includes:
[0007] Acquire the local pose of the IMU and the camera according to the image data acquired by the camera and the IMU data collected by the IMU;
[0008] When scanning the identification code in the external positioning device, obtaining the relative position and posture of the camera relative to the identification code according to the identification code;
[0009] Obtaining external parameters of the camera;
[0010] Obtaining the global position and posture of the external positioning device;
[0011] According to the local pose, the relative pose, the external parameter and the global pose, the AR device and the external positioning device are subjected to spatiotemporal calibration to obtain spatiotemporal calibration parameters.
[0012] Optionally, obtaining the external parameters of the camera includes:
[0013] measuring the external parameter; and / or,
[0014] The extrinsic parameter is estimated according to the relative pose and the global pose at adjacent first and second moments.
[0015] Optionally, estimating the extrinsic parameter according to the relative pose and the global pose at adjacent first and second moments includes:
[0016] According to the relative posture of the first moment and the second moment, obtaining a first relative posture of the identification code at the two adjacent moments;
[0017] Acquire a second relative posture of the external positioning device at the two adjacent moments according to the global postures at the first moment and the second moment;
[0018] Calculate the extrinsic parameters of the camera according to the first relative pose and the second relative pose.
[0019] Optionally, performing spatiotemporal calibration on the AR device and the external positioning device according to the local pose, the relative pose, the external reference and the global pose to obtain spatiotemporal calibration parameters includes:
[0020] According to the local posture and relative posture at a third moment, obtaining the target local posture of the identification code at the third moment; the time difference between the third moment and the first moment is less than a preset threshold;
[0021] Acquire the target global pose of the identification code at the first moment according to the global pose at the first moment and the external parameter;
[0022] Performing linear interpolation on the target local pose and the target global pose;
[0023] The spatiotemporal calibration parameters are obtained according to the interpolated local pose of the target and the global pose of the target.
[0024] Optionally, acquiring the spatiotemporal calibration parameters according to the interpolated local pose of the target and the global pose of the target includes:
[0025] Calculating an optimization factor according to the interpolated local pose of the target and the global pose of the target;
[0026] According to the optimization factor and the preset optimization algorithm, spatial calibration parameters and temporal calibration parameters are obtained.
[0027] Optionally, the spatiotemporal calibration parameters include space calibration parameters and time calibration parameters, and the method further includes:
[0028] According to the various spatial calibration parameters calculated in the sliding window, the yaw angle between the local pose and the global pose is obtained;
[0029] Calculating the variance of each acquired yaw angle and the variance of each time calibration parameter within the sliding window;
[0030] If the variance of the yaw angle and the variance of the time calibration parameter are both greater than corresponding thresholds, the time-space calibration parameter is updated according to the sliding window.
[0031] Optionally, when scanning the identification code in the external positioning device, acquiring the relative position and posture of the camera relative to the identification code according to the identification code includes:
[0032] Obtaining identification information of the identification code;
[0033] If the identification information is information in an identification database, obtaining vertex coordinates of the identification code;
[0034] The relative pose is calculated according to the vertex coordinates.
[0035] Optionally, the method further comprises:
[0036] If the identification information is not the information in the identification database, prompt information is displayed, and the prompt information is used to prompt to re-acquire the relative posture.
[0037] In a second aspect, a time-space calibration device is provided, the device comprising a memory and a processor, the memory storing at least one program instruction, the processor implementing the method as described in the first aspect by loading and executing the at least one program instruction.
[0038] According to a third aspect, a computer storage medium is provided, wherein at least one program instruction is stored in the computer storage medium, and the at least one program instruction is loaded and executed by a processor to implement the method according to the first aspect.
[0039] According to the image data obtained by the camera and the IMU data collected by the IMU, the local pose of the IMU and the camera is obtained; when scanning the identification code in the external positioning device, the relative pose of the camera relative to the identification code is obtained according to the identification code; the external parameters of the camera are obtained; the global pose of the external positioning device is obtained; according to the local pose, the relative pose, the external parameters and the global pose, the AR device and the external positioning device are spatiotemporally calibrated to obtain spatiotemporally calibration parameters. The problem that spatiotemporal calibration cannot be achieved in the prior art is solved, and the AR device and the external positioning device can be spatiotemporally calibrated to obtain a high-precision global pose to meet the effect of more application scenarios.
[0040] At the same time, this application does not require the two sets of equipment to be started at the same time, does not require hard synchronization, and is flexible to use. Before the external device is started, the AR glasses always provide local posture. When the external device is started, the user can use AR glasses to view the external device to achieve the conversion of local posture to global posture, and at the same time estimate the clock offset between the two sets of equipment online, providing an efficient and fast initialization method for the multi-sensor fusion algorithm, realizing high-precision global positioning and orientation of AR glasses, and providing technical support for global posture sharing and three-dimensional virtual object anchoring in AR application scenarios.
[0041] Due to the loose coupling method, high-precision local positioning and orientation can be achieved before the external device is started. The calibration process only requires the user to use AR glasses to watch the external device for a period of time. The entire calibration process is simple and efficient.
[0042] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 A schematic diagram of an implementation environment involved in a time-space calibration method provided in an embodiment of the present invention;
[0044] Figure 2 A schematic diagram of the conversion relationship between an AR device and an external positioning device in a spatiotemporal calibration method provided by an embodiment of the present invention;
[0045] Figure 3 A method flow chart of a time-space calibration method provided by an embodiment of the present invention;
[0046] Figure 4 A possible schematic diagram of a time offset between an external device and AR glasses in a spatiotemporal calibration method provided by an embodiment of the present invention;
[0047] Figure 5 A flowchart of data matching between an external device and AR glasses provided in accordance with an embodiment of the present invention. DETAILED DESCRIPTION
[0048] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0049] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.
[0050] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0051] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0052] First, in order to facilitate understanding, a brief introduction is given to the various terms involved in this application.
[0053] VIO: Visual-Inertial Odometry, an algorithm that fuses camera and IMU data to achieve pose estimation.
[0054] IMU: Inertial Measurement Unit. Used to measure the three-axis acceleration and angular velocity of an object.
[0055] GNSS: Global Navigation Satellite System. A space-based radio navigation and positioning system that can provide users with all-weather 3D coordinates, speed, and time information at any location on the Earth's surface or in near-Earth space.
[0056] Apriltag: A visual benchmark library that can quickly detect landmarks and calculate relative positions using specific identification codes.
[0057] OpenCV: An open source, cross-platform computer vision and machine learning software library that implements many common algorithms for image processing and computer vision.
[0058] Please refer to Figure 1, which shows a schematic diagram of a possible implementation scenario involved in various embodiments of the present application, such as Figure 1 As shown, the implementation scenario includes: AR equipment and external positioning equipment.
[0059] The AR device includes a camera and an inertial measurement unit IMU. The camera is used to collect surrounding image data, and the IMU is used to collect accelerometer and gyroscope data. In various possible embodiments of the present application, a VIO algorithm is run in the AR device, and the VIO algorithm can calculate the relative pose based on the image data and the IMU. When the coordinate system of the initial pose is a local coordinate system, the VIO algorithm can only provide the local pose in real time. When the coordinate system of the initial pose is a global coordinate system or after the conversion from the local pose to the global pose is completed, the VIO algorithm can provide the global pose in real time.
[0060] The AR device can communicate with the external positioning device through a wireless connection. The wireless connection method can be wifi (wireless fidelity), Bluetooth, infrared, NFC (Near Field Communication), etc., which is not limited in this application.
[0061] The figure takes the AR device as AR glasses as an example, and does not limit its specific product implementation.
[0062] The external positioning device includes an external device and an identification code. The external device is equipped with GNSS and IMU, which can provide high-precision global pose. The identification code Apriltag is placed on the external device, and the two are fixedly connected, and there is an external parameter. Among them, the external parameter refers to the coordinate conversion relationship between the external device and the identification code. In actual implementation, the external positioning device can send the obtained global pose to the AR device through a wireless connection.
[0063] Spatial calibration: In this application, it refers to the coordinate conversion relationship between the local reference coordinate system of the AR device and the global coordinate system of the external positioning device. Please refer to Figure 2 , which shows the conversion relationship between the local reference coordinate system of the AR device and the global coordinate system of the Northeast Sky, combined with Figure 2 , Z of the local reference coordinate system w The axis points vertically upward to the sky, X w Axis, Y w Axis and origin O w Determined by user definition or VIO algorithm; X of the Northeast Sky global coordinate system n Axis, Y n Axis and Z n The axes point to the east, north, and sky respectively, with the origin O nThe first data after the external device is started is used as the origin. There is only one yaw angle between the local reference coordinate system and the northeast sky global coordinate system, and the two coordinate systems are fixed after each system startup.
[0064] Time calibration: in this application, it refers to estimating the time offset between the external device and the AR device.
[0065] In addition, in the following embodiments, various symbols involved are described as follows:
[0066] d represents the coordinate system of the external device, m represents the coordinate system of the identification code, w represents the local reference coordinate system of VIO, n represents the global coordinate system of the northeast sky, b represents the IMU coordinate system, and c represents the camera coordinate system. T = [q, p] represents the transformation matrix, which includes two parts: rotation and translation. is the transformation matrix from the external device to the global coordinate system of the northeast sky at time i, where and Represents the rotation quaternion and translation vector respectively. represents quaternion multiplication, × represents matrix multiplication, and the superscript -1 represents the inverse operation.
[0067] Please refer to Figure 3 , which shows a method flow chart of a time-space calibration method provided by an embodiment of the present application. The method described in this embodiment is used for Figure 1 In the AR device shown, Figure 3 As shown, the method includes:
[0068] Step 301, obtaining the local pose of the IMU and the camera according to the image data obtained by the camera and the IMU data collected by the IMU;
[0069] Optionally, the AR device can calculate the local pose based on the image data and IMU data through the VIO algorithm.
[0070] In this application, after the AR device is turned on, the local position and posture can be obtained in real time. Unless otherwise specified, the following examples are all based on the AR device being AR glasses.
[0071] Step 302, when scanning the identification code in the external positioning device, obtaining the relative position and posture of the camera relative to the identification code according to the identification code;
[0072] Optionally, this step includes:
[0073] First, obtain the image data of the identification code through the camera;
[0074] When the user uses the AR glasses to view the identification code, the camera in the AR glasses can obtain image data. In actual implementation, the image data can be dedistorted while obtaining the image data. Optionally, the image dedistortion can be performed through the open source library OpenCV.
[0075] The AR glasses and the external positioning device can be turned on at the same time or asynchronously. When calibration is required, turn on the external positioning device and use the AR glasses to view the identification code in the external positioning device.
[0076] Second, obtain the vertex coordinates of the identification code according to the image data;
[0077] Optionally, Apriltag may be used to detect the identification code of the image and identify the pixel coordinates of the four corner points.
[0078] Third, calculate the relative pose based on the vertex coordinates.
[0079] Calculate the relative position of the camera based on the identified vertex coordinates and the custom 3D space coordinates Alternatively, the pose calculation can be performed using the SOLVE_PNP_IPPE_SQUARE method in the open source library OpenCV to obtain the relative pose.
[0080] It should be supplemented that, before obtaining the vertex coordinates, this step can also obtain the identification information of the identification code, and execute the step of obtaining the vertex coordinates when the identification information is information in the identification database. For example, obtain the ID of the identification code, and when the ID belongs to the ID database, obtain the vertex coordinates. Through the above scheme, the effect of matching the wrong identification code and affecting the accuracy of subsequent calibration is avoided. If the identification information is not the information in the identification database, in order to ensure the calibration accuracy, a prompt message can be displayed, which is used to prompt the user to re-acquire the relative posture, and the user re-trigger the subsequent operation after viewing the prompt message.
[0081] Step 303, obtaining the external parameters of the camera;
[0082] Optionally, this step includes at least one of the following two possible implementations:
[0083] In a first possible implementation, an external parameter is measured;
[0084] In a second possible implementation manner, the extrinsic parameter is estimated according to the relative pose and the global pose at adjacent first and second moments.
[0085] In actual implementation, when both are included, after the external parameters are measured, if the user triggers the estimation of the external parameters, the external parameter estimation step is performed. If the user triggers the use of the measured external parameters, the subsequent steps are directly executed, and unless otherwise specified, this embodiment is described by taking the external parameter estimation as an example.
[0086] Optionally, the extrinsic parameter estimation step includes:
[0087] First, according to the relative posture at the first moment and the second moment, a first relative posture of the identification code at two adjacent moments is obtained;
[0088] Figure 4 is the time offset t between the external device and the AR glasses d . The external device and AR glasses do not require hardware synchronization and can be started asynchronously. The global pose of the external device is only valid when the user wears AR glasses to view the identification code. In order to improve the flexibility of use, in this embodiment, AR glasses are used as the main device, and their camera clock is used as the reference system clock to align the clock of the external device. In this embodiment, it is assumed that the camera and IMU are hardware synchronized or the clock offset is known, that is, the time offset between the two is not considered. Under the reference system clock, the external device is at time t i The data is transmitted to the AR glasses via WIFI, and the time when the AR glasses receive the data is t j Since the two sets of equipment are not hardware synchronized, there are problems such as different trigger times, data transmission delays, and inconsistent clocks. There is a time offset t between the two sets of equipment clocks. o =t i -t j , then t c =t d +t o . Time offset t o Indicates the time required to offset the external device and the AR glasses clock. In actual implementation, time t j is known, and the online time offset estimation is to estimate the time t that matches it. i .
[0089] Since there is an unknown time offset between AR glasses and external devices, in order to avoid the influence of time offset on the external parameter estimation results, this application estimates the external parameters in an offline manner. When collecting data, when the AR glasses are stationary, the external positioning device is also stationary. After a period of time (1s or more), the data is acquired again to offset the influence of time offset on the data at different times. Since the amount of data collected offline is limited, and the amount of calculation and real-time performance are not considered, all collected data are used for external parameter estimation.
[0090] In this step, at adjacent times i and j, the AR glasses obtain the relative position at time i through the identification code and the relative position at time j The specific acquisition method is as described in step 302 and will not be repeated here.
[0091] The first relative posture is calculated according to the relative postures at the two moments i and j obtained. Optionally, the first relative posture includes:
[0092]
[0093] Second, according to the global poses at the first moment and the second moment, obtaining a second relative pose of the external positioning device at two adjacent moments;
[0094] At adjacent moments i and j, the AR glasses obtain the information provided by external devices at the two moments. and Among them, the global posture at each moment is sent by the external positioning device through a wireless connection.
[0095] The second relative posture is calculated according to the global postures at the two moments i and j obtained. Optionally, the second relative posture includes:
[0096]
[0097] Third, the external parameters of the camera are calculated according to the first relative pose and the second relative pose.
[0098] Optionally, the external parameters can be solved by a nonlinear optimization algorithm. Specifically, according to the hand-eye calibration algorithm
[0099]
[0100] From the above equation we can get:
[0101]
[0102]
[0103] The above formula is solved by least squares using nonlinear optimization to obtain the external parameter
[0104] This application estimates and calibrates the fixed external parameters of the external device and the identification code based on the hand-eye calibration method by utilizing the global pose provided by the external device and the local pose calculated by the VIO algorithm, thereby providing a way to obtain the external parameters and making the final calibration result more accurate.
[0105] Step 304, obtaining the global position and posture of the external positioning device;
[0106] Step 305: Perform spatiotemporal calibration on the AR device and the external positioning device according to the local pose, relative pose, external parameters and global pose to obtain spatiotemporal calibration parameters.
[0107] Optionally, this step includes:
[0108] First, according to the local posture and relative posture at the third moment, the target local posture of the identification code at the third moment is obtained; the time difference between the third moment and the first moment is less than a preset threshold;
[0109] The frequency of the external device is different from that of the AR glasses, and there is an unknown timestamp. Therefore, this embodiment selects data with a timestamp difference within a certain range for data matching, and then performs bilinear interpolation on the data between the external device and the AR glasses to obtain data under the same reference clock. Figure 5 The figure shows a data matching flow chart of the external device and the AR glasses when the external device is in motion. At adjacent moments i and j, adjacent moments k0 and k1 are the closest times.
[0110] The third moment is k0, and the local pose at k0 is obtained as The relative position is The local pose of the target obtained is:
[0111]
[0112] Among them, the external parameters of IMU and camera It is obtained by offline calibration or real-time estimation based on the VIO algorithm.
[0113] Second, according to the global pose and external parameters at the first moment, the target global pose of the identification code at the first moment is obtained;
[0114] The target global pose is:
[0115]
[0116] Among them, the camera's external parameters is estimated through the above steps.
[0117] Third, linear interpolation is performed on the target local pose and the target global pose;
[0118] In order to obtain the poses at k0 and j at the same time, linear interpolation is performed on the poses of the external device and the AR glasses respectively. The specific formula is as follows:
[0119]
[0120] in, It is a spherical linear interpolation of quaternions with a ratio of α, which is equivalent to the interpolation on the Lie algebra, that is R represents the rotation matrix corresponding to the quaternion, Exp and Log represent the exponential and logarithmic operations of Lie algebra respectively.
[0121] Fourth, the spatiotemporal calibration parameters are obtained according to the interpolated target local pose and target global pose.
[0122] Optionally, this step includes:
[0123] (1) Calculate the optimization factor based on the interpolated target local pose and target global pose;
[0124]
[0125] (2) According to the optimization factor and the preset optimization algorithm, the spatial calibration parameters and the temporal calibration parameters are obtained.
[0126] In actual implementation, the open source nonlinear optimization calculation library can be used to calculate the optimization factor and the corresponding Jacobian matrix and t d That is, the spatial calibration parameters and temporal calibration parameters are calculated.
[0127] It should be supplemented that, in each of the above embodiments, after obtaining the spatiotemporal calibration parameters, the above method may further include the following steps:
[0128] First, the yaw angle between the local pose and the global pose is obtained according to the various spatial calibration parameters calculated in the sliding window;
[0129] Euler angles and quaternions can be converted to each other. In this embodiment, the definition of Euler angles is ZYX, that is, first rotate around the Z axis (yaw angle), then rotate around the Y axis (pitch angle), and finally rotate around the X axis (roll angle). Considering that there is only a yaw angle between the global posture and the local posture, and the pitch angle and roll angle are both zero, only the yaw angle is optimized during optimization. Among them, the yaw angle can be obtained through the quaternion Convert to Euler angles and get it directly.
[0130] This step obtains the yaw angle and time offset of each sliding window.
[0131] Second, the variance of each yaw angle and each time calibration parameter in the sliding window is calculated;
[0132] Third, if the variances of the yaw angle and time calibration parameters are both greater than the corresponding thresholds, the spatiotemporal calibration parameters are updated according to the sliding window.
[0133] If the variance of the yaw angle and the time offset are both less than the corresponding threshold, it means that the current estimated parameter accuracy is high. At this time, in order to reduce the processing complexity of the AR device, the calibration can be terminated. On the contrary, if the variance is greater than the preset threshold, it means that the current error is small. At this time, the sliding window can be used to continuously calculate and update the spatiotemporal calibration parameters.
[0134] Bilinear interpolation is used to match data between AR glasses and external devices, and loose coupling and nonlinear iteration are used to simultaneously estimate time and space. It is flexible to use, the calibration process is simple and efficient, and the calibration results are highly accurate.
[0135] In summary, the local pose of the IMU and the camera is obtained according to the image data obtained by the camera and the IMU data collected by the IMU; when scanning the identification code in the external positioning device, the relative pose of the camera relative to the identification code is obtained according to the identification code; the external parameters of the camera are obtained; the global pose of the external positioning device is obtained; according to the local pose, the relative pose, the external parameters and the global pose, the AR device and the external positioning device are spatiotemporally calibrated to obtain spatiotemporally calibration parameters. The problem that spatiotemporal calibration cannot be achieved in the prior art is solved, and the AR device and the external positioning device can be spatiotemporally calibrated to obtain a high-precision global pose to meet the effect of more application scenarios.
[0136] At the same time, this application does not require the two sets of equipment to be started at the same time, does not require hard synchronization, and is flexible to use. Before the external device is started, the AR glasses always provide local posture. When the external device is started, the user can use AR glasses to view the external device to achieve the conversion of local posture to global posture, and at the same time estimate the clock offset between the two sets of equipment online, providing an efficient and fast initialization method for the multi-sensor fusion algorithm, realizing high-precision global positioning and orientation of AR glasses, and providing technical support for global posture sharing and three-dimensional virtual object anchoring in AR application scenarios.
[0137] Due to the loose coupling method, high-precision local positioning and orientation can be achieved before the external device is started. The calibration process only requires the user to use AR glasses to watch the external device for a period of time. The entire calibration process is simple and efficient.
[0138] The present application also provides a time-space calibration device, which includes a memory and a processor, wherein the memory stores at least one program instruction, and the processor implements the above-mentioned method by loading and executing the at least one program instruction.
[0139] The present application also provides a computer storage medium, in which at least one program instruction is stored. The at least one program instruction is loaded and executed by a processor to implement the method described above.
[0140] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0141] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.
Claims
1. A spatiotemporal calibration method, It is characterized in that Used in an AR device, the AR device includes a camera and an inertial measurement unit IMU, and the method includes: Acquire the local pose of the IMU and the camera according to the image data acquired by the camera and the IMU data collected by the IMU; When scanning the identification code in the external positioning device, obtaining the relative position and posture of the camera relative to the identification code according to the identification code; Obtaining external parameters of the camera; Obtaining the global position and posture of the external positioning device; According to the local posture, the relative posture, the external parameters and the global posture, the AR device and the external positioning device are subjected to spatiotemporal calibration to obtain spatiotemporal calibration parameters, including: obtaining the target local posture of the identification code at the third moment according to the local posture and the relative posture at the third moment; the time difference between the third moment and the first moment is less than a preset threshold; according to the target local posture at the first moment and the external parameters, obtaining the target global posture of the identification code at the first moment; linearly interpolating the target local posture and the target global posture; obtaining the spatiotemporal calibration parameters according to the interpolated target local posture and the target global posture; obtaining the spatiotemporal calibration parameters according to the interpolated target local posture and the target global posture, including: calculating the optimization factor according to the interpolated target local posture and the target global posture; obtaining the spatial calibration parameters and the temporal calibration parameters according to the optimization factor and the preset optimization algorithm.
2. The method according to claim 1, It is characterized in that The obtaining of the external parameters of the camera includes: measuring the external parameter; and / or, The extrinsic parameter is estimated according to the relative pose and the global pose at adjacent first and second moments.
3. The method according to claim 2, It is characterized in that The estimating the external parameter according to the relative pose and the global pose at adjacent first and second moments includes: According to the relative posture of the first moment and the second moment, obtaining a first relative posture of the identification code at the two adjacent moments; Acquire a second relative posture of the external positioning device at the two adjacent moments according to the global postures at the first moment and the second moment; Calculate the extrinsic parameters of the camera according to the first relative pose and the second relative pose.
4. The method according to any one of claims 1 to 3, It is characterized in that The spatiotemporal calibration parameters include space calibration parameters and time calibration parameters, and the method further includes: According to the various spatial calibration parameters calculated in the sliding window, the yaw angle between the local pose and the global pose is obtained; Calculating the variance of each acquired yaw angle and the variance of each time calibration parameter within the sliding window; If the variance of the yaw angle and the variance of the time calibration parameter are both greater than corresponding thresholds, the spatiotemporal calibration parameter is updated according to the sliding window.
5. The method according to any one of claims 1 to 3, It is characterized in that When scanning the identification code in the external positioning device, obtaining the relative position of the camera relative to the identification code according to the identification code includes: Obtaining identification information of the identification code; If the identification information is information in an identification database, obtaining vertex coordinates of the identification code; The relative pose is calculated according to the vertex coordinates.
6. The method according to claim 5, It is characterized in that The method further comprises: If the identification information is not the information in the identification database, prompt information is displayed, and the prompt information is used to prompt to re-acquire the relative posture.
7. A time-space calibration device, It is characterized in that The device includes a memory and a processor, wherein the memory stores at least one program instruction, and the processor implements the method according to any one of claims 1 to 6 by loading and executing the at least one program instruction.
