Multi-sensor calibration method, device and system and storage medium

Abstract

The invention relates to the field of sensors, and provides a multi-sensor calibration method, device and equipment and a computer storage medium. In the moving process of the intelligent glasses, image information is obtained through the visual sensor, and motion data information is obtained through the inertial measurement sensor; determining a position relation parameter and a time relation parameter between the visual sensor and the inertial measurement sensor according to the image information and the motion data information; and under the condition that the time relation parameter meets a preset condition, the intelligent glasses are calibrated according to the position relation parameter, so that the multi-sensor joint calibration efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of sensors, and in particular to a multi-sensor calibration method, device, system and storage medium. Background Art

[0002] Smart glasses are usually equipped with visual sensors, such as optical cameras to obtain the wearer's motion information so that it can be displayed according to the wearer's head movement. However, optical cameras are easily disturbed by rapid motion, lighting conditions and occlusion. Therefore, smart glasses are usually also equipped with inertial measurement sensors (Inertial Measurement Unit, IMU) to make up for the shortcomings of optical cameras. In order to make the IMU and optical cameras complement each other, it is necessary to jointly calibrate the visual sensors and inertial measurement sensors on the same smart glasses. However, the joint calibration requires a long time for manual data collection and equipment adjustment, etc., which has a certain complexity, resulting in low efficiency of joint calibration. How to improve the efficiency of multi-sensor joint calibration has become an urgent problem to be solved. Summary of the invention

[0003] The main purpose of this application is to provide a multi-sensor calibration method, device, equipment and computer storage medium, aiming to reduce the complexity of joint calibration and thus improve calibration efficiency.

[0004] In a first aspect, the present application provides a multi-sensor calibration method, wherein the terminal device is connected to a smart pair of glasses to be calibrated, the smart pair of glasses is disposed on a mechanical arm, the terminal device is further connected to the mechanical arm, the smart pair of glasses includes a visual sensor and an inertial measurement sensor, and the multi-sensor calibration method includes the following steps:

[0005] Controlling the mechanical arm to move along a preset trajectory so that the smart glasses acquire image information through the visual sensor and motion data information through the inertial measurement sensor during the movement;

[0006] Determine a position relationship parameter and a time relationship parameter between the visual sensor and the inertial measurement sensor according to the image information and the motion data information;

[0007] When the time relationship parameter satisfies a preset condition, the smart glasses are calibrated according to the position relationship parameter.

[0008] In a second aspect, the present application further provides a multi-sensor calibration device, the multi-sensor calibration device comprising:

[0009] An information acquisition unit, used to control the robot arm to move along a preset trajectory, so that the smart glasses can obtain image information through the visual sensor and motion data information through the inertial measurement sensor during the movement;

[0010] a parameter calculation unit, configured to determine a position relationship parameter and a time relationship parameter between the visual sensor and the inertial measurement sensor according to the image information and the motion data information;

[0011] The device calibration unit is used to calibrate the smart glasses according to the position relationship parameters when the time relationship parameters meet the preset conditions.

[0012] In a third aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the multi-sensor calibration method as described above is implemented.

[0013] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the multi-sensor calibration method as described above is implemented.

[0014] The present application provides a multi-sensor calibration method, device, equipment and computer storage medium. The present application controls the mechanical arm to move along a preset trajectory so that the smart glasses can obtain image information through the visual sensor and motion data information through the inertial measurement sensor during the movement; determine the position relationship parameters and time relationship parameters between the visual sensor and the inertial measurement sensor according to the image information and the motion data information; and calibrate the smart glasses according to the position relationship parameters when the time relationship parameters meet the preset conditions. Since the smart glasses to be calibrated are moved by the mechanical arm to collect data, and the collected image information and motion data information are automatically processed, the complexity of the joint calibration is reduced, and the efficiency of the joint calibration of the camera unit and the inertial measurement unit is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 A schematic diagram of a multi-sensor calibration method provided in one embodiment of the present application;

[0017] Figure 2 A diagram showing a usage scenario of a multi-sensor calibration method provided in one embodiment of the present application;

[0018] Figure 3 A schematic block diagram of a multi-sensor calibration device provided in one embodiment of the present application;

[0019] Figure 4 The present invention is a block diagram showing the structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0022] Embodiments of the present application provide a multi-sensor calibration method, apparatus, computer equipment, and computer-readable storage medium.

[0023] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0024] Please refer to Figure 1 , Figure 1 A schematic flow chart of a multi-sensor calibration method provided in an embodiment of the present application. The multi-sensor calibration method can be used in a terminal or a server to improve the efficiency of joint calibration of a visual sensor and an inertial measurement sensor.

[0025] Please refer to Figure 2 , Figure 2 This is a usage scenario diagram provided by an embodiment of the present application. Figure 2As shown, the multi-sensor calibration method is applied to a terminal device, the terminal device is connected to a mechanical arm provided with smart glasses, the smart glasses are provided with a visual sensor and an inertial measurement sensor, the smart glasses are controlled to move by the mechanical arm, and during the movement, motion data information is obtained by the inertial measurement sensor and image information of the calibration plate is obtained by the visual sensor, so as to perform joint calibration through the motion data information and the image information. The calibration plate can be, for example, a calibration plate with a black and white checkerboard pattern.

[0026] Among them, the inertial measurement sensor, referred to as IMU (Inertial Measurement Unit), is a sensor used to measure and report the speed, acceleration and other motion information of an object. Since IMU can respond quickly to the motion changes of an object and is not affected by lighting conditions, it can work in a completely dark environment, but it is easily affected by cumulative errors, especially in the case of long-term measurements. The camera unit provides relatively stable visual feedback and can provide rich environmental information under good lighting conditions, but it responds slowly to motion and is easily affected by lighting and occlusion. Therefore, IMU and visual sensors (such as visual cameras) can be used complementary in practical applications to provide more accurate and robust navigation and positioning algorithms.

[0027] In order to ensure that the data acquired by the IMU and the visual sensor can be accurately synchronized and integrated, it is necessary to jointly calibrate the IMU and the visual sensor so that the data collected by the two are in the same coordinate system. However, joint calibration is somewhat complex. When jointly calibrating the IMU and camera unit of smart glasses, data collection needs to be performed for each smart glasses separately, resulting in low data calibration efficiency. Based on this, the following multi-sensor calibration method is proposed.

[0028] like Figure 1 As shown, the multi-sensor calibration method provided in the embodiment of the present application includes steps S101 to S103.

[0029] Step S101: Control the robotic arm to move along a preset trajectory, so that the smart glasses acquire image information through the visual sensor and acquire motion data information through the inertial measurement sensor during the movement.

[0030] Exemplarily, since the calibration plate provides a known geometric shape and size, a reference coordinate system can be established through the calibration plate, thereby associating the motion data information of the IMU and the image information of the visual sensor into the same coordinate system, so that the data of the IMU and the camera unit can be compared and fused in the same coordinate system. Therefore, in order to obtain the image information of the calibration plate captured by the visual sensor at multiple angles and the motion data information of the IMU during the movement, the robotic arm is controlled to move along a preset trajectory and obtain image information and motion data information during the movement. Among them, the image information may include, for example, images or video frames captured by the visual camera, and the motion data information may include, for example, angular velocity information and acceleration information collected by the IMU during the movement of the smart glasses.

[0031] Exemplarily, the preset trajectory can be pre-set in the terminal device, and the instructions of the preset trajectory can be expressed in the joint space (Joint Space) or Cartesian space (Cartesian Space), wherein the joint space trajectory planning is directly performed in the joint coordinates of the robot arm, while the Cartesian space trajectory planning focuses on the path of the end effector of the robot arm in space, and the robot arm moves according to the preset trajectory pre-set in the terminal device. Since the robot arm is connected to the terminal device, the terminal device can control the robot arm to move along the preset trajectory. The robot arm is provided with a unit under test (UUT) fixture for mounting smart glasses, so that the smart glasses are clamped in the UUT fixture of the robot arm and move along the preset trajectory.

[0032] In some embodiments, before controlling the mechanical arm to move along a preset trajectory, the method further includes:

[0033] The device identification of the smart glasses is obtained, the preset trajectory and the starting position corresponding to the preset trajectory are determined according to the device identification, and the mechanical arm is initialized according to the preset trajectory and the starting position.

[0034] For example, although the installation positions of the IMU and camera unit in the smart glasses are usually fixed, in actual production, the IMU and camera unit in each smart glasses may have different degrees of deviation. The purpose of the multi-sensor calibration method provided in this embodiment is to determine the positional relationship between the IMU and camera unit in each smart glasses so that the two are located in the same coordinate system. Therefore, the device identification of the smart glasses is obtained so that after the joint calibration, the data obtained by the joint calibration can be written into each smart glasses according to the device identification for subsequent multi-sensor calibration.

[0035] For example, the device identification of the smart glasses can be obtained by scanning a barcode posted on the surface of the smart glasses, wherein the device identification can be a unique serial number (SN) for identifying and tracking the smart glasses. In addition, the model of the smart glasses can also be determined based on the device identification.

[0036] For example, since the positions of camera units of different models of smart glasses may be different, the preset trajectory and starting position can be set for each model according to actual needs, and the robotic arm is initialized when the test starts, and the preset trajectory and starting position are synchronized to the robotic arm, so that the robotic arm moves to the starting position of the preset trajectory, and the preset trajectory is read for subsequent movement according to the preset trajectory.

[0037] Step S102: determining a position relationship parameter and a time relationship parameter between the visual sensor and the inertial measurement sensor according to the image information and the motion data information.

[0038] Exemplarily, the purpose of multi-sensor joint calibration is to make different sensors located under the same spatial coordinates and reduce the time error between sensors. Therefore, it is necessary to calculate the position relationship parameters and time relationship parameters between the camera unit and the inertial measurement unit based on the image information and motion data information.

[0039] In some embodiments, determining the position relationship parameter and the time relationship parameter between the visual sensor and the inertial measurement sensor according to the image information and the motion data information includes:

[0040] determining visual parameters of the camera unit based on the image information;

[0041] Determining the position relationship parameter according to the visual parameter and the motion data information;

[0042] Determining a second timestamp of each of the motion data information according to the first timestamp of the image information;

[0043] The time difference between each of the first timestamps and the corresponding second timestamp is used as the time relationship parameter.

[0044] Exemplarily, the visual parameters include a first focal length, a second focal length, a first image principal point offset, and a second image principal point offset. In order to determine the positional relationship parameters between the visual sensor and the inertial measurement sensor, it is first necessary to determine the visual parameters of the visual sensor, wherein the visual parameters of the visual sensor include the first focal length f of the camera unit x , second focal length f y , the first image principal point offset c xand the second image principal point offset c y .

[0045] Specifically, the first focal length f x and the second focal length f y They represent the focal lengths of the visual sensor on the X-axis and Y-axis, which are related to the width and height of the image respectively; the offset of the first image principal point c x and the second image principal point offset c y They represent the X and Y coordinates of the imaging center on the camera imaging plane, respectively.

[0046] Exemplarily, the above-mentioned visual parameters of the visual sensor can be calculated based on the image information based on a computer vision algorithm (such as Zhang Zhengyou calibration method, etc.), which will not be elaborated here.

[0047] Exemplarily, the data acquired by the IMU and the visual sensor are all timestamped to identify the time when the data was collected. Since the IMU acquires data more frequently than the camera unit, the motion data information closest in time to the image information is determined through the first timestamp of the image information, and the second timestamp of the motion data information is acquired, and the time difference between the first timestamp and the second timestamp is used as the time relationship parameter.

[0048] In some implementations, the position relationship parameters include at least a rotation matrix and an offset vector, and determining the position relationship parameters according to the visual parameters and the motion data information includes:

[0049] A rotation matrix and an offset vector between the visual sensor and the inertial measurement sensor are determined according to the visual parameters and the motion data information.

[0050] For example, determining the rotation matrix and translation vector between the visual sensor and the IMU is a key step in sensor fusion and multi-sensor calibration.

[0051] Exemplarily, feature points on the calibration plate (such as the intersection of a chessboard) are extracted from the image information, and these feature points are matched in multiple images to determine their positions in the camera coordinate system, thereby calculating the positions of the feature points in the image information in the IMU coordinate system based on the motion information data and the known position of the calibration plate to obtain the rotation matrix and the offset vector The image information of the visual sensor is thus converted into the IMU coordinate system through the rotation matrix R and the offset vector t.

[0052] Specifically, assuming that P w is the position of the characteristic point of the calibration plate in the world coordinate system, P cis the position of the feature point in the camera coordinate system. The process of transformation using the rotation matrix R and the translation vector t can be expressed by the following formula: c =R·P w +t.

[0053] For example, the rotation matrix R and the translation vector t can be defined by the error function E(R, t) = ∑ i ||P c,i -(R.P w,i +t)|| 2 To calculate, where P c,i represents the predicted position of the feature point of the i-th calibration plate in the camera coordinate system, P w,i It represents the actual position of the feature point of the i-th calibration plate in the IMU coordinate system, where the IMU coordinate system can be the world coordinate system. Based on methods such as gradient descent and least squares method, the error function E(R, t) is minimized to find the optimal rotation matrix and translation vector. Of course, it is not limited to this and is not limited here.

[0054] It can be understood that the differences in time and position between the visual sensor and the IMU are structured through time relationship parameters and position relationship parameters so that the smart glasses can be calibrated according to the differences in time and position between the two.

[0055] Step S103: When the time relationship parameter meets a preset condition, calibrate the smart glasses according to the position relationship parameter.

[0056] In some implementations, calibrating the smart glasses according to the position relationship parameters includes:

[0057] According to the position relationship parameters, a coordinate conversion operation is performed on the image information acquired by the visual sensor and the motion data information acquired by the relationship measurement sensor, so that the image information and the motion data information are converted into the same target coordinate system.

[0058] Exemplarily, the positional relationship solved in step S102 is packaged in a .bin file format and stored in the storage unit of the corresponding smart glasses. Since the .bin file has a smaller storage space and a higher read and write speed, and can be accessed in different operating systems, it has good compatibility. Therefore, the positional relationship parameters are packaged in the .bin format. In this way, the smart glasses can convert the coordinate systems of the image information and the motion data information to each other during use, convert the image information to the coordinate system where the motion data information is located, or convert the motion data information to the coordinate system where the image information is located, or convert the image information and the motion data information to a preset reference coordinate system. The process of the coordinate conversion operation can refer to the coordinate conversion method in the relevant technology, which will not be repeated here. In this way, the data fusion of the image information and the motion data information is realized, and the effect of complementary functions of the visual sensor and the IMU is achieved.

[0059] In some implementations, when the time relationship parameter satisfies a preset condition, calibrating the smart glasses according to the position relationship parameter includes:

[0060] Determine the delay time between the image information and the motion data information according to the time relationship parameter;

[0061] When the delay duration is less than a preset duration threshold, the smart glasses are calibrated according to the position relationship parameters.

[0062] For example, since the frequencies of acquiring data between the IMU and the visual sensor are different, in order to ensure the accuracy of data fusion, before calibrating the smart glasses according to the position relationship parameters, it is necessary to determine whether the time error between the IMU and the visual sensor meets the preset conditions.

[0063] Specifically, when the delay duration represented by the time relationship parameter between the visual sensor and the IMU is less than the preset duration threshold, it is determined that the delay between the image information and the motion data information is small, and further joint calibration can be performed to ensure the time consistency of the image information and the motion data information, so that the image information and the motion data information can be accurately fused.

[0064] In some implementations, determining the delay duration between the image information and the motion data information according to the time relationship parameter includes:

[0065] The time relationship parameter with the largest value is determined as the delay duration.

[0066] It can be understood that if the time relationship parameter with the largest value, that is, the time relationship parameter with the largest time difference between the first timestamp and the second timestamp, is less than the preset time length threshold, it means that the time difference between all image information and motion data information can be maintained at a low level, and the image information and the motion data information can be considered to be synchronized; otherwise, it means that there is a certain delay between the image information and the motion data information, which means that the error between the image information and the motion data information is large, and the visual sensor or IMU needs to be re-debugged to improve the time accuracy of the smart glasses.

[0067] The multi-sensor calibration method provided in the above embodiment controls the mechanical arm to move along a preset trajectory so that the smart glasses acquire image information through the visual sensor and motion data information through the inertial measurement sensor during the movement; the position relationship parameters and time relationship parameters between the visual sensor and the inertial measurement sensor are determined according to the image information and the motion data information; and when the time relationship parameters meet the preset conditions, the smart glasses are calibrated according to the position relationship parameters. Since the smart glasses to be calibrated are moved by the mechanical arm to collect data, and the collected image information and motion data information are automatically processed, the efficiency of the joint calibration of the camera unit and the inertial measurement unit is improved.

[0068] See also Figure 3 , Figure 3 It is a schematic diagram of a multi-sensor calibration device provided in an embodiment of the present application. The multi-sensor calibration device can be configured in a terminal device to execute the aforementioned multi-sensor calibration method. The terminal device is connected to a smart glass to be calibrated with multiple sensors. The smart glasses are arranged on a robotic arm. The terminal device is also connected to the robotic arm. The smart glasses include a visual sensor and an inertial measurement sensor.

[0069] like Figure 3 As shown, the multi-sensor calibration device includes: an information acquisition unit 110, a parameter calculation unit 120, and a device calibration unit 130.

[0070] The information acquisition unit 110 is used to control the robot arm to move along a preset trajectory so that the smart glasses can obtain image information through the visual sensor and motion data information through the inertial measurement sensor during the movement;

[0071] A parameter calculation unit 120, configured to determine a position relationship parameter and a time relationship parameter between the visual sensor and the inertial measurement sensor according to the image information and the motion data information;

[0072] The device calibration unit 130 is used to calibrate the smart glasses according to the position relationship parameters when the time relationship parameters meet the preset conditions.

[0073] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0074] The method and apparatus of the present application can be used in many general or special computing system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0075] Exemplarily, the above method and apparatus may be implemented in the form of a computer program. The computer program may be implemented in Figure 4 Runs on the computer device shown.

[0076] See also Figure 4 , Figure 4 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. The computer device may be a server or a terminal.

[0077] like Figure 4 As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a storage medium and an internal memory.

[0078] The storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any multi-sensor calibration method.

[0079] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0080] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any multi-sensor calibration method.

[0081] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0082] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0083] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0084] Controlling the mechanical arm to move along a preset trajectory so that the smart glasses acquire image information through the visual sensor and motion data information through the inertial measurement sensor during the movement;

[0085] Determine a position relationship parameter and a time relationship parameter between the visual sensor and the inertial measurement sensor according to the image information and the motion data information;

[0086] When the time relationship parameter satisfies a preset condition, the smart glasses are calibrated according to the position relationship parameter.

[0087] In some implementations, the processor, in the process of determining the position relationship parameter and the time relationship parameter between the camera unit and the inertial measurement unit according to the image information and the motion data information, is configured to implement:

[0088] determining visual parameters of the camera unit based on the image information;

[0089] Determining the position relationship parameter according to the visual parameter and the motion data information;

[0090] Determining a second timestamp of each of the motion data information according to the first timestamp of the image information;

[0091] The time difference between each of the first timestamps and the corresponding second timestamp is used as the time relationship parameter.

[0092] In some implementations, the processor, in the process of calibrating the smart glasses according to the position relationship parameter when the time relationship parameter satisfies a preset condition, is used to implement:

[0093] Determine the delay time between the image information and the motion data information according to the time relationship parameter;

[0094] When the delay duration is less than a preset duration threshold, the smart glasses are calibrated according to the position relationship parameters.

[0095] In some implementations, the processor, in the process of determining the delay duration between the image information and the motion data information according to the time relationship parameter, is configured to implement:

[0096] The time relationship parameter with the largest value is determined as the delay duration.

[0097] In some implementations, the processor, in the process of determining the position relationship parameter according to the visual parameter and the motion data information, is configured to implement:

[0098] A rotation matrix and an offset vector between the visual sensor and the inertial measurement sensor are determined according to the visual parameters and the motion data information.

[0099] In some implementations, the processor, in the process of implementing the calibration of the smart glasses according to the position relationship parameters, is used to implement:

[0100] The position relationship parameters are packaged as a binary file and stored in the smart glasses, so that the smart glasses can obtain the position relationship parameters through the binary file when performing data fusion on the image information and the motion data information.

[0101] In some embodiments, the processor, in the process of controlling the robotic arm to move along a preset trajectory, is used to implement:

[0102] The device identification of the smart glasses is obtained, the preset trajectory and the starting position corresponding to the preset trajectory are determined according to the device identification, and the mechanical arm is initialized according to the preset trajectory and the starting position.

[0103] It should be noted that technical personnel in the relevant field can clearly understand that, for the convenience and brevity of description, the specific working process of the multi-sensor calibration described above can refer to the corresponding process in the aforementioned multi-sensor calibration control method embodiment, and will not be repeated here.

[0104] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the multi-sensor calibration method of the present application.

[0105] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0106] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0107] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0108] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A multi-sensor calibration method, the method is applied to a terminal device, the terminal device is connected to a smart pair of glasses to be calibrated, the smart pair of glasses is arranged on a mechanical arm, the terminal device is also connected to the mechanical arm, the smart pair of glasses at least includes a visual sensor and an inertial measurement sensor, and is characterized in that: The method comprises: Controlling the mechanical arm to move along a preset trajectory so that the smart glasses acquire image information through the visual sensor and motion data information through the inertial measurement sensor during the movement; Determine a position relationship parameter and a time relationship parameter between the visual sensor and the inertial measurement sensor according to the image information and the motion data information; When the time relationship parameter satisfies a preset condition, the smart glasses are calibrated according to the position relationship parameter.

2. The multi-sensor calibration method according to claim 1, characterized in that: The determining, according to the image information and the motion data information, the position relationship parameter and the time relationship parameter between the visual sensor and the inertial measurement sensor comprises: Determining visual parameters of the visual sensor according to the image information; Determining the position relationship parameter according to the visual parameter and the motion data information; Determining a second timestamp of each of the motion data information according to the first timestamp of the image information; The time difference between each of the first timestamps and the corresponding second timestamp is used as the time relationship parameter.

3. The multi-sensor calibration method according to claim 1, characterized in that: When the time relationship parameter satisfies a preset condition, calibrating the smart glasses according to the position relationship parameter includes: Determine the delay time between the image information and the motion data information according to the time relationship parameter; When the delay duration is less than a preset duration threshold, the smart glasses are calibrated according to the position relationship parameters.

4. The multi-sensor calibration method according to claim 3, characterized in that: The determining the delay duration between the image information and the motion data information according to the time relationship parameter includes: The time relationship parameter with the largest value is determined as the delay duration.

5. The multi-sensor calibration method according to claim 4, characterized in that: The position relationship parameters include at least a rotation matrix and an offset vector, and determining the position relationship parameters according to the visual parameters and the motion data information includes: A rotation matrix and an offset vector between the visual sensor and the inertial measurement sensor are determined according to the visual parameters and the motion data information.

6. The multi-sensor calibration method according to any one of claims 1 to 5, characterized in that: The calibrating the smart glasses according to the position relationship parameters includes: According to the position relationship parameters, a coordinate conversion operation is performed on the image information acquired by the visual sensor and the motion data information acquired by the relationship measurement sensor, so that the image information and the motion data information are converted to the same target coordinate system.

7. The multi-sensor calibration method according to any one of claims 1 to 5, characterized in that: Before controlling the mechanical arm to move along a preset trajectory, the method further includes: Obtain a device identification of the smart glasses, determine the preset trajectory and a starting position corresponding to the preset trajectory according to the device identification, and initialize the robotic arm according to the preset trajectory and the starting position.

8. A multi-sensor calibration device, characterized in that: The multi-sensor calibration device comprises: An information acquisition unit, used to control the robot arm to move along a preset trajectory, so that the smart glasses can obtain image information through the visual sensor and motion data information through the inertial measurement sensor during the movement; a parameter calculation unit, configured to determine a position relationship parameter and a time relationship parameter between the visual sensor and the inertial measurement sensor according to the image information and the motion data information; The device calibration unit is used to calibrate the smart glasses according to the position relationship parameters when the time relationship parameters meet the preset conditions.

9. A multi-sensor calibration system, characterized in that: The multi-sensor calibration system includes a terminal device, a robotic arm, a calibration board, a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein the terminal device is used to communicate with the smart glasses set on the robotic arm, wherein when the computer program is executed by the processor, the steps of the multi-sensor calibration method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the multi-sensor calibration method according to any one of claims 1 to 7 are implemented.

Images