Multi-sensor calibration method, calibration device, calibration system and storage medium

Abstract

The invention discloses a multi-sensor calibration method, a calibration device, a calibration system and a computer readable storage medium. The multi-sensor calibration method comprises the following steps: fixing a head-mounted display device on a mechanical arm, carrying out joint calibration on an outward camera and an inertial navigation sensor according to a first calibration plate and a second calibration plate, carrying out hand-eye calibration on an inward camera and the mechanical arm, and carrying out hand-eye calibration on the outward camera and the mechanical arm. According to the multi-sensor calibration method, calibration among the inward camera, the outward camera and the inertial navigation sensor is carried out, so that one-stop calibration of the inward camera, the outward camera and the inertial navigation sensor can be realized, and the calibration efficiency and the calibration precision of multi-sensor calibration can be improved.

Description

Technical Field

[0001] This application relates to the technical field of sensor calibration, and particularly relates to a multi-sensor calibration method, a calibration device, a calibration system, and a computer-readable storage medium. Background Art

[0002] With the progress of technology, in order to implement functions such as simultaneous localization, mapping, gesture recognition, and eye movement tracking, head-mounted display devices include more and more different types of cameras facing outwards and inwards, as well as inertial measurement units (IMUs). Multi-sensor fusion technology is the mainstream technology in the field of mixed reality (MR) currently. The prerequisite for multi-sensor perception and positioning systems of MR devices is multi-sensor calibration. However, current multi-sensor calibration technologies rarely involve one-stop calibration of different types of cameras facing outwards and inwards, and the calibration efficiency of multi-sensor calibration technologies is low. Summary of the Invention

[0003] Embodiments of this application provide a multi-sensor calibration method, a calibration device, a calibration system, and a computer-readable storage medium to solve at least one of the above-mentioned technical problems.

[0004] The multi-sensor calibration method of the embodiments of this application is applied to a head-mounted display device. The head-mounted display device includes an in-facing camera, an out-facing camera, and an inertial measurement unit. During the calibration process, the head-mounted display device is fixed relative to a robotic arm. The robotic arm is also provided with a second calibration plate that can deflect relative to the in-facing camera for calibrating the in-facing camera, and a first calibration plate that is fixedly arranged at a certain position away from the robotic arm. The multi-sensor calibration method includes:

[0005] Jointly calibrate the out-facing camera and the inertial measurement unit based on the first calibration plate to obtain a first conversion relationship between the out-facing camera and the inertial measurement unit;

[0006] Perform hand-eye calibration on the in-facing camera and the robotic arm based on the pose transformation relationship between the second calibration plate and the robotic arm at different times to obtain a second conversion relationship between the in-facing camera and the robotic arm base coordinate system;

[0007] Perform hand-eye calibration on the out-facing camera and the robotic arm to obtain a third conversion relationship between the out-facing camera and the robotic arm base coordinate system;

[0008] Determine a fourth conversion relationship between the in-facing camera and the out-facing camera according to the second conversion relationship and the third conversion relationship;

[0009] Determine a fifth transformation relationship among the inward-facing camera, the outward-facing camera, and the inertial navigation sensor according to the first transformation relationship and the fourth transformation relationship.

[0010] In some embodiments, the jointly calibrating the outward-facing camera and the inertial navigation sensor based on the first calibration board to obtain a first transformation relationship between the outward-facing camera and the inertial navigation sensor includes:

[0011] Perform static calibration on the inertial navigation sensor according to the static inertial data collected by the inertial navigation sensor to obtain static calibration data of the inertial navigation sensor;

[0012] Perform internal and external parameter calibration on the outward-facing camera according to a plurality of first calibration picture data obtained by the outward-facing camera photographing the first calibration board to obtain a first pose of the outward-facing camera in the first calibration board coordinate system;

[0013] Jointly calibrate the outward-facing camera and the inertial navigation sensor according to the static calibration data, the dynamic inertial data collected by the inertial navigation sensor, and the first pose to obtain a first transformation relationship between the outward-facing camera and the inertial navigation sensor.

[0014] In some embodiments, the hand-eye calibration of the inward-facing camera and the robotic arm based on the pose transformation relationship between the second calibration board and the robotic arm at different times to obtain a second transformation relationship between the inward-facing camera and the robotic arm base coordinate system includes:

[0015] Obtain a second pose of the robotic arm in the robotic arm base coordinate system according to the motion trajectory of the robotic arm;

[0016] Perform internal and external parameter calibration on the inward-facing camera according to a plurality of second calibration picture data obtained by the inward-facing camera photographing the second calibration board to obtain a third pose of the inward-facing camera in the second calibration board coordinate system;

[0017] Obtain the pose change of the second calibration board during the motion of the robotic arm according to the second pose and the third pose;

[0018] Solve for a fourth pose of the inward-facing camera in the robotic arm coordinate system according to the second pose, the third pose, and the pose change;

[0019] Determine a second transformation relationship between the inward-facing camera and the robotic arm base coordinate system according to the fourth pose and the second pose.

[0020] In some embodiments, the hand-eye calibration of the outward-facing camera and the robotic arm to obtain a third transformation relationship between the outward-facing camera and the robotic arm base coordinate system includes:

[0021] Obtain the fifth pose of the robotic arm in the robotic arm base coordinate system according to the motion trajectory of the robotic arm;

[0022] Solve for the sixth pose of the outward-facing camera in the robotic arm coordinate system according to the first pose and the fifth pose;

[0023] Determine the third transformation relationship between the outward-facing camera and the robotic arm base coordinate system according to the sixth pose and the fifth pose.

[0024] In some embodiments, the multi-sensor calibration method further includes:

[0025] Control the inward-facing camera, the outward-facing camera, and the inertial navigation sensor to synchronously collect calibration data;

[0026] Control the robotic arm to move along a predetermined trajectory so that the inward-facing camera, the outward-facing camera, and the inertial navigation sensor collect calibration data at different poses.

[0027] In some embodiments, the multi-sensor calibration method further includes:

[0028] Determine whether the frequencies at which the inward-facing camera, the outward-facing camera, and the inertial navigation sensor collect calibration data are the same;

[0029] When the frequencies are the same, use the calibration data collected by the inward-facing camera, the outward-facing camera, and the inertial navigation sensor for multi-sensor calibration;

[0030] When the frequencies are different, statistically analyze the calibration data collected by the inward-facing camera, the outward-facing camera, and the inertial navigation sensor according to timestamps, and use the calibration data corresponding to the common timestamps of the inward-facing camera, the outward-facing camera, and the inertial navigation sensor for multi-sensor calibration.

[0031] In some embodiments, one end of the robotic arm is fixedly installed on the arm base, and the other end of the robotic arm is connected to a fixture for fixing the head-mounted display device;

[0032] The controlling the robotic arm to move along a predetermined trajectory so that the inward-facing camera, the outward-facing camera, and the inertial navigation sensor collect calibration data at different poses includes:

[0033] Control the robotic arm to move along a predetermined trajectory so that the outward-facing camera captures a first calibration board fixed relative to the arm base to obtain a plurality of first calibration picture data at different poses; so that the inward-facing camera captures a second calibration board that can move relative to the fixture to obtain a plurality of second calibration picture data at different poses; so that the inertial navigation sensor collects dynamic inertial navigation data.

[0034] In some embodiments, when the robotic arm moves along a predetermined trajectory, the second calibration plate may deflect relative to the inward-facing camera under the action of inertial force or motor driving force.

[0035] In some embodiments, one end of the clamp is provided with a fixing plate, the head mounted display device is fixedly mounted on the fixing plate, and the other end of the clamp is provided with a sliding plate;

[0036] The second calibration plate is slidably disposed on the sliding plate along a first direction, and / or the sliding plate is slidably disposed on the fixture along a second direction;

[0037] The first direction is perpendicular to the second direction.

[0038] In some embodiments, one end of the clamp is provided with a fixing plate, the head mounted display device is fixedly mounted on the fixing plate, and the other end of the clamp is provided with a sliding plate;

[0039] The second calibration plate is rotatably arranged on the sliding plate along the third direction by a rotating bracket; and / or the second calibration plate includes a calibration plate body, a calibration plate inner frame and a calibration plate outer frame, the calibration plate body is fixedly arranged on the calibration plate inner frame, and the calibration plate inner frame is rotatably arranged on the calibration plate outer frame along the first direction;

[0040] The first direction is perpendicular to the third direction.

[0041] In some embodiments, a limiting bracket is further provided on the sliding plate, and the limiting bracket is located on the side of the second calibration plate facing away from the head-mounted display device, and is used to limit the rotation of the second calibration plate around the first direction and / or the third direction.

[0042] In some embodiments, before jointly calibrating the outward-facing camera and the inertial navigation sensor, the multi-sensor calibration method further includes:

[0043] Determining whether the inertial navigation sensor has completed static calibration;

[0044] When the inertial navigation sensor has not completed static calibration, the robotic arm is controlled to be stationary so that the inertial navigation sensor collects static inertial navigation data.

[0045] The multi-sensor calibration device according to the embodiment of the present application is applied to a head-mounted display device. The head-mounted display device includes an inward-facing camera, an outward-facing camera, and an inertial navigation sensor. During the calibration process, the head-mounted display device is fixed relative to a robotic arm. The robotic arm is further provided with a second calibration plate that can deflect relative to the inward-facing camera for calibrating the inward-facing camera, and a first calibration plate that is fixedly arranged at a certain position away from the robotic arm. The multi-sensor calibration device includes:

[0046] A calibration module, configured to jointly calibrate the outward-facing camera and the inertial navigation sensor based on the first calibration plate to obtain a first conversion relationship between the outward-facing camera and the inertial navigation sensor;

[0047] The calibration module is further configured to perform hand-eye calibration on the inward-facing camera and the robotic arm based on the pose transformation relationship between the second calibration plate and the robotic arm at different times to obtain a second conversion relationship between the inward-facing camera and the robotic arm base coordinate system;

[0048] The calibration module is further configured to perform hand-eye calibration on the outward-facing camera and the robotic arm to obtain a third conversion relationship between the outward-facing camera and the robotic arm base coordinate system;

[0049] A determination module, configured to determine a fourth conversion relationship between the inward-facing camera and the outward-facing camera according to the second conversion relationship and the third conversion relationship;

[0050] The determination module is further configured to determine a fifth conversion relationship between the inward-facing camera, the outward-facing camera, and the inertial navigation sensor according to the first conversion relationship and the fourth conversion relationship.

[0051] The multi-sensor calibration system according to the embodiment of the present application includes one or more processors and a memory. The memory stores a computer program. When the computer program is executed by the processor, the multi-sensor calibration method according to any of the above embodiments is implemented.

[0052] The computer-readable storage medium according to the embodiment of the present application has a computer program stored thereon. When the program is executed by a processor, the multi-sensor calibration method according to any of the above embodiments is implemented.

[0053] The multi-sensor calibration method, calibration device, calibration system, and computer-readable storage medium according to the embodiments of the present application fix the head-mounted display device on the robotic arm, and perform joint calibration on the outward-facing camera and the inertial navigation sensor, hand-eye calibration on the inward-facing camera and the robotic arm, and hand-eye calibration on the outward-facing camera and the robotic arm based on the first calibration board and the second calibration board, so as to perform calibration among the inward-facing camera, the outward-facing camera, and the inertial navigation sensor. In this way, one-stop calibration of the inward-facing camera, the outward-facing camera, and the inertial navigation sensor can be achieved, the calibration of multiple sensors can be efficiently completed, and the calibration accuracy of multi-sensor calibration can be improved.

[0054] Additional aspects and advantages of the embodiments of the present application will be given in part in the following description, become apparent in part from the following description, or be understood through the practice of the embodiments of the present application. Description of the Drawings

[0055] The above and / or additional aspects and advantages of the present application will become apparent and be easily understood from the description of the embodiments in conjunction with the following drawings, where:

[0056] Figure 1 is a schematic flowchart of the multi-sensor calibration method according to some embodiments of the present application;

[0057] Figure 2 is a schematic diagram of the connection state of the head-mounted display device and the second calibration board with the fixture according to some embodiments of the present application;

[0058] Figure 3 is a schematic diagram of the connection state of the head-mounted display device and the second calibration board with the fixture according to some embodiments of the present application;

[0059] Figure 4 is a schematic diagram of the structure of the robotic arm according to some embodiments of the present application;

[0060] Figure 5 is a schematic flowchart of the multi-sensor calibration method according to some embodiments of the present application;

[0061] Figure 6 is a schematic diagram of the calibration board according to some embodiments of the present application;

[0062] Figure 7 is a schematic flowchart of the multi-sensor calibration method according to some embodiments of the present application;

[0063] Figure 8 is a schematic flowchart of the multi-sensor calibration method according to some embodiments of the present application;

[0064] Figure 9 is a schematic flowchart of the multi-sensor calibration method according to some embodiments of the present application;

[0065] Figure 10 It is a schematic structural diagram of a fixture according to some embodiments of the present application;

[0066] Figure 11 It is a schematic diagram of the motion state of a robotic arm moving along a predetermined trajectory according to some embodiments of the present application;

[0067] Figure 12 It is a schematic flowchart of a multi-sensor calibration method according to some embodiments of the present application;

[0068] Figure 13 It is a schematic flowchart of a multi-sensor calibration method according to some embodiments of the present application;

[0069] Figure 14 It is a schematic flowchart of a multi-sensor calibration method according to some embodiments of the present application;

[0070] Figure 15 It is a schematic flowchart of a multi-sensor calibration method according to some embodiments of the present application;

[0071] Figure 16 It is a schematic module diagram of a multi-sensor calibration device according to some embodiments of the present application;

[0072] Figure 17 It is a schematic module diagram of a multi-sensor calibration system according to some embodiments of the present application;

[0073] Figure 18 It is a schematic diagram of the connection state between a computer-readable storage medium and a processor according to some embodiments of the present application. Specific embodiments

[0074] The following further describes the embodiments of the present application with reference to the accompanying drawings. The same or similar reference numerals in the drawings denote the same or similar elements or elements having the same or similar functions throughout. In addition, the embodiments of the present application described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and should not be construed as a limitation of the present application.

[0075] Please refer to Figures 1 to 5 , an embodiment of the present application provides a multi-sensor calibration method, which is applied to a head-mounted display device 100. The head-mounted display device 100 includes an inward-facing camera 10, an outward-facing camera 20, and an inertial navigation sensor 30. During the calibration process, the head-mounted display device 100 is fixed relative to the robotic arm 101. A second calibration plate 105 that can deflect relative to the inward-facing camera 10 for calibrating the inward-facing camera 10 and a first calibration plate fixedly arranged at a certain position away from the robotic arm 101 are further provided on the robotic arm 101. The multi-sensor calibration method includes:

[0076] 010: Calibrate the outward-facing camera 20 and the inertial navigation sensor 30 jointly based on the first calibration board to obtain the first conversion relationship between the outward-facing camera 20 and the inertial navigation sensor 30;

[0077] 020: Calibrate the in-facing camera 10 and the robotic arm 101 for hand-eye calibration based on the pose transformation relationship between the second calibration board 105 and the robotic arm 101 at different times to obtain the second conversion relationship between the in-facing camera 10 and the base coordinate system of the robotic arm;

[0078] 030: Calibrate the outward-facing camera 20 and the robotic arm 101 for hand-eye calibration to obtain the third conversion relationship between the outward-facing camera 20 and the base coordinate system of the robotic arm;

[0079] 040: Determine the fourth conversion relationship between the in-facing camera 10 and the outward-facing camera 20 according to the second conversion relationship and the third conversion relationship;

[0080] 050: Determine the fifth conversion relationship among the in-facing camera 10, the outward-facing camera 20, and the inertial navigation sensor 30 according to the first conversion relationship and the fourth conversion relationship.

[0081] In the multi-sensor calibration method of the embodiment of the present application, by fixing the head-mounted display device 100 on the robotic arm 101, jointly calibrating the outward-facing camera 20 and the inertial navigation sensor 30 based on the first calibration board and the second calibration board 105, calibrating the in-facing camera 10 and the robotic arm 101 for hand-eye calibration, and calibrating the outward-facing camera 20 and the robotic arm 101 for hand-eye calibration, the calibration among the in-facing camera 10, the outward-facing camera 20, and the inertial navigation sensor 30 is performed. In this way, one-stop calibration of the in-facing camera 10, the outward-facing camera 20, and the inertial navigation sensor 30 can be achieved, the calibration of multiple sensors is efficiently completed, and the calibration accuracy of multiple sensors is improved.

[0082] Among them, the in-facing camera 10 can be an eye movement camera, and the eye movement camera can include a left-eye movement camera and a right-eye movement camera. The outward-facing camera 20 can include a color (RGB) camera and / or a time-of-flight (TOF) camera and / or a simultaneous localization and mapping (SLAM) camera. For example, the outward-facing camera 20 can only include an RGB camera; or, the outward-facing camera 20 can only include a TOF camera; or, the outward-facing camera 20 can only include a SLAM camera; or the outward-facing camera 20 can include an RGB camera, a TOF camera, and a SLAM camera at the same time, which will not be exemplified one by one here. In the above various cases, the number of RGB cameras, TOF cameras, and SLAM cameras can be one or more, which is not limited here. Of course, the in-facing camera 10 and the outward-facing camera 20 can also include other types of cameras.

[0083] The first calibration board and the second calibration board 105 can adopt an apriltag calibration board as shown in Figure 6 the figure. Of course, other types of calibration boards can also be used, which are not limited here. When performing multi-sensor calibration, the external camera 20 and the inertial navigation sensor 30 can be jointly calibrated based on the first calibration board to obtain the first conversion relationship between the external camera 20 and the inertial navigation sensor 30.

[0084] The second calibration board 105 can deflect relative to the inward-facing camera 10. That is to say, the second calibration board 105 deflects relative to the overall body formed by the head-mounted display device 100 and the robotic arm 101 that is relatively fixed to the head-mounted display device 100 (i.e., relative to the robotic arm base coordinate system). Based on the pose transformation relationship between the second calibration board 105 and the robotic arm 101 at different times, the inward-facing camera 10 and the robotic arm 101 can be hand-eye calibrated to obtain the second conversion relationship between the inward-facing camera 10 and the robotic arm base coordinate system. The external camera 20 and the robotic arm 101 are hand-eye calibrated to obtain the third conversion relationship between the external camera 20 and the robotic arm base coordinate system.

[0085] Furthermore, according to the second conversion relationship between the inward-facing camera 10 and the robotic arm base coordinate system, and the third conversion relationship between the external camera 20 and the robotic arm base coordinate system, the fourth conversion relationship between the inward-facing camera 10 and the external camera 20 can be determined. Based on the first conversion relationship between the external camera 20 and the inertial navigation sensor 30, and the fourth conversion relationship between the inward-facing camera 10 and the external camera 20, the conversion relationship between the inward-facing camera 10 and the inertial navigation sensor 30 can be calculated, and then the fifth conversion relationship between the inward-facing camera 10, the external camera 20, and the inertial navigation sensor 30 is determined.

[0086] It should be noted that the focal length of the inward-facing camera 10 is small and the field of view is narrow; in contrast, the data acquisition range of the inertial navigation sensor 30 is wider. If the inward-facing camera 10 and the inertial navigation sensor 30 are directly jointly calibrated, it may lead to a large error in the calibration result. Therefore, in the implementation manner of this application, the first conversion relationship between the external camera 20 and the inertial navigation sensor 30, and the fourth conversion relationship between the inward-facing camera 10 and the external camera 20 are determined to determine the conversion relationship between the inward-facing camera 10 and the inertial navigation sensor 30, so as to achieve the calibration between the inward-facing camera 10 and the inertial navigation sensor 30. At the same time, the joint calibration process between the inward-facing camera 10 and the inertial navigation sensor 30 is reduced, and the calibration accuracy is improved. In this way, by fixing the head-mounted display device 100 on the robotic arm 101 to perform the calibration between the inward-facing camera 10, the external camera 20, and the inertial navigation sensor 30, the multi-sensor calibration is efficiently completed, and the calibration accuracy of the multi-sensor calibration is improved.

[0087] Please refer toFigures 2 to 5 , Figure 7 , in some embodiments, the multi-sensor calibration method further includes:

[0088] 060: Controlling the in-facing camera 10, the out-facing camera 20, and the inertial navigation sensor 30 to synchronously collect calibration data;

[0089] 070: Controlling the robotic arm 101 to move along a predetermined trajectory so that the in-facing camera 10, the out-facing camera 20, and the inertial navigation sensor 30 collect calibration data at different poses.

[0090] Specifically, the control terminal 102 determines the data to be collected for calibration and sends a trigger signal to trigger the in-facing camera 10, the out-facing camera 20, and the inertial navigation sensor 30 to start synchronously collecting calibration data at the same timestamp. When collecting calibration data, as Figure 4 and Figure 11 shown, the control terminal 102 controls the robotic arm 101 to move along a predetermined trajectory so that the in-facing camera 10, the out-facing camera 20, and the inertial navigation sensor 30 collect calibration data at different poses.

[0091] Please refer to Figures 2 to 4 、and Figure 8 , in some embodiments, the multi-sensor calibration method further includes:

[0092] 080: Judging whether the frequencies of collecting calibration data by the in-facing camera 10, the out-facing camera 20, and the inertial navigation sensor 30 are the same;

[0093] 090: When the frequencies are the same, using the calibration data collected by the in-facing camera 10, the out-facing camera 20, and the inertial navigation sensor 30 for multi-sensor calibration;

[0094] 0100: When the frequencies are different, statistically analyzing the calibration data collected by the in-facing camera 10, the out-facing camera 20, and the inertial navigation sensor 30 according to timestamps, and using the calibration data corresponding to the common timestamps of the in-facing camera 10, the out-facing camera 20, and the inertial navigation sensor 30 for multi-sensor calibration.

[0095] Specifically, since the in-facing camera 10, the out-facing camera 20, and the inertial navigation sensor 30 may collect calibration data at different frequencies, it is necessary to first judge whether the frequencies of collecting calibration data by the in-facing camera 10, the out-facing camera 20, and the inertial navigation sensor 30 are the same. If the frequencies are the same, the calibration data collected by the in-facing camera 10, the out-facing camera 20, and the inertial navigation sensor 30 can be directly used for multi-sensor calibration; if the frequencies are different, it is necessary to statistically analyze the calibration data collected by the in-facing camera 10, the out-facing camera 20, and the inertial navigation sensor 30 according to timestamps, and screen out the calibration data corresponding to the common timestamps of the in-facing camera 10, the out-facing camera 20, and the inertial navigation sensor 30.

[0096] It should be noted that there may be frame loss problems in the collected calibration data. To ensure the integrity of the calibration data, it is necessary to process the screened data for frame loss problems. The processing process can be as follows: Organize the calibration data of the inward-facing camera 10, the outward-facing camera 20, and the inertial navigation sensor 30 after screening into the same folder, and convert the format of the calibration data according to the type of sensor. Compare the converted calibration data through a list (list), traverse and compare the calibration data of multiple sensors, find the missing or lost calibration data points, and discard the remaining calibration data at the corresponding moments of the missing or lost calibration data points. Among the finally remaining calibration data, the calibration data at the same moment includes the calibration data of each sensor. In this way, the frame loss problem is effectively solved.

[0097] Please refer to Figures 2 to 4 、 Figure 9 and Figure 10 , in some embodiments, one end of the robotic arm 101 is fixedly installed on the arm base 103, the other end of the robotic arm 101 is connected to the fixture 104, and the fixture 104 is used to fix the head-mounted display device 100. Control the robotic arm 101 to move along a predetermined trajectory so that the inward-facing camera 10, the outward-facing camera 20, and the inertial navigation sensor 30 collect calibration data (i.e., 070) in different poses, including:

[0098] 071: Control the robotic arm 101 to move along a predetermined trajectory so that the outward-facing camera 20 takes pictures of the first calibration board fixed relative to the arm base 103 to obtain multiple first calibration picture data in different poses; make the inward-facing camera 10 take pictures of the second calibration board 105 that can move relative to the fixture 104 to obtain multiple second calibration picture data in different poses; make the inertial navigation sensor 30 collect dynamic inertial navigation data.

[0099] Specifically, as Figure 4 shown, one end of the robotic arm 101 is fixedly installed on the arm base 103, the other end of the robotic arm 101 is connected to the fixture 104 through a flange structure, and the head-mounted display device 100 is fixed on the fixture 104. It should be noted that the left-eye camera and the right-eye camera of the inward-facing camera 10 need to be calibrated separately. Therefore, the second calibration board 105 can include two calibration boards, and the left-eye camera and the right-eye camera respectively take pictures of the corresponding calibration boards to obtain multiple second calibration picture data in different poses. Compared with the specifications of the second calibration board 105, the specifications of the first calibration board are larger, and the specific specification sizes of the first calibration board and the second calibration board 105 can be determined according to the actual situation so that when the robotic arm 101 moves along a predetermined trajectory, the outward-facing camera 20 can take pictures of the complete and clear first calibration board, and the inward-facing camera 10 can take pictures of the complete and clear second calibration board 105.

[0100] During the calibration process, asFigure 4 and Figure 11 As shown in Figure 11 , the robotic arm 101 is controlled to move along a predetermined trajectory. Since the head-mounted display device 100 is fixed to the fixture 104, the outward-facing camera 20 can move along the predetermined trajectory with the robotic arm 101 via the fixture 104. The first calibration board is fixed relative to the arm base 103. The outward-facing camera 20 captures the first calibration board, and multiple first calibration picture data of different poses of the outward-facing camera 20 relative to the first calibration board can be obtained. The second calibration board 105 is also disposed on the fixture 104. When the robotic arm 101 moves along the predetermined trajectory, the second calibration board 105 and the inward-facing camera 10 can move along the predetermined trajectory with the robotic arm 101 via the fixture 104. The inward-facing camera 10 is fixed relative to the fixture 104, while the second calibration board 105 can move relative to the fixture 104. Therefore, the second calibration board 105 moves relative to the inward-facing camera 10. The inward-facing camera 10 captures the second calibration board 105, and multiple second calibration picture data of different poses of the inward-facing camera 10 relative to the second calibration board 105 can be obtained. During the movement of the robotic arm 101, the inertial sensor 30 collects dynamic inertial data, and the dynamic inertial data may include acceleration, angular velocity, etc.

[0101] In some embodiments, when the robotic arm 101 moves along the predetermined trajectory, the second calibration board 105 may deflect relative to the inward-facing camera 10 under the action of inertial force or motor driving force. It should be noted that when the second calibration board 105 deflects, it is always within the imaging range of the inward-facing camera 10 for calibration by the inward-facing camera 10. The second calibration board 105 can deflect in any direction, which is not limited herein.

[0102] For example, when the robotic arm 101 moves along the predetermined trajectory, under the action of inertial force or motor driving force: the second calibration board 105 can translate relative to the fixture 104 in the first direction and / or the second direction; wherein, the first direction is perpendicular to the second direction.

[0103] Specifically, as Figure 3As shown, the x-direction is the length direction of the fixture 104, the y-direction is the width direction of the fixture 104, and the z-direction is the height direction of the fixture 104. The three directions are perpendicular to each other pairwise. The first direction refers to the length direction of the fixture 104, the second direction refers to the width direction of the fixture 104, and the first direction is perpendicular to the second direction. When the robotic arm 101 moves along a predetermined trajectory, the second calibration plate 105 can translate relative to the fixture 104 along the first direction and / or the second direction under the action of the inertial force generated by the movement of the robotic arm 101; alternatively, motor control can be used to make the second calibration plate 105 translate relative to the fixture 104 along the first direction and / or the second direction under the action of the driving force of the motor. The second calibration plate 105 can translate only along the first direction; or, the second calibration plate 105 can translate only along the second direction; or, the second calibration plate 105 can translate along the first direction and the second direction simultaneously.

[0104] For another example, when the robotic arm 101 moves along a predetermined trajectory, under the action of the inertial force or the driving force of the motor: the second calibration plate 105 can rotate relative to the fixture 104 around the first direction and / or the third direction; wherein, the first direction is perpendicular to the third direction.

[0105] Specifically, as Figure 3 shown, the x-direction is the length direction of the fixture 104, the y-direction is the width direction of the fixture 104, and the z-direction is the height direction of the fixture 104. The three directions are perpendicular to each other pairwise. The first direction refers to the length direction of the fixture 104, the third direction refers to the height direction of the fixture 104, and the first direction is perpendicular to the third direction. When the robotic arm 101 moves along a predetermined trajectory, the second calibration plate 105 can rotate relative to the fixture 104 around the first direction and / or the third direction under the action of the inertial force generated by the movement of the robotic arm 101; alternatively, motor control can be used to make the second calibration plate 105 rotate relative to the fixture 104 around the first direction and / or the third direction under the action of the driving force of the motor. The second calibration plate 105 can rotate only around the first direction; or, the second calibration plate 105 can rotate only around the third direction; or, the second calibration plate 105 can rotate around the first direction and the third direction simultaneously.

[0106] Of course, the second calibration plate 105 can also deflect in other ways, which will not be exemplified one by one here.

[0107] Please refer to Figure 2 、 Figure 3 and Figure 10, in some embodiments, one end of the fixture 104 is provided with a fixing plate 1041, the head-mounted display device 100 is fixedly arranged on the fixing plate 1041, and the other end of the fixture 104 is provided with a sliding plate 1042. The second calibration plate 105 is slidably arranged on the sliding plate 1042 along a first direction, and / or the sliding plate 1042 is slidably arranged on the fixture 104 along a second direction.

[0108] Specifically, the head-mounted display device 100 is fixedly arranged on the fixing plate 1041 by a buckle, that is, the head-mounted display device 100 is fixed relative to the fixture 104. A card slot 1043 is arranged on the sliding plate 1042 along the first direction, and the second calibration plate 105 is slidably connected to the sliding plate 1042 through the card slot 1043, so that the second calibration plate 105 can translate relative to the fixture 104 along the first direction. A slide rail 1044 is arranged on the fixture 104 along the second direction, and the sliding plate 1042 is slidably connected to the fixture 104 through the slide rail 1044, so that the sliding plate 1042 can translate relative to the fixture 104 along the second direction, and further the second calibration plate 105 translates relative to the fixture 104 along the second direction. Among them, the second calibration plate 105 can be only slidably arranged on the sliding plate 1042 along the first direction; or, the sliding plate 1042 can be only slidably arranged on the fixture 104 along the second direction; or the second calibration plate 105 can be slidably arranged on the sliding plate 1042 along the first direction and the sliding plate 1042 can be slidably arranged on the fixture 104 along the second direction at the same time. It can be understood that the second calibration plate 105 is arranged facing the head-mounted display device 100.

[0109] It should be noted that the second calibration plate 105 is slidably arranged on the sliding plate 1042 along the first direction, and the distance between the two second calibration plates 105 in the first direction can be adjusted. In this way, the calibration of the head-mounted display device 100 with different pupil distances can be adapted.

[0110] Please refer to Figure 2 、 Figure 3 and Figure 10 , in some embodiments, one end of the fixture 104 is provided with a fixing plate 1041, the head-mounted display device 100 is fixedly arranged on the fixing plate 1041, and the other end of the fixture 104 is provided with a sliding plate 1042. The second calibration plate 105 is rotatably arranged on the sliding plate 1042 along a third direction through a rotating bracket 1045; and / or the second calibration plate 105 includes a calibration plate body, a calibration plate inner frame 1051 and a calibration plate outer frame 1052, the calibration plate body is fixedly arranged on the calibration plate inner frame 1051, and the calibration plate inner frame 1051 is rotatably arranged on the calibration plate outer frame 1052 along the first direction.

[0111] Specifically, the head-mounted display device 100 is fixedly arranged on the fixing plate 1041 through a buckle, that is, the head-mounted display device 100 is fixed relative to the fixture 104. The rotating bracket 1045 is arranged in the above-mentioned card slot 1043 along the third direction, the second calibration plate 105 is connected to the rotating bracket 1045, and the rotating bracket 1045 includes a bearing structure. The rotating bracket 1045 can translate relative to the card slot 1043 along the first direction, so that the second calibration plate 105 can achieve the above-mentioned translation along the first direction relative to the fixture 104. The second calibration plate 105 is rotatably arranged on the sliding plate 1042 along the third direction through the bearing structure of the rotating bracket 1045. The bearing structure can rotate around the third direction, so that the second calibration plate 105 can rotate around the third direction.

[0112] The second calibration plate 105 includes a calibration plate body, a calibration plate inner frame 1051 and a calibration plate outer frame 1052, and the calibration plate body is fixedly arranged on the calibration plate inner frame 1051. It can be understood that the calibration plate body is fixed on the side of the calibration plate inner frame 1051 facing the head-mounted display device 100. The calibration plate inner frame 1051 is rotatably arranged on the calibration plate outer frame 1052 along the first direction through two bearing structures. The two bearing structures are in the same position in the second direction and in the same height position in the third direction. The two bearing structures can rotate around the first direction, so that the calibration plate inner frame 1051 can rotate around the first direction; the calibration plate body is fixed on the calibration plate inner frame 1051 and can rotate around the first direction along with the calibration plate inner frame 1051.

[0113] Please refer to Figure 2 , Figure 3 and Figure 10 , in some embodiments, a limit bracket 1046 is further arranged on the sliding plate 1042. The limit bracket 1046 is located on the side of the second calibration plate 105 facing away from the head-mounted display device 100 and is used to limit the rotation of the second calibration plate 105 around the first direction and / or the third direction.

[0114] Specifically, as Figure 2 shown, the limit bracket 1046 can be a cylindrical structure. Of course, the limit bracket 1046 can also be other structures. For example, the limit bracket 1046 can be a cuboid structure or an irregular structure, which is not limited here. The number of the limit brackets 1046 is determined according to the actual situation, as Figure 2As shown, there are 4 limiting brackets 1046 provided on the sliding plate 1042. One side of the second calibration plate 105 faces the head-mounted display device 100, and the limiting brackets 1046 are located on the other side of the second calibration plate 105, that is, on the side of the second calibration plate 105 facing away from the head-mounted display device 100. The limiting brackets 1046 can limit the rotation range of the second calibration plate 105. When the second calibration plate 105 rotates around the first direction and / or the third direction to a certain angle, the limiting brackets 1046 can limit the second calibration plate 105 from continuing to rotate. In this way, it can prevent the rotation range of the second calibration plate 105 from being too large, resulting in the inward-facing camera 10 being unable to capture the front of the second calibration plate 105 and affecting the calibration accuracy.

[0115] Please refer to Figure 2 、 Figure 5 and Figure 12 In some embodiments, before jointly calibrating the outward-facing camera 20 and the inertial navigation sensor 30, the multi-sensor calibration method further includes:

[0116] 0110: Determine whether the inertial navigation sensor 30 has completed static calibration;

[0117] 0120: When the inertial navigation sensor 30 has not completed static calibration, control the robotic arm 101 to be stationary so that the inertial navigation sensor 30 can collect static inertial data.

[0118] Specifically, performing static calibration on the inertial navigation sensor 30 means making the inertial navigation sensor 30 collect static inertial data for a predetermined time in a stationary state, and analyzing the static inertial data to obtain white noise and random walk. Among them, the static inertial data can include angular velocity and acceleration. The predetermined time is determined according to the actual situation to ensure that sufficient static inertial data can be obtained.

[0119] Before jointly calibrating the outward-facing camera 20 and the inertial navigation sensor 30, it is necessary to first perform static calibration on the inertial navigation sensor 30. Performing static calibration on the inertial navigation sensor 30 requires collecting static inertial data. Before controlling the robotic arm 101 to move along a predetermined trajectory, the robotic arm 101 can be controlled to be stationary for a predetermined time to collect static inertial data and perform static calibration on the inertial navigation sensor 30. It can also be performed after controlling the robotic arm 101 to move along a predetermined trajectory, and there is no limitation here. To prevent missing the static calibration of the inertial navigation sensor 30, before jointly calibrating the outward-facing camera 20 and the inertial navigation sensor 30, it can be first determined whether the inertial navigation sensor 30 has completed static calibration. If the inertial navigation sensor 30 has completed static calibration, then start jointly calibrating the outward-facing camera 20 and the inertial navigation sensor 30. If the inertial navigation sensor 30 has not completed static calibration, then control the robotic arm 101 to be stationary for a predetermined time so that the inertial navigation sensor 30 can collect static inertial data.

[0120] Please refer to Figure 2 and Figure 13 , in some embodiments, the external-facing camera 20 and the inertial navigation sensor 30 are jointly calibrated based on the first calibration board to obtain a first conversion relationship (i.e., 010) between the external-facing camera 20 and the inertial navigation sensor 30, including:

[0121] 011: Perform static calibration on the inertial navigation sensor 30 according to the static inertial data collected by the inertial navigation sensor 30 to obtain the static calibration data of the inertial navigation sensor 30;

[0122] 012: Perform internal and external parameter calibration on the external-facing camera 20 according to a plurality of first calibration picture data captured by the external-facing camera 20 to obtain the first pose of the external-facing camera 20 in the first calibration board coordinate system;

[0123] 013: Perform joint calibration on the external-facing camera 20 and the inertial navigation sensor 30 according to the static calibration data, the dynamic inertial data collected by the inertial navigation sensor 30, and the first pose to obtain a first conversion relationship between the external-facing camera 20 and the inertial navigation sensor 30.

[0124] Specifically, if the inertial navigation sensor 30 has not completed static calibration before the joint calibration of the external-facing camera 20 and the inertial navigation sensor 30, perform static calibration on the inertial navigation sensor 30 according to the static inertial data to obtain the static calibration data of the inertial navigation sensor 30, including the above-mentioned white noise and random walk. Then, perform internal and external parameter calibration on the external-facing camera 20 according to a plurality of first calibration picture data collected by the external-facing camera 20. The process of internal and external parameter calibration is as follows: Extract corner points from the plurality of first calibration picture data. According to the pixel coordinates of the corner points in the image coordinate system and the calibration board coordinates of the corner points in the first calibration board coordinate system, the internal parameter matrix and the external parameter matrix of the external-facing camera 20 can be calculated. The external parameter matrix represents the pose transformation relationship of the external-facing camera 20 relative to the first calibration board coordinate system, that is, the external parameter matrix can represent the first pose of the external-facing camera 20 in the first calibration board coordinate system.

[0125] Based on the static calibration data and the first pose, the external-facing camera 20 and the inertial navigation sensor 30 can be jointly calibrated. Any camera-IMU joint calibration algorithm can be used, which is not limited here. For example, the conventional joint calibration process can be as follows: Estimate the pose information of the inertial navigation sensor 30 according to the dynamic inertial data of the inertial navigation sensor 30. Estimate an initial conversion relationship between the external-facing camera 20 and the inertial navigation sensor 30 according to the pose information of the inertial navigation sensor 30 and the first pose of the external-facing camera 20. Perform error compensation and correction on the initial conversion relationship through the static calibration data of the inertial navigation sensor 30 to obtain a first conversion relationship between the external-facing camera 20 and the inertial navigation sensor 30.

[0126] When there are multiple outward-facing cameras 20, first calibrate each outward-facing camera 20 and the inertial navigation sensor 30 to obtain the conversion relationship between each outward-facing camera 20 and the inertial navigation sensor 30. The multiple obtained conversion relationships represent the respective optimal conversion relationships between the multiple outward-facing cameras 20 and the inertial navigation sensor 30. Then, globally optimize these multiple conversion relationships to obtain the first conversion relationship (i.e., the globally optimal conversion relationship) between the multiple outward-facing cameras 20 and the inertial navigation sensor 30, achieving the calibration unification between the multiple outward-facing cameras 20 and the inertial navigation sensor 30.

[0127] Please refer to Figure 2 、 Figure 11 and Figure 14 In some embodiments, the in-facing camera 10 and the robotic arm 101 are hand-eye calibrated according to the pose transformation relationship between the second calibration board 105 and the robotic arm 101 at different times to obtain the second conversion relationship (i.e., 020) between the in-facing camera 10 and the robotic arm base coordinate system, including:

[0128] 021: According to the motion trajectory of the robotic arm 101, obtain the second pose of the robotic arm 101 in the robotic arm base coordinate system;

[0129] 022: Calibrate the internal and external parameters of the in-facing camera 10 according to the multiple second calibration picture data obtained by the in-facing camera 10 shooting the second calibration board 105 to obtain the third pose of the in-facing camera 10 in the second calibration board coordinate system;

[0130] 023: According to the second pose and the third pose, obtain the pose change of the second calibration board 105 during the motion of the robotic arm 101;

[0131] 024: Solve the fourth pose of the in-facing camera 10 in the robotic arm coordinate system according to the second pose, the third pose, and the pose change;

[0132] 025: Determine the second conversion relationship between the in-facing camera 10 and the robotic arm base coordinate system according to the fourth pose and the second pose.

[0133] Specifically, when the robotic arm 101 moves along a predetermined trajectory, the arm base 103 remains stationary, and the robotic arm base coordinate system represents the position of the arm base 103. The real-time motion trajectory of the robotic arm 101 can be output and saved through the Software Development Kit (SDK) of the robotic arm 101. According to the structure and joint parameters of the robotic arm 101, a kinematic model of the robotic arm 101 is established, including Denavit-Hartenberg (DH) parameters, geometric parameters of joints, kinematic parameters, etc. According to the real-time motion trajectory of the robotic arm 101, the pose transformation relationship of the robotic arm 101 relative to the arm base 103 at multiple moments is calculated using the forward kinematic equation, that is, the second pose of the robotic arm 101 in the robotic arm base coordinate system. Let the second pose at time t1 obtained by calculation be A1, the second pose at time t2 be A2, and so on. The corresponding second poses at subsequent multiple moments are A3, A4, A5, …, A n . It should be noted that the second pose of the robotic arm 101 in the robotic arm base coordinate system refers to the second pose of the end of the robotic arm 101 in the robotic arm base coordinate system.

[0134] The internal and external parameters of the inward-facing camera 10 are calibrated according to multiple sets of second calibration image data collected by the inward-facing camera 10. The process of internal and external parameter calibration is as follows: Corner points are extracted from multiple sets of second calibration image data. According to the pixel coordinates of the corner points in the image coordinate system and the calibration plate coordinates of the corner points in the second calibration plate coordinate system, the external parameter matrix of the inward-facing camera 10 at the corresponding moment of each set of second calibration image data can be calculated respectively. The external parameter matrix represents the pose transformation relationship of the inward-facing camera 10 relative to the second calibration plate coordinate system, that is, the external parameter matrix can represent the third pose of the inward-facing camera 10 in the second calibration plate coordinate system. Let the third pose at time t1 obtained by calculation be C1, the third pose at time t2 be C2, and so on. The corresponding third poses at subsequent multiple moments are C3, C4, C5, …, C n .

[0135] When the robotic arm 101 moves along a predetermined trajectory, the second calibration plate 105 will translate and rotate under the action of inertial force or motor driving force. Therefore, it is necessary to obtain the pose changes of the second calibration plate 105 at multiple moments during the movement of the robotic arm 101. The inward-facing camera 10 is fixedly connected to the robotic arm 101 through the fixture 104. Thus, the overall combination of the inward-facing camera 10 and the fixture 104 does not change its movement relative to the robotic arm 101. In other words, the change ΔB in the fourth pose of the overall combination of the inward-facing camera 10 and the fixture 104 relative to the robotic arm 101 from time t1 to time t2 is the identity matrix. Taking the second calibration plate 105 relative to the robotic arm base coordinate system as the reference benchmark, it is possible to obtain the pose changes of the second calibration plate 105 during the movement of the robotic arm 101 based on the second pose of the robotic arm 101 in the robotic arm base coordinate system and the third pose of the inward-facing camera 10 in the second calibration plate coordinate system. The pose change E1 at time t1 can be considered as the identity matrix, and the calculation formula for the pose change E2 at time t2 is as follows:

[0136]

[0137] And so on, the corresponding pose changes at subsequent multiple moments are E3, E4, E5, …, E n 。Based on the second pose, the third pose, and the pose changes, the fourth pose of the inward-facing camera 10 in the robotic arm coordinate system can be solved. The robotic arm coordinate system refers to the coordinate system at the end of the robotic arm 101. It can be understood that the head-mounted display device 100 is fixed on the fixture 104, that is, the inward-facing camera 10 is fixed relative to the robotic arm coordinate system. Therefore, the fourth pose remains unchanged and is denoted as B. An equation can be established based on the second poses, the third poses, the pose changes, and the fourth pose at time t1 and time t2 to solve for the fourth pose B, and the equation is as follows:

[0138]

[0139] After arrangement, we get:

[0140]

[0141] It can be understood that in the above equation, except for the fourth pose B which is unknown, the rest are all known, and the fourth pose B of the inward-facing camera 10 in the robotic arm coordinate system can be obtained through calculation. It should be noted that A1, A2, C1, C2, E1, and E2 are all 4×4 homogeneous transformation matrices. Therefore, the fourth pose B is a 4×4 homogeneous transformation matrix. The calculated fourth pose B can be expressed as:

[0142]

[0143] Among them, R1 is a 3×3 matrix representing the rotation matrix of the in-facing camera 10 relative to the robotic arm coordinate system; T1 is a 3×1 matrix representing the translation matrix of the in-facing camera 10 relative to the robotic arm coordinate system.

[0144] According to the fourth pose of the in-facing camera 10 in the robotic arm coordinate system and the second pose of the robotic arm 101 in the robotic arm base coordinate system, the second conversion relationship between the in-facing camera 10 and the robotic arm base coordinate system can be determined. It should be noted that multiple fourth poses can be calculated based on the above-mentioned second poses, third poses, and pose changes at multiple moments. An optimal second conversion relationship can be determined from multiple fourth poses and multiple second poses using an optimization algorithm. For example, the second conversion relationship can be determined using the least squares method or a non-linear optimization algorithm.

[0145] Please refer to Figure 2 and Figure 15 , in some embodiments, hand-eye calibration is performed on the out-facing camera 20 and the robotic arm 101 to obtain the third conversion relationship (i.e., 030) between the out-facing camera 20 and the robotic arm base coordinate system, including:

[0146] 031: According to the motion trajectory of the robotic arm 101, obtain the fifth pose of the robotic arm 101 in the robotic arm base coordinate system;

[0147] 032: Solve for the sixth pose of the out-facing camera 20 in the robotic arm coordinate system based on the first pose and the fifth pose;

[0148] 033: Determine the third conversion relationship between the out-facing camera 20 and the robotic arm base coordinate system based on the sixth pose and the fifth pose.

[0149] Specifically, it should be pointed out that the foregoing explanation of "According to the motion trajectory of the robotic arm 101, obtain the second pose of the robotic arm 101 in the robotic arm base coordinate system" in 021 also applies to the corresponding solution in 031 of the embodiments of the present application, and will not be elaborated here. It can be understood that the fifth pose is equivalent to the above-mentioned second pose. Let the fifth pose at the calculated t1 moment be A1, the fifth pose at the t2 moment be A2, and so on. The corresponding fifth poses at subsequent multiple moments are A3, A4, A5,..., A n . Let the first pose at the calculated t1 moment be G1, the first pose at the t2 moment be G2, and so on. The corresponding first poses at subsequent multiple moments are G3, G4, G5,..., G n .

[0150] According to the first pose of the outward-facing camera 20 in the first calibration board coordinate system and the fifth pose of the robotic arm 101 in the robotic arm base coordinate system, the sixth pose of the outward-facing camera 20 in the robotic arm coordinate system can be solved. The robotic arm coordinate system refers to the coordinate system at the end of the robotic arm 101. It can be understood that the head-mounted display device 100 is fixed on the fixture 104, that is, the outward-facing camera 20 is fixed relative to the robotic arm coordinate system. Therefore, the sixth pose remains unchanged and is denoted as F. An equation can be established based on the fifth poses, the first pose, and the sixth pose at time t1 and time t2 to solve for the sixth pose F. The equation is as follows:

[0151]

[0152] After arrangement, it is obtained:

[0153]

[0154] It can be understood that in the above equation, except for the sixth pose F which is unknown, the rest are known. By calculation, the sixth pose F of the outward-facing camera 20 in the robotic arm coordinate system can be obtained. It should be noted that A1, A2, G1, and G2 are all 4×4 homogeneous transformation matrices. Therefore, the sixth pose F is a 4×4 homogeneous transformation matrix. The calculated sixth pose F can be expressed as:

[0155]

[0156] Among them, R2 is a 3×3 matrix representing the rotation matrix of the outward-facing camera 20 relative to the robotic arm coordinate system; T2 is a 3×1 matrix representing the translation matrix of the outward-facing camera 20 relative to the robotic arm coordinate system.

[0157] According to the sixth pose of the outward-facing camera 20 in the robotic arm coordinate system and the fifth pose of the robotic arm 101 in the robotic arm base coordinate system, the third conversion relationship between the outward-facing camera 20 and the robotic arm base coordinate system can be determined. It should be noted that according to the fifth poses and the first pose at the above-mentioned multiple moments, multiple sixth poses can be calculated. Based on the multiple sixth poses and the multiple fifth poses, an optimal third conversion relationship can be determined using an optimization algorithm. For example, the least squares method or a non-linear optimization algorithm can be used to determine the third conversion relationship.

[0158] After determining the second transformation relationship between the in-facing camera 10 and the robotic arm base coordinate system, and the second transformation relationship between the out-facing camera 20 and the robotic arm base coordinate system, the fourth transformation relationship between the in-facing camera 10 and the out-facing camera 20 can be calculated based on the second transformation relationship and the third transformation relationship. Based on the first transformation relationship between the out-facing camera 20 and the inertial sensor 30, and the fourth transformation relationship between the in-facing camera 10 and the out-facing camera 20, the transformation relationship between the in-facing camera 10 and the inertial sensor 30 can be calculated, and then the fifth transformation relationship between the in-facing camera 10, the out-facing camera 20, and the inertial sensor 30 is determined.

[0159] Please refer to Figure 2 and Figure 16 In some embodiments, the present application further provides a multi-sensor calibration device 200, which is applied to a head-mounted display device 100. The head-mounted display device 100 includes an in-facing camera 10, an out-facing camera 20, and an inertial sensor 30. During the calibration process, the head-mounted display device 100 is fixed relative to the robotic arm 101. The robotic arm 101 is further provided with a second calibration plate 105 that can deflect relative to the in-facing camera 10 for calibrating the in-facing camera 10, and a first calibration plate that is fixedly arranged at a certain position away from the robotic arm 101. The multi-sensor calibration device 200 includes a calibration module 210 and a determination module 220. The calibration module 210 is used to jointly calibrate the out-facing camera 20 and the inertial sensor 30 based on the first calibration plate to obtain the first transformation relationship between the out-facing camera 20 and the inertial sensor 30. The calibration module 210 is further used to perform hand-eye calibration on the in-facing camera 10 and the robotic arm 101 based on the pose transformation relationship between the second calibration plate 105 and the robotic arm 101 at different times to obtain the second transformation relationship between the in-facing camera 10 and the robotic arm base coordinate system. The calibration module 210 is further used to perform hand-eye calibration on the out-facing camera 20 and the robotic arm 101 to obtain the third transformation relationship between the out-facing camera 20 and the robotic arm base coordinate system. The determination module 220 is used to determine the fourth transformation relationship between the in-facing camera 10 and the out-facing camera 20 according to the second transformation relationship and the third transformation relationship. The determination module 220 is further used to determine the fifth transformation relationship between the in-facing camera 10, the out-facing camera 20, and the inertial sensor 30 according to the first transformation relationship and the fourth transformation relationship.

[0160] In some embodiments, the calibration module 210 is further used to control the in-facing camera 10, the out-facing camera 20, and the inertial sensor 30 to synchronously collect calibration data; control the robotic arm 101 to move along a predetermined trajectory so that the in-facing camera 10, the out-facing camera 20, and the inertial sensor 30 collect calibration data with different poses.

[0161] In some embodiments, the calibration module 210 is further configured to determine whether the frequencies of collecting calibration data by the inward-facing camera 10, the outward-facing camera 20, and the inertial navigation sensor 30 are the same; when the frequencies are the same, the calibration data collected by the inward-facing camera 10, the outward-facing camera 20, and the inertial navigation sensor 30 is used for multi-sensor calibration; when the frequencies are different, the calibration data collected by the inward-facing camera 10, the outward-facing camera 20, and the inertial navigation sensor 30 is statistically analyzed according to timestamps, and the calibration data corresponding to the common timestamps of the inward-facing camera 10, the outward-facing camera 20, and the inertial navigation sensor 30 is used for multi-sensor calibration.

[0162] In some embodiments, one end of the robotic arm 101 is fixedly installed on the arm base 103, and the other end of the robotic arm 101 is connected to the fixture 104, and the fixture 104 is used to fix the head-mounted display device 100.

[0163] The calibration module 210 is specifically configured to control the robotic arm 101 to move along a predetermined trajectory, so that the outward-facing camera 20 captures a first calibration plate fixed relative to the arm base 103 to obtain a plurality of first calibration picture data with different poses; the inward-facing camera 10 captures a second calibration plate 105 that can move relative to the fixture 104 to obtain a plurality of second calibration picture data with different poses; and the inertial navigation sensor 30 is made to collect dynamic inertial navigation data.

[0164] In some embodiments, when the robotic arm 101 moves along a predetermined trajectory, the second calibration plate 105 can deflect relative to the inward-facing camera 10 under the action of inertial force or motor driving force.

[0165] In some embodiments, one end of the fixture 104 is provided with a fixing plate, the head-mounted display device 100 is fixedly arranged on the fixing plate, and the other end of the fixture 104 is provided with a sliding plate. The second calibration plate 105 is slidably arranged on the sliding plate along a first direction, and / or the sliding plate is slidably arranged on the fixture 104 along a second direction.

[0166] In some embodiments, one end of the fixture 104 is provided with a fixing plate, the head-mounted display device 100 is fixedly arranged on the fixing plate, and the other end of the fixture 104 is provided with a sliding plate. The second calibration plate 105 is rotatably arranged on the sliding plate along a third direction through a rotating bracket; and / or the second calibration plate 105 includes a calibration plate body, a calibration plate inner frame, and a calibration plate outer frame. The calibration plate body is fixedly arranged on the calibration plate inner frame, and the calibration plate inner frame is rotatably arranged on the calibration plate outer frame along a first direction.

[0167] In some embodiments, a limiting bracket is further arranged on the sliding plate, and the limiting bracket is located on the side of the second calibration plate 105 facing away from the head-mounted display device 100 and is used to limit the rotation of the second calibration plate 105 around the first direction and / or the third direction.

[0168] In some embodiments, before jointly calibrating the outward-facing camera 20 and the inertial navigation sensor 30, the calibration module 210 is further configured to determine whether the inertial navigation sensor 30 has completed static calibration; when the inertial navigation sensor 30 has not completed static calibration, the manipulator 101 is controlled to be stationary so that the inertial navigation sensor 30 can collect static inertial data.

[0169] In some embodiments, the calibration module 210 is specifically configured to perform static calibration on the inertial navigation sensor 30 according to the static inertial data collected by the inertial navigation sensor 30 to obtain static calibration data of the inertial navigation sensor 30; perform internal and external parameter calibration on the outward-facing camera 20 according to a plurality of first calibration picture data obtained by the outward-facing camera 20 photographing the first calibration board to obtain the second pose of the outward-facing camera 20 in the first calibration board coordinate system; perform joint calibration on the outward-facing camera 20 and the inertial navigation sensor 30 according to the static calibration data, the dynamic inertial data collected by the inertial navigation sensor 30, and the first pose to obtain the first conversion relationship between the outward-facing camera 20 and the inertial navigation sensor 30.

[0170] In some embodiments, the calibration module 210 is specifically configured to obtain the second pose of the manipulator 101 in the manipulator base coordinate system according to the motion trajectory of the manipulator 101; perform internal and external parameter calibration on the inward-facing camera 10 according to a plurality of second calibration picture data obtained by the inward-facing camera 10 photographing the second calibration board 105 to obtain the third pose of the inward-facing camera 10 in the second calibration board coordinate system; obtain the pose change of the second calibration board 105 during the motion of the manipulator 101 according to the second pose and the third pose; solve for the fourth pose of the inward-facing camera 10 in the manipulator coordinate system according to the second pose, the third pose, and the pose change; determine the second conversion relationship between the inward-facing camera 10 and the manipulator base coordinate system according to the fourth pose and the second pose.

[0171] In some embodiments, the calibration module 210 is specifically configured to obtain the fifth pose of the manipulator 101 in the manipulator base coordinate system according to the motion trajectory of the manipulator 101; solve for the sixth pose of the outward-facing camera 20 in the manipulator coordinate system according to the first pose and the fifth pose; determine the third conversion relationship between the outward-facing camera 20 and the manipulator base coordinate system according to the sixth pose and the fifth pose.

[0172] It should be noted that the explanations of the multi-sensor calibration method in the foregoing embodiments are equally applicable to the multi-sensor calibration device 200 of the embodiments of the present application, and will not be elaborated herein.

[0173] Please refer to Figure 17 , the embodiments of the present application further provide a multi-sensor calibration system 300, including one or more processors 310 and a memory 320. The memory 320 stores a computer program, and when the computer program is executed by the processor 310, the multi-sensor calibration method of any of the foregoing embodiments is implemented.

[0174] For example, when the computer program is executed by the processor 310, a multi-sensor calibration method as follows is implemented:

[0175] 010: Jointly calibrate the outward-facing camera 20 and the inertial navigation sensor 30 based on the first calibration board to obtain the first conversion relationship between the outward-facing camera 20 and the inertial navigation sensor 30;

[0176] 020: Perform hand-eye calibration on the inward-facing camera 10 and the robotic arm 101 based on the pose transformation relationship between the second calibration board 105 and the robotic arm 101 at different times to obtain the second conversion relationship between the inward-facing camera 10 and the base coordinate system of the robotic arm;

[0177] 030: Perform hand-eye calibration on the outward-facing camera 20 and the robotic arm 101 to obtain the third conversion relationship between the outward-facing camera 20 and the base coordinate system of the robotic arm;

[0178] 040: Determine the fourth conversion relationship between the inward-facing camera 10 and the outward-facing camera 20 according to the second conversion relationship and the third conversion relationship;

[0179] 050: Determine the fifth conversion relationship among the inward-facing camera 10, the outward-facing camera 20, and the inertial navigation sensor 30 according to the first conversion relationship and the fourth conversion relationship.

[0180] For another example, when the computer program is executed by the processor 310, a multi-sensor calibration method as follows is implemented:

[0181] 060: Control the inward-facing camera 10, the outward-facing camera 20, and the inertial navigation sensor 30 to synchronously collect calibration data;

[0182] 070: Control the robotic arm 101 to move along a predetermined trajectory so that the inward-facing camera 10, the outward-facing camera 20, and the inertial navigation sensor 30 collect calibration data at different poses.

[0183] It should be noted that the explanations of the multi-sensor calibration method and the multi-sensor calibration device 200 in the foregoing embodiments are equally applicable to the multi-sensor calibration system 300 of the embodiments of the present application, and will not be elaborated here.

[0184] Please refer to Figure 18 , the embodiments of the present application further provide a computer-readable storage medium 400, on which a computer program 410 is stored. When the program is executed by the processor 420, the multi-sensor calibration method of any of the foregoing embodiments is implemented.

[0185] For example, when the program is executed by the processor 310, a multi-sensor calibration method as follows is implemented:

[0186] 010: Calibrate the outward-facing camera 20 and the inertial navigation sensor 30 jointly based on the first calibration board to obtain the first conversion relationship between the outward-facing camera 20 and the inertial navigation sensor 30;

[0187] 020: Calibrate the inward-facing camera 10 and the robotic arm 101 for hand-eye calibration based on the pose transformation relationship between the second calibration board 105 and the robotic arm 101 at different times to obtain the second conversion relationship between the inward-facing camera 10 and the base coordinate system of the robotic arm;

[0188] 030: Calibrate the outward-facing camera 20 and the robotic arm 101 for hand-eye calibration to obtain the third conversion relationship between the outward-facing camera 20 and the base coordinate system of the robotic arm;

[0189] 040: Determine the fourth conversion relationship between the inward-facing camera 10 and the outward-facing camera 20 according to the second conversion relationship and the third conversion relationship;

[0190] 050: Determine the fifth conversion relationship among the inward-facing camera 10, the outward-facing camera 20, and the inertial navigation sensor 30 according to the first conversion relationship and the fourth conversion relationship.

[0191] For another example, when the program is executed by the processor 310, the following multi-sensor calibration method is implemented:

[0192] 060: Control the inward-facing camera 10, the outward-facing camera 20, and the inertial navigation sensor 30 to synchronously collect calibration data;

[0193] 070: Control the robotic arm 101 to move along a predetermined trajectory so that the inward-facing camera 10, the outward-facing camera 20, and the inertial navigation sensor 30 collect calibration data at different poses.

[0194] It should be noted that the explanations of the multi-sensor calibration method and the multi-sensor calibration device 200 in the foregoing embodiments are equally applicable to the computer-readable storage medium 400 of the embodiments of the present application, and will not be elaborated herein.

[0195] In summary, the multi-sensor calibration method, the multi-sensor calibration device 200, the multi-sensor calibration system 300, and the computer-readable storage medium 400 of the embodiments of the present application fix the head-mounted display device 100 on the robotic arm 101, and perform joint calibration on the outward-facing camera 20 and the inertial navigation sensor 30, hand-eye calibration on the inward-facing camera 10 and the robotic arm 101, and hand-eye calibration on the outward-facing camera 20 and the robotic arm 101 based on the first calibration board and the second calibration board 105 to perform calibration among the inward-facing camera 10, the outward-facing camera 20, and the inertial navigation sensor 30. In this way, one-stop calibration of the inward-facing camera 10, the outward-facing camera 20, and the inertial navigation sensor 30 can be achieved, the calibration of multiple sensors is efficiently completed, and the calibration accuracy of the multi-sensor calibration is improved.

[0196] In the description of this specification, the descriptions with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0197] Any process or method description represented in the flowchart or described in other ways herein can be understood to represent a module, segment, or part of code including one or more executable instructions for implementing a specific logical function or process, and the scope of the preferred embodiments of the present application includes additional implementations, where the functions may be executed in a manner other than shown or discussed, including in a substantially simultaneous manner according to the functions involved or in a reverse order, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0198] The logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a sequenced list of executable instructions for implementing a logical function, and can be specifically implemented in any computer-readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in connection with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a computer-readable storage medium can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection part with one or more wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable storage medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other suitable processing as necessary, and then stored in a computer memory.

[0199] It should be understood that each part of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application specific integrated circuits with appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0200] Those of ordinary skill in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments. In addition, in each embodiment of the present application, each functional unit can be integrated in a processing module, or each unit can exist physically alone, or two or more units can be integrated in a module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. When the above integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage media mentioned above can be read-only memories, magnetic disks, optical disks, etc.

[0201] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application, and the scope of the present application is defined by the claims and their equivalents.

Claims

1. A multi-sensor calibration method, characterized in that, Applied to a head-mounted display device, the head-mounted display device includes an in-facing camera, an out-facing camera, and an inertial navigation sensor. During the calibration process, the head-mounted display device is fixed relative to the robotic arm. There is also a second calibration plate on the robotic arm that can deflect relative to the in-facing camera for calibrating the in-facing camera, and a first calibration plate fixedly arranged at a certain position from the robotic arm. The multi-sensor calibration method includes: Jointly calibrating the out-facing camera and the inertial navigation sensor based on the first calibration plate to obtain a first conversion relationship between the out-facing camera and the inertial navigation sensor; Performing hand-eye calibration on the in-facing camera and the robotic arm based on the pose transformation relationships of the second calibration plate and the robotic arm at different times to obtain a second conversion relationship between the in-facing camera and the robotic arm base coordinate system; Performing hand-eye calibration on the out-facing camera and the robotic arm to obtain a third conversion relationship between the out-facing camera and the robotic arm base coordinate system; Determining a fourth conversion relationship between the in-facing camera and the out-facing camera according to the second conversion relationship and the third conversion relationship; Determining a fifth conversion relationship between the in-facing camera, the out-facing camera, and the inertial navigation sensor according to the first conversion relationship and the fourth conversion relationship.

2. The multi-sensor calibration method according to claim 1, wherein, The step of jointly calibrating the out-facing camera and the inertial navigation sensor based on the first calibration plate to obtain a first conversion relationship between the out-facing camera and the inertial navigation sensor includes: Performing static calibration on the inertial navigation sensor according to the static inertial data collected by the inertial navigation sensor to obtain static calibration data of the inertial navigation sensor; Performing internal and external parameter calibration on the out-facing camera according to multiple first calibration picture data obtained by the out-facing camera photographing the first calibration plate to obtain a first pose of the out-facing camera in the first calibration plate coordinate system; Jointly calibrating the out-facing camera and the inertial navigation sensor according to the static calibration data, the dynamic inertial data collected by the inertial navigation sensor, and the first pose to obtain a first conversion relationship between the out-facing camera and the inertial navigation sensor.

3. The multi-sensor calibration method according to claim 1, characterized in that The step of performing hand-eye calibration on the in-facing camera and the robotic arm based on the pose transformation relationships of the second calibration plate and the robotic arm at different times to obtain a second conversion relationship between the in-facing camera and the robotic arm base coordinate system includes: Obtaining a second pose of the robotic arm in the robotic arm base coordinate system according to the motion trajectory of the robotic arm; Performing internal and external parameter calibration on the in-facing camera according to multiple second calibration picture data obtained by the in-facing camera photographing the second calibration plate to obtain a third pose of the in-facing camera in the second calibration plate coordinate system; Obtaining the pose change of the second calibration plate during the motion of the robotic arm according to the second pose and the third pose; Solving for a fourth pose of the in-facing camera in the robotic arm coordinate system according to the second pose, the third pose, and the pose change; Determining a second conversion relationship between the in-facing camera and the robotic arm base coordinate system according to the fourth pose and the second pose.

4. The multi-sensor calibration method according to claim 2, wherein, Performing hand-eye calibration on the outward-facing camera and the robotic arm to obtain a third transformation relationship between the outward-facing camera and the base coordinate system of the robotic arm, including: Obtaining a fifth pose of the robotic arm in the base coordinate system of the robotic arm according to the motion trajectory of the robotic arm; Solving for a sixth pose of the outward-facing camera in the coordinate system of the robotic arm according to the first pose and the fifth pose; Determining a third transformation relationship between the outward-facing camera and the base coordinate system of the robotic arm according to the sixth pose and the fifth pose.

5. The multi-sensor calibration method according to claim 1, wherein The multi-sensor calibration method further includes: Controlling the inward-facing camera, the outward-facing camera, and the inertial navigation sensor to synchronously collect calibration data; Controlling the robotic arm to move along a predetermined trajectory so that the inward-facing camera, the outward-facing camera, and the inertial navigation sensor collect calibration data at different poses.

6. The multi-sensor calibration method according to claim 5, wherein The multi-sensor calibration method further includes: Judging whether the frequencies of the calibration data collected by the inward-facing camera, the outward-facing camera, and the inertial navigation sensor are the same; When the frequencies are the same, using the calibration data collected by the inward-facing camera, the outward-facing camera, and the inertial navigation sensor for multi-sensor calibration; When the frequencies are different, statistically analyzing the calibration data collected by the inward-facing camera, the outward-facing camera, and the inertial navigation sensor according to timestamps, and using the calibration data corresponding to the common timestamps of the inward-facing camera, the outward-facing camera, and the inertial navigation sensor for multi-sensor calibration.

7. The multi-sensor calibration method according to claim 5, characterized in that, One end of the robotic arm is fixedly installed on the arm base, and the other end of the robotic arm is connected to a fixture for fixing the head-mounted display device; The controlling the robotic arm to move along a predetermined trajectory so that the inward-facing camera, the outward-facing camera, and the inertial navigation sensor collect calibration data at different poses includes: Controlling the robotic arm to move along a predetermined trajectory so that the outward-facing camera captures a first calibration board fixed relative to the arm base to obtain a plurality of first calibration picture data at different poses; enabling the inward-facing camera to capture a second calibration board movable relative to the fixture to obtain a plurality of second calibration picture data at different poses; enabling the inertial navigation sensor to collect dynamic inertial navigation data.

8. The multi-sensor calibration method according to claim 7, wherein When the robotic arm moves along a predetermined trajectory, the second calibration board can deflect relative to the inward-facing camera under the action of inertial force or motor driving force.

9. The multi-sensor calibration method according to claim 7, characterized in that One end of the fixture is provided with a fixed plate, the head-mounted display device is fixedly arranged on the fixed plate, and the other end of the fixture is provided with a sliding plate; The second calibration board is slidably arranged on the sliding plate along a first direction, and / or the sliding plate is slidably arranged on the fixture along a second direction; Wherein, the first direction is perpendicular to the second direction.

10. The multi-sensor calibration method according to claim 7, wherein One end of the fixture is provided with a fixed plate, the head-mounted display device is fixedly arranged on the fixed plate, and the other end of the fixture is provided with a sliding plate; The second calibration plate is rotatably arranged on the sliding plate along the third direction by a rotating bracket; and / or the second calibration plate includes a calibration plate body, a calibration plate inner frame and a calibration plate outer frame, the calibration plate body is fixedly arranged on the calibration plate inner frame, and the calibration plate inner frame is rotatably arranged on the calibration plate outer frame along the first direction; The first direction is perpendicular to the third direction.

11. The multi-sensor calibration method according to claim 10, wherein The sliding plate is further provided with a limiting bracket, which is located on the side of the second calibration plate facing away from the head-mounted display device and is used to limit the rotation of the second calibration plate around the first direction and / or the third direction.

12. The multi-sensor calibration method according to claim 7, wherein, Before jointly calibrating the outward-facing camera and the inertial navigation sensor, the multi-sensor calibration method further includes: Determining whether the inertial navigation sensor has completed static calibration; When the inertial navigation sensor has not completed static calibration, the robotic arm is controlled to be stationary so that the inertial navigation sensor collects static inertial navigation data.

13. A multi-sensor calibration device, characterized in that, The multi-sensor calibration device is applied to a head-mounted display device, the head-mounted display device including an inward-facing camera, an outward-facing camera, and an inertial navigation sensor. During the calibration process, the head-mounted display device is fixed relative to a robotic arm, and the robotic arm is further provided with a second calibration plate that can deflect relative to the inward-facing camera for calibration of the inward-facing camera, and a first calibration plate fixedly disposed at a predetermined distance from the robotic arm. The multi-sensor calibration device includes: a calibration module, configured to jointly calibrate the outward-facing camera and the inertial navigation sensor based on the first calibration plate to obtain a first conversion relationship between the outward-facing camera and the inertial navigation sensor; The calibration module is further configured to perform hand-eye calibration on the inward-facing camera and the robotic arm according to the posture transformation relationship between the second calibration plate and the robotic arm at different times, to obtain a second transformation relationship between the inward-facing camera and the robotic arm base coordinate system; The calibration module is further configured to perform hand-eye calibration on the outward-facing camera and the robotic arm to obtain a third transformation relationship between the outward-facing camera and the robotic arm base coordinate system; a determining module, configured to determine a fourth conversion relationship between the inward-facing camera and the outward-facing camera according to the second conversion relationship and the third conversion relationship; The determining module is further configured to determine a fifth conversion relationship between the inward-facing camera, the outward-facing camera, and the inertial navigation sensor according to the first conversion relationship and the fourth conversion relationship.

14. A multi-sensor calibration system, characterized in that, The system comprises one or more processors and memories, wherein the memories store a computer program, and when the computer program is executed by the processors, the multi-sensor calibration method according to any one of claims 1 to 12 is implemented.

15. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and when the program is executed by a processor, the multi-sensor calibration method according to any one of claims 1 to 12 is implemented.

Images