Device calibration method, apparatus and system, and storage medium

By deploying calibration boards within the target space range in augmented reality, virtual reality and mixed reality devices, the calibration data is obtained and optimized, and the calibration accuracy of the equipment is improved during rapid movement.

WO2024230451A1PCT designated stage expired Publication Date: 2024-11-14ARCSOFT CORP LTD

Patent Information

Application Number
PCT/CN2024/088663
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-05
Filing Date
2024-04-18
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

In the prior art, the augmented reality, virtual reality and mixed reality equipment during parameter calibration process, due to the insufficient anti-fuzzing ability of the calibration plate, the images collected by the camera are blurred, especially the fish-eye lens is prone to detection of edge corner points. Failure leads to inaccurate calibration of camera internal parameters, especially distortion parameters, which affects the working accuracy of the equipment.

Method used

By deploying at least one calibration board within the target space range, the calibration data, including image data and optimization data, the location information of the marked feature points is detected, and the optimization data is combined with the optimization data to be synchronized parameter optimization, the calibration of the calibration device is obtained by using the calibration device to be calibrated using the calibration device to be calibrated. Parameters to ensure the accuracy of calibration parameters.

Benefits of technology

It improves the accuracy of equipment parameter calibration, solves the problem of inaccurate calibration, and enhances the calibration accuracy of the equipment during rapid movement.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024088663_14112024_PF_FP_ABST
    Figure CN2024088663_14112024_PF_FP_ABST
Patent Text Reader

Abstract

Disclosed in the present application are a device calibration method, apparatus and system, and a storage medium. The device calibration method comprises: determining a target space range from among candidate space ranges according to the type of a device to be calibrated, wherein at least one calibration plate is deployed within the candidate space range, and the device to be calibrated comprises a first device and a second device; acquiring calibration data by means of the devices to be calibrated and within the target space range, wherein the calibration data comprises image data, which is collected by the first device with respect to the calibration plate, and optimization data, which is collected by the second device; performing detection on the image data in the calibration data, and determining position information of a marked feature point; and performing synchronous parameter optimization in combination with the position information and the optimization data, so as to obtain calibration parameters for the device to be calibrated.
Need to check novelty before this filing date? Find Prior Art

Description

Device calibration method, device, system and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 5, 2023, with priority number 202310508727.X, and invention name “Calibration method, device, system and storage medium for equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of data processing technology, and in particular to a device calibration method, apparatus, system, and storage medium. Background Art

[0003] With the rise of the "metaverse" concept, demand for augmented reality (AR), virtual reality (VR), and mixed reality (MR / XR) devices is continuing to surge. These devices are typically equipped with multiple cameras and inertial measurement units (IMUs) for scene perception, device positioning, virtual-reality integration, and user interaction. To avoid camera imaging distortion, pre-calibration of the camera and IMU is often required.

[0004] In the related art, in the joint calibration of multiple cameras + IMU, a pattern array flat plate with a fixed spacing is usually used as a calibration plate for calibration. However, the anti-blur capability of this type of calibration plate is weak. When the device moves rapidly, the image of the calibration plate captured by the camera will produce motion blur. If the camera uses a fisheye lens, it is easy for the edge corner detection of the picture to fail, resulting in inaccurate calibration of the camera's internal parameters, especially the distortion parameters, thereby affecting the accuracy of the camera's working results.

[0005] To address the above-mentioned problems, no effective solutions have been proposed so far.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide a device calibration method, apparatus, system, and storage medium to at least solve the technical problem of inaccurate device parameter calibration in related technologies.

[0008] According to one aspect of an embodiment of the present application, a device calibration method is provided. The method may include: determining a target spatial range from a candidate spatial range based on the type of the device to be calibrated, wherein at least one calibration plate is deployed within the candidate spatial range, and the device to be calibrated includes a first device and a second device; obtaining calibration data within the target spatial range by the device to be calibrated, wherein the calibration data includes image data collected by the first device from the calibration plate and optimized data collected by the second device; detecting the image data in the calibration data to determine position information of marked feature points; and performing synchronous parameter optimization based on the position information and the optimized data to obtain calibration parameters for the device to be calibrated.

[0009] Optionally, determining the target space range from the candidate space range according to the type of the device to be calibrated includes: the candidate space range includes a first target space range and a second target space range, wherein at least one first camera calibration plate is deployed within the first target space range, and at least one first camera calibration plate and at least one second camera calibration plate are deployed within the second target space range; when the type of the second device is an inertial measurement component, the first target space range is selected as the target space range; when the type of the second device is an image acquisition device, the second target space range is selected as the target space range.

[0010] Optionally, the first target space range includes a number of preset landmarks, and the device to be calibrated moves according to the preset landmarks, and calibration data is collected during the movement.

[0011] Optionally, in the second target space range, the first camera calibration plate and the second camera calibration plate are parallel, and the calibration plate patterns are arranged relative to each other.

[0012] Optionally, the marked feature points on the calibration plate correspond to index numbers, and the image data in the calibration data are detected to determine the position information of the marked feature points in the image data, including: obtaining the index number of the marked feature point on the calibration plate by detecting the image data in the calibration data; determining the position coordinates of the marked feature point in the three-dimensional space coordinate system based on the index number; determining the position coordinates of the marked feature point in the image coordinate system based on the position of the marked feature point in the image data; and determining the position coordinates of the marked feature point in the three-dimensional space coordinate system and the position coordinates of the marked feature point in the image coordinate system as the position information of the marked feature point.

[0013] Optionally, obtaining calibration data through the device to be calibrated includes: sending a data acquisition instruction to the device to be calibrated, wherein the data acquisition instruction is used to instruct the device to be calibrated to turn on a data acquisition function to collect image data and optimization data; receiving image data and optimization data fed back by the device to be calibrated, wherein the image data and optimization data constitute calibration data.

[0014] Optionally, the calibration plate includes common feature points and marked feature points, wherein the difference between the marking information of the common feature points and the marking information of the marked feature points includes at least one of the following: marking structure, marking color, marking gray value, and marking shape.

[0015] Optionally, the marking information of the marking feature points in different calibration plates is different; the arrangement angle between multiple calibration plates is determined according to the arrangement position and field of view angle between the first device and the second device to ensure that both the first device and the second device can collect the complete marking information of the marking feature points.

[0016] Optionally, the image acquisition device is a rolling shutter camera, and the calibration parameters include time compensation parameters of the rolling shutter.

[0017] Optionally, the image acquisition device is a fisheye camera, and the calibration parameters include vignetting correction parameters.

[0018] Optionally, the calibration method of the device includes: determining the grayscale value of the marked feature point in the image data and the position coordinates of the marked feature point in the image coordinate system; performing polynomial fitting on the grayscale value and the position coordinate to obtain the vignetting correction parameter.

[0019] According to another aspect of the embodiment of the present application, a device calibration apparatus is also provided. The apparatus includes: a first determination component, configured to determine a target spatial range from a candidate spatial range based on the type of the device to be calibrated, wherein at least one calibration plate is deployed within the candidate spatial range, and the device to be calibrated includes a first device and a second device; an acquisition component, configured to acquire calibration data within the target spatial range through the device to be calibrated, wherein the calibration data includes image data collected by the first device on the calibration plate and optimization data collected by the second device; a second determination component, configured to detect the image data in the calibration data and determine the position information of the marked feature points; and an optimization component, configured to perform synchronous parameter optimization based on the position information and the optimization data to obtain calibration parameters of the device to be calibrated.

[0020] According to another aspect of an embodiment of the present application, a device calibration system is also provided. The calibration system may include: a control component, a device to be calibrated, and a calibration plate, wherein the device to be calibrated includes an image acquisition device and an inertial measurement component; the image acquisition device is configured to acquire image data of the calibration plate within a target spatial range; the inertial measurement component is deployed within the device to be calibrated and is configured to acquire spatial data of the device to be calibrated moving within the target spatial range; and the control component is connected to the image acquisition device and the inertial measurement component and is configured to convert the image data and spatial data into calibration parameters for the device to be calibrated.

[0021] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to perform a calibration method of a device of any one of the above items.

[0022] According to another aspect of an embodiment of the present application, a computer program product is further provided, including computer instructions, which, when executed by a processor, implement the steps of the device calibration method of the embodiment of the present application.

[0023] In an embodiment of the present application, a target space range is determined from a candidate space range according to the type of the device to be calibrated, wherein at least one calibration plate is deployed in the candidate space range, and the device to be calibrated includes a first device and a second device; within the target space range, calibration data is obtained by the device to be calibrated, wherein the calibration data includes image data collected by the first device on the calibration plate and optimization data collected by the second device; the image data in the calibration data is detected to determine the position information of the marked feature points; and synchronous parameter optimization is performed in combination with the position information and the optimization data to obtain the calibration parameters of the device to be calibrated. That is to say, in the embodiment of the present application, at least one calibration plate is deployed within the target space range, and the calibration data within the target space range can be obtained by the device to be calibrated, and the image data in the calibration data can be detected, and the position information of the marked feature points on the calibration plate included in the image data can be determined. Since the position information of the marked feature points can represent the position coordinates of the marked feature points in the three-dimensional space coordinate system, based on this, the parameters of the device to be calibrated are optimized based on the position information of the marked feature points and the optimization data collected by the second device, and the calibration parameters of the device to be calibrated are relatively accurate, thereby achieving the technical effect of improving the accuracy of the calibration parameters, and thus solving the technical problem of inaccurate equipment parameter calibration in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the embodiments of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] FIG1 is a flow chart of a device calibration method according to an embodiment of the present application;

[0026] FIG2 is a schematic diagram of a calibration plate according to an embodiment of the present application;

[0027] FIG3 is a schematic diagram of a marking position according to an embodiment of the present application;

[0028] FIG4 is a schematic diagram of a device calibration system according to an embodiment of the present application;

[0029] FIG5 is a schematic diagram of images captured by a VR headset at several landmark points according to an embodiment of the present application;

[0030] FIG6 is a schematic diagram of using two calibration plates to adapt to the calibration requirements of VR glasses according to an embodiment of the present application;

[0031] FIG7 is a schematic diagram of an eye-tracking camera calibration environment according to an embodiment of the present application;

[0032] FIG8 is a schematic diagram of an image of a calibration plate captured by a camera within a second target space according to an embodiment of the present application;

[0033] FIG9 is a schematic diagram of a calibration apparatus for a device according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or components is not necessarily limited to those steps or components clearly listed, but may include other steps or components that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] To facilitate those skilled in the art to understand the present application, some of the terms or nouns involved in the embodiments of the present application are explained below:

[0037] Image acquisition equipment, composed of a lens and an area array sensor, can be a camera, a video camera, etc.

[0038] Inertial Measurement Unit, referred to as IMU, is a device mainly used to detect and measure the three-axis attitude angle (or angular rate) and acceleration of an object;

[0039] Camera calibration, in image measurement and machine vision applications, is a process that determines the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image. A geometric model of the camera's imaging must be established. These geometric model parameters are the camera parameters. Under most conditions, these parameters (intrinsic parameters, extrinsic parameters, and distortion parameters) must be obtained through experimentation and calculation. This process of determining these parameters is called camera calibration.

[0040] Joint calibration is performed to obtain the intrinsic and extrinsic parameters of each camera, the intrinsic and extrinsic parameters of the IMU, the camera rolling shutter compensation, and the vignetting correction parameters. The camera's intrinsic parameters include focal length, optical center coordinates, distortion, and other parameters, which are used to describe the imaging law. The IMU's intrinsic parameters include zero bias, scale factor error, installation error, and other parameters, which are used to describe the IMU's intrinsic properties. The camera / IMU's extrinsic parameters represent the camera / IMU's rotation and translation in the world coordinate system and are used to describe the transformation relationship from the camera / IMU coordinates to the unified world coordinate system.

[0041] The following embodiments of the present application can be applied to a variety of display devices that can realize augmented reality, virtual reality, or mixed reality functions, for example, head-mounted display devices such as Holens (AR glasses) and Metaquest2 (VR glasses), or image acquisition devices such as cameras.

[0042] According to an embodiment of the present application, an embodiment of a calibration method for a device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0043] FIG1 is a flow chart of a device calibration method according to an embodiment of the present application. As shown in FIG1 , the method includes the following steps:

[0044] Step S101 : determining a target space range from candidate space ranges according to the type of the device to be calibrated.

[0045] In the technical solution of the above-mentioned step S101 of the present application, the device to be calibrated includes a first device and a second device, wherein the first device can be an image acquisition device, such as a camera or a video camera, and the second device can be an inertial measurement unit (IMU) or an image acquisition device. Based on this, the target space range can be determined from the candidate space range according to the type of the device to be calibrated, wherein at least one calibration plate is deployed in the candidate space range.

[0046] In this embodiment, the candidate spatial range includes a first target spatial range and a second target spatial range, wherein at least one first camera calibration plate is deployed within the first target spatial range, and at least one first camera calibration plate and at least one second camera calibration plate are deployed within the second target spatial range. Based on this, when the type of the second device is an inertial measurement component, the first target spatial range can be selected as the target spatial range, and when the type of the second device is an image acquisition device, the second target spatial range can be selected as the target spatial range.

[0047] For example, when the device to be calibrated is one of AR / VR / MR / XR glasses, since the AR / VR / MR / XR glasses include a first camera, a second camera and an inertial measurement unit, where the first camera can be an outward-facing camera and the second camera can be an eye-movement camera, based on this, when calibrating the first camera and the inertial measurement unit, the candidate space range where at least one first camera calibration plate is deployed can be used as the target space range; when calibrating the first camera and the second camera, the candidate space range where at least one first camera calibration plate and at least one second camera calibration plate are deployed can be used as the target space range.

[0048] Optionally, the device to be calibrated can also be a mobile phone, robot, drone, or other device equipped with both a camera and an IMU.

[0049] Step S102: obtaining calibration data through the device to be calibrated within the target space.

[0050] In the technical solution of the above-mentioned step S102 of the present application, after determining the target space range, calibration data can be obtained within the target space range through the device to be calibrated, wherein the calibration data includes image data collected by the calibration plate of the first device and optimized data collected by the second device.

[0051] In this embodiment, since the device to be calibrated includes a first device and a second device, wherein the first device can be a camera or a video camera, and the second device can be an inertial measurement unit or an image acquisition device, based on this, when the second device is an inertial measurement unit, at least one first camera calibration plate is deployed within the target space range. Based on this, when using the device to be calibrated to obtain calibration data, the device to be calibrated can be fixed on the fixture of the robotic arm, and the robotic arm is controlled to move with six degrees of freedom within the target space range, so as to drive the device to be calibrated to move within the target space range and collect calibration data, wherein the calibration data includes image data collected by the first device and optimized data collected by the second device, and the optimized data includes at least spatial data of the device to be calibrated moving within the target space range, and the spatial data includes posture data and velocity data of the device to be calibrated moving within the target space range. Exemplarily, the spatial data collected by the inertial measurement unit includes at least three-axis acceleration and three-axis angular velocity.

[0052] In addition, the device to be calibrated usually also has an eye-tracking camera for capturing human eye movements and line of sight directions. The eye-tracking camera is usually installed on the inner side of the glasses to shoot towards the human eye, while the outward-facing camera is usually installed on the outer side of the glasses to shoot towards the application scene. Calibrating the external parameters between the eye-tracking camera and other outward-facing cameras is a necessary condition for converting the line of sight in the eye-tracking camera coordinate system to the glasses coordinate system, and is also a prerequisite for subsequent interaction. However, because the eye-tracking camera and the outward-facing camera are back-to-back designs, that is, the viewing angles of the two are in opposite directions and there is no overlapping area, it is impossible to complete the external parameter calibration between multiple cameras with opposite viewing angles during the joint calibration of the first camera and the inertial measurement unit. Therefore, the present application also deploys a multi-camera calibration scene within the target space to achieve one-stop calibration.

[0053] Optionally, when the second device is an image acquisition device, since the first device and the second device in the device to be calibrated are designed back-to-back, based on this, at least one first camera calibration plate and at least one second camera calibration plate can be deployed within the target space range, wherein the at least one first camera calibration plate and the at least one second camera calibration plate include marked feature points, and the one side of the at least one first camera calibration plate and the at least one second camera calibration plate containing the marked feature points are arranged relative to each other, so that when the device to be calibrated moves between the first camera calibration plate and the second camera calibration plate, the first device collects image data of the first camera calibration plate, and the second device collects image data of the second camera calibration plate.

[0054] For example, the device to be calibrated can be fixed on the clamp of the robotic arm, and the robotic arm can be controlled to move the device to be calibrated between the first camera calibration plate and the second camera calibration plate, so that the first device collects image data of the first camera calibration plate, and the second device collects image data of the second camera calibration plate. That is, the calibration data includes the image data of the first camera calibration plate collected by the first device and the image data of the second camera calibration plate collected by the second device.

[0055] Optionally, when acquiring calibration data, the control component may first send a data acquisition instruction to the device to be calibrated. This data acquisition instruction is used to control the device to be calibrated to enable a data acquisition function, that is, to control the first and second devices in the device to be calibrated to enable a data acquisition function. The control component may then control the robotic arm to move the device to be calibrated within the target space to collect calibration data within the target space. The collected calibration data may then be transmitted to the control component. Alternatively, the device to be calibrated may transmit the collected calibration data to the control component in real time. The control component may receive the calibration data sent by the device to be calibrated and determine calibration parameters for the device to be calibrated based on the calibration data.

[0056] In this embodiment, the calibration plate can be a first camera calibration plate or a second camera calibration plate, and the calibration plate includes multiple feature points, which are divided into ordinary feature points and marked feature points, wherein the difference between the marking information of the ordinary feature points and the marking information of the marked feature points includes at least one of the following: marking structure, marking color, marking grayscale value, and marking shape.

[0057] For example, the ordinary feature points on the calibration plate can be composed of patterns of the same size and fixed spacing, while the marking feature points can be distinguished from the ordinary feature points by marking information, and the marking feature points constitute the marking position on the calibration plate. For example, the ordinary feature points on the calibration plate can be black dots of the same size and fixed spacing, while the marking feature points in the marking position can be dots of different sizes, dots and rings, dots and squares, dots and triangles, etc., wherein the color of the marking feature points can also be distinguished from the color of the ordinary feature points. Figure 2 is a schematic diagram of a calibration plate according to an embodiment of the present application. As shown in Figure 2, the marking feature points in the marking position on the same calibration plate are obviously distinguished from the ordinary marking points on the calibration plate. Figure 2 only shows one type of marking feature point. In addition, the marking feature points in the marking position can also be distinguished by other shapes or colors. Figure 3 is a schematic diagram of a marking position according to an embodiment of the present application. As shown in Figure 3, the marking feature points in the marking position can also be distinguished from the ordinary feature points by structure, color, shape, or dots / rings.

[0058] Since the equipment needs to rotate and translate significantly in the joint calibration of the camera and the inertial measurement unit, when the angle between the camera optical axis and the normal of the calibration plate is large, the marked feature points will often appear at the edge of the image, or even move out of the image, resulting in the failure or error of the detection of the marked feature points on the calibration plate, leading to the problem of calibration failure. Therefore, on the one hand, this application adopts a scheme of arranging multiple calibration plates at appropriate angles according to the camera arrangement characteristics of the device to be calibrated. On the other hand, the marked feature points are arranged in the middle area of ​​the calibration plate, so that the image data collected by the camera meets the conditions of containing complete marked feature points, thereby avoiding the subsequent false detection and missed detection of feature points. Even when the camera only captures a part of the calibration plate, it can still infer the serial number of the detected dots through the marked position, establish a correct mapping relationship with the calibration plate in the real three-dimensional world, and obtain the calibration parameters through the optimization algorithm.

[0059] It should be noted that, under the premise of ensuring that the image acquisition device can capture the marked feature points on the calibration plate, the number of marking positions contained in a calibration plate can be determined according to the camera arrangement and field of view of the actual device to be calibrated. For example, there can be one or more marking positions on a calibration plate, but the design of the marking positions needs to ensure the uniqueness of the orientation of the calibration plate, that is, the placement direction of the calibration plate can be distinguished based on the marking positions on the calibration plate. This application does not limit the number of groups of marking positions on the calibration plate.

[0060] Step S103: Detect the image data in the calibration data to determine the position information of the marked feature points.

[0061] In the technical solution of the above-mentioned step S103 of the present application, it can be seen from the above introduction that the calibration data includes the image data collected by the first device and the optimization data collected by the second device, wherein, when the second device is an inertial measurement component, the optimization data at least includes the spatial data of the device to be calibrated moving within the target space range, wherein the spatial data includes the posture data and speed data of the device to be calibrated moving within the target space range, and when the second device is an image acquisition device, the optimization data at least includes the image data collected from the second camera calibration plate. Based on this, after obtaining the calibration data collected by the device to be calibrated, the image data collected by the first device in the calibration data can be first detected to determine the position information of the marked feature points, wherein the position information is used to indicate the position coordinates of the marked feature points in three-dimensional space and the position coordinates of the marked feature points in the image coordinate system.

[0062] In this embodiment, after obtaining the image data collected by the first device, the image data can be input into the visual detection model for detection. Since the image data includes the image data of the first camera calibration plate, and the first camera calibration plate includes marked feature points, and the marked feature points have characteristics that are obviously different from ordinary feature points, based on this, the position information of the marked feature points in the image data can be determined by the visual detection model.

[0063] For example, each feature point on the camera calibration plate corresponds to an index number, and the index number is used to determine the position of the feature point on the camera calibration plate. Based on this, after the image data is input into the visual detection model, the visual detection model can determine the position coordinates of each marked feature point on the camera calibration plate in the image coordinate system, as well as the index number of each marked feature point, wherein the index number of the marked feature point is used to indicate the position of the marked feature point on the camera calibration plate. After determining the position of each marked feature point on the camera calibration plate, the position coordinates of each marked feature point in the three-dimensional space coordinates can be further determined in combination with the index number of the feature point, and then the position coordinates of each marked feature point in the image coordinate system and the position coordinates of each feature point in the three-dimensional space are combined to form the position information of each marked feature point.

[0064] Step S104 , performing synchronous parameter optimization in combination with the position information and the optimization data to obtain calibration parameters of the device to be calibrated.

[0065] In the technical solution of the above-mentioned step S104 of the present application, after determining the position information of each marking feature point on the calibration plate, synchronous parameter optimization can be performed in combination with the position information and optimization data to obtain the calibration parameters of the device to be calibrated, wherein the calibration parameters include the intrinsic parameters and extrinsic parameters of the first device (image acquisition device). When the second device is an inertial measurement component, the calibration parameters also include the intrinsic parameters and extrinsic parameters of the second device; when the second device is an image acquisition device, the calibration parameters include the intrinsic parameters and extrinsic parameters of the second device.

[0066] In this embodiment, the position coordinates of each marked feature point in the image data collected by the first device in the image coordinate system and the position coordinates in the three-dimensional space coordinate system can be used as a pair of feature point pairs, and then the feature point pairs corresponding to the multiple marked feature points can be used to form a point pair set. When the second device is an inertial measurement unit, the optimization data collected by the second device at least includes spatial data of the device to be calibrated moving within the target space. Based on this, the point pair set consisting of the multiple feature point pairs and the spatial data collected by the second device can be input into a parameter optimization model for parameter optimization calculation to obtain calibration parameters of the device to be calibrated, wherein the calibration parameters include the intrinsic and extrinsic parameters of the first device, as well as the intrinsic and extrinsic parameters of the second device.

[0067] For example, when the parameter optimization model performs parameter optimization calculations based on multiple feature point pairs and spatial data collected by the second device to obtain the calibration parameters of the device to be calibrated, the movement trajectory of the device to be calibrated within the first target space range can be determined based on multiple feature point pairs, and then the determined movement trajectory can be fitted into a continuous trajectory. The continuous trajectory and the trajectory determined according to the spatial data collected by the IMU are time-synchronized and calculated. A nonlinear optimization problem of the first device (image acquisition device) and the IMU can be constructed. By solving the nonlinear optimization problem, the external parameters between the first device and the IMU camera can be obtained, and the internal parameters of the IMU can be obtained at the same time.

[0068] Optionally, when the second device is an image acquisition device, the method described in the aforementioned step S103 can be referred to to determine the position coordinates of each marked feature point in the image data captured by the second device in the three-dimensional space coordinate system and the position coordinates in the image coordinate system, and then the position coordinates of each marked feature point in the three-dimensional space coordinates and the position coordinates in the image coordinate system constitute a feature point pair to obtain multiple feature point pairs, and the multiple feature point pairs constitute a point pair set, and then the point pair set corresponding to the multiple feature points on the calibration plate in the image data captured by the first device and the point pair set corresponding to the multiple feature points on the calibration plate in the image data captured by the second device are input into the parameter optimization model for optimization calculation to obtain the calibration parameters of the device to be calibrated, wherein the calibration parameters include the intrinsic parameters and extrinsic parameters of the first device, and the intrinsic parameters and extrinsic parameters of the second device.

[0069] Through the above steps, at least one calibration plate is deployed within the target space range, and the calibration data within the target space range can be obtained through the device to be calibrated. The calibration data includes image data and optimization data collected from the calibration plate. The image data in the calibration data is detected, and the position information of the marked feature points on the calibration plate included in the image data can be determined. Since the position information of the marked feature points can represent the position coordinates of the marked feature points in the three-dimensional space coordinate system and the position coordinates of the marked feature points in the image coordinate system, the position of the marked feature points can be accurately determined based on the position information. Based on this, the parameters of the device to be calibrated are optimized according to the position information of the marked feature points and the optimization data collected by the second device, and the calibration parameters of the device to be calibrated are relatively accurate, thereby achieving the technical effect of improving the accuracy of the calibration parameters, thereby solving the technical problem of inaccurate equipment parameter calibration in the related technology.

[0070] The above method of this embodiment is further introduced below with examples.

[0071] As an optional implementation, step S101, determining the target space range from the candidate space range according to the type of the device to be calibrated, including: the candidate space range includes a first target space range and a second target space range, wherein at least one first camera calibration plate is deployed in the first target space range, and at least one first camera calibration plate and at least one second camera calibration plate are deployed in the second target space range; when the type of the second device is an inertial measurement component, the first target space range is selected as the target space range; when the type of the second device is an image acquisition device, the second target space range is selected as the target space range.

[0072] In this embodiment, since the device to be calibrated includes a first device and a second device, and the first device is an image acquisition device, and the second device can be an inertial measurement component or an image acquisition device, based on this, when determining the target space range from the candidate space range according to the type of the device to be calibrated, when the second device is an inertial measurement component, the first target space range deployed with at least one first camera calibration plate can be used as the target space range; when the second device is an image acquisition device, the second target space range deployed with at least one first camera calibration plate and at least one second camera calibration plate can be used as the target space range.

[0073] For example, when the device to be calibrated is one of AR / VR / MR / XR glasses, since the AR / VR / MR / XR glasses include a first camera, a second camera, and an inertial measurement unit, where the first camera can be an outward-facing camera, and there can be multiple first cameras, and the second camera can be an eye-tracking camera, based on this, when calibrating the first camera and the inertial measurement unit, there is usually not enough common view area between the multiple first cameras, that is, the orientation of each camera is very different. In this case, the coverage area of ​​the calibration plate is usually required to be large enough. Based on this, the size of a single calibration plate can be increased. However, when the angle between the camera optical axis and the normal of the calibration plate is large, it is easy to fail to detect the marked feature points on the calibration plate or detect errors, which in turn leads to calibration failure. In this case, a solution of arranging multiple calibration plates at appropriate angles can be adopted to adapt to the calibration requirements of the device to be calibrated. Therefore, when calibrating the first camera and the inertial measurement unit, the first target space range where at least one first camera calibration plate is deployed can be determined as the target space range.

[0074] For another example, when calibrating the first camera and the second camera, since the first camera and the second camera are designed back-to-back in the device to be calibrated, that is, the viewing directions of the two cameras are opposite to each other and there is no overlapping area, the candidate space range where at least one first camera calibration plate and at least one second camera calibration plate are deployed can be used as the target space range, and the one side including the marked feature points on the at least one first camera calibration plate and the at least one second camera calibration plate are arranged relative to each other. In this way, when the device to be calibrated moves between the first camera calibration plate and the second camera calibration plate, the first camera can collect image data of the first camera calibration plate, and the second camera can collect image data of the second camera calibration plate.

[0075] As an optional implementation manner, the first target space range includes a number of preset landmarks, and the device to be calibrated moves according to the preset landmarks, and the calibration data is collected during the movement.

[0076] In this embodiment, the first target space contains several preset landmarks. The line connecting these landmarks forms the movement route of the device to be calibrated within the first target space. As the device to be calibrated moves along this movement route, calibration data can be collected during the movement. Furthermore, the calibration device moves along the landmarks in a predetermined order.

[0077] For example, the robotic arm can be controlled to drive the device to be calibrated to move from an initial position through several pre-set landmark points. During the movement along the moving route composed of landmark points in a predetermined order, the control component can send data collection instructions to the device to be calibrated. After receiving the data collection instructions, the device to be calibrated can collect calibration data.

[0078] As an optional implementation, in the second target space range, the first camera calibration plate and the second camera calibration plate are parallel, and the calibration plate patterns are arranged relative to each other.

[0079] In this embodiment, since at least one first camera calibration plate and at least one second camera calibration plate are deployed within the second target space range, and both the first camera calibration plate and the second camera calibration plate include marked feature points, the marked feature points are the calibration plate patterns on the calibration plates. Since the first camera and the second camera are designed back-to-back in the device to be calibrated, their viewing directions are opposite to each other. Therefore, in order to ensure that the device to be calibrated moves to the second target space range, the first camera and the second camera can both collect the marked feature points on the calibration plate. The first camera calibration plate and the second camera calibration plate within the second target space range can be arranged in parallel, and the calibration plate patterns on the first camera calibration plate and the second camera calibration plate can be arranged relative to each other.

[0080] As an optional implementation, the marked feature points on the calibration plate correspond to index numbers. Step S103 detects the image data in the calibration data to determine the position information of the marked feature points in the image data, including: obtaining the index number of the marked feature point on the calibration plate by detecting the image data in the calibration data; determining the position coordinates of the marked feature point in the three-dimensional space coordinate system based on the index number; determining the position coordinates of the marked feature point in the image coordinate system based on the position of the marked feature point in the image data; and determining the position coordinates of the marked feature point in the three-dimensional space coordinate system and the position coordinates of the marked feature point in the image coordinate system as the position information of the marked feature point.

[0081] In this embodiment, the marking feature points on the calibration plate correspond to index numbers. Based on this, when detecting the image data in the calibration data, the image data can be input into the visual detection model for detection, wherein the visual detection model is a model pre-trained based on image samples, and based on the visual detection model, the index numbers of each marking feature point on the calibration plate in the image data can be detected. Specifically, the visual detection model can infer which of the many marking positions the detected marking position is based on the local combination structure, and the indexes and coordinates of other dots on the image captured by the camera can be correctly inferred through the marking position. After detecting the index numbers of each marking feature point in the image data, the position coordinates of the marking feature point in the three-dimensional space coordinate system can be determined based on the index numbers corresponding to the marking feature points on the calibration plate.

[0082] In this embodiment, since the image data is collected from the calibration plate deployed within the target space, the position of the marked feature point in the image can be clearly determined based on the image data, and then an image coordinate system can be established to determine the position coordinates of the marked feature point in the image coordinate system.

[0083] After determining the position coordinates of the marker feature point in the three-dimensional space coordinate system and the position coordinates of the marker feature point in the image coordinate system, the position coordinates of the marker feature point in the three-dimensional space coordinate system and the position coordinates of the marker feature point in the image coordinate system can be determined as the position information of the marker feature point. According to the above method, the position information of each marker feature point on the calibration plate can be determined. After determining the position information of each marker feature point on the calibration plate, the position coordinates of each marker feature point in the three-dimensional space coordinate system and the position coordinates of the marker feature point in the image coordinate system can be regarded as a coordinate point pair, thereby obtaining multiple coordinate point pairs, and the multiple coordinate point pairs are regarded as a point pair set.

[0084] As an optional implementation, step S101, obtaining calibration data through the device to be calibrated, includes: sending a data acquisition instruction to the device to be calibrated, wherein the data acquisition instruction is used to instruct the device to be calibrated to turn on the data acquisition function to collect image data and optimization data; receiving image data and optimization data fed back by the device to be calibrated, wherein the image data and optimization data constitute calibration data.

[0085] In this embodiment, a communication connection can be established between the control component and the device to be calibrated via a data line and / or a wireless network. Based on this, when obtaining calibration data, the control component can send data acquisition instructions to the device to be calibrated, and receive image data and optimization data returned by the device to be calibrated, wherein the image data and optimization data can constitute the calibration data.

[0086] For example, when the device to be calibrated moves within the first target space range, the control component can send a data acquisition instruction to the device to be calibrated via a data line and / or a wireless network. After receiving the data acquisition instruction, the device to be calibrated can start the data acquisition function. For example, the device to be calibrated can start the image acquisition function of the first device to collect image data, and start the second device (inertial measurement component) to collect spatial data of the device to be calibrated when it moves within the target space range, wherein the spatial data includes posture data and speed data of the device to be calibrated moving within the target space range. After collecting the image data and spatial data, the device to be calibrated can transmit the collected image data and spatial data to the control component via a data line and / or a wireless network. The control component can receive the image data and spatial data fed back by the device to be calibrated, and determine the image data and spatial data as calibration data.

[0087] For another example, when the device to be calibrated is within the second target space, the control component can send a data acquisition instruction to the device to be calibrated. After receiving the data acquisition instruction, the device to be calibrated can turn on the image acquisition function of the first device to collect image data of the first camera calibration plate within the second target space, and turn on the image acquisition function of the second device (image acquisition device) to collect image data of the second camera calibration plate within the second target space. After that, the image data collected by the first device and the second device can be sent to the control component. The control component can receive the image data returned by the device to be calibrated, and use the received image data as calibration data.

[0088] As an optional embodiment, the calibration plate includes ordinary feature points and marked feature points, wherein the difference between the marking information of the ordinary feature points and the marking information of the marked feature points includes at least one of the following: marking structure, marking color, marking grayscale value, and marking shape.

[0089] In this embodiment, the calibration plate includes common feature points and marked feature points, wherein the marking information of the common feature points is different from the marking information of the marked feature points. For example, the common feature points may be composed of patterns of the same size and fixed spacing, while the marked feature points may be distinguished from the common feature points by marking information such as structure, color, grayscale value, shape, etc. The marked feature points on the calibration plate may constitute a marking position. For example, the common feature points on the calibration plate may be black dots of the same size and fixed spacing, while the marked feature points in the marking position may be dots of different sizes, dots and rings, dots and squares, dots and triangles, etc., wherein the color of the marked feature points may also be different from the color of the common feature points.

[0090] As an optional implementation, the marking information of the marking feature points in different calibration plates is different; the arrangement angle between multiple calibration plates is determined according to the arrangement position and field of view angle between the first device and the second device to ensure that both the first device and the second device can collect the complete marking information of the marking feature points.

[0091] In this embodiment, the marking information of the marker feature points on different calibration plates can be different. For example, calibration plates A and B can have different marking information, where the difference in marking information is reflected in at least: marker structure, marker color, marker grayscale value, or marker shape. The arrangement angle between the multiple calibration plates can be determined based on the arrangement position and field of view of the first and second devices to ensure that both the first and second devices can collect complete marking information of the marker feature points.

[0092] For example, when the second device is an image acquisition device, since the first device and the second device in the device to be calibrated are deployed back to back, the side containing the marking feature points of at least one first camera calibration plate deployed within the second target space can be arranged opposite to the side containing the marking feature points of at least one second camera calibration plate to ensure that when the device to be calibrated is moved between the first camera calibration plate and the second camera calibration plate, both the first device and the second device can collect the complete marking information of the marking feature points on the calibration plate.

[0093] As an optional implementation, the image acquisition device is a rolling shutter camera, and the calibration parameters include time compensation parameters of the rolling shutter.

[0094] In this embodiment, when the image acquisition device on the device to be calibrated is a rolling shutter camera, the rolling shutter camera can also collect image data of the calibration plate when the device to be calibrated moves within the target space. After collecting the image data, the image data can be processed based on the visual detection model and the parameter optimization model to obtain the time compensation parameters of the rolling shutter.

[0095] As an optional implementation, the image acquisition device is a fisheye camera, and the calibration parameters include vignetting correction parameters.

[0096] In this embodiment, when the image acquisition device on the device to be calibrated is a fisheye camera, since the fisheye camera usually has dark corners on the edges of the image when acquiring image data of the calibration plate, when the device to be calibrated moves within the target space, the mark bits in the image data of the calibration plate acquired by the fisheye camera may appear at various positions in the image. Based on this, the image data acquired by the fisheye camera is processed by a visual detection model and a parameter optimization model to obtain dark corner correction parameters to overcome the problem of dark corners on the edges of the image acquired by the fisheye camera.

[0097] As an optional implementation, the grayscale value of the marked feature point in the image data and the position coordinates of the marked feature point in the image coordinate system are determined; the grayscale value and the position coordinates are polynomially fitted to obtain the vignetting correction parameters.

[0098] In this embodiment, when determining the vignetting correction parameters, the grayscale value of the white area inside the mark position formed by the mark feature points in each frame of the image data captured by the fisheye camera and the position coordinates of the mark feature points in the mark position in the image coordinate system can be determined first. Then, the determined grayscale value and the position coordinates are polynomially fitted to obtain the vignetting correction parameters.

[0099] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0100] According to an embodiment of the present application, a device calibration system is also provided, comprising a control component, a device to be calibrated, and a calibration plate, wherein the device to be calibrated comprises an image acquisition device and an inertial measurement component. The image acquisition device is configured to acquire image data of the calibration plate within a target spatial range; the inertial measurement component is deployed within the device to be calibrated and is configured to acquire spatial data of the device to be calibrated as it moves within the target spatial range, wherein the spatial data includes posture data and velocity data of the device to be calibrated as it moves within the target spatial range; and the control component is connected to the image acquisition device and the inertial measurement component and is configured to convert the image data and spatial data into calibration parameters for the device to be calibrated.

[0101] Figure 4 is a schematic diagram of a calibration system for a device according to an embodiment of the present application. As shown in Figure 4, the calibration system 400 of the device includes a control component 401, a device to be calibrated 402 and a calibration plate 403, wherein the device to be calibrated 402 includes an image acquisition device 4021 and an inertial measurement component 4022, the image acquisition device 4021 includes an outward-facing camera 40211 and an eye-tracking camera 40212, the calibration plate 403 includes a calibration plate 4031 and a calibration plate 4032 deployed within a first target space range, and an outward-facing camera calibration plate 4033 and an eye-tracking camera calibration plate 4034 deployed within a second target space range, wherein the calibration plates 4031 and 4032 within the first target space range are deployed at a certain angle, and the outward-facing camera calibration plate 4033 within the second target space range is arranged opposite to a side of the eye-tracking camera calibration plate 4034 containing marked feature points.

[0102] The device calibration system 400 also includes a robotic arm 404, wherein one end of the robotic arm 404 is connected to the control component 401, and the other end is connected to the device to be calibrated 402. The robotic arm 404 is used to drive the device to be calibrated 402 to move within the first target space range under the control of the control component 401, or to move the device to be calibrated 402 to the second target space range.

[0103] When the robotic arm 404 is located at the joint calibration position of the outward-facing camera and the IMU, the device to be calibrated 402 is located within the first target space range, and the robotic arm 404 can drive the device to be calibrated to move within the first target space range. During the movement, the outward-facing camera 40211 can collect image data of the calibration plate 4031 and the calibration plate 4032 within the first target space range, and the IMU 4022 can collect spatial data of the device to be calibrated during the movement of the device to be calibrated, wherein the spatial data includes posture data and speed data of the device to be calibrated moving within the first target space range; when the robotic arm 404 is located at the joint calibration position of the eye-tracking camera and the IMU, the device to be calibrated 402 is located within the second target space range, and the robotic arm 404 can move the device to be calibrated to the second target space range, and the outward-facing camera 40211 can collect image data of the outward-facing camera calibration plate 4033 within the second target space range, and the eye-tracking camera 40212 can collect image data of the eye-tracking camera calibration plate 4034 within the second target space range.

[0104] After collecting image data and spatial data within the first target space range, the collected image data and spatial data can be used as calibration data, and then the calibration data can be processed to determine the intrinsic parameters and extrinsic parameters of the outward-facing camera 40211, as well as the intrinsic parameters and extrinsic parameters of the IMU. The process of processing the calibration data can refer to the methods introduced in the aforementioned steps S103 and S104, which will not be repeated here. Similarly, after collecting image data within the second target space range, the collected image data can be used as calibration data, and then the calibration data can be processed to determine the intrinsic parameters and extrinsic parameters of the eye-tracking camera. The process of processing the calibration data can also refer to the methods introduced in the aforementioned steps S103 and S104, which will not be repeated here.

[0105] The technical solutions of the embodiments of the present application are illustrated below with reference to preferred implementation methods.

[0106] With the rise of the concept of the "metaverse," the demand for augmented reality devices, virtual reality devices, and mixed reality devices has continued to rise. These devices are usually equipped with multiple cameras and IMUs for scene perception, device positioning, virtual-reality integration, user interaction, etc. In order to prevent distortion in the imaging of virtual reality devices, it is usually necessary to pre-calibrate the intrinsic and extrinsic parameters of each camera in the virtual reality device, as well as the intrinsic and extrinsic parameters of the IMU. Among them, the intrinsic parameters of the camera include parameters such as focal length, optical center coordinates, and distortion, which are used to describe the imaging laws of the camera; the intrinsic parameters of the IMU include zero bias, scale factor error, installation error, etc., which are used to describe the intrinsic properties of the IMU; the extrinsic parameters of the camera / IMU are used to describe the rotation and translation of the camera / IMU in the world coordinate system, and are used to describe the conversion relationship between the coordinate system of the camera / IMU and the world coordinate system.

[0107] At present, in the joint calibration of multiple cameras + IMU, a pattern array flat plate with a fixed spacing is usually used as the calibration plate. The handheld device to be calibrated is moved in front of the calibration plate. The movement includes translation and rotation. However, due to the weak anti-blur capability of this type of calibration plate, the image of the calibration plate captured by the camera will produce motion blur during the rapid movement of the device to be calibrated. In addition, if the camera uses a fisheye lens, it is easy to fail to detect the edge corners of the picture, resulting in inaccurate calibration of the camera's intrinsic parameters, especially the distortion parameters, thereby affecting the accuracy of the camera's working results.

[0108] However, an embodiment of the present application provides a device calibration method, in which at least one calibration plate is deployed within the target space range, and calibration data within the target space range can be obtained through the device to be calibrated, and the calibration data includes image data and optimization data collected from the calibration plate. The image data in the calibration data is detected, and the position information of the marked feature points on the calibration plate included in the image data can be determined. Since the position information of the marked feature points can represent the position coordinates of the marked feature points in the three-dimensional space coordinate system and the position coordinates of the marked feature points in the image coordinate system, and since the position information can accurately determine the position of the marked feature points, based on this, the parameters of the device to be calibrated are optimized based on the position information of the marked feature points and the optimization data collected by the second device, and the calibration parameters of the device to be calibrated are relatively accurate, thereby achieving the technical effect of improving the accuracy of the calibration parameters, thereby solving the technical problem of inaccurate device parameter calibration in the related art.

[0109] In this embodiment, when the device to be calibrated is one of the AR / VR / MR / XR glasses, since the AR / VR / MR / XR glasses include an outward-facing camera, an eye-tracking camera and an inertial measurement unit, wherein the outward-facing camera is deployed on the outside of the AR / VR / MR / XR glasses and the eye-tracking camera is deployed on the inside of the AR / VR / MR / XR glasses, based on this, the calibration of the AR / VR / MR / XR glasses includes: calibration of the outward-facing camera and the inertial measurement unit, as well as external parameter calibration of the eye-tracking camera.

[0110] The following is a further introduction to the calibration methods of the outward-facing camera and inertial measurement unit.

[0111] In this embodiment, two calibration plates can be deployed within the first target space, as shown in Figure 4. The two calibration plates can be arranged at a certain angle. The calibration plates include marked feature points. The control component and the device to be calibrated (AR / VR / MR / XR glasses) are connected via a data cable or wirelessly. Based on this, the AR / VR / MR / XR glasses can be fixed to the head of a robotic arm, which is controlled by the control component and can drive the device to be calibrated to move freely in six degrees of freedom within the first target space. For example, the control component can control the robotic arm to drive the device to be calibrated from an initial position through several pre-set waypoints to a final position. As the AR / VR / MR / XR glasses move along this movement route, they can collect calibration data during the movement. The outward-facing camera in the AR / VR / MR / XR glasses can collect image data of the calibration plates within the first target space, and the inertial measurement unit in the AR / VR / MR / XR glasses can collect spatial data of the device to be calibrated moving within the first target space. The spatial data includes posture data and velocity data of the device to be calibrated during the movement. After collecting image data and spatial data, the AR / VR / MR / XR glasses can transmit the collected image data and spatial data to the control component. After receiving the image data, the control component can input the image data into the visual detection model. The visual detection model can detect the index number of the marked feature point of the calibration plate included in the image data. After determining the index number, the three-dimensional coordinates of the marked feature point in the real world can be determined. Based on the position of the marked feature point in the image data, the image coordinates of the marked feature point in the image coordinate system can be determined. After obtaining the image coordinates and three-dimensional coordinates of the marked feature point, the image coordinates and three-dimensional coordinates corresponding to each marked feature point can be determined as a feature point pair, and then multiple feature point pairs are obtained. The multiple feature point pairs are formed into a point pair set. Afterwards, the point pair set and the spatial data collected by the inertial measurement component are input into the parameter optimization model to determine the intrinsic and extrinsic parameters of the outward-facing camera, as well as the intrinsic and extrinsic parameters of the inertial measurement component.

[0112] FIG5 is a schematic diagram of images captured by a VR pair of glasses at several landmark points according to an embodiment of the present application, as well as visual detection results of feature points on a calibration plate.

[0113] Optionally, if the outward-facing camera carried by the device to be calibrated is a rolling shutter camera, the time compensation parameters of the rolling shutter camera can also be determined based on the above method.

[0114] Optionally, if the outward-facing camera on the device to be calibrated is a fisheye camera, the fisheye camera generally has dark corners at the edges of the image. Based on the above calibration method, the dark corner correction parameters of the fisheye camera can also be determined. For example, when the robot arm drives the device to be calibrated to move, the markers composed of the marking feature points may appear in various positions of the image. Since the objective brightness of the markers in the real world is fixed, for a single camera, the algorithm records the grayscale value of the white area inside the marker of each frame {gray i}, and the image coordinate values ​​{x i ,y i}, polynomial fitting can be performed through the following formula to obtain the image vignetting correction parameters. predict (u, V) = gray measure (u,V)*(1+k1*r+k2*r 2 +k3*r 3 )

[0115] Wherein, graypredict(u, v) is the grayscale value after vignetting correction, graymeasure(u, v) is the grayscale value of the original image, r = sqrt((u-u0)2+(v-v0)2), u0 and v0 are the brightness centers of the image to be fitted, and k1, k2, and k3 are fitting parameters.

[0116] It should be noted that the calibration plate in the calibration environment described above includes both common feature points and marker feature points. The differences in marker information between common feature points and marker feature points include at least one of the following: marker structure, marker color, marker grayscale value, and marker shape. For devices to be calibrated (such as mobile phones and some AR glasses) with multiple cameras facing essentially the same direction, a single calibration plate of appropriate size is typically sufficient. As shown in Figure 2, the calibration plate is composed of dots, including two marker positions. The marker positions consist of larger dots and rings that are distinct from ordinary dots. The algorithm can identify unique dots within the marker positions and, based on their local structure, infer which of the many marker positions a detected marker position represents. The marker positions can then be used to correctly infer the indices and coordinates of other dots in the image captured by the camera, and this information can then be used to estimate calibration parameters. Figure 3 shows some examples of marker positions. Marker positions can also be distinguished using discriminative information such as color and shape.

[0117] It should be noted that some devices to be calibrated (such as VR glasses) usually do not have enough common viewing area between the multiple cameras, that is, the camera orientations are quite different. In this case, the coverage area of ​​the calibration plate is required to be larger. Based on this, the size of a single calibration plate can be increased. However, when the angle between the camera optical axis and the normal of the calibration plate is large, it is easy for the feature point detection on the calibration plate to fail or be detected incorrectly, resulting in calibration failure. Therefore, it is usually possible to arrange multiple calibration plates at appropriate angles according to the camera arrangement characteristics of the device to be calibrated. Figure 6 is a schematic diagram of using two calibration plates to adapt to the calibration requirements of VR glasses according to an embodiment of the present application. As shown in Figure 6, the ring-dot distribution of the pattern of the mark position on the calibration plate is identified by the algorithm, and the two calibration plates can be distinguished. For example, the mark position of a ring plus two large dots belongs to the type-A calibration plate, and the mark position of two rings plus one large dot belongs to the type-B calibration plate. The calibration plates are essentially distinguished by the marking positions, so the calibration plates can also be distinguished by distinguishing information such as color, shape, and structure of the marking positions.

[0118] The following is a further introduction to the calibration method of the eye-tracking camera.

[0119] In this embodiment, at least one outward-facing camera calibration plate and at least one eye-tracking camera calibration plate can be deployed within the second target space. Since the outward-facing camera and the eye-tracking camera in the device to be calibrated are designed back-to-back, their viewing angles are opposite to each other. Based on this, the at least one outward-facing camera calibration plate and the at least one eye-tracking camera calibration plate can be arranged relative to each other on one side containing the marked feature points. Figure 7 is a schematic diagram of an eye-tracking camera calibration environment according to an embodiment of the present application. As shown in Figure 7, the eye-tracking camera is used to capture image data of the eye-tracking camera calibration plate, and the outward-facing camera is used to capture image data of the outward-facing camera calibration plate. The outward-facing camera calibration plate and the eye-tracking camera calibration plate include marked feature points, and after the outward-facing camera calibration plate and the eye-tracking camera calibration plate are designed and installed, the relative position relationship between the two calibration plates can be obtained by measurement, and then the 3D coordinates of the marked feature points on the two calibration plates in the same coordinate system can be obtained. For example, the image coordinates of the marked feature points on the calibration plate and their corresponding index numbers on the calibration plate can be detected by the visual detection model. Based on the index numbers, the three-dimensional coordinates of the marked feature points in the real world can be inferred. According to this method, a set of image coordinate-three-dimensional coordinate pairs corresponding to multiple marked feature points on the calibration plate can be obtained; the parameter optimization model can use the set of image coordinate-three-dimensional coordinate pairs to solve the external parameters of the eye-tracking camera and the outward-facing camera. It should be noted that in this embodiment, the outward-facing camera can be any of the cameras mentioned in the aforementioned joint calibration of the camera and IMU. Usually, the most suitable camera can be selected as the outward-facing camera used in this embodiment based on its installation position and installation angle.

[0120] Figure 8 is a schematic diagram of a calibration plate image captured by a camera within a second target space range according to an embodiment of the present application. As shown in Figure 8, the left image 801 is an image of the calibration plate captured by an outward-facing camera in the VR glasses, the middle image 802 is an image of the calibration plate captured by an eye-tracking camera, and the right image 803 is an image of the calibration plate captured by another eye-tracking camera.

[0121] In the above embodiment, the device to be calibrated is driven by a robotic arm to move according to multiple preset landmarks. The device to be calibrated simultaneously collects image data and IMU data, and the calibration parameters of the camera and IMU are obtained through the calibration algorithm, including the internal and external parameters of the camera and the internal and external parameters of the IMU. In addition, the calibration scheme can also simultaneously realize the rolling shutter time compensation calibration and the image vignetting correction calibration; the device to be calibrated is moved to the calibration position of the eye-tracking camera by the robotic arm, which can meet the external parameter calibration of the eye-tracking camera and the outward-facing camera of the AR / VR / MR glasses, and provide a coordinate conversion relationship for the eye-tracking device in the virtual scene.

[0122] According to an embodiment of the present application, a device calibration apparatus is also provided. It should be noted that the device calibration apparatus can be used to execute the device calibration method in Example 1.

[0123] FIG9 is a schematic diagram of a device calibration apparatus according to an embodiment of the present application. As shown in FIG9 , the device calibration apparatus 900 may include: a first determination component 901 , an acquisition component 902 , a second determination component 903 , and an optimization component 904 .

[0124] The first determining component 901 is configured to determine a target space range from a candidate space range according to the type of the device to be calibrated, wherein at least one calibration plate is deployed in the candidate space range, and the device to be calibrated includes a first device and a second device.

[0125] The acquisition component 902 is configured to acquire calibration data through the device to be calibrated within the target space, wherein the calibration data includes image data collected by the first device on the calibration plate and optimization data collected by the second device.

[0126] The second determining component 903 is configured to detect the image data in the calibration data to determine the position information of the marked feature points.

[0127] The optimization component 904 is configured to perform synchronous parameter optimization in combination with the position information and the optimization data to obtain calibration parameters of the device to be calibrated.

[0128] Optionally, the candidate spatial range includes a first target spatial range and a second target spatial range, wherein at least one first camera calibration plate is deployed within the first target spatial range, and at least one first camera calibration plate and at least one second camera calibration plate are deployed within the second target spatial range. The first determination component 901 includes: a first selection component, configured to select the first target spatial range as the target spatial range when the type of the second device is an inertial measurement component; and a second selection component, configured to select the second target spatial range as the target spatial range when the type of the second device is an image acquisition device.

[0129] Optionally, the marked feature points on the calibration plate correspond to index numbers, and the image data in the calibration data is detected. The second determination component 903 includes: an acquisition component, which is configured to obtain the index number of the marked feature point on the calibration plate by detecting the image data in the calibration data; a first determination component, which is configured to determine the position coordinates of the marked feature point in the three-dimensional space coordinate system based on the index number; a second determination component, which is configured to determine the position coordinates of the marked feature point in the image coordinate system based on the position of the marked feature point in the image data; and a third determination component, which is configured to determine the position coordinates of the marked feature point in the three-dimensional space coordinate system and the position coordinates of the marked feature point in the image coordinate system as the position information of the marked feature point.

[0130] Optionally, the acquisition component 901 includes: a sending component, configured to send a data acquisition instruction to the device to be calibrated, wherein the data acquisition instruction is configured to instruct the device to be calibrated to turn on the data acquisition function to collect image data and optimization data; a receiving component, configured to receive image data and optimization data fed back by the device to be calibrated, wherein the image data and optimization data constitute calibration data.

[0131] Optionally, the device 900 also includes: a third determination component, configured to determine the grayscale value of the marked feature point in the image data and the position coordinates of the marked feature point in the image coordinate system; a fitting component, configured to perform polynomial fitting on the grayscale value and the position coordinate to obtain the vignetting correction parameter.

[0132] In this embodiment, a robotic arm drives the device to be calibrated to move according to multiple preset landmarks. The device to be calibrated simultaneously collects image data and IMU data, and the calibration parameters of the camera and IMU are obtained through the calibration algorithm, including the internal and external parameters of the camera and the internal and external parameters of the IMU. In addition, this calibration scheme can also simultaneously realize rolling shutter time compensation calibration and image vignetting correction calibration; the robotic arm moves the device to be calibrated to the calibration position of the eye-tracking camera, which can meet the external parameter calibration of the eye-tracking camera and the outward-facing camera of the AR / VR / MR glasses, and provide a coordinate conversion relationship for the eye-tracking device in the virtual scene.

[0133] According to an embodiment of the present application, a computer-readable storage medium is also provided, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the calibration method of the device in Example 1.

[0134] According to an embodiment of the present application, a processor is further provided, which is used to run a program, wherein the calibration method of the device in Example 1 is executed when the program is run.

[0135] An embodiment of the present application further provides a computer program product, including computer instructions, which, when executed by a processor, implement the device calibration method provided in the embodiment of the present application.

[0136] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0137] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the components can be a logical function division. In actual implementation, there may be other division methods, such as multiple components or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of components, which can be electrical or other forms.

[0139] The components described as separate parts may or may not be physically separate, and the components shown as components may or may not be physical components, that is, they may be located in one place or distributed across multiple components. Some or all of the components may be selected according to actual needs to achieve the purpose of the present embodiment.

[0140] In addition, the functional components in the various embodiments of the present application may be integrated into a processing component, or each component may exist physically separately, or two or more components may be integrated into a single component. The above-mentioned integrated components may be implemented in the form of hardware or software functional components.

[0141] If the integrated components are implemented in the form of software functional components and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0142] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application. Industrial Applicability

[0143] The solution provided by the embodiment of the present application can be applied to the calibration process of a mixed reality device, and the target space range is determined from the candidate space range according to the type of the device to be calibrated, wherein at least one calibration plate is deployed in the candidate space range, and the device to be calibrated includes a first device and a second device; within the target space range, calibration data is obtained by the device to be calibrated, wherein the calibration data includes image data collected by the first device on the calibration plate and optimization data collected by the second device; the image data in the calibration data is detected to determine the position information of the marked feature points; synchronous parameter optimization is performed in combination with the position information and the optimization data to obtain the calibration parameters of the device to be calibrated. Since the calibration parameters of the device to be calibrated are obtained by optimizing the parameters of the device to be calibrated based on the position information of the marked feature points and the optimization data collected by the second device, the technical effect of improving the accuracy of the calibration parameters can be achieved, thereby solving the technical problem of inaccurate device parameter calibration in the related art.

Claims

1. A method for calibrating a device, comprising: Determine a target space range from a candidate space range according to the type of the device to be calibrated, wherein at least one calibration plate is deployed in the candidate space range, and the device to be calibrated includes a first device and a second device; Acquiring calibration data through the device to be calibrated within the target space, wherein the calibration data includes image data acquired by the first device on the calibration plate and optimization data acquired by the second device; Detecting the image data in the calibration data to determine the position information of the marked feature points; The position information and the optimization data are combined to perform synchronous parameter optimization to obtain calibration parameters of the device to be calibrated.

2. The calibration method according to claim 1, wherein: Determining the target space range from the candidate space range according to the type of device to be calibrated includes: The candidate space range includes a first target space range and a second target space range, wherein at least one first camera calibration plate is deployed within the first target space range, and at least one first camera calibration plate and at least one second camera calibration plate are deployed within the second target space range; When the type of the second device is an inertial measurement unit, selecting the first target space range as the target space range; When the type of the second device is an image acquisition device, the second target space range is selected as the target space range.

3. The calibration method according to claim 2, wherein: The first target space range includes a number of preset landmarks, and the device to be calibrated moves according to the preset landmarks, and the calibration data is collected during the movement.

4. The calibration method according to claim 2, wherein: In the second target space range, the first camera calibration plate and the second camera calibration plate are parallel, and the calibration plate patterns are arranged opposite to each other.

5. The calibration method according to claim 1, wherein: The marked feature points on the calibration plate correspond to index numbers, and the detecting of the image data in the calibration data to determine the position information of the marked feature points in the image data includes: By detecting the image data in the calibration data, an index number of the marked feature point on the calibration plate is obtained; Determine the position coordinates of the marked feature point in a three-dimensional space coordinate system based on the index number; Determine the position coordinates of the marked feature point in the image coordinate system based on the position of the marked feature point in the image data; The position coordinates of the mark feature point in the three-dimensional space coordinate system and the position coordinates of the mark feature point in the image coordinate system are determined as the position information of the mark feature point.

6. The calibration method according to claim 1, wherein: The step of obtaining calibration data through the device to be calibrated includes: Sending a data acquisition instruction to the device to be calibrated, wherein the data acquisition instruction is used to instruct the device to be calibrated to start a data acquisition function to acquire the image data and the optimization data; The image data and the optimization data fed back by the device to be calibrated are received, wherein the image data and the optimization data constitute the calibration data.

7. The calibration method according to claim 1, wherein: The calibration plate includes common feature points and the marked feature points, wherein the difference between the marking information of the common feature points and the marking information of the marked feature points includes at least one of the following: marking structure, marking color, marking gray value, and marking shape.

8. The calibration method according to claim 5, wherein: The marking information of the marking feature points in different calibration plates is different; the arrangement angle between multiple calibration plates is determined according to the arrangement position and field of view angle between the first device and the second device to ensure that the first device and the second device can both collect the complete marking information of the marking feature points.

9. The calibration method according to claim 1, wherein: The image acquisition device is a rolling shutter camera, and the calibration parameters include time compensation parameters of the rolling shutter.

10. The calibration method according to claim 1, wherein: The image acquisition device is a fisheye camera, and the calibration parameters include vignetting correction parameters.

11. The calibration method according to claim 10, wherein: The method further comprises: Determine the grayscale value of the marked feature point in the image data and the position coordinates of the marked feature point in the image coordinate system; The grayscale value and the position coordinate are subjected to polynomial fitting to obtain the vignetting correction parameter.

12. A device for calibrating an equipment, wherein: include: The first determination component is configured to determine the target space from the candidate space range according to the type of the device to be calibrated. A candidate space range is defined, wherein at least one calibration plate is deployed within the candidate space range, and the device to be calibrated includes a first device and a second device; an acquisition component, configured to acquire calibration data through the device to be calibrated within the target space, wherein the calibration data includes image data acquired by the first device on the calibration plate and optimization data acquired by the second device; A second determination component is configured to detect the image data in the calibration data to determine the position information of the marked feature points; The optimization component is configured to perform synchronous parameter optimization in combination with the position information and the optimization data to obtain calibration parameters of the device to be calibrated.

13. A calibration system for a device, wherein: The calibration system comprises: a control component, a device to be calibrated and a calibration board, wherein the device to be calibrated comprises an image acquisition device and an inertial measurement component; The image acquisition device is configured to acquire image data of a calibration plate within a target space; The inertial measurement assembly is deployed in the device to be calibrated and is configured to collect spatial data of the device to be calibrated moving within the target space; The control component is connected to the image acquisition device and the inertial measurement component, and is configured to convert the image data and the spatial data into calibration parameters of the device to be calibrated.

14. A computer-readable storage medium, wherein: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the calibration method of the device according to any one of claims 1 to 11.

15. A computer program product, comprising computer instructions, which, when executed by a processor, implement the device calibration method according to any one of claims 1 to 11.

Citation Information

Patent Citations

  • Vehicle-mounted camera auxiliary calibration method, device and equipment and medium

    CN110276807A

  • Calibration method and device for relative postures of camera and inertial measurement unit

    CN110378968A

  • Camera calibration board, calibration data acquisition method, distortion correction method and device

    CN110458898A

  • Calibration method and calibration device of augmented reality equipment and terminal equipment

    CN110782499A

  • Multi-sensor joint calibration method and device, processor and storage medium

    CN114545373A

Cited By

  • Automatic calibration method of multi-view camera, electronic equipment and storage medium

    CN120782875A

  • Virtual-real superposition precision test method and system, and storage medium

    CN120909861A

  • 3D printing modular structural mechanics intelligent STEM teaching method and system

    CN121302795A

  • Head-mounted device, calibration method and device of head-mounted device, and medium

    CN121564111A