Calibration method and system, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides a calibration method and system, electronic equipment and a storage medium. The calibration system comprises a motion assembly, a camera assembly and a processing assembly. The motion assembly is fixedly connected with the augmented reality device and drives the augmented reality device to move synchronously. The camera assembly is fixedly connected with the augmented reality device and collects a plurality of images, displayed by the augmented reality device, of the target plate in the synchronous movement process. The processing component calibrates external parameters of the camera component based on the plurality of images and the motion pose sequence of the motion component.

Description

Technical Field

[0001] The present disclosure relates to the technical field of extended reality devices, and in particular to a calibration method, a calibration system, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the development of science and technology, higher requirements are placed on the performance of extended reality devices such as virtual reality devices, augmented reality devices, and mixed reality devices.

[0003] Taking the head-mounted display (HMD) as an example, the extended reality device needs to respond to the user's head movement in real time. Although the user's head movement can be measured by a variety of sensors, the HMD cannot immediately display the corresponding image to the user due to the time constraints of transmission and calculation, which will cause image delay. This delay will cause the difference between visual motion perception and vestibular motion perception, which will not only reduce the sense of presence of the extended reality device, but also increase the burden on the brain and cause dizziness and nausea. MTP (motion-to-photon) delay can include the time from the start of the user's movement to the display of the corresponding image on the screen. It represents the delay time between the image seen by the user in the extended reality device such as HMD and the user's head movement, and can quantify the degree of match between visual observation and user head movement. Therefore, for the user, the smaller the MTP delay, the better the user's immersion; the larger the MTP delay, the stronger the user's dizziness.

[0004] Considering that human head movements are mostly continuous three-dimensional (three-axis) compound movements, for extended display devices that support motion prediction algorithms, analyzing the delay changes in continuous three-dimensional compound movements is a key indicator for evaluating their motion prediction algorithms. In order to compare the motion posture changes in the real world and the virtual world, these two types of motion posture changes need to be unified in the same coordinate system.

[0005] The motion changes in the real world use the motion component as the reference coordinate system, and the motion changes in the virtual world use the camera component as the reference coordinate system. Therefore, it is necessary to calibrate the camera component and the motion component for coordinate system conversion, which is also called hand-eye calibration.

[0006] Therefore, how to unify the motion information of the motion component and the visual component in the same coordinate system and provide a coordinate system reference for measuring the delay of the extended reality device in continuous three-dimensional compound motion has become one of the research hotspots in the field of extended reality device technology. Summary of the invention

[0007] The embodiments of the present disclosure provide a calibration method, a calibration system, an electronic device, a computer-readable storage medium, and a computer program product, which provide a coordinate system reference for measuring the delay of an extended reality device in continuous three-dimensional compound motion.

[0008] In a first aspect, an embodiment of the present disclosure proposes a calibration system, which includes: a motion component, which is fixedly connected to an extended reality device and drives the extended reality device to move synchronously; a camera component, which is fixedly connected to the extended reality device and collects multiple images of a target plate displayed by the extended reality device during the synchronous movement; and a processing component, which calibrates the external parameters of the camera component based on the multiple images and the motion posture sequence of the motion component.

[0009] In some embodiments of the present disclosure, the processing component includes: a first conversion unit, which extracts pixel coordinates of feature points of a target based on multiple images; a second conversion unit, which converts the pixel coordinates into a visual pose sequence according to a spatial geometric constraint relationship between preset feature points; and a calibration unit, which calibrates the external parameters of the camera component based on the visual pose sequence and the motion pose sequence.

[0010] In some embodiments of the present disclosure, the calibration unit includes: a linear transformation subunit, which obtains the initial values ​​of the extrinsic parameters of the camera component based on the visual pose sequence and the motion pose sequence through a direct linear transformation method; and a nonlinear transformation subunit, which optimizes the initial values ​​of the extrinsic parameters of the camera component based on minimizing the reprojection error through a nonlinear transformation method to obtain the extrinsic parameters of the camera component.

[0011] In some embodiments of the present disclosure, a visual posture sequence includes a plurality of first time-pose information arranged in chronological order, the first time-pose information including first time information and a first posture angle corresponding to the first time information, wherein the first posture angle includes a first roll angle, a first yaw angle, and a first pitch angle; and a motion posture sequence includes a plurality of second time-pose information arranged in chronological order, the second time-pose information including second time information and a second posture angle corresponding to the second time information, wherein the second posture angle includes a second roll angle, a second yaw angle, and a second pitch angle.

[0012] In some embodiments of the present disclosure, the calibration system further includes: a time component, including a clock source, wherein the clock source is used to perform time synchronization between the motion component and the camera component before and during the synchronous movement.

[0013] In some embodiments of the present disclosure, during the synchronous movement of the camera assembly, when the extended reality device passes through a waypoint, the camera assembly captures an image of a target displayed by the extended reality device.

[0014] In some embodiments of the present disclosure, the camera assembly continuously captures images of the target displayed by the extended reality device during the synchronous movement.

[0015] In some embodiments of the present disclosure, the processing component further includes: a time correction unit for performing time correction on the visual pose sequence.

[0016] In some embodiments of the present disclosure, the time correction unit includes: an angular velocity calculation subunit, which determines a first angular velocity based on a motion posture sequence, and determines a second angular velocity based on a visual posture sequence; a cross-correlation subunit, which aligns the first angular velocity and the second angular velocity based on a cross-correlation method to determine a time offset of the visual posture sequence; and a correction subunit, which performs time correction on the visual posture sequence based on the time offset.

[0017] In some embodiments of the present disclosure, the time correction unit further includes: an interpolation subunit, which performs interpolation processing on the visual pose sequence before determining the second angular velocity so that the sampling frequencies of the motion pose sequence and the visual pose sequence are the same.

[0018] In some embodiments of the present disclosure, the time correction unit further includes: a filtering subunit, which performs filtering processing on the visual pose sequence before interpolating the visual pose sequence.

[0019] In some embodiments of the present disclosure, the processing component further includes: an image processing unit, which processes multiple images before extracting pixel coordinates of feature points of the target based on the multiple images, and the processing includes at least one of downsampling processing and Gaussian blur processing.

[0020] In some embodiments of the present disclosure, the target is a virtual target, wherein the virtual target includes a visual marker calibration plate in a visual reference library.

[0021] In some embodiments of the present disclosure, the processing component further includes: an adjustment unit, which adjusts the positional relationship between the camera component and the extended reality device so that multiple feature points specified in the target plate are shown in multiple images.

[0022] In the second aspect, an embodiment of the present disclosure proposes a calibration method, which includes: obtaining a motion posture sequence of a motion component, wherein the motion component is fixedly connected to an extended reality device and drives the extended reality device to move synchronously; using a camera component fixedly connected to the extended reality device, during the synchronous movement, collecting multiple images of a target plate displayed by the extended reality device; and calibrating the external parameters of the camera component based on the multiple images and the motion posture sequence of the motion component.

[0023] In some embodiments of the present disclosure, based on multiple images and a motion pose sequence of a motion component, calibrating the external parameters of a camera component includes: extracting pixel coordinates of feature points of a target plate based on multiple images; converting the pixel coordinates into a visual pose sequence according to a preset spatial geometric constraint relationship between the feature points; and calibrating the external parameters of the camera component based on the visual pose sequence and the motion pose sequence.

[0024] In some embodiments of the present disclosure, calibrating the extrinsic parameters of a camera component based on a visual pose sequence and a motion pose sequence includes: obtaining initial values ​​of the extrinsic parameters of the camera component based on the visual pose sequence and the motion pose sequence through a direct linear transformation method; and optimizing the initial values ​​of the extrinsic parameters of the camera component based on minimizing the reprojection error through a nonlinear transformation method to obtain the extrinsic parameters of the camera component.

[0025] In some embodiments of the present disclosure, a visual posture sequence includes a plurality of first time-pose information arranged in chronological order, the first time-pose information including first time information and a first posture angle corresponding to the first time information, wherein the first posture angle includes a first roll angle, a first yaw angle, and a first pitch angle; and a motion posture sequence includes a plurality of second time-pose information arranged in chronological order, the second time-pose information including second time information and a second posture angle corresponding to the second time information, wherein the second posture angle includes a second roll angle, a second yaw angle, and a second pitch angle.

[0026] In some embodiments of the present disclosure, the calibration method further includes: utilizing a clock source to perform time synchronization on the motion component and the camera component before and during the synchronous movement.

[0027] In some embodiments of the present disclosure, during the synchronous movement, collecting multiple images of the target board displayed by the extended reality device includes: during the synchronous movement, the camera component collects the image of the target board displayed by the extended reality device when the extended reality device passes a waypoint.

[0028] In some embodiments of the present disclosure, during the synchronous movement, capturing multiple images of the target displayed by the extended reality device includes: the camera component continuously captures the images of the target displayed by the extended reality device during the synchronous movement.

[0029] In some embodiments of the present disclosure, the calibration method further includes: performing time correction on the visual pose sequence.

[0030] In some embodiments of the present disclosure, performing temporal correction on the visual pose sequence includes: A first angular velocity is determined based on a motion pose sequence, and a second angular velocity is determined based on a visual pose sequence; based on a cross-correlation method, the first angular velocity and the second angular velocity are aligned to determine a time offset of the visual pose sequence; and based on the time offset, a time correction is performed on the visual pose sequence.

[0031] In some embodiments of the present disclosure, performing time correction on the visual pose sequence further includes: before determining the second angular velocity, performing interpolation processing on the visual pose sequence so that the sampling frequencies of the motion pose sequence and the visual pose sequence are the same.

[0032] In some embodiments of the present disclosure, performing time correction on the visual pose sequence further includes: performing filtering processing on the visual pose sequence before performing interpolation processing on the visual pose sequence.

[0033] In some embodiments of the present disclosure, the calibration method further includes: before extracting the pixel coordinates of the feature points of the target plate based on the multiple images, processing the multiple images, wherein the processing includes at least one of downsampling processing and Gaussian blur processing.

[0034] In some embodiments of the present disclosure, the target is a virtual target, wherein the virtual target includes a visual marker calibration plate in a visual reference library.

[0035] In some embodiments of the present disclosure, the calibration method further includes: before acquiring multiple images of the target plate displayed by the extended reality device, adjusting the positional relationship between the camera assembly and the extended reality device so that multiple feature points specified in the target plate are shown in the multiple images.

[0036] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the calibration method described in any implementation method in the second aspect when executing the instructions.

[0037] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the calibration method described in any implementation manner in the second aspect when executed.

[0038] In a fifth aspect, an embodiment of the present disclosure provides a computer program product including a computer program, which, when executed by a processor, can implement the calibration method described in any implementation manner in the second aspect.

[0039] It should be understood that the content described in this section is not intended to identify the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Other features, objectives and advantages of the disclosed embodiments will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings, in which: Figure 1 A schematic diagram of a calibration system according to an exemplary embodiment of the present disclosure is shown; Figure 2 A schematic diagram showing an extended reality device according to an exemplary embodiment of the present disclosure is shown; Figure 3 A schematic diagram of a target according to an exemplary embodiment of the present disclosure is shown; Figure 4 A schematic diagram showing one of multiple images displayed by an extended reality device according to an exemplary embodiment of the present disclosure; Figure 5 A schematic diagram of a motion assembly according to an exemplary embodiment of the present disclosure is shown; Figure 6 A schematic diagram of the structure of a processing component according to an exemplary embodiment of the present disclosure is shown; Figure 7 A schematic diagram of a process of capturing multiple images by a camera assembly according to an exemplary embodiment of the present disclosure is shown; Figure 8 A schematic structural diagram of a time correction unit according to an exemplary embodiment of the present disclosure is shown; Fig. 9 A time correction flow chart according to an exemplary embodiment of the present disclosure is shown; Fig.10 A schematic diagram showing alignment of a first angular velocity and a second angular velocity based on a cross-correlation method according to an exemplary embodiment of the present disclosure is shown; Fig.11 A schematic diagram of a process of extrinsic calibration of a camera assembly according to an exemplary embodiment of the present disclosure is shown; Fig.12 A schematic diagram of a process for calibrating the external parameters of a camera component based on multiple images and a motion pose sequence of a motion component according to an exemplary embodiment of the present disclosure is shown; Fig.13 A schematic diagram showing motion information of a motion component and a calibrated camera component in the same coordinate system according to an exemplary embodiment of the present disclosure is shown; Fig.14 A flow chart of a calibration method provided in an embodiment of the present disclosure; Fig.15 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0041] In order to better understand the present disclosure, various aspects of the present disclosure will be described in more detail with reference to the accompanying drawings. It should be understood that these detailed descriptions are only descriptions of exemplary embodiments of the present disclosure and are not intended to limit the scope of the present disclosure in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0042] It should be noted that in this specification, the expressions first, second, etc. are only used to distinguish one feature from another feature, and do not represent any limitation on the feature. Therefore, without departing from the teaching of the present disclosure, the first conversion unit discussed below may also be referred to as the second conversion unit, and the first time-pose curve may also be referred to as the second time-pose curve.

[0043] It should also be understood that expressions such as "include", "comprising", "having", "including" and / or "comprising" are open rather than closed expressions in this specification, which indicate the presence of the stated features, elements and / or components, but do not exclude the presence of one or more other features, elements, components and / or combinations thereof. In addition, the word "exemplarily" is intended to refer to an example or illustration.

[0044] Unless otherwise defined, all terms (including technical and scientific terms) used in the present disclosure have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs. It should also be understood that terms (such as those defined in commonly used dictionaries) should be understood to have the same meaning as they have in the context of the relevant technology, and should not be interpreted in an idealized or overly formal sense, unless explicitly defined in the present disclosure.

[0045] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other. In addition, unless explicitly limited or contradictory to the context, the specific steps included in the method described in the present disclosure are not necessarily limited to the order described, but can be performed in any order or in parallel. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0046] Figure 1 A schematic diagram of a calibration system 1000 according to an exemplary embodiment of the present disclosure is shown. Figure 2 A schematic diagram of an extended reality device 400 according to an exemplary embodiment of the present disclosure is shown. Figure 3A schematic diagram of a target 410 according to an exemplary embodiment of the present disclosure is shown. Figure 4 A schematic diagram showing one image 420 among a plurality of images displayed by an extended reality device 400 according to an exemplary embodiment of the present disclosure is shown.

[0047] like Figure 1-Figure 4 As shown, the present disclosure provides a calibration system 1000, which may include a motion component 100, a camera component 200, and a processing component 300. The motion component 100 is fixedly connected to the extended reality device 400, and drives the extended reality device 400 to move synchronously. The camera component 200 is also fixedly connected to the extended reality device 400, and during the synchronous movement, it collects multiple images of a target plate 410 displayed by the extended reality device 400, such as Figure 4 An image 420 is shown in the plurality of images. The processing component 300 determines the external parameters of the camera component 200 based on the plurality of images and the motion pose sequence of the motion component 100. The external parameters of the camera component 200 can be understood as parameters describing the position and pose of the camera component 200 in the world coordinate system with the motion component 100 as a reference, and may include a rotation matrix and a translation vector.

[0048] According to the calibration system provided by at least one embodiment of the present disclosure, the motion component that moves synchronously with the extended reality device can simulate the continuous compound three-dimensional motion of the human body. In addition, the camera component can be used to obtain multiple images of the target plate displayed by the extended reality device. Therefore, the external parameters of the camera component can be obtained through the multiple images of the target plate displayed by the extended reality device acquired by the camera component and the motion pose sequence of the motion component, so as to unify the motion information of the camera component and the motion component in the same coordinate system, and provide a coordinate system reference for measuring the delay of the extended reality device in the continuous three-dimensional compound motion.

[0049] Specifically, in some embodiments of the present disclosure, the extended reality device 400 may include a virtual reality device, an augmented reality device, or a mixed reality device. It should be understood that the calibration system 1000 can also be applied to application scenarios of calibration of other display devices, which is not limited here.

[0050] The motion component 100 is fixedly connected to the extended reality device 400, for example, rigidly fixedly connected. In addition, the motion component 100 can drive the extended reality device 400 to move synchronously. The motion component 100 may include any suitable structural components, for example, the motion component 100 may include a mechanical arm and a controller for controlling the movement of the mechanical arm, etc. The motion component 100 can perform various translational movements and rotational movements in the world coordinate system. Taking the extended reality device 400 as an HMD as an example, the motion component 100 fixedly connected to the HMD can simulate the rotational movement of the human head in the sagittal plane, the coronal plane and the horizontal plane, etc.

[0051] The camera assembly 200 can be used to simulate the scene of human eyes receiving the image displayed by the extended reality device 400. Optionally, the camera assembly 200 may include at least one of a camera and a video camera, and the camera assembly may include an IR camera (Infrared Camera, infrared camera assembly), an RGB camera (RedGreenBlue Camera, red, green, blue camera assembly) or a black and white camera, etc. In addition, the camera assembly 200 may also include other structures such as photoelectric sensors, and the present disclosure does not limit the specific internal structure of the camera assembly 200.

[0052] Optionally, the target plate 410 may be a virtual target plate located in the extended reality device 400. The virtual target plate may include a visual marker calibration plate in the visual reference library. For example, a two-dimensional code calibration plate such as ApriTag and ArUco. Using a virtual target plate for calibration can simplify the structure of the calibration system, reduce the difficulty of implementing the calibration, and reduce resource consumption.

[0053] The processing component 300 can be various general and / or special processing components with processing and computing capabilities. Some examples of the processing component 300 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processing component 300 determines the external parameters of the camera component 200 based on the multiple images collected by the camera component 200 and the motion pose sequence of the motion component 100.

[0054] Optionally, the processing component 300 may include various suitable communication buses and input / output interfaces for achieving connection with the motion component 100 and the camera component 200 .

[0055] Reference again Figure 1 In some embodiments of the present disclosure, the calibration system 1000 further includes a time component 500, which may include a clock source, wherein the clock source is used to perform time synchronization on the motion component 100 and the camera component 200 before and during the synchronous movement.

[0056] Specifically, the clock source can provide unified time synchronization to the motion component 100 and the camera component 200 through the PTP (Precision Time Protocol) protocol. A clock source with high precision and accuracy is used as the master clock to send a timestamp to the motion component 100 and the camera component 200 as slave clocks. The slave clock compares the timestamp with its own local time and adjusts the time according to the difference, thereby ensuring that the subsequent motion posture sequence obtained based on the motion component 100 is synchronized with the time of multiple images collected based on the camera component 200.

[0057] Figure 5 A schematic diagram of a motion assembly 100 according to an exemplary embodiment of the present disclosure is shown. Figure 6 A schematic structural diagram of a processing component 300 according to an exemplary embodiment of the present disclosure is shown.

[0058] Combination Figure 1-Figure 6 In some embodiments of the present disclosure, the processing component 300 may include: a first conversion unit 320, a second conversion unit 330, and a calibration unit 350. The first conversion unit 320 may extract the pixel coordinates of the feature points of the target plate 410 based on the multiple images of the target plate 410 displayed by the extended reality device 400 collected by the camera component 200 during the synchronous movement. The second conversion unit 330 converts the pixel coordinates into a visual pose sequence according to the spatial geometric constraint relationship between the preset feature points. The calibration unit 350 calibrates the external parameters of the camera component 200 based on the visual pose sequence and the motion pose sequence.

[0059] Optionally, the processing component 300 may further include an image processing unit 310. The image processing unit 310 may process the multiple images before extracting the pixel coordinates of the feature points of the target plate 410 based on the multiple images. The processing includes at least one of downsampling processing and Gaussian blur processing. Since the appearance of moiré patterns may lead to inaccurate or even impossible extraction of 2D feature points when the screen of the extended reality device 400 is photographed using the camera component 200, it is necessary to process the multiple acquired images. By at least one of downsampling processing and Gaussian blur processing, the influence of moiré patterns on the above images can be reduced.

[0060] In addition, the processing component 300 may further include an adjustment unit 360, which may adjust the positional relationship between the camera component 200 and the extended reality device 400 so that the plurality of feature points specified in the target plate 410 are shown in the plurality of images. This may improve the quality of the generation of the plurality of images and ensure high stability and high accuracy of the calibration results.

[0061] like Figure 5 As shown, as an option, the motion component 100 may include a robotic arm, which may be fixedly connected to the extended reality device 400 and drive the extended reality device 400 to move synchronously. A predetermined motion trajectory may be designed and encoded into the robotic arm, so that the robotic arm can drive the extended reality device 400 to move in various directions according to the predetermined motion trajectory. In addition, the predetermined motion trajectory may be optimized to ensure that various rotations and accelerations of the extended reality device 400 are activated in various directions, thereby reducing the blurring of the target image due to excessive movement speed, which can ensure high stability and high accuracy of the calibration result.

[0062] Figure 7 A schematic diagram of a process of capturing multiple images by the camera assembly 200 according to an exemplary embodiment of the present disclosure is shown.

[0063] like Figure 1 , Figure 3 and Figure 7 As shown, the camera assembly 200 can have multiple acquisition modes to acquire multiple images of the target 410 displayed by the extended reality device 400. At the beginning of data acquisition for multiple images, the target (e.g., virtual target) 410 located in the extended reality device 400 can be opened. Afterwards, the data acquisition mode can be selected. As an option, the camera assembly 200 can acquire the image of the target 410 displayed by the extended reality device 400 only when the extended reality device 400 passes through the waypoint during the synchronous movement. As another option, the camera assembly 200 can continuously acquire the image of the target 410 displayed by the extended reality device 400 during the entire synchronous movement. Afterwards, the multiple image data acquired by the camera assembly 200 and the three-axis encoder data of the motion assembly 100 can be stored, wherein the three-axis encoder data of the motion assembly 100 can be used as the motion posture sequence of the motion assembly 100.

[0064] Waypoints can be understood as a series of key position points preset during the movement of the robot arm, and the robot arm needs to pass through these key position points in sequence to complete the task. In other words, multiple key position points can be selected as waypoints when the predetermined motion trajectory is pre-set. Multiple waypoints can fully activate the movement of the extended reality device 400 in all directions, so that it includes various rotation and acceleration motion information. Therefore, as an option, during the synchronous movement, the image of the target plate 410 displayed by the extended reality device 400 can be collected only when the extended reality device 400 passes the waypoint.

[0065] In this case, the synchronous motion is a discrete motion between different waypoints, and the camera assembly 200 is stationary at the waypoint for acquisition. For example, the synchronous motion can be stationary at the waypoint for a few seconds and then move to the next waypoint. Therefore, the time when the camera assembly 200 acquires the image is the same as the time when the extended reality device 400 (or, the motion assembly 100) synchronously moves to the corresponding waypoint, and there is no need to correct the acquisition time.

[0066] In another alternative, the camera assembly 200 can continuously capture the image of the target plate 410 displayed by the extended reality device 400 during the entire process of synchronous motion, wherein the synchronous motion is a continuous motion. Therefore, although the timestamp of the image of the target plate 410 captured by the camera assembly 200 and the trajectory timestamp of the motion assembly 100 (or the extended reality device) are under the same clock source, due to the existence of MTP, the process of capturing the image has a certain delay, so time correction is required.

[0067] Optionally, refer again to Figure 6 The processing component 300 may also include: a time correction unit 340, which can perform time correction on the visual pose sequence converted from pixel coordinates by the second conversion unit 330 according to the spatial geometric constraint relationship between the preset feature points.

[0068] It should be noted that, for the sake of clarity, Figure 6 The processing platform 300 shown shows the optional image processing unit 310, time correction unit 340 and adjustment unit 360. It can be distinguished that the optional image processing unit 310, time correction unit 340 or adjustment unit 360 is shown with a dotted line frame, and the connection between the image processing unit 310, time correction unit 340 or adjustment unit 360 and other units is shown with a dotted connecting line when the image processing unit 310, time correction unit 340 or adjustment unit 360 is included.

[0069] Figure 8 FIG. 4 shows a schematic structural diagram of a time correction unit 340 according to an exemplary embodiment of the present disclosure. Fig. 9 A time correction flow chart according to an exemplary embodiment of the present disclosure is shown.

[0070] like Figure 1 and Figure 8 As shown, in some embodiments of the present disclosure, the time correction unit 340 may include: an angular velocity calculation subunit 343, a cross-correlation subunit 344 and a correction subunit 345. The angular velocity calculation subunit 343 may determine a first angular velocity based on the motion pose sequence of the motion component 100, and determine a second angular velocity based on a visual pose sequence converted from a plurality of images collected by the camera component 200. The cross-correlation subunit 343 may align the first angular velocity and the second angular velocity based on a cross-correlation method, thereby determining a time offset (Time_offset) of the visual pose sequence. The correction subunit 345 may perform time correction on the visual pose sequence based on the time offset.

[0071] Combination Figure 1 , Figure 2 , Figure 8 and Fig. 9 , the visual posture sequence may include a plurality of first time-pose information arranged in time order, the first time-pose information includes the first time information and the first posture angle corresponding to the first time information, wherein the first posture angle includes the first roll angle, the first yaw angle and the first pitch angle. For example, the first time-pose information may include: timestamp_roll_pitch_yaw, wherein 000_10_20_30 may represent the angle postures of roll, pitch and yaw at the starting time (10°, 20°, 30°), respectively. In the right-handed Cartesian coordinate system in three-dimensional space, roll is the roll angle, around the z-axis; pitch is the pitch angle, around the x-axis; yaw is the yaw angle, around the y-axis.

[0072] The motion posture sequence may include a plurality of second time-posture information arranged in time order, and the second time-posture information may include second time information and a second posture angle corresponding to the second time information, wherein the second posture angle includes a second roll angle, a second yaw angle, and a second pitch angle. Similarly, the second time-posture information may also include: timestamp_roll_pitch_yaw.

[0073] Optionally, the fixed connection between the motion component 100 and the extended reality device 400 can be considered as a rigid connection of a rigid body, and the fixed connection between the camera component 200 and the extended reality device 400 can also be considered as a rigid connection of a rigid body. According to the principle of rotational invariance of a rigid body, the time offset can be obtained by aligning the first angular velocity of the rotation of the motion platform 100 and the second angular velocity estimated by the posture of the camera component 200.

[0074] Fig.10 A schematic diagram showing alignment of a first angular velocity and a second angular velocity based on a cross-correlation method according to an exemplary embodiment of the present disclosure is shown.

[0075] like Fig.10 As shown in the figure of the original signal, the first signal can represent the first angular velocity determined based on the motion posture sequence of the motion component 100, and the second signal can represent the second angular velocity determined based on the visual posture sequence converted from the multiple images collected by the camera component 200. Fig.10 The first figure shows the amplitudes of the first angular velocity and the second angular velocity at different times. From the first figure, it can be observed that the first angular velocity and the second angular velocity have different times at the same amplitude, or in other words, the amplitudes of the first angular velocity and the second angular velocity corresponding to the same time are different.

[0076] Fig.10The second figure shows a cross correlation function (Cross Correlation). Here, the cross correlation method is used to align the two angular velocities. The accuracy of the classic cross correlation estimation method depends on the sampling frequency of the two angular velocity sequences. Therefore, in order to improve the accuracy and precision of aligning the two angular velocities using the cross correlation method, the second angular velocity determined based on the visual posture sequence converted from the multiple images collected by the camera assembly 200 can be interpolated.

[0077] Reference again Figure 8 The time correction unit 340 may further include: an interpolation subunit 342, which performs interpolation processing on the visual pose sequence before determining the second angular velocity so that the sampling frequencies of the motion pose sequence and the visual pose sequence are the same.

[0078] In addition, since the second angular velocity determined based on the sampling of the camera component is estimated by the target plate (for example, a virtual target plate) in the extended reality device, and the quality of the target plate image will be affected by the accuracy of the positioning and tracking (slam) system of the extended reality device itself, filtering processing can be performed before interpolation, and then the visual pose sequence is interpolated to the same sampling frequency as the motion pose sequence through the above interpolation processing, and then the time offset of the visual pose sequence is determined based on the cross-correlation method, and the relevant data is aligned for time compensation. For example, refer to Figure 8 The time correction unit 340 may further include: a filtering subunit 341, and the filtering subunit 341 performs filtering processing on the visual pose sequence before interpolating the visual pose sequence.

[0079] It should be noted that, for the sake of clarity, Figure 8 The illustrated time correction unit 340 shows the optional filtering subunit 341 and interpolation subunit 342. It can be distinguished that the optional filtering subunit 341 or interpolation subunit 342 is shown in a dotted line frame, and in addition, the connection between the filtering subunit 341 or interpolation subunit 342 and other subunits is shown by a dotted connecting line when the filtering subunit 341 or interpolation subunit 342 is included.

[0080] Reference again Fig.10 In the aligned signal diagram, the first signal and the second signal are processed using a cross-correlation function to obtain an aligned first signal and an aligned second signal, wherein the aligned first signal can characterize the aligned first angular velocity, and the aligned second signal can characterize the aligned second angular velocity. Fig.10The third figure shows the amplitudes of the aligned first angular velocity and the aligned second angular velocity at different times after the first angular velocity and the second angular velocity are aligned based on the cross-correlation method. It can be observed from the third figure that the time of the aligned first angular velocity and the aligned second angular velocity at the same amplitude is approximately the same, or that the amplitudes of the aligned first angular velocity and the aligned second angular velocity corresponding to the same time are approximately the same.

[0081] Fig.11 A schematic diagram of a process for extrinsic calibration of a camera assembly according to an exemplary embodiment of the present disclosure is shown.

[0082] refer to Figure 1 , Figure 6 and Fig.11 The calibration unit 350 can calibrate the external parameters of the camera assembly 200 based on the acquired visual pose sequence and motion pose sequence; or can calibrate the external parameters of the camera assembly 200 based on the visual pose sequence and motion pose sequence corrected by at least one of the above-mentioned embodiments.

[0083] Specifically, at the beginning of calibration, a target board (eg, Figure 3 The robot arm of the motion assembly 100 can be fixedly connected to the extended reality device 400 and drive the extended reality device 400 to move synchronously. By presetting a predetermined motion trajectory and encoding it to the robot arm, the robot arm can be driven to move the extended reality device 400 in various directions according to the predetermined motion trajectory.

[0084] The camera assembly 200 may collect multiple images of the target plate displayed by the extended reality device 400 during the synchronous movement. The data collection of the camera assembly 200 for multiple images may include multiple data collection modes. As an option, the camera assembly 200 may collect the image of the target plate 410 displayed by the extended reality device 400 only at the moment when the extended reality device 400 passes through the waypoint during the synchronous movement. As another option, the camera assembly 200 may continuously collect the image of the target plate 410 displayed by the extended reality device 400 during the entire synchronous movement. Afterwards, the multiple image data collected by the camera assembly 200 and the three-axis encoder data of the motion assembly 100 can be stored, wherein the three-axis encoder data of the motion assembly 100 can be used as the motion posture sequence of the motion assembly 100.

[0085] In another alternative, although the timestamp of the image captured by the camera component 200 and the timestamp of the trajectory of the motion component 100 (or the extended reality device) are under the same clock source, due to the existence of MTP, the process of capturing the image has a certain delay, so time correction is required to align the time of the motion component 100 and the camera component 200. After the visual pose sequence and the motion pose sequence are obtained, the calibration unit 350 can calibrate the external parameters of the camera component 200 based on the visual pose sequence and the motion pose sequence.

[0086] The motion changes in the real world use the motion component as the reference coordinate system, and the motion changes in the virtual world use the camera component as the reference coordinate system, so it is necessary to calibrate the camera component and the motion component for coordinate system conversion, that is, hand-eye calibration. The external parameters of the calibrated camera component 200 can be understood as hand-eye calibration.

[0087] Optionally, the external parameters of the camera assembly 200 can be obtained by solving the hand-eye calibration equation group AX=XB, where A is the attitude transformation matrix of the camera assembly, B is the attitude transformation matrix of the end effector of the motion assembly (e.g., a robotic arm), and X is the hand-eye matrix to be solved. By solving the eye calibration equation group, the coordinate transformation relationship between the camera assembly 200 and the end effector of the motion assembly 100 can be obtained.

[0088] Fig.12 A schematic diagram of a process for calibrating the external parameters of a camera component based on multiple images and a motion pose sequence of a motion component according to an exemplary embodiment of the present disclosure is shown.

[0089] Combination Figure 1 , Figure 6 and Fig.12 In some embodiments of the present disclosure, the first conversion unit 320 of the processing component 300 can extract the pixel coordinates of the feature points of the target plate 410 based on the multiple images of the target plate 410 displayed by the extended reality device 400 collected during the synchronous movement of the camera component 200 (extraction of 2D (two-dimensional) coordinates of the target plate feature points). The second conversion unit 330 of the processing component 300 can convert the pixel coordinates into a visual pose sequence according to the preset spatial geometric constraint relationship between the feature points, such as the preset target plate feature point ID resolution, where the ID can be understood as the spatial position sequence number of the target plate feature point. An ID-3D mapping table can be established through the preset target plate feature point ID resolution, and the three-dimensional spatial position of the target plate feature point can be obtained by searching the ID-3D (three-dimensional) mapping table, and then a 2D-3D point pair relationship is established to convert the pixel coordinates into a visual pose sequence. The calibration unit 350 of the processing component 300 can calibrate the external parameters of the camera component 200 based on the visual pose sequence and the motion pose sequence.

[0090] Optionally, the calibration unit 350 may include a linear transformation subunit 351 and a nonlinear transformation subunit 352. The linear transformation subunit 351 may obtain the initial values ​​of the extrinsic parameters of the camera component 200 based on the visual pose sequence and the motion pose sequence through a direct linear transformation method. The nonlinear transformation subunit 352 may optimize the initial values ​​of the extrinsic parameters of the camera component 200 based on minimizing the reprojection error through a nonlinear transformation method to obtain the extrinsic parameters of the camera component 200.

[0091] Specifically, the direct linear transformation method can be divided into two steps. The first step is to calculate the projection matrix M, which represents the projection relationship between 2D-3D point pairs. As shown in formula (1), u and v represent the coordinates of 2D points, X, Y and Z represent the coordinates of 3D points, K represents the internal parameters of the camera component 200, and R and T represent the external parameters of the camera component 200 and the target in the world coordinate system (the second coordinate system with the motion component as the reference). According to a series of point pairs, the QM equation group is constructed (as shown in formula (2)), and the matrix M is obtained by singular value decomposition. Since the external parameter R is an orthogonal matrix, the matrix M can be decomposed by QR to obtain the internal parameter K and the external parameters R and T as shown in formula (3).

[0092] (1)

[0093] (2)

[0094] (3)

[0095] (4)

[0096] As shown in formula (4), when the initial intrinsic parameter K, extrinsic parameters R and T are known, the distortion D can be preset to 0, and nonlinear joint optimization is performed based on minimizing the reprojection error (shown in formula 2) to obtain the extrinsic parameters of the camera assembly 200.

[0097] In order to compare the real-world motion gesture changes with the augmented reality device 400 (such as Figure 5 The motion changes in the virtual world (as shown in FIG. 2 ) need to be unified in the same coordinate system. The motion changes in the virtual world can be determined in a first coordinate system (e.g., a camera assembly coordinate system) with the camera assembly 200 as a reference, and the motion changes in the real world can be determined in a second coordinate system (e.g., a world coordinate system) with the motion assembly 100 as a reference. Therefore, the motion assembly 100 and the camera assembly 200 need to be calibrated for coordinate system conversion.

[0098] Since different extended reality devices have different eye reliefs, the positional relationship between the camera assembly 200 and the extended reality device 400 needs to be adjusted before measurement, which causes the positional relationship between the motion assembly 100 and the camera assembly 200 to change. In order to solve this problem, a feasible solution is to calibrate the external parameters of the camera assembly in the camera assembly 200 based on the virtual standard plate displayed by the extended reality device photographed by the camera assembly 200.

[0099] In the process of calibrating the external parameters of the camera assembly 200 by solving the AX=XB hand-eye calibration equation group, the external parameters can be solved according to traditional hand-eye calibration algorithms such as the T-SAI method. However, it should be noted that the present disclosure does not limit the algorithm for solving the external parameters. In the solution process, the posture transformation matrix of the camera assembly represented by A can be understood as the visual posture sequence obtained by any of the above-mentioned embodiments, and the posture transformation matrix of the end effector of the motion assembly (for example, a robotic arm) represented by B can be understood as the motion posture sequence obtained by any of the above-mentioned embodiments. The external parameters of the camera assembly 200 can be solved by A and B, or in other words, by solving the eye calibration equation group, the coordinate transformation relationship between the camera assembly 200 and the end effector of the motion assembly 100 can be obtained.

[0100] Fig.13 A schematic diagram of motion information of a motion component and a calibrated camera component in the same coordinate system according to an exemplary embodiment of the present disclosure is shown.

[0101] like Fig.13 As shown, the motion information of the motion component in the same coordinate system is a compound three-dimensional motion (3DOF Compound Motion), which can be represented by yaw motion, pitch motion, and roll motion. In the three-dimensional right-handed Cartesian coordinate system, roll is the roll angle around the z-axis; pitch is the pitch angle around the x-axis; yaw is the yaw angle around the y-axis. The motion information of the calibrated camera component in the same coordinate system is also a compound three-dimensional motion, which can be represented by yaw motion information (yaw photon), pitch motion information (pitch photon), and roll motion information (roll photon). Similarly, in the three-dimensional right-handed Cartesian coordinate system, roll is the roll angle around the z-axis; pitch is the pitch angle around the x-axis; yaw is the yaw angle around the y-axis.

[0102] pass Fig.13It can be observed that after delay correction is performed on the motion component and the calibrated camera component, the degree of motion information of the delay-corrected motion component in the same coordinate system and the degree of motion information of the calibrated camera component in the same coordinate system in all directions (roll, pitch and yaw) at different times are approximately the same.

[0103] Therefore, according to the calibration system provided by at least one embodiment of the present disclosure, the motion component that moves synchronously with the extended reality device can simulate the continuous compound three-dimensional motion of the human body, and in addition, the camera component can be used to obtain multiple images of the target plate displayed by the extended reality device. Therefore, the external parameters of the camera component can be obtained through the multiple images of the target plate displayed by the extended reality device acquired by the camera component and the motion pose sequence of the motion component, so as to unify the motion information of the camera component and the motion component in the same coordinate system, and provide a coordinate system reference for measuring the delay of the extended reality device in the continuous three-dimensional compound motion.

[0104] Fig.14 A flow chart of a calibration method 2000 provided in an embodiment of the present disclosure, wherein the calibration method 2000 may include the following steps: S1: Obtain a motion posture sequence of a motion component, wherein the motion component is fixedly connected to an extended reality device and drives the extended reality device to move synchronously.

[0105] S2: Using a camera assembly fixedly connected to the extended reality device, during synchronous movement, multiple images of a target plate displayed by the extended reality device are collected.

[0106] S3: Calibrate the extrinsic parameters of the camera component based on multiple images and the motion pose sequence of the motion component.

[0107] The various steps of the calibration method 2000 will be described in detail below with reference to the accompanying drawings. Step S1

[0108] refer to Figure 1 , Figure 5 and Fig.14 In some embodiments of the present disclosure, the motion component 100 is fixedly connected to the extended reality device 400, for example, rigidly fixedly connected. In addition, the motion component 100 can drive the extended reality device 400 to move synchronously. The motion component 100 may include any suitable structural components and may perform various translational and rotational movements in the world coordinate system. Taking the extended reality device 400 as an HMD as an example, the motion component 100 fixedly connected to the HMD can simulate the rotational movement of the human head in the sagittal plane, coronal plane and horizontal plane, etc.

[0109] Optionally, the camera assembly 200 is used to obtain the target plate 410 (such as Figure 3 The calibration method 2000 further includes: before and during the synchronous movement, performing time synchronization on the motion component 100 and the camera component 200.

[0110] For example, a clock source is used to synchronize the motion component 100 and the camera component 200. During the time synchronization process, the motion component 100 and the camera component 200 can be synchronized uniformly through the PTP protocol. A clock source with high precision and high accuracy is used as the master clock to send a timestamp to the motion component 100 and the camera component 200 as slave clocks. The slave clock compares the timestamp with its own local time and adjusts the time according to the difference, thereby ensuring that the subsequent motion pose sequence obtained based on the motion component 100 is synchronized with the visual pose sequence obtained based on the camera component 200. Step S2

[0111] refer to Figure 1-Figure 5 , Fig.14 In some embodiments of the present disclosure, a camera assembly 200 fixedly connected to an extended reality device 400 may be used to collect multiple images of a target plate 410 displayed by the extended reality device 400 during synchronous movement, for example Figure 4 One image 420 of the plurality of images is shown.

[0112] The camera assembly 200 can be used to simulate the scene of human eyes receiving the image displayed by the extended reality device 400. Optionally, the camera assembly 200 may include at least one of a camera and a video camera, and the camera assembly may include an IR camera assembly (Infrared Camera), an RGB camera assembly (RedGreenBlue Camera), or a black and white camera assembly. In addition, the camera assembly 200 may also include other structures such as photoelectric sensors, and the present disclosure does not limit the specific internal structure of the camera assembly 200.

[0113] Optionally, the target plate 410 may be a virtual target plate located in the extended reality device 400. The virtual target plate may include a visual marker calibration plate in the visual reference library. For example, a two-dimensional code calibration plate such as ApriTag and ArUco. Using a virtual target plate for calibration can simplify the structure of the calibration system and reduce the difficulty of implementing the calibration. Step S3

[0114] In some embodiments of the present disclosure, step S3 calibrates the external parameters of the camera component based on multiple images and a motion pose sequence of the motion component, and may, for example, include: extracting the pixel coordinates of the feature points of the target plate based on the multiple images; converting the pixel coordinates into a visual pose sequence according to a preset spatial geometric constraint relationship between the feature points; and calibrating the external parameters of the camera component based on the visual pose sequence and the motion pose sequence.

[0115] Specifically, refer to Figure 1 , Figure 5 and Fig.14 , the processing component 300 can be used to obtain a visual pose sequence of an extended reality device based on multiple images. The processing component 300 can be various general and / or special processing components with processing and computing capabilities. Some examples of the processing component 300 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc.

[0116] The processing component 300 obtains a visual pose sequence of the extended reality device 400 based on the multiple images, and determines the delay of the extended reality device 400 based on the visual pose sequence and the motion pose sequence of the motion component 100. Optionally, the processing component 300 may include various suitable communication buses and input / output interfaces for realizing connection with the motion component 100 and the camera component 200.

[0117] In addition, based on the multiple images, the target plate 410 (such as Figure 3 Before obtaining the pixel coordinates of the feature points of the extended reality device 400 (as shown in the figure), the multiple images may be processed, and the processing may include at least one of downsampling processing and Gaussian blur processing. Since the appearance of moiré when shooting the screen of the extended reality device 400 will cause inaccurate extraction of 2D feature points, or even failure to extract them, it is necessary to process the multiple images obtained, and the influence of moiré on the above images may be reduced by at least one of downsampling processing and Gaussian blur processing.

[0118] Optionally, before collecting multiple images of the target plate 410 displayed by the extended reality device 400, the positional relationship between the camera assembly 200 and the extended reality device 400 can be adjusted so that multiple feature points specified in the target plate 410 are shown in the multiple images. This can improve the generation quality of the multiple images and ensure high stability and high accuracy of the calibration results.

[0119] In addition, in order to compare the motion posture changes in the real world with the motion posture changes in the virtual world of the extended reality device 400, it is necessary to unify the two motion posture changes in the same coordinate system. The motion changes in the virtual world can be determined in a first coordinate system (e.g., a camera assembly coordinate system) with the camera assembly 200 as a reference, and the motion changes in the real world can be determined in a second coordinate system (e.g., a world coordinate system) with the motion assembly 100 as a reference, so it is necessary to calibrate the motion assembly 100 and the camera assembly 200 for coordinate system conversion.

[0120] Since different extended reality devices have different eye reliefs, the positional relationship between the camera assembly 200 and the extended reality device 400 needs to be adjusted before measurement, which causes the positional relationship between the motion assembly 100 and the camera assembly 200 to change. In order to solve this problem, a feasible solution is to calibrate the external parameters of the camera assembly in the camera assembly 200 based on the virtual standard plate displayed by the extended reality device photographed by the camera assembly 200.

[0121] Optionally, the camera assembly 200 may collect multiple images of the target plate displayed by the extended reality device 400 during the synchronous movement. The data collection of the camera assembly 200 for multiple images may include multiple data collection modes. As an option, the camera assembly 200 may collect the image of the target plate 410 displayed by the extended reality device 400 only at the moment when the extended reality device 400 passes through a waypoint during the synchronous movement. As another option, the camera assembly 200 may continuously collect the image of the target plate 410 displayed by the extended reality device 400 during the entire synchronous movement. Afterwards, the multiple image data collected by the camera assembly 200 and the three-axis encoder data of the motion assembly 100 can be stored, wherein the three-axis encoder data of the motion assembly 100 can be used as the motion posture sequence of the motion assembly 100.

[0122] In another alternative, although the timestamp of the image of the standard plate captured by the camera component 200 and the timestamp of the trajectory of the motion component 100 (or the extended reality device) are under the same clock source, due to the existence of MTP, the process of capturing the image has a certain delay, so it is necessary to perform time correction on the visual pose sequence to align the time of the motion component 100 and the camera component 200. Optionally, the visual pose sequence is time corrected based on the time offset of the visual pose sequence.

[0123] like Figure 2As shown, the visual posture sequence may include a plurality of first time-pose information arranged in time order, the first time-pose information including the first time information and the first posture angle corresponding to the first time information, wherein the first posture angle includes the first roll angle, the first yaw angle and the first pitch angle. For example, the first time-pose information may include: timestamp_roll_pitch_yaw, wherein 000_10_20_30 may represent that the angle postures of roll, pitch and yaw at the starting time are (10°, 20°, 30°) respectively, and in the right-handed Cartesian coordinate system of three-dimensional space, roll is the roll angle, around the z-axis; pitch is the pitch angle, around the x-axis; yaw is the yaw angle, around the y-axis.

[0124] The motion posture sequence may include a plurality of second time-posture information arranged in time order, and the second time-posture information may include second time information and a second posture angle corresponding to the second time information, wherein the second posture angle includes a second roll angle, a second yaw angle, and a second pitch angle. Similarly, the second time-posture information may also include: timestamp_roll_pitch_yaw.

[0125] refer to Figure 1 , Figure 5 and Fig.14 In some embodiments of the present disclosure, the fixed connection between the motion component 100 and the extended reality device 400 can be considered as a rigid connection of a rigid body, and the fixed connection between the camera component 200 and the extended reality device 400 can also be considered as a rigid connection of a rigid body. According to the principle of rigid body rotation invariance, the time offset can be obtained by aligning the first angular velocity of the motion platform 100 and the second angular velocity estimated by the posture of the camera component 200.

[0126] For example, a first angular velocity is determined based on a motion posture sequence of the motion component 100, and a second angular velocity is determined based on a visual posture sequence converted from a plurality of images acquired by the camera component 200; based on a cross-correlation method, the first angular velocity and the second angular velocity are aligned to determine a time offset of the visual posture sequence; and based on the time offset, a time correction is performed on the visual posture sequence.

[0127] In addition, performing time correction on the visual pose sequence may also include performing interpolation processing on the visual pose sequence before determining the second angular velocity, so that the sampling frequencies of the motion pose sequence and the visual pose sequence are the same.

[0128] Optionally, since the second angular velocity determined based on the sampling of the camera component is estimated by a target (e.g., a virtual target) in the extended reality device, and the quality of the target image will be affected by the accuracy of the extended reality device's own positioning and tracking (slam) system, filtering processing may be performed before interpolation, and then the visual pose sequence is interpolated to the same sampling frequency as the motion pose sequence through the above-mentioned interpolation processing, and then the time offset of the visual pose sequence is determined based on the cross-correlation method, and the relevant data is time-compensated and aligned.

[0129] In some embodiments of the present disclosure, the external parameters of the camera assembly 200 can be calibrated based on the acquired visual pose sequence and motion pose sequence; or the external parameters of the camera assembly 200 can be calibrated based on the visual pose sequence and motion pose sequence corrected by at least one of the above embodiments.

[0130] Specifically, at the beginning of calibration, the target board (for example, Figure 3 The robot arm of the motion assembly 100 can be fixedly connected to the extended reality device 400 and drive the extended reality device 400 to move synchronously. By presetting a predetermined motion trajectory and encoding it to the robot arm, the robot arm can be driven to move the extended reality device 400 in various directions according to the predetermined motion trajectory.

[0131] The camera assembly 200 may collect multiple images of the target plate displayed by the extended reality device 400 during the synchronous movement. The data collection of the camera assembly 200 for multiple images may include multiple data collection modes. As an option, the camera assembly 200 may collect the image of the target plate 410 displayed by the extended reality device 400 only at the moment when the extended reality device 400 passes through the waypoint during the synchronous movement. As another option, the camera assembly 200 may continuously collect the image of the target plate 410 displayed by the extended reality device 400 during the entire synchronous movement. Afterwards, the multiple image data collected by the camera assembly 200 and the three-axis encoder data of the motion assembly 100 can be stored, wherein the three-axis encoder data of the motion assembly 100 can be used as the motion posture sequence of the motion assembly 100.

[0132] In another alternative, although the timestamp of the image captured by the camera component 200 and the timestamp of the trajectory of the motion component 100 (or the extended reality device) are under the same clock source, due to the existence of MTP, the process of capturing the image has a certain delay, so time correction is required to align the time of the motion component 100 and the camera component 200. After the visual pose sequence and the motion pose sequence are obtained, the external parameters of the camera component 200 can be calibrated based on the visual pose sequence and the motion pose sequence.

[0133] The motion changes in the real world use the motion component as the reference coordinate system, and the motion changes in the virtual world use the camera component as the reference coordinate system, so it is necessary to calibrate the camera component and the motion component for coordinate system conversion, that is, hand-eye calibration. The external parameters of the calibrated camera component 200 can be understood as hand-eye calibration.

[0134] Optionally, the external parameters of the camera assembly 200 can be obtained by solving the hand-eye calibration equation group AX=XB, where A is the attitude transformation matrix of the camera assembly, B is the attitude transformation matrix of the end effector of the motion assembly (e.g., a robotic arm), and X is the hand-eye matrix to be solved. By solving the eye calibration equation group, the coordinate transformation relationship between the camera assembly 200 and the end effector of the motion assembly 100 can be obtained.

[0135] In other words, in the process of calibrating the external parameters of the camera assembly 200 by solving the AX=XB hand-eye calibration equation group, the external parameters can be solved according to traditional hand-eye calibration algorithms such as the T-SAI method. However, it should be noted that the present disclosure does not limit the algorithm for solving the external parameters. In the solution process, the posture transformation matrix of the camera assembly represented by A can be understood as the visual posture sequence obtained by any of the above-mentioned embodiments, and the posture transformation matrix of the end effector of the motion assembly (for example, a robotic arm) represented by B can be understood as the motion posture sequence obtained by any of the above-mentioned embodiments. The external parameters of the camera assembly 200 can be solved by A and B, or in other words, by solving the eye calibration equation group, the coordinate transformation relationship between the camera assembly 200 and the end effector of the motion assembly 100 can be obtained.

[0136] Optionally, the initial values ​​of the extrinsic parameters of the camera component 200 can be obtained based on the visual pose sequence and the motion pose sequence through a direct linear transformation method; then, the initial values ​​of the extrinsic parameters of the camera component 200 can be optimized based on minimizing the reprojection error through a nonlinear transformation method to obtain the extrinsic parameters of the camera component 200.

[0137] Therefore, according to the calibration method provided by at least one embodiment of the present disclosure, the motion component that moves synchronously with the extended reality device can simulate the continuous compound three-dimensional motion of the human body, and the camera component can be used to obtain multiple images of the target plate displayed by the extended reality device. Through the multiple images of the target plate displayed by the extended reality device acquired by the camera component and the motion pose sequence of the motion component, the external parameters of the camera component can be obtained, so that the motion information of the camera component and the motion component are unified in the same coordinate system, providing a coordinate system reference for measuring the delay of the extended reality device in the continuous three-dimensional compound motion.

[0138] Fig.15A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0139] like Fig.15 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0140] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0141] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the calibration method. For example, in some embodiments, the calibration method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the calibration method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the calibration method in any other appropriate manner (e.g., by means of firmware).

[0142] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0144] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0145] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0146] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0147] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client-server relationship is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private server (VPS) services.

[0148] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0149] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A calibration system, characterized in that: include: A motion component, fixedly connected to the extended reality device, and driving the extended reality device to move synchronously; A camera assembly fixedly connected to the extended reality device and collecting multiple images of the target displayed by the extended reality device during the synchronous movement; as well as A processing component calibrates the external parameters of the camera component based on the multiple images and the motion posture sequence of the motion component.

2. The system according to claim 1, wherein: The processing components include: A first conversion unit extracts pixel coordinates of feature points of the target based on the multiple images; A second conversion unit converts the pixel coordinates into a visual pose sequence according to a preset spatial geometric constraint relationship between the feature points; and A calibration unit is used to calibrate the external parameters of the camera assembly based on the visual pose sequence and the motion pose sequence.

3. The system according to claim 2, wherein: The calibration unit comprises: A linear transformation subunit, which obtains an initial value of an extrinsic parameter of the camera assembly based on the visual pose sequence and the motion pose sequence by a direct linear transformation method; and The nonlinear transformation subunit optimizes the initial value of the extrinsic parameter of the camera component based on minimizing the reprojection error through a nonlinear transformation method to obtain the extrinsic parameter of the camera component.

4. The system according to claim 2, wherein: The visual posture sequence includes a plurality of first time-pose information arranged in time order, the first time-pose information includes first time information and a first posture angle corresponding to the first time information, wherein the first posture angle includes a first roll angle, a first yaw angle and a first pitch angle; and The motion posture sequence includes a plurality of second time-posture information arranged in chronological order, the second time-posture information including second time information and a second posture angle corresponding to the second time information, wherein the second posture angle includes a second roll angle, a second yaw angle and a second pitch angle.

5. The system according to claim 1, wherein: The system further comprises: A time component comprises a clock source, wherein the clock source is used to perform time synchronization between the motion component and the camera component before and during the synchronous movement.

6. The system according to claim 1, wherein: During the synchronous movement, the camera component collects an image of the target plate displayed by the extended reality device when the extended reality device passes a waypoint.

7. The system according to claim 2, wherein: During the synchronous movement, the camera assembly continuously captures the image of the target displayed by the extended reality device.

8. The system according to claim 7, wherein: The processing component also includes: A time correction unit performs time correction on the visual pose sequence.

9. The system according to claim 8, wherein: The time correction unit comprises: an angular velocity calculation subunit, which determines a first angular velocity based on the motion posture sequence, and determines a second angular velocity based on the visual posture sequence; a cross-correlation subunit, aligning the first angular velocity and the second angular velocity based on a cross-correlation method to determine a time offset of the visual pose sequence; and A correction subunit performs temporal correction on the visual pose sequence based on the temporal offset.

10. The system according to claim 9, wherein: The time correction unit also includes: The interpolation subunit performs interpolation processing on the visual pose sequence before determining the second angular velocity, so that the sampling frequency of the motion pose sequence and the visual pose sequence is the same.

11. The system according to claim 9, wherein: The time correction unit also includes: The filtering subunit performs filtering processing on the visual pose sequence before performing interpolation processing on the visual pose sequence.

12. The system according to claim 2, wherein: The processing component also includes: The image processing unit processes the multiple images before extracting the pixel coordinates of the feature points of the target based on the multiple images, wherein the processing includes at least one of downsampling processing and Gaussian blur processing.

13. The system of claim 1, wherein: The target plate is a virtual target plate, wherein the virtual target plate includes a visual marker calibration plate in a visual reference library.

14. The system of claim 1, wherein: The processing component also includes: An adjustment unit adjusts the positional relationship between the camera assembly and the extended reality device so that a plurality of feature points specified in the target plate are shown in the plurality of images.

15. A calibration method, characterized in that: include: Acquire a motion posture sequence of a motion component, wherein the motion component is fixedly connected to an extended reality device and drives the extended reality device to move synchronously; Using a camera assembly fixedly connected to the extended reality device, during the synchronous movement, a plurality of images of a target displayed by the extended reality device are collected; as well as Based on the multiple images and the motion pose sequence of the motion component, the external parameters of the camera component are calibrated.

16. The calibration method according to claim 15, wherein: Based on the multiple images and the motion pose sequence of the motion component, calibrating the external parameters of the camera component includes: Extracting pixel coordinates of feature points of the target based on the multiple images; According to the preset spatial geometric constraint relationship between the feature points, the pixel coordinates are converted into a visual pose sequence; and Based on the visual pose sequence and the motion pose sequence, extrinsic parameters of the camera assembly are calibrated.

17. The calibration method according to claim 16, wherein: Based on the visual pose sequence and the motion pose sequence, calibrating the external parameters of the camera assembly includes: Obtaining initial values ​​of extrinsic parameters of the camera assembly based on the visual pose sequence and the motion pose sequence by a direct linear transformation method; and The initial value of the extrinsic parameter of the camera component is optimized based on minimizing the reprojection error through a nonlinear transformation method to obtain the extrinsic parameter of the camera component.

18. The calibration method according to claim 16, wherein: The visual posture sequence includes a plurality of first time-pose information arranged in time order, the first time-pose information includes first time information and a first posture angle corresponding to the first time information, wherein the first posture angle includes a first roll angle, a first yaw angle and a first pitch angle; and The motion posture sequence includes a plurality of second time-posture information arranged in chronological order, the second time-posture information including second time information and a second posture angle corresponding to the second time information, wherein the second posture angle includes a second roll angle, a second yaw angle and a second pitch angle.

19. The calibration method according to claim 15, wherein: The calibration method further comprises: The motion component and the camera component are time synchronized by using a clock source before and during the synchronous movement.

20. The calibration method according to claim 15, wherein: During the synchronous movement, collecting a plurality of images of the target displayed by the extended reality device includes: During the synchronous movement, the camera component collects an image of the target plate displayed by the extended reality device when the extended reality device passes a waypoint.

21. The calibration method according to claim 16, wherein: During the synchronous movement, collecting a plurality of images of the target displayed by the extended reality device includes: During the synchronous movement, the camera assembly continuously captures the image of the target displayed by the extended reality device.

22. The calibration method according to claim 21, wherein: The calibration method also includes: performing time correction on the visual posture sequence.

23. The calibration method according to claim 22, wherein: Performing temporal correction on the visual pose sequence includes: Determining a first angular velocity based on the motion pose sequence, and determining a second angular velocity based on the visual pose sequence; Aligning the first angular velocity and the second angular velocity based on a cross-correlation method to determine a time offset of the visual pose sequence; and Based on the time offset, the visual pose sequence is time-corrected.

24. The calibration method according to claim 23, wherein: Temporally correcting the visual pose sequence further includes: Before determining the second angular velocity, interpolation processing is performed on the visual pose sequence so that the sampling frequencies of the motion pose sequence and the visual pose sequence are the same.

25. The calibration method according to claim 23, wherein: Temporally correcting the visual pose sequence further includes: Before interpolating the visual pose sequence, filtering is performed on the visual pose sequence.

26. The calibration method according to claim 16, wherein: The calibration method further includes: before extracting the pixel coordinates of the feature points of the target plate based on the multiple images, processing the multiple images, wherein the processing includes at least one of downsampling processing and Gaussian blur processing.

27. The calibration method according to claim 15, wherein: The target plate is a virtual target plate, wherein the virtual target plate includes a visual marker calibration plate in a visual reference library.

28. The calibration method according to claim 15, wherein: The calibration method further comprises: Before acquiring a plurality of images of a target displayed via the extended reality device, a positional relationship between the camera assembly and the extended reality device is adjusted so that a plurality of feature points specified in the target are shown in the plurality of images.

29. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the calibration method according to any one of claims 15-28.

30. A computer-readable storage medium storing a computer program, characterized in that: The computer instructions are used to enable the computer to execute the calibration method according to any one of claims 15-28.

31. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the calibration method according to any one of claims 15-28 is implemented.

Images