A data calibration method, device, equipment and medium of an extended reality device

Abstract

The present application relates to the technical field of extended reality devices, and discloses a data calibration method, device, equipment and medium of an extended reality device. The method obtains first target coordinates of any two cameras through a first matching mode and second target coordinates of any two cameras through a second matching mode when calibration trigger data and memory occupation data meet preset conditions, and calibrates the extrinsic parameter data of any two cameras of the target XR device. The method can not only meet the calibration point requirement of the extrinsic parameter calibration of the XR device, but also corrects the camera and IMU calibration data without user awareness, ensures the positioning and tracking effect of the XR device, and is relatively simple. Even after the XR device is shipped, the calibration can be quickly realized to ensure the normal operation of the XR device when the extrinsic parameter changes due to the deformation of the shell caused by long-term use.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of extended reality devices (XR devices), and in particular to a data calibration method and device for an extended reality device, an extended reality device, and a medium. BACKGROUND

[0002] Visual and inertial SLAM is the mainstream scheme for realizing positioning of an XR device at present. By calibrating the internal and external parameters of a camera and an inertial sensor (IMU), the corresponding information can be fused with each other, so that the XR device can more accurately perceive the surrounding objects, thereby ensuring the accuracy of positioning of the XR device and the safety of use.

[0003] The related art calibrates the internal and external parameters of an XR device by using a calibration board to achieve parameter calibration. In order to accurately correct the internal and external parameters of the XR device, the calibration board needs to be calibrated with high precision in factory production, which results in a relatively cumbersome calibration method. In addition, after long-term use of the XR device, the shell is prone to deformation, and accidental falling during use can also cause the external parameters between sensors to change, thereby affecting the normal work of the XR device after leaving the factory. SUMMARY

[0004] Therefore, the present application provides a data calibration method and device for an extended reality device, an extended reality device, and a medium, to solve the problems of a relatively cumbersome calibration method and changes in external parameters due to shell deformation or falling after long-term use, thereby affecting the normal work of the XR device after leaving the factory.

[0005] According to a first aspect, the present application provides a data calibration method for an extended reality device, the method comprising:

[0006] obtaining camera data, internal parameter data, and external parameter data of any two cameras of a target XR device, and IMU raw data of an inertial sensor, calibration trigger data and memory occupation data of the target XR device;

[0007] when the calibration trigger data and the memory occupation data satisfy a preset condition, calculating the similarity between the actual running track and the preset running track of the target XR device based on the camera data and the IMU raw data;

[0008] when the similarity is less than or equal to a first preset threshold, extracting a target feature point from the camera data by a first matching method, and matching a first target coordinate corresponding to the target feature point;

[0009] based on the internal parameter data and the external parameter data of the any two cameras, matching the camera data of the any two cameras by a second matching method to obtain a second target coordinate;

[0010] Based on the first target coordinate and the second target coordinate, the extrinsic parameter data of any two cameras is optimized in an iterative manner until the optimization result is less than or equal to the second preset threshold, and the calibration parameters of the target XR device are output.

[0011] By executing the above-mentioned implementation, when the calibration trigger data and the memory occupation data meet the preset condition, the first target coordinate of any two cameras is obtained by the first matching manner, and the second target coordinate of any two cameras is obtained by the second matching manner. The extrinsic parameter data of any two cameras of the target XR device is calibrated, which not only meets the demand of the XR device extrinsic parameter calibration point, but also corrects the camera and IMU calibration data without the user's awareness, ensures the positioning and tracking effect of the XR device, and the calibration method is relatively simple. Even after the XR device is out of the factory, the extrinsic parameter changes due to the deformation of the shell caused by long-term use, and the calibration can be quickly realized to ensure the normal work of the XR device.

[0012] In an optional implementation, the calibration trigger data and the memory occupation data meet the preset condition, comprising:

[0013] Obtaining the current falling distance and the current falling acceleration of the target XR device;

[0014] When the current falling distance of the target XR device is greater than or equal to the third preset threshold, the current falling acceleration of the target XR device is obtained;

[0015] When the current falling acceleration of the target XR device is equal to the free-fall acceleration, the current state and the memory occupation data of the target XR device are obtained;

[0016] When the current state of the target XR device is in the home interface state, and the memory occupation data is less than or equal to the fourth preset threshold, the current occlusion condition of the target XR device is obtained;

[0017] When the target XR device is not currently occluded, the target XR device meets the preset condition.

[0018] By executing the above-mentioned implementation, whether the current falling distance, the current falling acceleration, the current state, the memory occupation data, and the current occlusion condition meet the preset condition is judged in sequence, which is conducive to subsequent accurate calibration of the extrinsic parameter of the target XR device.

[0019] In an optional implementation, the calibration trigger data and the memory occupation data meet the preset condition, further comprising:

[0020] Obtaining the current use duration of the target XR device;

[0021] When the current use duration of the target XR device is greater than or equal to the preset period, the current state and the memory occupation data of the target XR device are obtained;

[0022] When the current state of the target XR device is in the home interface state, and the memory occupation data is less than or equal to the fourth preset threshold, the current occlusion of the target XR device is obtained;

[0023] When the target XR device is not currently occluded, the target XR device meets the preset condition.

[0024] By executing the above-mentioned implementation, whether the current use duration, the current state and the memory occupation data of the target XR device meet the preset condition is judged in sequence, which is beneficial to subsequent accurate calibration of the external parameters of the target XR device.

[0025] In an optional implementation, the similarity between the actual running track of the target XR device and the preset running track is calculated based on the camera data and the IMU raw data, including:

[0026] The actual running track of the target XR device is calculated based on the camera data and the IMU raw data through a visual SLAM algorithm;

[0027] The preset running track of the target XR device is obtained;

[0028] The actual running track and the preset running track are compared to obtain the similarity therebetween.

[0029] By executing the above-mentioned implementation, the similarity between the actual running track of the target XR device and the preset running track is calculated, so as to accurately calibrate the external parameters of the target XR device subsequently.

[0030] In an optional implementation, the target feature point is extracted from the camera data through a first matching manner, and the first target coordinate corresponding to the target feature point is matched, including:

[0031] The target feature point is extracted from the camera data through a SIFT feature manner;

[0032] The first target coordinate corresponding to the target feature point is matched through a SIFT matching algorithm.

[0033] By executing the above-mentioned implementation, the target feature point corresponding to the camera data is matched through the first matching manner, and the first target coordinate corresponding to the target feature point is obtained, which is beneficial to subsequent accurate calibration of the external parameters of the target XR device.

[0034] In an optional implementation, the camera data of any two cameras is matched through a second matching manner to obtain a second target coordinate, including:

[0035] The camera data of any two cameras is matched through a second matching manner based on the intrinsic parameter data and the extrinsic parameter data of the any two cameras to obtain second target coordinates.

[0036] By executing the above-mentioned embodiments, the target feature points corresponding to the camera data are matched through the second matching manner, and the second target coordinates corresponding to the target feature points are obtained, which is beneficial to subsequent accurate calibration of the extrinsic parameters of the target XR device.

[0037] In an optional embodiment, the data calibration method of the extended reality device in the embodiment further includes:

[0038] When the calibration trigger data and the memory occupation data do not satisfy the preset condition, the target XR device is not triggered to perform the data calibration action.

[0039] According to a second aspect, the embodiment further provides a data calibration apparatus of an extended reality device, and the apparatus includes:

[0040] The device data acquisition module is configured to acquire camera data, intrinsic parameter data, extrinsic parameter data of any two cameras of a target XR device, and IMU original data of an inertial sensor, calibration trigger data and memory occupation data of the target XR device,

[0041] The device data calculation module is configured to, when the calibration trigger data and the memory occupation data satisfy a preset condition, calculate a similarity between an actual running track and a preset running track of the target XR device based on the camera data and the IMU original data.

[0042] The first coordinate matching module is configured to, when the similarity is less than or equal to a first preset threshold, extract a target feature point from the camera data through a first matching manner, and match first target coordinates corresponding to the target feature point.

[0043] The second coordinate matching module is configured to, through a second matching manner, match camera data of any two cameras based on intrinsic parameter data and extrinsic parameter data of the any two cameras to obtain second target coordinates.

[0044] The calibration parameter output module is configured to, based on the first target coordinates, optimize the extrinsic parameter data of the any two cameras in an iterative manner until an optimization result is less than or equal to a second preset threshold, and output calibration parameters of the target XR device.

[0045] According to a third aspect, the embodiment further provides a computer device, which includes:

[0046] The memory and the processor are in communication connection with each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the data calibration method of the extended reality device in the first aspect or any embodiment of the first aspect.

[0047] According to a fourth aspect, the embodiments further provide a computer readable storage medium having stored thereon computer instructions for causing a computer to execute the data calibration method of the extended reality device according to the first aspect or any implementation of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the specific embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0049] Figure 1 is a flowchart of the data calibration method of the extended reality device according to an embodiment of the present application;

[0050] Figure 2 is a flowchart of another data calibration method of the extended reality device according to an embodiment of the present application;

[0051] Figure 3 is a flowchart of still another data calibration method of the extended reality device according to an embodiment of the present application;

[0052] Figure 4 is a structural block diagram of the data calibration device of the extended reality device according to an embodiment of the present application;

[0053] Figure 5 is a hardware structure diagram of the computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0055] According to an embodiment of the present application, a data calibration method of an extended reality device is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0056] A data calibration method of an extended reality device is provided in the embodiment, which can be used for mobile terminals such as mobile phones, tablets and the like (the execution subject is described in combination with the actual situation), Figure 1 A flowchart of the data calibration method of the extended reality device according to the embodiment of the present application is shown in FIG. 1, which includes the following steps: Figure 1

[0057] In step S101, the camera data, the intrinsic data, the extrinsic data of any two cameras of a target XR device, and the IMU original data of an inertial sensor, the calibration trigger data and the memory occupation data of the target XR device are obtained.

[0058] Specifically, a plurality of cameras and one inertial sensor are provided on the target XR device, and the camera data of any two cameras can be obtained from the plurality of cameras. In computer vision, the intrinsic data of the camera is a parameter related to the characteristics of the camera itself, such as the focal length and pixel size of the camera, and the extrinsic parameter of the camera is a parameter in the world coordinate system, such as the position and rotation direction of the camera.

[0059] Exemplarily, for the extrinsic data of the camera: the intrinsic matrix is given below, it should be noted that the real camera also has radial and tangential distortion, and these distortions belong to the intrinsic parameters of the camera. Camera intrinsic matrix:

[0060]

[0061] Where fx, fy are focal lengths, generally equal, x0, y0 are principal point coordinates (relative to the imaging plane), s is the coordinate axis tilt parameter, which is 0 in the ideal case.

[0062] Exemplarily, for the extrinsic data of the camera: the rotation and translation of the camera belong to the extrinsic parameter, which is used to describe the motion of the camera in the static scene, or the rigid motion of the moving object when the camera is fixed. Therefore, in image stitching or three-dimensional reconstruction, the extrinsic data is needed to calculate the relative motion between several images, so as to transfer them to the same coordinate system. The extrinsic matrix of the camera includes the rotation matrix and the translation matrix, which together describe how to convert the point from the world coordinate system to the camera coordinate system. The rotation matrix describes the direction of the coordinate axis of the world coordinate system relative to the camera coordinate axis, and the translation matrix describes the position of the space origin in the camera coordinate system.

[0063] Specifically, the IMU original data refers to the bias noise data. The calibration trigger data of the target XR device includes the current falling distance and the current falling acceleration of the target XR device, and the calibration trigger data of the target XR device also includes the current use time length of the target XR device. ​

[0064] Step S102, when the calibration trigger data and the memory occupation data meet the preset condition, the similarity between the actual running track of the target XR device and the preset running track is calculated based on the camera data and the IMU raw data.

[0065] Specifically, the preset condition is determined based on the calibration trigger data and the memory occupation data, and the actual running track of the target XR device is calculated, which is beneficial to calibrate the actual situation of the target XR device.

[0066] Specifically, when the actual situation of the target XR device is similar to the preset situation, the actual situation of the target XR device is calibrated, which is beneficial to improve the accuracy of calibration.

[0067] Step S103, when the similarity is less than or equal to the first preset threshold, the target feature point is extracted from the camera data by the first matching mode, and the first target coordinate corresponding to the target feature point is matched.

[0068] Specifically, the first matching mode is a SIFT matching mode, which is a very robust matching algorithm in the image matching process. The SIFT feature matching algorithm is generally implemented in two stages: the first stage is the extraction of SIFT feature vectors, that is, the feature vectors independent of scale, rotation and brightness changes are extracted from multiple images to be matched; the second stage is the matching of SIFT feature vectors. In this embodiment, the target feature point is extracted from the camera data of the first camera and the camera data of the second camera, and matched to obtain the corresponding first target coordinate.

[0069] Step S104, based on the intrinsic data and the extrinsic data of any two cameras, the camera data of any two cameras is matched by the second matching mode to obtain the second target coordinate.

[0070] Specifically, the second matching mode is an extreme matching mode, which is calculated based on the intrinsic data and the extrinsic data of any two cameras.

[0071] Step S105, based on the first target coordinate and the second target coordinate, the extrinsic data of any two cameras is optimized in an iterative manner until the optimization result is less than or equal to the second preset threshold, and the calibration parameter of the target XR device is output.

[0072] Specifically, the matching result obtained by the first matching mode is taken as a target function, the difference data of the first target coordinate and the second target coordinate is calculated, the extrinsic data of any two cameras is optimized in an iterative manner, and the calibration parameter of the target XR device is output.

[0073] The data calibration method for extended reality devices in this embodiment, under the preset conditions that the calibration trigger data and the memory usage data meet, obtains the first target coordinates of any two cameras through a first matching method and the second target coordinates of any two cameras through a second matching method, and calibrates the extrinsic data of any two cameras of the target XR device. This not only meets the extrinsic calibration point requirements of the XR device, allowing users to correct the camera and IMU calibration data without their awareness, thus ensuring the positioning and tracking effect of the XR device, but also has a relatively simple calibration method. Even if the XR device undergoes shell deformation due to long-term use after leaving the factory, resulting in changes in extrinsic parameters, calibration can be quickly achieved to ensure the normal operation of the XR device.

[0074] This embodiment provides a data calibration method for extended reality devices, which can be used in the aforementioned mobile terminals, such as mobile phones and tablets. Figure 2 This is a flowchart of a data calibration method for an extended reality device according to an embodiment of the present invention, such as... Figure 2 As shown, in step S102 above, when the calibration trigger data and memory usage data meet preset conditions, the process includes the following steps:

[0075] Step S201: Obtain the current drop distance and current drop acceleration of the target XR device.

[0076] Specifically, the current drop distance of the target XR device represents the distance the target XR device has fallen. This current drop distance is detected by a distance sensor, and the current drop acceleration can also be detected by an acceleration sensor.

[0077] Step S202: When the current drop distance of the target XR device is greater than or equal to the third preset threshold, obtain the current drop acceleration of the target XR device.

[0078] Specifically, for example, if the current drop distance of the target XR device is 0.6m and the third preset threshold is 0.5m, that is, the current drop distance is greater than the third preset threshold, then the current drop acceleration of the target XR device can be obtained.

[0079] Step S203: When the current drop acceleration of the target XR device is equal to the free fall acceleration, obtain the current state and memory usage data of the target XR device.

[0080] Specifically, the acceleration due to free fall is 9.8 m / s². 2 When the current drop acceleration of the target XR device equals its free fall acceleration, it indicates that the target XR device has experienced a free fall during use, requiring further calibration. The current state of the target XR device indicates its operational status, while the memory usage data indicates CPU memory utilization.

[0081] Step S204, when the current state of the target XR device is in the main interface state, and the memory occupation data is less than or equal to the fourth preset threshold, the current occlusion of the target XR device is obtained.

[0082] Specifically, the main interface state of the target XR device is the launcher interface state, in which state each camera and inertial sensor is in the starting state. In this launcher interface state, the current occlusion of the target XR device is detected by the sensor so as to accurately calibrate the target XR device.

[0083] Step S205, when the target XR device is not currently occluded, the target XR device meets the preset condition.

[0084] Under the conditions met by the above steps S201-S205, it is considered that the target XR device meets the preset condition to be calibrated, and based on the preset condition, the external parameters of the target XR device are accurately calibrated.

[0085] In this embodiment, a data calibration method of an extended reality device is provided, which can be used in the above-mentioned mobile terminal, such as a mobile phone, a tablet computer, etc. Figure 3 is a flowchart of the data calibration method of the extended reality device according to the embodiment of the present application, as shown in Figure 3 Step S102, when the calibration trigger data and the memory occupation data meet the preset condition, the flow includes the following steps:

[0086] Step S301, the current use duration of the target XR device is obtained.

[0087] Specifically, for example, the current use duration of the user is 4 months (more than 10 minutes of daily use is counted as effective use of one day).

[0088] Step S302, when the current use duration of the target XR device is greater than or equal to a preset period, the current state of the target XR device and the memory occupation data are obtained.

[0089] Specifically, the preset period is 3 months (more than 10 minutes of daily use is counted as effective use of one day), and the current state of the target XR device indicates the working condition of the target XR device, and the memory occupation data indicates the CPU memory occupation rate.

[0090] Step S303, when the current state of the target XR device is in the main interface state, and the memory occupation data is less than or equal to the fourth preset threshold, the current occlusion of the target XR device is obtained.

[0091] Specifically, the main interface state of the target XR device is a launcher interface state, in which each camera and inertial sensor is in a starting state. In the launcher interface state, the fourth preset threshold can be 20% of the CPU occupancy rate, and when the memory occupancy data is less than or equal to 20% of the CPU occupancy rate, the current occlusion of the target XR device can be obtained.

[0092] The current occlusion of the target XR device is detected by the sensor, so as to accurately calibrate the target XR device.

[0093] In step S304, when the target XR device is not currently occluded, the target XR device meets the preset condition.

[0094] Under the conditions met in steps S301-S304, it is considered that the target XR device meets the preset condition to be calibrated, and based on the preset condition, the external parameters of the target XR device are accurately calibrated.

[0095] In some optional embodiments, in step S102, the similarity between the actual running track of the target XR device and the preset running track is calculated based on the camera data and the IMU raw data, including:

[0096] In step a1, the actual running track of the target XR device is calculated based on the camera data and the IMU raw data by a visual SLAM algorithm.

[0097] In step a2, the preset running track of the target XR device is obtained.

[0098] In step a3, the actual running track and the preset running track are compared to obtain the similarity therebetween.

[0099] Specifically, the actual running track (Relative Pose Error, RPE) of the target XR device is calculated based on the camera data and the IMU raw data of any two cameras. The relative pose error is used to calculate the difference in the change amount of the pose within the same two time stamps. Similarly, after alignment with the time stamps, the real pose and the estimated pose each calculate the change amount of the pose at a fixed time interval, and then the change amount is subtracted to obtain the relative pose error. This calibration is suitable for estimating the drift relative pose error of the system, and mainly describes the accuracy of the difference between two poses at a fixed time interval (compared with the real pose), which is equivalent to directly measuring the error of the odometer.

[0100] Therefore, the RPE of the i-th frame is defined as follows:

[0101] Wherein, E i represents the estimated pose state of the camera data of the i-th frame, Q ia real pose of the i-th frame of the camera data;

[0102] Given the total number n and the interval Δ, we can get m = n - Δ RPEs, then this embodiment can use the root mean square error RMSE to count this error and get a total value,

[0103]

[0104] where trans(E i ) represents the translation part of the relative position error. Of course, there is also no RMSE, but using the average value, even the median to describe the relative error situation. It should be noted that RPE contains two parts of error, which are rotation error and translation error, and the translation error is usually used for evaluation, but if necessary, the error of the rotation angle can also be counted using the same method. In order to comprehensively measure the algorithm performance, the average value of all RMSEs can be calculated.

[0105] And calculate a fixed number of RPE samples to calculate an estimated value as the final result, that is, the actual running trajectory of the target XR device in the above.

[0106] And the preset running trajectory in the above (Absolute Trajectory Error, ATE) directly calculates the difference between the real value of the camera pose and the estimated value of the SLAM system. First, the real value and the estimated value are aligned according to the timestamp of the pose, then the difference between each pair of poses is calculated, and finally output in the form of a chart. This standard is very suitable for evaluating the performance of a visual SLAM system.

[0107] The absolute trajectory error is the direct difference between the estimated pose and the real pose, which can very intuitively reflect the algorithm accuracy and the global consistency of the trajectory. It should be noted that the estimated pose and the real pose are usually not in the same coordinate system, so this embodiment needs to align the two first. For binocular SLAM and RGB-D SLAM, the scale is unified, so a conversion matrix from the estimated pose to the real pose needs to be calculated by the least square method.

[0108] For monocular cameras, there is scale uncertainty, and a similarity conversion S ∈ SE(3) from the estimated pose to the real pose needs to be calculated.

[0109] Therefore, the ATE of the first frame is defined as follows:

[0110]

[0111] Similar to RPE, it is recommended to use RMSE to count ATE,

[0112] Of course, ATE can also be reflected by using average, median, etc. Now many evaluation tools will give RMSE, Mean, Median. In summary, it should be noted that RPE error contains translation and rotation error, while ATE only contains translation error (from the full name of the two), both have strong correlation, but also not the same. Still need to combine the actual situation, choose the right indicators to evaluate the algorithm.

[0113] By the above specific calculation method, the actual running track is compared with the preset running track to obtain the similarity between them.

[0114] In some optional embodiments, in step S103, the target feature points are extracted from the camera data by a first matching method, and the first target coordinates corresponding to the target feature points are matched, including:

[0115] In step b1, the target feature points are extracted from the camera data by a SIFT feature method.

[0116] In step b2, the first target coordinates corresponding to the target feature points are matched by a SIFT matching algorithm.

[0117] Specifically, SIFT (Scale-Invariant Feature Transform) is a feature extraction algorithm of computer vision, which is used to detect and describe local features in images. In essence, it searches for key points (feature points) in different scale spaces and calculates the direction of the key points.

[0118] In the process of extracting target feature points according to the SIFT feature method, mainly through the steps of extreme value detection of scale space, key point positioning, direction assignment, key point descriptor, etc.; and the matching method of the SIFT matching algorithm is described as follows:

[0119] For example, for the camera data in any two cameras, which generates the descriptors of two images A and B (k1*128 dimensions and k2*128 dimensions, respectively), the descriptors of each scale (all scales) in the two images are matched, and the 128 dimensions are matched to indicate that the two feature points are matched. After the SIFT feature vectors of the two images are generated, the next step of the embodiment uses the Euclidean distance of the key point feature vectors as the similarity determination measure of the key points in the two images. Take a key point in image 1, and find the first two key points in image 2 with the closest Euclidean distance. In the two key points, if the ratio of the nearest distance to the second nearest distance is less than a certain proportion threshold, the pair of matching points is accepted. Reducing the proportion threshold, the number of SIFT matching points will decrease, but it will be more stable. The method of the nearest neighbor distance and the second nearest neighbor distance considers that the distance ratio ratio is less than a certain threshold as correct matching. The recommended ratio value principle is as follows:

[0120] ratio = 0.4, for high accuracy matching;

[0121] ratio = 0.6, for matching with a relatively large number of matching points;

[0122] ratio = 0.5, in general, or according to the following principle: ratio = 0.6 when the nearest neighbor distance is less than 200, otherwise ratio = 0.4. The ratio value strategy can eliminate incorrect matching points. Finally, the first target coordinate corresponding to the target feature point matched by the SIFT matching algorithm is output.

[0123] In some optional embodiments, step S104, based on the intrinsic data and extrinsic data of any two cameras, the camera data of any two cameras is matched by a second matching method to obtain a second target coordinate, comprising:

[0124] Step c1, based on the intrinsic data and extrinsic data of any two cameras, the camera data of any two cameras is matched by an epipolar matching method to obtain a second target coordinate.

[0125] Specifically, after obtaining the intrinsic data (intrinsic matrix and distortion coefficient) of the two cameras and the extrinsic data (relative rotation and translation matrix) between any two cameras, the camera data of any two cameras is matched by an epipolar matching method, which is equivalent to performing binocular parallel correction. The images of the same object taken by the two cameras are processed in a certain way, so that the two images have the same size and the horizontal is on a straight line.

[0126] The epipolar line matching mode is performed on the camera coordinate system corresponding to the image pair. When the epipolar line correction is performed, the camera coordinate systems corresponding to the left and right images are respectively corrected by rotation matrices R1 and R2. The steps are as follows: the source image pixel coordinate system is converted into a camera coordinate system (compared with the image physical coordinate system, the scaling and Z axis are more) by an intrinsic matrix, the parallel epipolar line correction is performed by rotation matrices R1 and R2, then the camera coordinates of the image are corrected by distortion coefficients, and the camera coordinate system is converted into an image pixel coordinate system after correction, and the pixel value of the source image coordinate is assigned to the new image coordinate, that is, the camera data of any two cameras is matched by the epipolar line matching mode to obtain the second target coordinate.

[0127] The data calibration method of the extended reality device in the embodiment can not only meet the demand of XR device external parameter calibration points, but also correct the camera and IMU calibration data without user awareness, so as to ensure the positioning and tracking effect of the XR device, and the calibration process is relatively simple.

[0128] In the embodiment, a data calibration device of an extended reality device is also provided. The device is used to implement the above embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware is also possible and is contemplated.

[0129] The embodiment provides a data calibration device of an extended reality device, as shown in Figure 4 The device includes:

[0130] The device data acquisition module 41 is configured to acquire camera data of any two cameras of the target XR device, intrinsic data, extrinsic data, and IMU raw data of an inertial sensor, calibration trigger data of the target XR device, and memory occupation data.

[0131] The device data calculation module 42 is configured to calculate the similarity between the actual running track and the preset running track of the target XR device based on the camera data and the IMU raw data when the calibration trigger data and the memory occupation data meet the preset condition.

[0132] The first coordinate matching module 43 is configured to extract a target feature point from the camera data by the first matching mode and match a first target coordinate corresponding to the target feature point when the similarity is less than or equal to a first preset threshold.

[0133] The second coordinate matching module 44 is configured to match the camera data of any two cameras by a second matching manner based on the intrinsic data and the extrinsic data of the any two cameras to obtain a second target coordinate.

[0134] The calibration parameter output module 45 is configured to optimize the extrinsic data of the any two cameras in an iterative manner based on the first target coordinate until an optimization result is less than or equal to a second preset threshold, and output a calibration parameter of the target XR device.

[0135] In some optional embodiments, the device data calculation module 42 comprises:

[0136] The first data acquisition submodule is configured to acquire a current falling distance and a current falling acceleration of the target XR device.

[0137] The second data acquisition submodule is configured to acquire the current falling acceleration of the target XR device when the current falling distance of the target XR device is greater than or equal to a third preset threshold.

[0138] The third data acquisition submodule is configured to acquire a current state and memory occupation data of the target XR device when the current falling acceleration of the target XR device is equal to a free-fall acceleration.

[0139] The fourth data acquisition submodule is configured to acquire a current occlusion condition of the target XR device when the current state of the target XR device is in a home interface state and the memory occupation data is less than or equal to a fourth preset threshold.

[0140] The preset condition determination submodule is configured to determine that the target XR device satisfies a preset condition when the target XR device is not currently occluded.

[0141] In some optional embodiments, the device data calculation module 42 comprises:

[0142] The fifth data acquisition submodule is configured to acquire a current use duration of the target XR device.

[0143] The sixth data acquisition submodule is configured to acquire a current state and memory occupation data of the target XR device when the current use duration of the target XR device is greater than or equal to a preset period.

[0144] The seventh data acquisition submodule is configured to acquire a current occlusion condition of the target XR device when the current state of the target XR device is in a home interface state and the memory occupation data is less than or equal to a fourth preset threshold.

[0145] The preset condition determination submodule is configured to determine that the target XR device satisfies a preset condition when the target XR device is not currently occluded.

[0146] In some optional embodiments, the device data calculation module 42 comprises:

[0147] The running trajectory calculation sub-module is configured to calculate the actual running trajectory of the target XR device based on the camera data and the IMU raw data by using a visual SLAM algorithm.

[0148] The running trajectory acquisition sub-module is configured to acquire the preset running trajectory of the target XR device.

[0149] The eighth data acquisition sub-module is configured to compare the actual running trajectory with the preset running trajectory and acquire the similarity therebetween.

[0150] In some optional embodiments, the first coordinate matching module 43 comprises:

[0151] The target feature matching sub-module is configured to extract target feature points from the camera data by using a SIFT feature method.

[0152] The first coordinate matching sub-module is configured to match the first target coordinates corresponding to the target feature points by using a SIFT matching algorithm.

[0153] In some optional embodiments, the first coordinate matching module 43 comprises:

[0154] The second coordinate matching sub-module is configured to match the camera data of any two cameras by using a epipolar matching method based on the intrinsic data and the extrinsic data of the any two cameras to obtain the second target coordinates.

[0155] In some optional embodiments, the data calibration apparatus of the extended reality device further comprises:

[0156] The calibration closing module is configured to not trigger the target XR device to perform the data calibration action when the calibration trigger data and the memory occupation data do not satisfy the preset condition.

[0157] Further function descriptions of the above modules and units are the same as those of the corresponding embodiments, and thus are not described herein.

[0158] The data calibration apparatus of the extended reality device in the embodiments is presented in the form of functional units, and the units refer to ASIC (Application Specific Integrated Circuit) circuits, processors and memories executing one or more software or fixed programs, and / or other devices that can provide the above functions.

[0159] The embodiments of the present application also provide a computer device with the data calibration apparatus of the extended reality device.

[0160] Please refer toFigure 5 , Figure 5 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 5 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 5 Take a processor 10 as an example.

[0161] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0162] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0163] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0164] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0165] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 can be connected through a bus or other means, and are connected through a bus in FIG. X as an example.

[0166] The input device 30 can receive inputted digital or character information, and generate key signal input related to user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, a pointing stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 can include a display device, an auxiliary lighting device (e.g., an LED), a tactile feedback device (e.g., a vibration motor), etc. The display device includes but is not limited to a liquid crystal display, a light emitting diode, a display, and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0167] The embodiments of the present application also provide a computer readable storage medium, and the method according to the embodiments of the present application can be implemented in hardware, firmware, or recorded in a storage medium, or be implemented by computer code stored in a remote storage medium or a non-transitory machine readable storage medium and downloaded from a network and stored in a local storage medium, so that the method described herein can be processed by such software on a storage medium using a general purpose computer, a special purpose processor, or programmable or special purpose hardware. The storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk or a solid state disk, etc. Further, the storage medium can also include a combination of the above-mentioned types of memories. It can be understood that the computer, the processor, the microprocessor controller or the programmable hardware includes a storage component that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, the processor or the hardware, the method shown in the above embodiments is implemented.

[0168] Although the embodiments of the present application are described in conjunction with the accompanying drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the scope defined by the appended claims.

Claims

1. A data calibration method for an extended reality device, characterized in that, The method includes: Acquire the video data, intrinsic parameter data, and extrinsic parameter data of any two cameras of the target XR device, as well as the raw IMU data of the inertial sensor, calibration trigger data, and memory usage data of the target XR device; When the calibration trigger data and the memory usage data meet the preset conditions, the similarity between the actual running trajectory of the target XR device and the preset running trajectory is calculated based on the camera data and the IMU raw data. When the similarity is less than or equal to a first preset threshold, target feature points are extracted from the camera data using a first matching method, and the first target coordinates corresponding to the target feature points are matched. Based on the intrinsic and extrinsic data of any two cameras, the camera data of any two cameras are matched using a second matching method to obtain the coordinates of the second target. Based on the first target coordinates and the second target coordinates, the extrinsic data of any two cameras are optimized in an iterative manner until the optimization result is less than or equal to a second preset threshold, and the calibration parameters of the target XR device are output.

2. The method according to claim 1, characterized in that, The calibration trigger data and the memory usage data meet preset conditions, including: Obtain the current drop distance and current drop acceleration of the target XR device; When the current drop distance of the target XR device is greater than or equal to a third preset threshold, the current drop acceleration of the target XR device is obtained; When the current drop acceleration of the target XR device equals the free fall acceleration, obtain the current state and memory usage data of the target XR device; When the target XR device is currently in the main interface state and the memory usage data is less than or equal to the fourth preset threshold, the current occlusion status of the target XR device is obtained. If the target XR device is not currently obstructed, then the target XR device meets the preset condition.

3. The method according to claim 1, characterized in that, The calibration trigger data and the memory usage data meet preset conditions, and further include: Obtain the current usage time of the target XR device; When the current usage time of the target XR device is greater than or equal to a preset period, the current status and memory usage data of the target XR device are obtained; When the target XR device is currently in the main interface state and the memory usage data is less than or equal to the fourth preset threshold, the current occlusion status of the target XR device is obtained. If the target XR device is not currently obstructed, then the target XR device meets the preset condition.

4. The method according to claim 1, characterized in that, Based on the camera data and IMU raw data, the similarity between the actual running trajectory of the target XR device and the preset running trajectory is calculated, including: Based on the camera data and IMU raw data, the actual running trajectory of the target XR device is calculated using a visual SLAM algorithm; Obtain the preset operating trajectory of the target XR device; The actual running trajectory is compared with the preset running trajectory to obtain the similarity between the two.

5. The method according to claim 1, characterized in that, Extracting target feature points from the camera data using a first matching method and matching the first target coordinates corresponding to those target feature points includes: Target feature points are extracted from the camera data using the SIFT feature extraction method; The SIFT matching algorithm is used to match the first target coordinates corresponding to the target feature point.

6. The method according to claim 1, characterized in that, The second target coordinates are obtained by matching the camera data of any two cameras using a second matching method, including: Based on the intrinsic and extrinsic data of any two cameras, the camera data of any two cameras are matched using epipolar matching to obtain the coordinates of the second target.

7. The method according to claim 1, characterized in that, Also includes: If the calibration trigger data and the memory usage data do not meet the preset conditions, the target XR device will not be triggered to perform data calibration.

8. A data calibration device for an extended reality device, characterized in that, The device includes: The device data acquisition module is used to acquire video data, intrinsic parameter data, and extrinsic parameter data from any two cameras of the target XR device, as well as the raw IMU data from the inertial sensor, calibration trigger data, and memory usage data of the target XR device. The device data calculation module is used to calculate the similarity between the actual running trajectory of the target XR device and the preset running trajectory based on the camera data and the IMU raw data when the calibration trigger data and the memory usage data meet the preset conditions. The first coordinate matching module is used to extract target feature points from the camera data through a first matching method when the similarity is less than or equal to a first preset threshold, and match the first target coordinates corresponding to the target feature points. The second coordinate matching module is used to match the camera data of any two cameras with the intrinsic parameter data and the extrinsic parameter data of any two cameras through a second matching method to obtain the second target coordinates; The calibration parameter output module is used to optimize the extrinsic data of any two cameras in an iterative manner based on the first target coordinates until the optimization result is less than or equal to a second preset threshold, and output the calibration parameters of the target XR device.

9. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the data calibration method for the extended reality device according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the data calibration method for the extended reality device according to any one of claims 1 to 7.

Images