Data calibration method and device for augmented reality equipment, equipment and medium

Abstract

The invention relates to the technical field of augmented reality equipment, and discloses a data calibration method and device for augmented reality equipment, equipment and a medium. The method comprises the following steps: when calibration trigger data and memory occupation data meet a preset condition, obtaining first target coordinates of any two cameras through a first matching mode, obtaining second target coordinates of any two cameras through a second matching mode, and calibrating external parameter data of any two cameras of target XR equipment, according to the method, the requirement of XR equipment external parameter calibration points can be met, a user can correct camera and IMU calibration data without perception, the positioning tracking effect of the XR equipment is guaranteed, the calibration mode is simple, and even if the XR equipment leaves a factory, the XR equipment is subjected to shell deformation and external parameter change due to long-time use, the calibration accuracy of the XR equipment is improved. And the calibration can be quickly realized to ensure the normal work of the XR equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of extended reality devices (XR devices), and in particular to a data calibration method, device, equipment and medium for extended reality devices. Background Art

[0002] Vision and inertial navigation SLAM is the mainstream solution for achieving XR device positioning. By calibrating the internal and external parameters of the camera and inertial sensor (IMU), the corresponding information can be integrated with each other, so that the XR device can perceive the surrounding objects more accurately, thus ensuring the accuracy of XR device positioning and safety in use.

[0003] The related technology calibrates the internal and external parameters of the XR device through a calibration plate to achieve parameter calibration. In order to accurately correct the internal and external parameters of the XR device, the calibration plate needs to be calibrated with high precision during factory production, which makes the calibration method more cumbersome. After using the XR device for a long time, the shell is prone to deformation. Accidental falling during use can also easily cause changes in the external parameters between sensors, thereby affecting the normal operation of the XR device after leaving the factory. Summary of the invention

[0004] In view of this, the present invention provides a data calibration method, device, equipment and medium for an extended reality device, so as to solve the problem that the calibration method is relatively cumbersome and the external parameters change due to deformation or falling off of the shell after long-term use, thereby affecting the normal operation of the XR device after leaving the factory.

[0005] According to the first aspect, this embodiment provides a data calibration method for an extended reality device, the method comprising:

[0006] Obtain the camera data, internal parameter data, external parameter data, and IMU raw data of any two cameras of the target XR device, as well as the calibration trigger data and memory usage data of the target XR device;

[0007] When the calibration trigger data and memory usage data meet the preset conditions, the similarity between the actual running trajectory of the target XR device and the preset running trajectory is calculated based on the camera data and IMU raw data;

[0008] When the similarity is less than or equal to a first preset threshold, extracting a target feature point from the camera data by a first matching method, and matching the first target coordinates corresponding to the target feature point;

[0009] Based on the intrinsic reference data and the extrinsic reference data of any two cameras, the camera data of any two cameras are matched by a second matching method to obtain a second target coordinate;

[0010] Based on the first target coordinates and the second target coordinates, the external parameter data of any two cameras are optimized in an iterative manner until the optimization result is less than or equal to the second preset threshold, and the calibration parameters of the target XR device are output.

[0011] By executing the above implementation, when the calibration trigger data and the memory occupancy data meet the preset conditions, the first target coordinates of any two cameras are obtained through the first matching method, and the second target coordinates of any two cameras are obtained through the second matching method, and the external parameter data of any two cameras of the target XR device are calibrated. This not only meets the calibration point requirements of the XR device external parameter calibration, but also enables the user to correct the camera and IMU calibration data without perception, thereby ensuring the positioning and tracking effect of the XR device. In addition, the calibration method is relatively simple. Even after leaving the factory, if the XR device has a shell deformation due to long-term use, resulting in changes in external parameters, calibration can be quickly achieved to ensure the normal operation of the XR device.

[0012] In an optional implementation manner, the calibration trigger data and the memory occupancy data meet preset conditions, including:

[0013] Get the current drop distance and current drop acceleration of the target XR device;

[0014] When the current falling distance of the target XR device is greater than or equal to the third preset threshold, obtaining the current falling acceleration of the target XR device;

[0015] When the current falling acceleration of the target XR device is equal to the free fall acceleration, the current state and memory usage data of the target XR device are obtained;

[0016] When the current state of the target XR device is in the main interface state and the memory usage data is less than or equal to the fourth preset threshold, the current occlusion status of the target XR device is obtained;

[0017] When the target XR device is not currently blocked, the target XR device meets the preset condition.

[0018] By executing the above implementation, it is possible to determine whether the current falling distance, the current falling acceleration, the current state, the memory usage data, and the current occlusion situation meet the preset conditions one by one, which is conducive to the subsequent accurate calibration of the external parameters of the target XR device.

[0019] In an optional implementation manner, the calibration trigger data and the memory occupancy data meet a preset condition, further comprising:

[0020] Get the current usage time of the target XR device;

[0021] When the current usage time of the target XR device is greater than or equal to the preset period, the current state and memory usage data of the target XR device are obtained;

[0022] When the current state of the target XR device is in the main interface state and the memory usage data is less than or equal to the fourth preset threshold, obtain the current occlusion status of the target XR device;

[0023] When the target XR device is not currently blocked, the target XR device meets the preset condition.

[0024] By executing the above implementation, it is determined one by one whether the current usage time, current state and memory usage data of the target XR device meet the preset conditions, which is conducive to the subsequent accurate calibration of the external parameters of the target XR device.

[0025] In an optional implementation, based on the camera data and the IMU raw data, calculating the similarity between the actual running trajectory of the target XR device and the preset running trajectory includes:

[0026] Based on the camera data and IMU raw data, the actual running trajectory of the target XR device is calculated through the visual SLAM algorithm;

[0027] Get the preset running trajectory of the target XR device;

[0028] Compare the actual running trajectory with the preset running trajectory to obtain the similarity between the two.

[0029] By executing the above implementation, the similarity between the actual running trajectory of the target XR device and the preset running trajectory is calculated, so as to accurately calibrate the external parameters of the target XR device later.

[0030] In an optional implementation, extracting target feature points from camera data by a first matching method and matching first target coordinates corresponding to the target feature points include:

[0031] Extract target feature points from camera data using SIFT feature method;

[0032] The first target coordinates corresponding to the target feature point are matched using the SIFT matching algorithm.

[0033] By executing the above implementation, the target feature points corresponding to the camera data are matched by the first matching method, and the first target coordinates corresponding to the target feature points are obtained, which is conducive to the subsequent accurate calibration of the external parameters of the target XR device.

[0034] In an optional implementation, matching the camera data of any two cameras by a second matching method to obtain the second target coordinates includes:

[0035] Based on the intrinsic parameter data and extrinsic parameter data of any two cameras, the camera data of any two cameras are matched by epipolar matching to obtain the second target coordinates.

[0036] By executing the above implementation, the target feature points corresponding to the camera data are matched by the second matching method, and the second target coordinates corresponding to the target feature points are obtained, which is conducive to the subsequent accurate calibration of the external parameters of the target XR device.

[0037] In an optional implementation, the data calibration method of the extended reality device in this embodiment further includes:

[0038] When the calibration trigger data and memory usage data do not meet the preset conditions, the target XR device will not be triggered to perform data calibration actions.

[0039] According to the second aspect, this embodiment further provides a data calibration device for an extended reality device, the device comprising:

[0040] The device data acquisition module is used to obtain the camera data, internal parameter data, external parameter data, and IMU raw data of the inertial sensor of any two cameras of the target XR device, the calibration trigger data and memory usage data of the target XR device,

[0041] The device data calculation module is used to calculate the similarity between the actual running trajectory of the target XR device and the preset running trajectory based on the camera data and IMU raw data when the calibration trigger data and memory usage data meet the preset conditions;

[0042] A first coordinate matching module, used for extracting target feature points from the camera data by a first matching method and matching first target coordinates corresponding to the target feature points when the similarity is less than or equal to a first preset threshold;

[0043] A second coordinate matching module is used to match the camera data of any two cameras through a second matching method based on the internal reference data and external reference data of any two cameras to obtain a second target coordinate;

[0044] The calibration parameter output module is used to optimize the external parameter data of any two cameras in an iterative manner based on the first target coordinates until the optimization result is less than or equal to the second preset threshold, and output the calibration parameters of the target XR device.

[0045] According to the third aspect, this embodiment further provides a computer device, including:

[0046] The memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data calibration method of the extended reality device in the first aspect or any embodiment of the first aspect by executing the computer instructions.

[0047] According to the fourth aspect, this embodiment also provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to enable a computer to execute the data calibration method of the extended reality device in the first aspect or any implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 is a flow chart of a data calibration method for an extended reality device according to an embodiment of the present invention;

[0050] Figure 2 is a flow chart of a data calibration method for another extended reality device according to an embodiment of the present invention;

[0051] Figure 3 is a flow chart of a data calibration method for another extended reality device according to an embodiment of the present invention;

[0052] Figure 4 is a structural block diagram of a data calibration device for an extended reality device according to an embodiment of the present invention;

[0053] Figure 5 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0055] According to an embodiment of the present invention, an embodiment of a data calibration method for an extended reality device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0056] In this embodiment, a data calibration method for an extended reality device is provided, which can be used in a mobile terminal, such as a mobile phone, a tablet computer, etc. (the execution subject is described in combination with actual conditions). Figure 1 is a flow chart of a data calibration method for an extended reality device according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0057] Step S101, obtaining the camera data, internal parameter data, external parameter data, and IMU raw data of any two cameras of the target XR device, the calibration trigger data and memory usage data of the target XR device.

[0058] Specifically, the target XR device is equipped with multiple cameras and an inertial sensor, and the camera data of any two cameras can be obtained from the multiple cameras. In computer vision, the intrinsic parameter data of the camera are parameters related to the camera's own characteristics, such as the focal length and pixel size of the camera, while the extrinsic parameters of the camera are parameters in the world coordinate system, such as the position and rotation direction of the camera.

[0059] For example, for the camera external parameter data: the internal parameter matrix is ​​given below. It should be noted that the real camera will also have radial and tangential distortions, and these distortions belong to the camera's internal parameters. The camera internal parameter matrix:

[0060]

[0061] Among them, fx and fy are focal lengths, which are generally equal, x0 and y0 are the principal point coordinates (relative to the imaging plane), and s is the coordinate axis tilt parameter, which is 0 in an ideal case.

[0062] Exemplarily, for the camera extrinsic parameter data: the rotation and translation of the camera belong to the extrinsic parameters, which are used to describe the movement of the camera in a static scene, or the rigid movement of a moving object when the camera is fixed. Therefore, in image stitching or three-dimensional reconstruction, it is necessary to use extrinsic parameter data to calculate the relative movement between several images, so as to transfer them to the same coordinate system. The extrinsic parameter matrix of the camera includes the rotation matrix and the translation matrix. The rotation matrix and the translation matrix together describe how to transform a point from the world coordinate system to the camera coordinate system. The rotation matrix: describes the direction of the coordinate axis of the world coordinate system relative to the camera coordinate axis. The translation matrix: describes the position of the origin of space in the camera coordinate system.

[0063] Specifically, the IMU raw data refers to the bias noise data. The calibration trigger data of the target XR device includes the current fall distance and the current fall acceleration of the target XR device, and the calibration trigger data of the target XR device also includes the current usage time of the target XR device.

[0064] Step S102: When the calibration trigger data and the memory occupancy data meet the preset conditions, the similarity between the actual running trajectory of the target XR device and the preset running trajectory is calculated based on the camera data and the IMU raw data.

[0065] Specifically, the preset conditions are determined based on the calibration trigger data and the memory usage data, and the actual running trajectory and the preset running trajectory of the target XR device are calculated, which is conducive to calibrating the actual situation of the target XR device.

[0066] Specifically, when the actual situation of the target XR device is similar to the preset situation, calibrating the actual situation of the target XR device is beneficial to improving the accuracy of the calibration.

[0067] Step S103: when the similarity is less than or equal to the first preset threshold, extracting target feature points from the camera data by a first matching method, and matching the first target coordinates corresponding to the target feature points.

[0068] Specifically, the first matching method is a SIFT matching method. In the image matching process, the first matching method is a highly robust matching algorithm. The SIFT feature matching algorithm is generally implemented in two stages: the first stage is the extraction of SIFT feature vectors, that is, extracting feature vectors that are independent of scale scaling, rotation, and brightness changes from multiple images to be matched; the second stage is the matching of SIFT feature vectors. In this embodiment, target feature points are extracted from the camera data of the first camera and the camera data of the second camera, and the corresponding first target coordinates are obtained by matching.

[0069] Step S104, based on the intrinsic reference data and the extrinsic reference data of any two cameras, the camera data of any two cameras are matched by a second matching method to obtain the second target coordinates.

[0070] Specifically, the second matching method is an extreme matching method, which is obtained by transforming and calculating the intrinsic parameter data and the extrinsic parameter data of any two cameras.

[0071] Step S105 , based on the first target coordinates and the second target coordinates, optimize the external parameter data of any two cameras in an iterative manner until the optimization result is less than or equal to a second preset threshold, and output the calibration parameters of the target XR device.

[0072] Specifically, the matching result obtained by the first matching method is used as the objective function, the difference data between the first target coordinates and the second target coordinates is calculated, the external parameter data of any two cameras is optimized in an iterative manner, and the calibration parameters of the target XR device are output.

[0073] The data calibration method for the extended reality device in this embodiment obtains the first target coordinates of any two cameras through a first matching method and obtains the second target coordinates of any two cameras through a second matching method when the calibration trigger data and the memory usage data meet the preset conditions, and calibrates the external parameter data of any two cameras of the target XR device. This not only meets the calibration point requirements of the external parameter calibration of the XR device, but also enables the user to correct the camera and IMU calibration data without perception, thereby ensuring the positioning and tracking effect of the XR device. In addition, the calibration method is relatively simple. Even after leaving the factory, if the external parameters of the XR device change due to shell deformation caused by long-term use, calibration can be quickly achieved to ensure the normal operation of the XR device.

[0074] In this embodiment, a data calibration method for an extended reality device is provided, which can be used for the above-mentioned mobile terminal, such as a mobile phone, a tablet computer, etc. Figure 2 is a flow chart of a data calibration method for an extended reality device according to an embodiment of the present invention. Figure 2 As shown, in the above step S102, when the calibration trigger data and the memory occupancy data meet the preset conditions, the process includes the following steps:

[0075] Step S201, obtaining the current falling distance and current falling acceleration of the target XR device.

[0076] Specifically, the current drop distance of the target XR device represents the distance the target XR device slides down. The current drop distance is detected by a distance sensor, and the current drop acceleration can also be detected by an acceleration sensor.

[0077] Step S202: When the current drop distance of the target XR device is greater than or equal to a third preset threshold, the current drop acceleration of the target XR device is obtained.

[0078] Specifically, for example, the current falling distance of the target XR device is 0.6 m, and the third preset threshold is 0.5 m, that is, the current falling distance is greater than the third preset threshold. At this time, the current falling acceleration of the target XR device can be obtained.

[0079] Step S203: When the current drop acceleration of the target XR device is equal to the free fall acceleration, the current state and memory usage data of the target XR device are obtained.

[0080] Specifically, the free fall acceleration is 9.8 m / s 2 When the current drop acceleration of the target XR device is equal to the free fall acceleration, it means that the target XR device has fallen freely during use, and the target XR device needs to be further calibrated. The current state of the target XR device indicates the working status of the target XR device, and the memory usage data indicates the CPU memory usage.

[0081] Step S204: When the current state of the target XR device is in the main interface state and the memory usage data is less than or equal to the fourth preset threshold, the current occlusion status of the target XR device is obtained.

[0082] Specifically, the main interface state of the target XR device is the launcher interface state, in which each camera and inertial sensor are in the startup state. In the launcher interface state, the current occlusion of the target XR device is detected by the sensor so as to accurately calibrate the target XR device.

[0083] Step S205: When the target XR device is not currently blocked, the target XR device meets the preset condition.

[0084] Under the conditions that the above steps S201 to S205 are successively met, it is considered that the target XR device meets the preset conditions to be calibrated, and based on the preset conditions, the external parameters of the target XR device can be accurately calibrated.

[0085] In this embodiment, a data calibration method for an extended reality device is provided, which can be used for the above-mentioned mobile terminal, such as a mobile phone, a tablet computer, etc. Figure 3 is a flow chart of a data calibration method for an extended reality device according to an embodiment of the present invention. Figure 3 As shown, in the above step S102, when the calibration trigger data and the memory occupancy data meet the preset conditions, the process includes the following steps:

[0086] Step S301, obtaining the current usage time of the target XR device.

[0087] Specifically, for example, the user's current usage duration is 4 months (daily usage of more than 10 minutes is counted as one day of effective use).

[0088] Step S302: When the current usage duration of the target XR device is greater than or equal to a preset period, the current state and memory usage data of the target XR device are obtained.

[0089] Specifically, the preset period is 3 months (daily usage of more than 10 minutes is counted as one day of effective use). The current state of the target XR device indicates the working status of the target XR device, and the memory usage data indicates the CPU memory usage rate.

[0090] Step S303: When the current state of the target XR device is in the main interface state and the memory usage data is less than or equal to the fourth preset threshold, the current occlusion status of the target XR device is obtained.

[0091] Specifically, the main interface state of the target XR device is the launcher interface state, in which each camera and inertial sensor are in the startup state. In the launcher interface state, the fourth preset threshold value may be 20% of the CPU occupancy rate, and when the memory occupancy data is less than or equal to 20% of the CPU occupancy rate, the current occlusion status of the target XR device may be obtained.

[0092] The current occlusion status of the target XR device is detected by the sensor so as to accurately calibrate the target XR device.

[0093] Step S304: When the target XR device is not currently blocked, the target XR device meets a preset condition.

[0094] Under the conditions that the above steps S301 to S304 are successively met, it is considered that the target XR device meets the preset conditions to be calibrated, and based on the preset conditions, the external parameters of the target XR device can be accurately calibrated.

[0095] In some optional implementations, step S102, based on the camera data and the IMU raw data, calculates the similarity between the actual running trajectory of the target XR device and the preset running trajectory, including:

[0096] Step a1: Based on the camera data and IMU raw data, the actual running trajectory of the target XR device is calculated through the visual SLAM algorithm.

[0097] Step a2: Obtain the preset running trajectory of the target XR device.

[0098] Step a3: compare the actual running trajectory with the preset running trajectory to obtain the similarity between the two.

[0099] Specifically, based on the video data of any two cameras and the original data of the IMU, the actual running trajectory (relative pose error, Relative Pose Error, RPE) of the target XR device is calculated. The relative pose error is used to calculate the difference in pose change within the same two timestamps. Similarly, after aligning the timestamps, the true pose and the estimated pose are calculated at the same interval of time, and then the change is subtracted to obtain the relative pose error. This calibration is suitable for estimating the drift relative pose error of the system, which mainly describes the accuracy of the two pose differences separated by a fixed time difference (compared with the true pose), which is equivalent to directly measuring the error of the odometer.

[0100] Therefore, the RPE of the i-th frame is defined as follows:

[0101] Among them, E i represents the estimated position and posture of the camera data of the i-th frame, Q irepresents the true pose of the camera data of the i-th frame;

[0102] When the total number n and the interval Δ are known, m=n-Δ RPEs can be obtained. Then, in this embodiment, the root mean square error RMSE can be used to calculate the error and obtain an overall value.

[0103]

[0104] Among them, trans(E i ), which represents the translation part of the relative position error. Of course, there are also methods that do not use RMSE, but directly use the average value or even the median to describe the relative error. It should be noted that RPE contains two parts of error, namely rotation error and translation error. Usually, it is sufficient to use the translation error for evaluation, but if necessary, the rotation angle error can also be counted using the same method. In order to comprehensively measure the performance of the algorithm, the average value of all RMSEs can be calculated.

[0105] And calculate a fixed number of RPE samples to calculate estimated values ​​as the final result, which is the actual running trajectory of the target XR device mentioned above.

[0106] The preset running trajectory mentioned above (Absolute Trajectory Error (ATE)) directly calculates the difference between the true value of the camera pose and the estimated value of the SLAM system. First, the true value and the estimated value are aligned according to the timestamp of the pose, and then the difference between each pair of poses is calculated and finally output in the form of a chart. This standard is very suitable for evaluating the performance of the visual SLAM system.

[0107] The absolute trajectory error is the direct difference between the estimated pose and the true pose, which can very intuitively reflect the algorithm accuracy and the global consistency of the trajectory. It should be noted that the estimated pose and the true pose are usually not in the same coordinate system. Therefore, this embodiment needs to align the two first. For binocular SLAM and RGB-D SLAM, the scale is unified. Therefore, it is necessary to calculate a transformation matrix from the estimated pose to the true pose by the least squares method.

[0108] For a monocular camera, there is scale uncertainty, and it is necessary to calculate a similarity transformation moment S∈SE(3) from the estimated pose to the true pose.

[0109] Therefore, the ATE of frame 1 is defined as follows:

[0110]

[0111] Similar to RPE, it is recommended to use RMSE to calculate ATE.

[0112] Of course, it is also possible to use the mean, median, etc. to reflect ATE. Now many evaluation tools will give RMSE, Mean, and Median. In summary, it should be noted that the RPE error includes the errors of both translation and rotation, while ATE only includes the translation error (as can be seen from the full names of the two). The two are strongly correlated, but not exactly the same. It is still necessary to combine the actual situation and select appropriate indicators for algorithm evaluation.

[0113] By using the above specific calculation method, the actual running trajectory is compared with the preset running trajectory to obtain the similarity between the two.

[0114] In some optional implementations, step S103, extracting target feature points from the camera data by a first matching method, and matching first target coordinates corresponding to the target feature points, includes:

[0115] Step b1, extracting target feature points from the camera data by using SIFT feature method.

[0116] Step b2: Match the first target coordinates corresponding to the target feature point by using a SIFT matching algorithm.

[0117] Specifically, SIFT (Scale-Invariant Feature Transform) is a feature extraction algorithm for computer vision, which is used to detect and describe local features in images. In essence, it searches for key points (feature points) in different scale spaces and calculates the direction of the key points.

[0118] In the process of extracting target feature points according to the SIFT feature method, the main steps are extreme value detection in scale space, key point positioning, direction allocation, key point descriptor, etc. The matching method of the SIFT matching algorithm is described as follows:

[0119] For example, for the video data from any two cameras, it generates descriptors for two images A and B (k1*128 dimensions and k2*128 dimensions respectively), and then matches the descriptors of each scale (all scales) in the two images. Matching 128 dimensions means that the two feature points match. After the SIFT feature vectors of the two images are generated, the next step of this embodiment is to use the Euclidean distance of the key point feature vectors as a similarity determination measure for the key points in the two images. Take a key point in image 1 and find the first two key points with the closest Euclidean distance to image 2. Among these two key points, if the closest distance divided by the next closest distance is less than a certain ratio threshold, then accept this pair of matching points. Lowering this ratio threshold will reduce the number of SIFT matching points, but it will be more stable. In the method of nearest neighbor distance and next nearest neighbor distance, a distance ratio ratio less than a certain threshold is considered to be a correct match. The recommended ratio value principle is as follows:

[0120] ratio = 0.4, for matching with high accuracy requirements;

[0121] ratio=0.6, for matches requiring a relatively large number of matching points;

[0122] ratio=0.5. In general, the following principle can be followed: when the nearest neighbor distance is <200, ratio=0.6, otherwise ratio=0.4. The value strategy of ratio can sort out the wrong matching points. Finally, the first target coordinates corresponding to the target feature point matched by the SIFT matching algorithm are output.

[0123] In some optional implementations, step S104, based on the intrinsic reference data and the extrinsic reference data of any two cameras, matching the camera data of any two cameras by a second matching method to obtain the second target coordinates, includes:

[0124] Step c1, based on the intrinsic parameter data and extrinsic parameter data of any two cameras, the camera data of any two cameras are matched by epipolar matching to obtain the second target coordinates.

[0125] Specifically, after obtaining the intrinsic parameter data (intrinsic parameter matrix and distortion coefficient) of the two cameras and the extrinsic parameter data (relative rotation and translation matrix) between any two cameras, the image data of any two cameras are matched by epipolar lines, which is equivalent to binocular parallel correction. The images of the same object taken by two cameras are processed in a certain way so that the two images are the same size and horizontally in a straight line.

[0126] The epipolar matching method is performed on the camera coordinate system corresponding to the image. During epipolar correction, after conversion to the camera coordinate system, the camera coordinate systems corresponding to the left and right images are respectively corrected for parallel epipolar lines through the rotation matrices R1 and R2. The steps are as follows: the source image pixel coordinate system is converted into the camera coordinate system through the intrinsic parameter matrix (compared with the image physical coordinate system, there are more scaling and Z axis), parallel epipolar correction is performed through the rotation matrices R1 and R2, and then the camera coordinates of the image are corrected through the distortion coefficient. After correction, the camera coordinate system is converted into the image pixel coordinate system through the intrinsic parameter matrix, and the pixel values ​​of the source image coordinates are assigned to the new image coordinates, that is, the camera data of any two cameras are matched through the epipolar matching method to obtain the second target coordinates.

[0127] The data calibration method for the extended reality device in this embodiment, through the first matching method and the second matching method mentioned above, can not only meet the calibration point requirements of the XR device external parameter calibration, but also enable the user to correct the camera and IMU calibration data without perception, thereby ensuring the positioning and tracking effect of the XR device, and the calibration process is relatively simple.

[0128] In this embodiment, a data calibration device for an extended reality device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" may be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0129] This embodiment provides a data calibration device for an extended reality device, such as Figure 4 As shown, including:

[0130] The device data acquisition module 41 is used to acquire the camera data, internal parameter data, external parameter data, and IMU raw data of the inertial sensor of any two cameras of the target XR device, the calibration trigger data and memory usage data of the target XR device,

[0131] The device data calculation module 42 is used to calculate the similarity between the actual running trajectory of the target XR device and the preset running trajectory based on the camera data and the IMU raw data when the calibration trigger data and the memory occupancy data meet the preset conditions;

[0132] A first coordinate matching module 43, configured to extract a target feature point from the camera data by a first matching method and match a first target coordinate corresponding to the target feature point when the similarity is less than or equal to a first preset threshold;

[0133] A second coordinate matching module 44 is used to match the camera data of any two cameras through a second matching method based on the internal reference data and external reference data of any two cameras to obtain a second target coordinate;

[0134] The calibration parameter output module 45 is used to optimize the external parameter data of any two cameras in an iterative manner based on the first target coordinates until the optimization result is less than or equal to the second preset threshold, and output the calibration parameters of the target XR device.

[0135] In some optional implementations, the device data calculation module 42 includes:

[0136] A first data acquisition submodule, used to acquire a current falling distance and a current falling acceleration of a target XR device;

[0137] A second data acquisition submodule, configured to acquire a current drop acceleration of the target XR device when the current drop distance of the target XR device is greater than or equal to a third preset threshold;

[0138] A third data acquisition submodule is used to acquire the current state and memory usage data of the target XR device when the current drop acceleration of the target XR device is equal to the free fall acceleration;

[0139] The fourth data acquisition submodule is used to obtain the current occlusion status of the target XR device when the current state of the target XR device is in the main interface state and the memory usage data is less than or equal to the fourth preset threshold;

[0140] The preset condition determination submodule is used to determine that when the target XR device is not currently blocked, the target XR device meets the preset condition.

[0141] In some optional implementations, the device data calculation module 42,

[0142] A fifth data acquisition submodule is used to obtain the current usage time of the target XR device;

[0143] A sixth data acquisition submodule, configured to acquire the current state and memory usage data of the target XR device when the current usage duration of the target XR device is greater than or equal to a preset period;

[0144] A seventh data acquisition submodule is used to obtain the current occlusion status of the target XR device when the current state of the target XR device is in the main interface state and the memory usage data is less than or equal to the fourth preset threshold;

[0145] The preset condition determination submodule is used to determine that when the target XR device is not currently blocked, the target XR device meets the preset condition.

[0146] In some optional implementations, the device data calculation module 42 includes:

[0147] The running trajectory calculation submodule is used to calculate the actual running trajectory of the target XR device through the visual SLAM algorithm based on the camera data and IMU raw data;

[0148] The running trajectory acquisition submodule is used to obtain the preset running trajectory of the target XR device;

[0149] The eighth data acquisition submodule is used to compare the actual running trajectory with the preset running trajectory to obtain the similarity between the two.

[0150] In some optional implementations, the first coordinate matching module 43 includes:

[0151] The target feature matching submodule is used to extract target feature points from the camera data using the SIFT feature method;

[0152] The first coordinate matching submodule is used to match the first target coordinates corresponding to the target feature point by using a SIFT matching algorithm.

[0153] In some optional implementations, the first coordinate matching module 43 includes:

[0154] The second coordinate matching submodule is used to match the camera data of any two cameras through epipolar matching based on the intrinsic parameter data and extrinsic parameter data of any two cameras to obtain the second target coordinates.

[0155] In some optional implementations, the data calibration apparatus for an extended reality device further includes:

[0156] The calibration shutdown module is used to not trigger the target XR device to trigger the data calibration action when the calibration trigger data and memory usage data do not meet the preset conditions.

[0157] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0158] The data calibration device of the extended reality device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0159] An embodiment of the present invention further provides a computer device having the data calibration device of the above-mentioned extended reality device.

[0160] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 5 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0161] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0162] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0163] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0164] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0165] The computer device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means, and FIG. X takes the bus connection as an example.

[0166] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0167] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0168] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A data calibration method for an extended reality device, characterized in that: The method comprises: Obtain the camera data, internal parameter data, external parameter data, and IMU raw data of any two cameras of the target XR device, as well as the calibration trigger data and memory usage data of the target XR device; When the calibration trigger data and the memory occupancy data meet preset conditions, the similarity between the actual running trajectory of the target XR device and the preset running trajectory is calculated based on the camera data and the IMU raw data; When the similarity is less than or equal to a first preset threshold, extracting a target feature point from the camera data by a first matching method, and matching a first target coordinate corresponding to the target feature point; Based on the intrinsic reference data and the extrinsic reference data of the any two cameras, matching the camera data of the any two cameras in a second matching manner to obtain second target coordinates; Based on the first target coordinates and the second target coordinates, the external parameter data of the arbitrary two cameras are optimized in an iterative manner until the optimization result is less than or equal to a second preset threshold, and the calibration parameters of the target XR device are output.

2. The method according to claim 1, characterized in that The calibration trigger data and the memory occupancy data meet preset conditions, including: Obtaining the current falling distance and current falling acceleration of the target XR device; When the current falling distance of the target XR device is greater than or equal to a third preset threshold, obtaining a current falling acceleration of the target XR device; When the current falling acceleration of the target XR device is equal to the free fall acceleration, obtaining the current state and memory usage data of the target XR device; When the current state of the target XR device is in the main interface state, and the memory usage data is less than or equal to a fourth preset threshold, obtaining a current occlusion condition of the target XR device; When the target XR device is not currently blocked, the target XR device meets the preset condition.

3. The method according to claim 1, characterized in that The calibration trigger data and the memory occupancy data meet a preset condition, further comprising: Obtain the current usage time of the target XR device; When the current usage duration of the target XR device is greater than or equal to a preset period, obtaining the current state and memory usage data of the target XR device; When the current state of the target XR device is in the main interface state, and the memory usage data is less than or equal to a fourth preset threshold, obtaining a current occlusion condition of the target XR device; When the target XR device is not currently blocked, the target XR device meets the preset condition.

4. The method according to claim 1, characterized in that Based on the camera data and the IMU raw data, calculating the similarity between the actual running trajectory of the target XR device and the preset running trajectory, including: Based on the camera data and the IMU raw data, the actual running trajectory of the target XR device is calculated by a visual SLAM algorithm; Obtaining a preset running trajectory of the target XR device; The actual running trajectory is compared with the preset running trajectory to obtain the similarity between the two.

5. The method according to claim 1, characterized in that Extracting target feature points from the camera data by a first matching method, and matching first target coordinates corresponding to the target feature points, including: Extracting target feature points from the camera data by using SIFT feature method; The first target coordinates corresponding to the target feature point are matched using the SIFT matching algorithm.

6. The method according to claim 1, characterized in that Matching the camera data of the arbitrary two cameras in a second matching manner to obtain a second target coordinate includes: Based on the intrinsic parameter data and the extrinsic parameter data of the any two cameras, the camera data of the any two cameras are matched by epipolar matching to obtain second target coordinates.

7. The method according to claim 1, characterized in that Also includes: When the calibration trigger data and the memory usage data do not meet the preset conditions, the target XR device is not triggered to trigger the execution of the data calibration action.

8. A data calibration device for an extended reality device, characterized in that: The device comprises: The device data acquisition module is used to obtain the camera data, internal parameter data, external parameter data, and IMU raw data of any two cameras of the target XR device, the calibration trigger data and memory usage data of the target XR device, A device data calculation module, configured to calculate the similarity between the actual running trajectory of the target XR device and the preset running trajectory based on the camera data and the IMU raw data when the calibration trigger data and the memory occupancy data meet the preset conditions; A first coordinate matching module, configured to extract a target feature point from the camera data by a first matching method and match a first target coordinate corresponding to the target feature point when the similarity is less than or equal to a first preset threshold; A second coordinate matching module, configured to match the video data of the any two cameras in a second matching manner based on the internal reference data and the external reference data of the any two cameras to obtain a second target coordinate; A calibration parameter output module is used to optimize the external parameter data of the arbitrary two cameras in an iterative manner based on the first target coordinates until the optimization result is less than or equal to a second preset threshold, and output the calibration parameters of the target XR device.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data calibration method of the extended reality device according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data calibration method for an extended reality device according to any one of claims 1 to 7.

Images