Calibration system, calibration method and non-transient computer readable storage medium
By utilizing the data of the tracking camera and dynamic sensor, the correction parameters are calculated, and the problem of inaccurate tracking of the handheld controller is solved, and the high-precision interaction and dynamic sensor correction in immersive experience are realized. .
Patent Information
- Application Number
- CN202210041512.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-12-07
- Filing Date
- 2022-01-14
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-01-14
AI Technical Summary
The prior art is difficult to accurately and instantly track the movement trajectory of the handheld controller, resulting in inaccurate interaction of virtual objects in the immersive experience, and the distortion of the dynamic sensor is gradually accumulated in repeated corrections.
Using a correction system, the camera captures the streaming images of the handheld controller and the mobile data generated by the built-in dynamic sensor, calculates the correction parameters, and transmits them to the handheld controller to correct their motion data, and uses external perspectives to correct them to avoid distortion. accumulation.
Accurate correction of handheld controller motion data is achieved, interactive accuracy in immersive experience is improved, dynamic sensor distortion accumulation is avoided, and more objective correction results are ensured.
Smart Images

Figure CN114764849B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a calibration system and a calibration method, and in particular to a calibration system and a calibration method for calibrating a handheld controller in an immersive system. Background Art
[0002] Various virtual reality (VR), augmented reality (AR), substitute reality (SR) and / or mixed reality (MR) devices are designed to provide users with a realistic immersive experience. When a user wears a corresponding head-mounted device, the immersive content displayed by the head-mounted device will cover the user's field of view. The so-called immersive content can display a specific virtual background and multiple virtual objects in a specific immersive scene.
[0003] In some applications, users can operate handheld controllers as an input method. In order to provide users with a realistic immersive experience, the immersive system must be able to track the movement trajectory of the handheld controller and render and display the handheld controller at the corresponding position in the immersive content. In this case, the user can manipulate the handheld controller to interact with other virtual objects in the immersive scene. For example, the user can swing a virtual sword in the immersive scene to fight against the virtual monster in the immersive scene. Therefore, it is very important to be able to accurately and instantly track the movement trajectory of the handheld controller. Summary of the invention
[0004] One embodiment of the present disclosure discloses a calibration system for calibrating a handheld controller, the calibration system comprising a tracking camera, a display, and a processing unit. The tracking camera is used to capture multiple streaming images related to the handheld controller. The display is used to display a calibration test instruction for moving the handheld controller along a predetermined path. The processing unit is communicatively connected to the tracking camera and the display, and the processing unit is used to: receive a first movement data generated by the handheld controller when the handheld controller moves along the predetermined path; receive the streaming images captured by the tracking camera when the handheld controller moves along the predetermined path; calculate a second movement data based on the streaming images; calculate a plurality of calibration parameters by comparing the first movement data with the second movement data; and transmit the calibration parameters to the handheld controller, wherein the handheld controller uses the calibration parameters to generate a third movement data.
[0005] In some embodiments of the present disclosure, the first movement data includes multiple first angle rotation amounts relative to three directional axes when the handheld controller moves along the predetermined path, the second movement data includes multiple second angle rotation amounts relative to the three directional axes when the handheld controller moves along the predetermined path, and the correction parameters include a first correction matrix for aligning the first angle rotation amounts with the second angle rotation amounts.
[0006] In some embodiments of the present disclosure, the first movement data is generated based on an inertial measurement unit or a gyroscope sensor built into the handheld controller. The inertial measurement unit or the gyroscope sensor is used to detect three angular velocities relative to the three direction axes. The first angular rotation amounts in the first movement data are determined based on the integral values of the angular velocities over time.
[0007] In some embodiments of the present disclosure, each of the streaming images includes a characteristic pattern disposed on the handheld controller or an outline of the handheld controller, and the processing unit calculates the second angle rotation amounts in the second motion data by tracking the characteristic pattern or the outline in the streaming images.
[0008] In some embodiments of the present disclosure, the first movement data includes multiple first displacements relative to three directional axes when the handheld controller moves along the predetermined path, the second movement data includes multiple second displacements relative to the three directional axes when the handheld controller moves along the predetermined path, and the correction parameters include a second correction matrix for aligning the first displacements with the second displacements.
[0009] In some embodiments of the present disclosure, the first movement data is generated based on an inertial measurement unit or an accelerometer built into the handheld controller, and the inertial measurement unit or the accelerometer is used to detect three accelerations relative to the three directional axes. The first displacements in the first movement data are determined based on the double integral values of the accelerations over time.
[0010] In some embodiments of the present disclosure, each of the streaming images includes a characteristic pattern disposed on the handheld controller or an outline of the handheld controller, and the processing unit calculates the second displacements in the second motion data by tracking the characteristic pattern or the outline in the streaming images.
[0011] In some embodiments of the present disclosure, the handheld controller is movably positioned within a field of view of the tracking camera, the tracking camera and the handheld controller are separately disposed, and the streaming images display a position and a rotation of the handheld controller in the field of view of the tracking camera.
[0012] In some embodiments of the present disclosure, the tracking camera is disposed on a head-mounted display or at a fixed position.
[0013] Another embodiment of the present disclosure discloses a calibration method, comprising: when a handheld controller moves along a predetermined path, a dynamic sensor built into the handheld controller generates a first motion data; when the handheld controller moves along the predetermined path, a tracking camera captures a plurality of streaming images involving the handheld controller; a second motion data is calculated based on the streaming images; a plurality of calibration parameters are calculated by comparing the first motion data with the second motion data; and the calibration parameters are transmitted to the handheld controller, wherein the handheld controller uses the calibration parameters to generate a third motion data.
[0014] In some embodiments of the present disclosure, the first movement data includes multiple first angle rotation amounts relative to three directional axes when the handheld controller moves along the predetermined path, and the second movement data includes multiple second angle rotation amounts relative to the three directional axes when the handheld controller moves along the predetermined path. The step of calculating the correction parameters includes: calculating a first correction matrix to align the first angle rotation amounts with the second angle rotation amounts.
[0015] In some embodiments of the present disclosure, the first movement data is generated based on an inertial measurement unit or a gyroscope sensor built into the handheld controller. The inertial measurement unit or the gyroscope sensor is used to detect three angular velocities relative to the three direction axes. The first angular rotation amounts in the first movement data are determined based on the integral values of the angular velocities over time.
[0016] In some embodiments of the present disclosure, each of the streaming images includes a characteristic pattern disposed on the handheld controller or an outline of the handheld controller, and the second angle rotation amounts in the second motion data are calculated by tracking the characteristic pattern or the outline in the streaming images.
[0017] In some embodiments of the present disclosure, the first movement data includes a plurality of first displacements relative to the three directional axes when the handheld controller moves along the predetermined path, and the second movement data includes a plurality of second displacements relative to the three directional axes when the handheld controller moves along the predetermined path. The step of calculating the correction parameters includes: calculating a second correction matrix for aligning the first displacements with the second displacements.
[0018] In some embodiments of the present disclosure, the first movement data is generated based on an inertial measurement unit or an accelerometer built into the handheld controller, and the inertial measurement unit or the accelerometer is used to detect three accelerations relative to the three directional axes. The first displacements in the first movement data are determined based on the double integral values of the accelerations over time.
[0019] In some embodiments of the present disclosure, each of the streaming images includes a characteristic pattern disposed on the handheld controller or an outline of the handheld controller, and the second displacements in the second motion data are calculated by tracking the characteristic pattern or the outline in the streaming images.
[0020] In some embodiments of the present disclosure, the handheld controller is movably positioned within a field of view of the tracking camera, the tracking camera and the handheld controller are separately disposed, and the streaming images display a position and a rotation of the handheld controller in the field of view of the tracking camera.
[0021] Another embodiment of the present disclosure discloses a non-transitory computer-readable medium comprising at least one instruction program, which is executed by a processing unit to implement a calibration method, which includes: when a handheld controller moves along a predetermined path, a dynamic sensor built into the handheld controller generates a first motion data; when the handheld controller moves along the predetermined path, a tracking camera captures a plurality of streaming images related to the handheld controller; a second motion data is calculated based on the streaming images; a plurality of calibration parameters are calculated by comparing the first motion data with the second motion data; and the calibration parameters are transmitted to the handheld controller, wherein the handheld controller uses the calibration parameters to generate a third motion data.
[0022] In some embodiments of the present disclosure, the first movement data includes multiple first angle rotation amounts relative to three directional axes when the handheld controller moves along the predetermined path, and the second movement data includes multiple second angle rotation amounts relative to the three directional axes when the handheld controller moves along the predetermined path. The step of calculating the correction parameters includes: calculating a first correction matrix to align the first angle rotation amounts with the second angle rotation amounts.
[0023] In some embodiments of the present disclosure, the first movement data includes a plurality of first displacements relative to the three directional axes when the handheld controller moves along the predetermined path, and the second movement data includes a plurality of second displacements relative to the three directional axes when the handheld controller moves along the predetermined path. The step of calculating the correction parameters includes: calculating a second correction matrix for aligning the first displacements with the second displacements.
[0024] Thereby, the calibration of the motion sensor built into the handheld controller can be performed according to the tracking camera of the external perspective, the calibration result is more objective, and the distortion of the motion sensor will not be gradually accumulated with repeated calibration. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 A schematic diagram showing an immersive system and a correction system used in conjunction therewith according to an embodiment of the present disclosure is shown;
[0026] Figure 2 Shows Figure 1 A schematic diagram of the immersion system and the calibration system during a calibration procedure;
[0027] Figure 3 A schematic diagram showing the appearance of a handheld controller according to some embodiments of the present disclosure is shown;
[0028] Figure 4 A method flow chart of a correction method according to some embodiments of the present disclosure is shown;
[0029] Figure 5 A schematic diagram showing a handheld controller moving along a predetermined path in an illustrative example is shown;
[0030] Figure 6 shows that when the handheld controller is moved along Figure 5 A schematic diagram of a plurality of first angular velocities relative to three direction axes sensed by a dynamic sensor built into the handheld controller when the handheld controller moves along a predetermined path in FIG.
[0031] Figure 7 Shows Figure 1 A schematic diagram of the immersion system and the calibration system after the calibration procedure is completed; and
[0032] Figure 8 A schematic diagram showing movement of a handheld controller along a predetermined path in one illustrative example is shown.
[0033] Explanation of symbols:
[0034] 100: Immersive System
[0035] 120: Head mounted display device
[0036] 122,260: Display
[0037] 140:Handheld Controller
[0038] 142: Motion sensor
[0039] 200: Calibration system
[0040] 220:Tracking Camera
[0041] 222: The first tracking camera
[0042] 224: Second tracking camera
[0043] 240: Processing unit
[0044] 242:Graphing Calculator
[0045] 244:Calibration Calculator
[0046] 300: Calibration method
[0047] S310, S321, S322, S330, S340, S350: Steps
[0048] S360, S370: Steps
[0049] SIMG: Streaming Video
[0050] MD1: First Mobile Data
[0051] MD2: Second Mobile Data
[0052] MD3: Third Mobile Data
[0053] CP: Correction Parameter
[0054] INST: Calibration test instruction
[0055] RT1,RT2: scheduled path DETAILED DESCRIPTION
[0056] The following disclosure provides many different embodiments or examples for implementing different features of the present disclosure. The components and configurations in the specific examples are used to simplify the present disclosure in the following discussion. Any examples discussed are used for illustrative purposes only and do not limit the scope and significance of the present disclosure or its examples in any way. Where appropriate, the same reference numerals are used between the drawings and in the corresponding text descriptions to represent the same or similar components.
[0057] See also Figure 1 , which shows a schematic diagram of an immersive system 100 and a correction system 200 used in conjunction therewith according to an embodiment of the present disclosure. Figure 1 As shown, the immersive system 100 includes a head mounted display device 120 and a handheld controller 140 located in a spatial area SA. For example, the spatial area SA may be Figure 1 The bedroom or conference room in the real world is shown, but the present disclosure is not limited thereto. In some other embodiments, the space area SA may also be a specific area in an outdoor space (not shown in the figure).
[0058] In some embodiments, the head mounted display device 120 can be worn on the user's head, and the handheld controller 140 can be held in one of the user's hands. In some embodiments, the head mounted display device 120 can provide immersive content to the user, such as virtual reality (VR), augmented reality (AR), substitute reality (SR) and / or mixed reality (MR) scenes. In order to provide immersive content to the user, the immersive system 100 is used to track the head mounted display device 120 to detect the position and rotation of the user's movement trajectory. There are many methods (such as outside-in tracking algorithms based on optical signals or ultrasonic signals, inside-out tracking algorithms based on image recognition) that can be used to track the head mounted display device 120. Those skilled in the art understand how to track the head mounted display device 120, so the tracking method of the head mounted display device 120 will not be further discussed in the subsequent paragraphs.
[0059] In order to provide immersive content to the user, the immersive system 100 also needs to track the handheld controller 140 to detect the position and rotation of the user's gesture and its movement trajectory. The user can manipulate the handheld controller 140 to perform various gestures (such as slapping, stabbing, waving, tapping, or other gestures). In some embodiments, the tracking of the movement trajectory of the handheld controller 140 is performed based on the first movement data generated by the built-in motion sensor in the handheld controller 140. Please refer to Figure 2 , which shows Figure 1 FIG. 1 is a schematic diagram of the immersive system 100 and the calibration system 200 during a calibration procedure.
[0060] like Figure 2 As shown, the handheld controller 140 may include a motion sensor 142, and the motion sensor 142 may detect the first motion data MD1 when the handheld controller 140 moves. In some embodiments, the motion sensor 142 may include at least one of an inertial measurement unit (IMU), a gyroscope sensor, and an accelerometer built into the handheld controller 140.
[0061] In some embodiments, when the handheld controller 140 moves in the spatial area SA, the inertial measurement unit (or gyroscope sensor) in the dynamic sensor 142 can be used to sense multiple first angular velocities relative to the three directional axes. The multiple first angular velocities of the three directional axes can be integrated with time respectively, and the integrated values can be used to determine multiple first angular rotation amounts relative to the three directional axes. In this example, the first movement data MD1 can include multiple first angular rotation amounts relative to the three directional axes.
[0062] In some embodiments, when the handheld controller 140 moves in the spatial area SA, the inertial measurement unit (or accelerometer) in the dynamic sensor 142 can be used to sense multiple first accelerations relative to the three directional axes. The multiple first accelerations of the three directional axes can be double-integrated with respect to time, and the multiple first displacements relative to the three directional axes can be determined by the double integral values. In this example, the first movement data MD1 can include multiple first displacements relative to the three directional axes.
[0063] In order to track the handheld controller 140, it is very important to ensure the correctness and accuracy of the first movement data MD1 detected by the motion sensor 142. Generally speaking, due to manufacturing errors, the first movement data MD1 detected by the motion sensor 142 may have a certain degree of detection distortion. Therefore, the motion sensor 142 needs to be calibrated to ensure that the motion sensor 142 can work properly.
[0064] In some cases, the calibration of the motion sensor is based on the motion data generated by the motion sensor itself. For example, the motion sensor generates motion data repeatedly in multiple different rounds, and then the motion sensor calibrates itself based on the motion data generated in multiple rounds. However, if the motion data generated by the motion sensor is already inaccurate from the beginning, the motion sensor calibration performed in this way will not achieve the desired effect. In some extreme cases, the distortion of the motion sensor will even gradually accumulate during the repeated calibration, and the above calibration method may even make the motion sensor more inaccurate.
[0065] In some embodiments, the calibration system 200 calibrates the motion sensor 142 built into the handheld controller 140 with reference to another motion data generated by an external source (ie, other than the motion sensor 142 itself), so that the motion sensor 142 can be properly calibrated according to the external reference standard.
[0066] like Figure 1 as well as Figure 2In the illustrated embodiment, the calibration system 200 includes a tracking camera 220 and a processing unit 240. The tracking camera 220 is used to capture a plurality of streaming images SIMG related to the movement trajectory of the handheld controller 140. The handheld controller 140 can move relatively within the field of view of the tracking camera 220. The tracking camera 220 is separated from the handheld controller 140, so that the handheld controller 140 can be observed from an external perspective. Figure 1 As shown, the tracking camera 220 includes a first tracking camera 222, which is integrated with the head mounted display device 120 and is disposed on the front surface of the head mounted display device 120. Thus, the first tracking camera 222 can observe the handheld controller 140 according to the viewing angle of the head mounted display device 120. Figure 1 As shown in FIG. 1 , the tracking camera 220 includes a second tracking camera 224, which is a stand-alone camera installed at a fixed position. In this example, the second tracking camera 224 can observe the handheld controller 140 from a top-down perspective. The tracking camera 220 in the calibration system 200 can be implemented by at least one of the first tracking camera 222 and the second tracking camera 224. Figure 1 In some other embodiments, the tracking camera 220 may also be implemented using other equivalent camera configurations, wherein the camera configuration includes at least one tracking camera independently disposed outside the handheld controller 140 .
[0067] The plurality of streaming images SIMG captured by the tracking camera 220 show the position and rotation of the handheld controller 140 in the viewing angle of the tracking camera 220. Figure 2 As shown, the processing unit 240 is used to receive a plurality of stream images SIMG captured by the tracking camera 220. Figure 2 As shown, the processing unit 240 includes a graphic calculator 242 and a calibration calculator 244. The graphic calculator 242 in the processing unit 240 is used to analyze a plurality of streaming images SIMG to generate second mobile data MD2.
[0068] Please also read Figure 3 , which shows a schematic diagram of the appearance of a handheld controller 140 according to some embodiments of the present disclosure. Figure 3In some embodiments, the handheld controller 140 includes a built-in motion sensor 142 and a characteristic pattern 144 (which may be an icon, a luminous pattern, or a color pattern) disposed on the handheld controller 140. The graphic calculator 242 is used to identify and track the characteristic pattern 144 or the outline of the handheld controller 140 in multiple frames of the streaming image SIMG, and then calculate the second movement data MD2. For example, the graphic calculator 242 can calculate multiple second angular rotation amounts (and / or multiple second displacement amounts relative to the three directional axes) when the handheld controller 140 moves along a predetermined path by comparing the positions of the same characteristic pattern 144 appearing in different frames.
[0069] The correction calculator 244 is used to compare the first motion data MD1 (generated by the handheld controller 140) and the second motion data MD2 (derived from the observation result of the tracking camera 220) to generate a correction parameter CP to correct the potential distortion that may exist in the first motion data MD1. Further details on calculating the correction parameter CP will be described in subsequent paragraphs.
[0070] In some embodiments, the processing unit 240 may be a processor, a central processing unit, a graphics processor, a tensor processing unit (TPU), or an application specific integrated circuit (ASIC) disposed in a computer or a smart phone, and disposed separately outside the head mounted display device 120. In other embodiments, the processing unit 240 may also be a processor, a central processing unit, a graphics processor, a tensor processing unit, or an application specific integrated circuit integrated in the head mounted display device 120. In some embodiments, the graphics calculator 242 and the correction calculator 244 may be implemented by software instructions executed by the processing unit 240, or by an application specific integrated circuit in the processing unit 240.
[0071] Please also read Figure 4 , which shows a method flow chart of a correction method 300 according to some embodiments of the present disclosure. Figure 4 The calibration method 300 shown can be performed by Figure 1 and Figure 2 The immersive system 100 and the correction system 200 are implemented. Figure 1 , Figure 2 and Figure 4In the illustrated embodiment, when the calibration procedure starts, step S310 is executed, and the processing unit 240 generates a calibration test indication INST, which is transmitted to the display 122 in the head mounted display device 120 or to the single display 260 (such as the single display 260) which is separately provided from the head mounted display device 120. Figure 1 and Figure 2 ).
[0072] If the user is not wearing the head mounted display device 120, the calibration test indication INST may be transmitted and displayed on the single display 260, which may be a TV, a smart TV, a smart display or other similar devices that can communicate with the processing unit 240. In this case, it is convenient for the user to perform the calibration procedure when not wearing the head mounted display device 120.
[0073] The calibration test instruction INST is used to guide the user to move the handheld controller 140 along a predetermined path, so as to improve the efficiency of calibration. For example, the calibration test instruction INST may suggest the user to perform a pitch gesture (i.e., rotation around a side-to-side axis), a roll gesture (i.e., rotation around a forward-to-backward axis), or a yaw gesture (i.e., rotation around a vertical axis). The calibration test instruction INST is displayed by the display 122 (or the display 260), and the user may move the handheld controller 140 accordingly according to the predetermined path required by the calibration test instruction INST. In some embodiments, the calibration test instruction INST may include text instructions, graphic instructions, or animation guidance. In addition, the calibration test instruction INST may also include voice instructions played along with the above-mentioned visual instructions.
[0074] When the handheld controller 140 moves along the predetermined path, steps S321 and S322 are synchronously executed. In step S321, the dynamic sensor built into the handheld controller 140 generates first movement data MD1. In step S322, the tracking camera 220 captures a streaming image SIMG related to the handheld controller 140.
[0075] In step S330, the processing unit 240 receives the first movement data MD1 from the handheld controller 140. In step S340, the processing unit 240 receives the streaming image SIMG from the tracking camera 220. In step S350, the graphic calculator 242 recognizes and tracks the feature pattern 144 (refer to Figure 3 ) or the outline of the handheld controller 140, and then calculate the second movement data MD2.
[0076] In step S360 , the calibration calculator 244 in the processing unit 240 compares the first mobile data MD1 and the second mobile data MD2 to generate a calibration parameter CP.
[0077] In an illustrative example, when the first mobile data MD1 includes a plurality of first angle rotations relative to three directional axes, and the second mobile data MD2 includes a plurality of second angle rotations relative to three directional axes, the correction parameter CP calculated by the correction calculator 244 in step S360 includes a first correction matrix, which is used to align the plurality of first angle rotations with the plurality of second angle rotations. Figure 5 as well as Figure 6 , Figure 5 FIG. 1 is a schematic diagram showing a handheld controller 140 moving along a predetermined path RT1 in an illustrative example. Figure 6 As shown in FIG. 1 , when the handheld controller 140 is moved along Figure 5 Schematic diagram of multiple first angular velocities relative to three direction axes sensed by the dynamic sensor 142 built into the handheld controller 140 when moving along the predetermined path RT1 in FIG.
[0078] like Figure 5 As shown in FIG. 1 , the handheld controller 140 is moving along a downward pitch gesture. In other words, the front end of the handheld controller 140 is rotating toward the floor. Figure 6 In the example, corresponding to the moving trajectory MOV of the handheld controller 140, the inertial measurement unit or the gyroscope sensor is used to sense the three angular velocities relative to the three direction axes. Figure 6 As shown, the angular velocity GYROx along the X-axis remains zero; the angular velocity GYROy along the Y-axis and the angular velocity GYROz along the Z-axis represent downward rotation. The angular velocity GYROx can be integrated over time to obtain a first angle rotation amount R_Ximu along the X-axis; the angular velocity GYROy can be integrated over time to obtain another first angle rotation amount R_Yimu along the Y-axis; the angular velocity GYROz can be integrated over time to obtain another first angle rotation amount R_Zimu along the Z-axis. The first mobile data MD1 includes the above three first angle rotation amounts R_Ximu, R_Yimu and R_Zimu.
[0079] On the other hand, the graphic calculator 242 is used to generate second movement data MD2 according to the streaming image SIMG. The second movement data MD2 includes three second rotation angles R_Xcam, R_Ycam and R_Zcam relative to three direction axes when the handheld controller 140 moves along the predetermined path RT1.
[0080] In some embodiments, the calibration parameters CP include a first calibration matrix CM1, and the first calibration matrix CM1 is used to align the first angle rotations R_Ximu, R_Yimu, and R_Zimu with the second angle rotations R_Xcam, R_Ycam, and R_Zcam. In some embodiments, the calibration calculator 244 calculates the first calibration matrix CM1 according to the following formula (1):
[0081]
[0082] In formula (1), The first correction matrix CM1 is used to perform matrix conversion on the first angle rotation amount (R_Ximu, R_Yimu, R_Zimu) and align with the second angle rotation amount (R_Xcam, R_Ycam, R_Zcam) as a reference.
[0083] Assume that the handheld controller 140 is moved along Figure 5 The predetermined path RT1 in the moving, the second angle rotation amount (R_Xcam, R_Ycam, R_Zcam) is equal to (0, -90, -90). Therefore, formula (1) can be updated to the following formula (2):
[0084]
[0085] In formula (2), the first angle rotation (R_Ximu, R_Yimu, R_Zimu) is known, and the second angle rotation (R_Xcam, R_Ycam, R_Zcam) is also known to be (0, -90, -90), so the calibration calculator 244 can calculate the value of the first calibration matrix CM1 accordingly. As described above, the calculated first calibration matrix CM1 can be used to calibrate the first angle rotation (R_Ximu, R_Yimu, R_Zimu) generated by the dynamic sensor 142, so that the calibration result can be aligned with the second angle rotation (R_Xcam, R_Ycam, R_Zcam) obtained based on the streaming image SIMG captured by the tracking camera 220. In this way, the calibration method 300 can calibrate the first motion data MD1 generated by the dynamic sensor 142 based on other reference signals (i.e., the second motion data MD2) other than the first motion data MD1 generated by the handheld controller 140 itself. In this case, the calibration of the motion sensor 142 built into the handheld controller 140 can be more objective, and the distortion of the motion sensor 142 will not be gradually accumulated as the calibration is repeated.
[0086] This disclosure document is not limited to Figure 5The downward pitch gesture is shown as the predetermined path RT1. In other examples, the calibration test indication INST may also suggest the user to adopt other predetermined paths, such as a rightward swing gesture.
[0087] Assume that the handheld controller 140 moves along the rightward yaw gesture, and the second angle rotation amount (R_Xcam, R_Ycam, R_Zcam) is equal to (90, 0, 90). Therefore, formula (1) can be updated to the following formula (3):
[0088]
[0089] In formula (3), the first angle rotation amount (R_Ximu, R_Yimu, R_Zimu) is known, and the second angle rotation amount (R_Xcam, R_Ycam, R_Zcam) is also known to be (90, 0, 90), so the correction calculator 244 can calculate the value of the first correction matrix CM1 accordingly.
[0090] In some embodiments, the first correction matrix CM1 may be calculated for each of the different predetermined paths, and the results of the respective calculations may be averaged to serve as the correction parameter CP.
[0091] After calculating the calibration parameter CP in step S360 , step S370 may be performed. In step S370 , the processing unit 240 returns the calibration parameter CP to the handheld controller 140 , and the handheld controller 140 may use the calibration parameter CP to calibrate the first motion data MD1 sensed by the motion sensor 142 .
[0092] Please also read Figure 7 , which shows Figure 1 FIG. 1 is a schematic diagram of the immersive system 100 and the calibration system 200 after the calibration process is completed. Figure 7 As shown, the handheld controller 140 may generate the third mobile data MD3 according to the calibration parameter CP, and the third mobile data MD3 may be the product of the first mobile data MD1 and the calibration parameter CP (eg, the first calibration matrix CM1 ).
[0093] In the above embodiment, the first correction matrix CM1 is used to correct the first angle rotation amount in the first movement data MD1 sensed by the inertial measurement unit or the gyroscope sensor built into the handheld sensor 140. However, the present disclosure is not limited thereto.
[0094] In some embodiments, the first movement data MD1 may include a plurality of first displacements relative to three direction axes when the handheld sensor 140 moves along a predetermined path. Figure 8A schematic diagram showing the handheld controller 140 moving along a predetermined path RT2 in one illustrative example is shown.
[0095] like Figure 2 and Figure 8 In the example shown, the dynamic sensor 142 (including an inertial measurement unit or an accelerometer) built into the handheld controller 140 can be used to sense three acceleration values relative to three directional axes. The displacement of the handheld controller 140 relative to the three directional axes can be calculated based on the double integral value of the acceleration value with respect to time. In this example, the first movement data MD1 can include multiple first displacements, which are respectively determined according to the acceleration values sensed by the dynamic sensor 142.
[0096] At the same time, each of the plurality of streaming images SIMG includes a feature pattern on the handheld controller 140 or an outline of the handheld controller 140. The graphic calculator 242 in the processing unit 240 is used to track the position / size of the feature pattern on the handheld controller 140 or the outline of the handheld controller 140 in different frames of the streaming image SIMG, thereby generating a plurality of second displacements relative to the three direction axes in the second movement data MD2.
[0097] That is, the first movement data MD1 includes a plurality of first displacements relative to the three direction axes when the handheld sensor 140 moves along the predetermined path RT2. Meanwhile, the second movement data MD2 also includes a plurality of second displacements relative to the three direction axes when the handheld sensor 140 moves along the predetermined path RT2.
[0098] The correction calculator 244 is used to calculate the second correction matrix CM2 in the correction parameter CP, which is used to align the first displacement amount with the second displacement amount.
[0099] Assume that the handheld controller 140 is moved along Figure 8 The first displacements detected by the dynamic sensor 142 relative to the three direction axes are (D_Ximu, D_Yimu, D_Zimu), and the second displacements calculated based on the tracking camera 220 are (0, 20, -20). The second correction matrix CM2 can be calculated by the following formula (4):
[0100]
[0101] In formula (4), is the second correction matrix CM2, which is used to align the first displacement (D_Ximu, D_Yimu, D_Zimu) with the second displacement (0, 20, -20) as a reference.
[0102] In formula (4), the first displacement (D_Ximu, D_Yimu, D_Zimu) is known, and the second displacement is also known to be (0, 20, -20), so the correction calculator 244 can calculate the correction values DX-DZ in the second correction matrix CM2. In some embodiments, the correction parameter may include the second correction matrix CM2, and the correction parameter CP may be sent back to the handheld controller 140 to correct the displacement detected by the handheld controller.
[0103] In some embodiments, the calibration parameters CP may include both the first calibration matrix CM1 and the second calibration matrix CM2 in the above embodiments. The calibration parameters CP may be sent back to the handheld controller 140 to calibrate the angular rotation and displacement detected by the handheld controller.
[0104] Another embodiment of the present disclosure is a non-transitory computer-readable storage medium that graphically stores at least one instruction program for a processing unit (refer to Figure 1 and Figure 2 and the embodiments discussed above) are executed to implement Figure 4 A calibration method 300 is shown.
[0105] Although the present disclosure has disclosed the specific contents of the above embodiments, the above embodiments are not intended to limit the present disclosure. Without departing from the principles and spirit of the present disclosure, a person skilled in the art may adopt various substitutions and improvements. Therefore, the protection scope of the present disclosure is determined by the scope of the attached patent application.
Claims
1. A calibration system, characterized in that: The calibration system is used to calibrate a handheld controller, and the calibration system comprises: a tracking camera for capturing a plurality of streaming images related to the handheld controller; a display for displaying a calibration test instruction for moving the handheld controller along a predetermined path; a processing unit in communication with the tracking camera and the display, the processing unit being configured to: When the handheld controller moves along the predetermined path, receiving a first movement data generated by the handheld controller; When the handheld controller moves along the predetermined path, receiving the streaming images captured by the tracking camera; Calculate a second motion data according to the streaming images; calculating a plurality of calibration parameters by comparing the first motion data with the second motion data; and The calibration parameters are transmitted to the handheld controller, wherein the handheld controller uses the calibration parameters to generate a third movement data.
2. A correction system according to claim 1, wherein the first movement data includes multiple first angle rotation amounts relative to three direction axes when the handheld controller moves along the predetermined path, the second movement data includes multiple second angle rotation amounts relative to the three direction axes when the handheld controller moves along the predetermined path, and the correction parameters include a first correction matrix for aligning the first angle rotation amounts with the second angle rotation amounts.
3. The calibration system according to claim 2, wherein the first movement data is generated based on an inertial measurement unit or a gyroscope sensor built into the handheld controller, the inertial measurement unit or the gyroscope sensor is used to detect three angular velocities relative to the three direction axes, and the first angle rotation amounts in the first movement data are determined based on the integral values of the angular velocities over time.
4. The correction system according to claim 2, wherein each of the streaming images includes a characteristic pattern disposed on the handheld controller or an outline of the handheld controller, and the processing unit calculates the second angle rotation amounts in the second motion data by tracking the characteristic pattern or the outline in the streaming images.
5. A correction system according to claim 1, wherein the first movement data includes a plurality of first displacements relative to three directional axes when the handheld controller moves along the predetermined path, the second movement data includes a plurality of second displacements relative to the three directional axes when the handheld controller moves along the predetermined path, and the correction parameters include a second correction matrix for aligning the first displacements with the second displacements.
6. The calibration system according to claim 5, wherein the first movement data is generated based on an inertial measurement unit or an accelerometer built into the handheld controller, the inertial measurement unit or the accelerometer is used to detect three accelerations relative to the three directional axes, and the first displacements in the first movement data are determined based on double integral values of the accelerations over time.
7. The correction system according to claim 5, wherein each of the streaming images includes a characteristic pattern disposed on the handheld controller or an outline of the handheld controller, and the processing unit calculates the second displacements in the second motion data by tracking the characteristic pattern or the outline in the streaming images.
8. The calibration system according to claim 1, wherein the handheld controller is movably positioned in a field of view of the tracking camera, the tracking camera and the handheld controller are separately disposed, and the streaming images display a position and a rotation of the handheld controller in the field of view of the tracking camera.
9. The correction system according to claim 8, wherein the tracking camera is disposed on a head mounted display or at a fixed position.
10. A calibration method, characterized in that: The calibration method includes: Displaying a calibration test instruction to move a handheld controller along a predetermined path; When the handheld controller moves along the predetermined path, a dynamic sensor built into the handheld controller generates a first movement data; When the handheld controller moves along the predetermined path, a tracking camera captures a plurality of streaming images related to the handheld controller; Calculate a second motion data according to the streaming images; calculating a plurality of calibration parameters by comparing the first motion data with the second motion data; and The calibration parameters are transmitted to the handheld controller, wherein the handheld controller uses the calibration parameters to generate a third movement data.
11. The calibration method according to claim 10, wherein the first movement data comprises a plurality of first angular rotation amounts relative to three directional axes when the handheld controller moves along the predetermined path, the second movement data comprises a plurality of second angular rotation amounts relative to the three directional axes when the handheld controller moves along the predetermined path, and the step of calculating the calibration parameters comprises: A first correction matrix is calculated to align the first angle rotation amounts with the second angle rotation amounts.
12. The calibration method according to claim 11, wherein the first movement data is generated based on an inertial measurement unit or a gyroscope sensor built into the handheld controller, the inertial measurement unit or the gyroscope sensor is used to detect three angular velocities relative to the three direction axes, and the first angle rotation amounts in the first movement data are determined based on the integral values of the angular velocities over time.
13. The correction method according to claim 11, wherein each of the streaming images comprises a characteristic pattern disposed on the handheld controller or an outline of the handheld controller, and the second angle rotation amounts in the second motion data are calculated by tracking the characteristic pattern or the outline in the streaming images.
14. The calibration method according to claim 10, wherein the first movement data comprises a plurality of first displacements relative to three directional axes when the handheld controller moves along the predetermined path, the second movement data comprises a plurality of second displacements relative to the three directional axes when the handheld controller moves along the predetermined path, and the step of calculating the calibration parameters comprises: A second correction matrix is calculated to align the first displacements with the second displacements.
15. The calibration method according to claim 14, wherein the first movement data is generated based on an inertial measurement unit or an accelerometer built into the handheld controller, the inertial measurement unit or the accelerometer is used to detect three accelerations relative to the three direction axes, and the first displacements in the first movement data are determined based on double integral values of the accelerations over time.
16. The calibration method according to claim 14, wherein each of the streaming images comprises a characteristic pattern disposed on the handheld controller or an outline of the handheld controller, and the second displacements in the second motion data are calculated by tracking the characteristic pattern or the outline in the streaming images.
17. The calibration method according to claim 10, wherein the handheld controller is movably located in a field of view of the tracking camera, the tracking camera and the handheld controller are separately arranged, and the streaming images display a position and a rotation of the handheld controller in the field of view of the tracking camera.
18. A non-transitory computer readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores at least one instruction program executed by a processing unit to implement a calibration method, the calibration method comprising: Displaying a calibration test instruction to move a handheld controller along a predetermined path; When the handheld controller moves along the predetermined path, a dynamic sensor built into the handheld controller generates a first movement data; When the handheld controller moves along the predetermined path, a tracking camera captures a plurality of streaming images related to the handheld controller; Calculate a second motion data according to the streaming images; calculating a plurality of calibration parameters by comparing the first motion data with the second motion data; and The calibration parameters are transmitted to the handheld controller, wherein the handheld controller uses the calibration parameters to generate a third movement data.
19. The non-transitory computer-readable storage medium according to claim 18, wherein the first movement data comprises a plurality of first angle rotations relative to three directional axes when the handheld controller moves along the predetermined path, the second movement data comprises a plurality of second angle rotations relative to the three directional axes when the handheld controller moves along the predetermined path, and the step of calculating the correction parameters comprises: calculating a first correction matrix for aligning the first angle rotations with the second angle rotations.
20. The non-transitory computer-readable storage medium of claim 18, wherein the first movement data comprises a plurality of first displacements relative to three directional axes when the handheld controller moves along the predetermined path, the second movement data comprises a plurality of second displacements relative to the three directional axes when the handheld controller moves along the predetermined path, and the step of calculating the correction parameters comprises: A second correction matrix is calculated to align the first displacements with the second displacements.
