Method for calibrating virtual object and calibration device

Abstract

The embodiment of the invention provides a method for calibrating a virtual object and a calibration device. The method comprises the following steps: determining a reference direction related to a first virtual object in a virtual world; obtaining current motion data of the input device corresponding to the first virtual object from a motion detection circuit of the input device; acquiring a predetermined orientation of a motion detection circuit of the input device; determining a current posture of the first virtual object relative to the input device based on the predetermined orientation of the motion detection circuit and the current motion data of the input device; determining a calibration factor based on the reference direction and the current attitude of the first virtual object; and calibrating an object pose of the first virtual object based on the calibration factor.

Description

Technical Field

[0001] The present invention relates to a calibration mechanism, and in particular to a method and a calibration device for calibrating a virtual object. Background Art

[0002] Reference Figure 1 , which shows a schematic diagram of using an input device to interact with a virtual world. Figure 1 In the present invention, a user may use an input device 101 worn on a finger (e.g., a wearable device such as a smart ring) to interact with a virtual world, wherein the virtual world may be a virtual environment provided by a real service in which the user is immersed, such as a virtual reality (VR) world.

[0003] exist Figure 1 In the virtual world, the virtual object 102 may be rendered based on the tracked posture of the input device 101, and the virtual object 102 may be further rendered with a ray 103, wherein the user may, for example, use the ray 103 to point to a desired location in the virtual world for interaction.

[0004] However, as the user is immersed in the virtual world for a longer time, the posture tracking errors of the input device 101 may accumulate, so that the virtual object 102 and the ray 103 may not be accurately rendered.

[0005] like Figure 1 As shown, the virtual object 102 and ray 103 in the virtual world may drift and fail to correctly correspond to the user's hand in the real world, which may prevent the user from accurately performing the desired indication / pointing action. Summary of the invention

[0006] In view of this, the present invention provides a method and a calibration device for calibrating a virtual object, which can be used to solve the above technical problems.

[0007] An embodiment of the present invention provides a method for calibrating a virtual object, which is applied to a calibration device. The method includes: determining a reference direction related to a first virtual object in a virtual world by the calibration device; obtaining current motion data of an input device corresponding to the first virtual object from a motion detection circuit of the input device by the calibration device; obtaining a predetermined orientation of the motion detection circuit of the input device by the calibration device; determining the current posture of the first virtual object relative to the input device by the calibration device based on the predetermined orientation of the motion detection circuit and the current motion data of the input device; determining a calibration factor by the calibration device based on the reference direction and the current posture of the first virtual object; and calibrating the object posture of the first virtual object by the calibration device based on the calibration factor.

[0008] An embodiment of the present invention provides a calibration device, including a storage circuit and a processor. The storage circuit stores program code. The processor is coupled to the storage circuit and accesses the program code to execute: determining a reference direction related to a first virtual object in a virtual world; obtaining current motion data of an input device corresponding to the first virtual object from a motion detection circuit of the input device; obtaining a predetermined orientation of the motion detection circuit of the input device; determining a current posture of the first virtual object relative to the input device based on the predetermined orientation of the motion detection circuit and the current motion data of the input device; determining a calibration factor based on the reference direction and the current posture of the first virtual object; and calibrating the object posture of the first virtual object based on the calibration factor. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings are included to provide a further understanding of the present invention and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the present invention and together with the description serve to explain the principles of the present invention.

[0010] Figure 1 A schematic diagram illustrating using an input device to interact with a virtual world.

[0011] Figure 2 FIG. 4 is a schematic diagram of a calibration device according to an embodiment of the present invention.

[0012] Figure 3 A flow chart of a method for calibrating a virtual object according to an embodiment of the present invention is shown.

[0013] Figure 4A FIG. 4 is a schematic diagram showing how to determine a reference direction according to a first embodiment of the present invention.

[0014] Figure 4B A schematic diagram illustrating the relative positions between a first coordinate system of an optical sensor of an input device and a second coordinate system of a motion detection circuit.

[0015] Figure 5 Another schematic diagram of determining a designated direction according to the first embodiment of the present invention is shown.

[0016] Fig. 6A FIG. 4 is a schematic diagram showing how to determine a reference direction according to a second embodiment of the present invention.

[0017] Figure 6B Drawing basis Fig. 6A Schematic diagram for deriving the reference directions.

[0018] Fig. 7A and Figure 7B According to Figure 4A and Figure 5 A schematic diagram of a calibrated first virtual object and its related indicators is shown.

[0019] Fig. 8A A system diagram of an input device and a host according to a first embodiment of the present invention is shown.

[0020] Figure 8B A system diagram of an input device and a host according to a second embodiment of the present invention is shown. DETAILED DESCRIPTION

[0021] Reference will now be made in detail to exemplary embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Whenever possible, the same reference numerals are used in the drawings and the description to refer to the same or like parts.

[0022] Reference Figure 2 , which is a schematic diagram of a calibration device according to an embodiment of the present invention.

[0023] In various embodiments, the calibration device 200 may be any smart device and / or computer device. In some embodiments, the calibration device 200 may be any input device that can be worn by a user to perform a specific operation.

[0024] exist Figure 2 In the embodiment, the storage circuit 202 is a combination of one or more static or mobile random access memories (RAMs), read-only memories (ROMs), flash memories, hard disks, or any other similar devices, which records multiple modules and / or program codes that can be executed by the processor 204.

[0025] The processor 204 can be coupled to the storage circuit 202, and the processor 204 can be, for example, a general-purpose processor, a special-purpose processor, a traditional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASICs), a field programmable gate array (FPGAs) circuit, any other type of integrated circuit (IC), a state machine, etc.

[0026] In a first embodiment, the calibration device 200 may be an input device, and the calibration device 200 may be connected to a host that provides visual content of a reality service, wherein the host may be, for example, a head-mounted display (HMD), and the reality service may be a virtual reality (VR) service, an augmented reality (AR) service, a mixed reality (MR) service, and / or an extended reality (XR) service, etc.

[0027] In this case, the input device (which may be a wearable device worn by the user) can be used by the user to interact with the visual content (e.g., VR world) provided by the host. For example, the calibration device 200 may be Figure 1 The input device 101 in the embodiment of the present invention is not limited thereto.

[0028] In a second embodiment, the calibration device 200 may be a host (e.g., an HMD) that can display the considered reality service visual content for the user to view. In a second embodiment, the user may also use an input device (e.g., Figure 1 The input device 101 in interacts with the visual content displayed by the connected host.

[0029] In the first and / or second embodiments, the host may render a virtual object corresponding to the input device in the visual content, e.g. Figure 1 102 in FIG. 103. However, as described above, virtual objects (and / or their associated rays) may drift over time.

[0030] Therefore, an embodiment of the present invention provides a method for calibrating a virtual object, which can be used to solve this problem.

[0031] In an embodiment of the present invention, the processor 204 may access the modules and / or program codes stored in the storage circuit 202 to implement the method for calibrating a virtual object provided by the present invention, which will be further described below.

[0032] See also Figure 3 , which depicts a flow chart of a method for calibrating a virtual object according to an embodiment of the present invention. The method of this embodiment can be Figure 2 The calibration device 200 in the embodiment is performed, and the following will be supplemented by Figure 2 Description of components shown Figure 3 Details of each step.

[0033] In step S310, the processor 204 determines a reference direction (given by V ref express).

[0034] In one embodiment, the first virtual object has an indicator in the virtual world, and the reference direction V ref Corresponds to the specified direction of the indicator.

[0035] In various embodiments, the reference direction V ref It can be determined in different ways.

[0036] See also Figure 4A , which illustrates a schematic diagram of determining a reference direction according to a first embodiment of the present invention.

[0037] exist Figure 4A In the present invention, it is assumed that the user wears an input device 301 (e.g., a smart ring) on ​​the index finger, and uses the input device 301 to interact with a virtual world (e.g., a VR world) of a real service provided by a host (e.g., a head-mounted display) connected to the input device 301.

[0038] In the first embodiment, since the calibration device 200 may be an input device as described above, it can be assumed that the calibration device 200 is Figure 4A The input device 301 in the embodiment of the present invention is not limited thereto.

[0039] In this embodiment, the host may render a first virtual object 302 (eg, a virtual model of the input device 301 ) according to the tracking posture of the input device 301 , and the first virtual object 302 may have an indicator 303 (eg, a ray) in the virtual world.

[0040] As described above, the first virtual object 302 and its related index 303 may drift over time. In this case, the user can start the calibration function of the host, wherein the calibration function can be used to calibrate the object posture of the first virtual object 302.

[0041] In one embodiment, after the calibration function is activated, the host may provide a calibration mode, which instructs the user to input a sliding direction 399. In this embodiment, the sliding direction 399 may correspond to the direction of the indicator 303 required by the user.

[0042] From another perspective, the user may determine any desired direction that the indicator 303 should indicate (eg, point to) as the sliding direction 399 , but the present invention is not limited thereto.

[0043] In one embodiment, the input device 301 (eg, the calibration device 200 ) may be configured with an optical sensor capable of sensing a sliding operation input by a user. In this case, the user may perform a sliding operation corresponding to the sliding direction 399 on the optical sensor to input the sliding direction 399 to the calibration device 200 .

[0044] In one embodiment, the optical sensor may be an optical finger navigation (OFN) sensor. In this case, the user may use, for example, a thumb to slide on the OFN sensor in a sliding direction 399 so that the OFN sensor may detect the sliding direction 399.

[0045] However, since the object posture of the first virtual object 302 may be mainly determined based on motion data provided by a motion detection circuit (such as an inertial measurement unit (IMU)) configured in the input device 301, it is necessary to derive the representation of the sliding direction 399 relative to the motion detection circuit.

[0046] In this embodiment, the representation of the sliding direction 399 relative to the motion detection circuit can be understood as the designated direction of the indicator 303, but the present invention is not limited thereto.

[0047] Therefore, after the input device 301 (e.g., the calibration device 200) detects the sliding direction 399 using an optical sensor, the calibration device 200 can convert the sliding direction 399 into a specified direction based on the relative position between the first coordinate system of the optical sensor and the second coordinate system of the motion detection circuit (e.g., an inertial measurement unit (IMU)) of the input device 301.

[0048] See also Figure 4B , which is a schematic diagram illustrating the relative positions between a first coordinate system of an optical sensor of an input device and a second coordinate system of a motion detection circuit.

[0049] In this embodiment, it is assumed that the first coordinate system 310 of the optical sensor and the second coordinate system 320 of the motion detection circuit have Figure 4B The relative positions shown.

[0050] In this case, the calibration device 200 can convert the sliding direction detected by the optical sensor into the corresponding designated direction accordingly.

[0051] For example, if the sliding direction detected in the first coordinate system 310 is (0, 1, 0) (i.e., corresponding to the direction of the Y axis of the first coordinate system 310), the corresponding specified direction in the second coordinate system will be (1, 0, 0) (i.e., corresponding to the direction of the X axis of the second coordinate system 320).

[0052] To give another example, if the sliding direction detected in the first coordinate system 310 is (1, 0, 0) (i.e., corresponding to the direction of the X-axis of the first coordinate system 310), the corresponding specified direction in the second coordinate system will be (0, -1, 0) (i.e., corresponding to the direction of the negative Y-axis of the second coordinate system 320).

[0053] exist Figure 4B In the embodiment of the present invention, the conversion mechanism for converting the sliding direction into the corresponding specified direction can be Q convert The quaternion is represented by (w, x, y, z)=(0.7071068, 0, -0.7071068, 0) (expressed in quaternion form), but the present invention is not limited thereto.

[0054] In the first embodiment, the calibration device 200 may determine a specified direction as the reference direction V considered in step S310. ref .

[0055] See also Figure 5 , which shows another schematic diagram of determining a designated direction according to the first embodiment of the present invention.

[0056] In this embodiment, due to the diversity of human hand shapes / contours, the pointer 303 may not originate from the front end of the first virtual object 302. Figure 4A The sliding direction 399 is detected in the calibration mode as described in the discussion, and the calibration device 200 is still able to convert the sliding direction 399 into a corresponding specified direction relative to the motion detection circuit, which will not be repeated here.

[0057] Likewise, the calibration device 200 may determine a designated direction as a reference direction V considered in step S310. ref .

[0058] See also Fig. 6A , which shows a schematic diagram of determining a reference direction according to a second embodiment of the present invention.

[0059] In the second embodiment, the calibration device 200 may be a host (eg, a head mounted display), and the calibration device 200 may track a gesture 520 of a hand 510 having a first joint on which the input device 301 is worn.

[0060] exist Fig. 6A In the present invention, it is assumed that the user wears an input device 301 (e.g., a smart ring) on ​​the first joint 511 of the hand 510, and uses the input device 301 to interact with a virtual world (e.g., a VR world) of a real service provided by a host (e.g., a calibration device 200).

[0061] In this embodiment, the gesture 520 of the hand 510 may be represented by a corresponding skeleton graph, but the present invention is not limited thereto.

[0062] In one embodiment, the calibration device 200 may determine whether the gesture 520 has performed a target gesture. If so, the calibration device 200 may obtain a joint posture 521 of the first joint 511 of the hand 510 .

[0063] In some embodiments, the target gesture may be determined according to a designer's requirements, such as one of the predetermined gestures 531 - 533 , but the present invention is not limited thereto.

[0064] For better understanding, a predetermined gesture 532 (eg, an OK gesture) will be used as an example of a target gesture, but the present invention is not limited thereto.

[0065] In this case, when the calibration device 200 determines that the gesture 520 indicates that the hand 510 has performed the predetermined gesture 532 , the calibration device 200 may obtain the joint posture 521 of the first joint 511 according to the tracked gesture 520 .

[0066] Reference Figure 6B , which is displayed according to Fig. 6A Schematic diagram for deriving the reference directions.

[0067] exist Figure 6B In the embodiment, since the hand 510 has performed the predetermined gesture 532, the calibration device 200 can obtain the joint posture 521 of the first joint 511, and derive the reference direction 599 according to the joint posture 521 of the first joint 511. In this embodiment, the reference direction 599 and the joint posture 521 of the first joint 511 may have a predetermined relative posture.

[0068] exist Figure 6B In the scenario of, the predetermined relative posture between the reference direction 599 and the joint posture 521 of the first joint 511 can be represented by a fixed angle (e.g., a 90-degree angle) between the reference direction 599 and the joint posture 521 of the first joint 511. In this case, once the joint posture 521 of the first joint 511 is obtained, the calibration device 200 can determine the reference direction 599 by rotating the direction of the joint posture 521 of the first joint 511 by the fixed angle (e.g., 90 degrees), but the present invention is not limited thereto.

[0069] In this case, the reference direction 599 may be regarded as the reference direction V considered in step S310. ref , but the present invention is not limited to this.

[0070] In an embodiment using other predetermined gestures as the target gesture, the predetermined relative posture between the reference direction 599 and the joint posture 521 of the first joint 511 may be adjusted accordingly, but the present invention is not limited thereto.

[0071] In step S320 , the processor 204 obtains current motion data of the input device 301 corresponding to the first virtual object 302 from the motion detection circuit of the input device 301 .

[0072] In this embodiment, the current motion data may be the current reading of the motion detection circuit. In the embodiment where the motion detection circuit of the input device 301 is an IMU, the current motion data may be the current IMU reading provided by the IMU of the input device 301, which may be in the form of a quaternion (Q w , Q x ,Qy,Q z ), but the present invention is not limited thereto.

[0073] In step S330 , the processor 204 obtains a predetermined orientation of the motion detection circuit of the input device 301 .

[0074] In one embodiment, the predetermined orientation of the motion detection circuit may also be understood as the initial orientation of the motion detection circuit, which may be characterized as:

[0075]

[0076] In one embodiment, the predetermined direction of the motion detection circuit may be, for example, (0, 0, 1), but the present invention is not limited thereto.

[0077] In step S340, the processor 204 determines the direction of the motion detection circuit based on the predetermined direction (eg, V init ) and the current motion data of the input device 301 to determine the current posture of the first virtual object 302 relative to the input device 301.

[0078] In one embodiment, the processor 204 may derive a transformation matrix based on the current motion data of the input device 301 .

[0079] In one embodiment, the conversion matrix can be represented as:

[0080]

[0081] However, the present invention is not limited thereto.

[0082] Thereafter, the processor 204 may use a transformation matrix (eg, Q) to transform the predetermined orientation of the motion detection circuit into a current posture of the first virtual object 302 relative to the input device 301 .

[0083] In one embodiment, the current posture of the first virtual object 302 relative to the input device 301 can be represented as:

[0084]

[0085] However, the present invention is not limited thereto.

[0086] In step S350 , the processor 204 determines a calibration factor based on the reference direction and the current posture of the first virtual object 302 .

[0087] In one embodiment, the processor 204 may be based on the reference direction V ref The inner product result of the current posture of the first virtual object 302 determines the first angle.

[0088] For example, the first angle can be represented by θ=cos -1 (V ref ·V imu ), but the present invention is not limited to this.

[0089] In addition, the processor 204 may be configured to generate a reference direction V ref The normal direction is determined by the outer product result of the current posture of the first virtual object 302.

[0090] For example, the normal direction can be represented as However, the present invention is not limited thereto.

[0091] Next, the processor 204 may determine a calibration factor based on the first angle and the normal direction.

[0092] In one embodiment, the calibration factor can be represented as Q fix =(cos(θ / 2),n x sinθ / 2,n y sinθ / 2,n z sinθ / 2), but the present invention is not limited thereto.

[0093] In step S360, the processor 204 calculates the calibration factor Q based on the calibration factor Q. fix The object pose of the first virtual object 302 is calibrated.

[0094] In one embodiment, the processor 204 may calibrate the factor Q fix The direction of the first virtual object 302 is calibrated.

[0095] In the first embodiment where the calibration device 200 is the input device 301 (eg, the calibration device 200 and the input device 301 are the same device), the calibration device 200 may provide a reference direction V to the host (eg, HMD). ref and a calibrated object pose of the first virtual object 302 .

[0096] In this case, the host may display the first virtual object 302 having the calibrated object posture and display the reference direction V accordingly. ref An indicator 303 of the first virtual object 302 .

[0097] Reference Fig. 7A and Figure 7B ,in Fig. 7A and Figure 7B According to Figure 4A and Figure 5 A schematic diagram of a calibrated first virtual object and its related indicators is shown.

[0098] exist Fig. 7A and Figure 7B In FIG. 3 , it can be seen that the first virtual object 302 with the calibrated object posture has accurately reflected the movement of the input device 301, and the indicator 303 indicates the reference direction V ref , which corresponds to the sliding direction 399 in the real world.

[0099] In the second embodiment where the calibration device 200 is a host connected to the input device 301, the calibration device 200 may display the first virtual object 302 having the calibrated object posture and display an indication of the reference direction V ref The index 303 of the first virtual object 302. Related results can be found in Fig. 7A and Figure 7B , I will not go into details here.

[0100] Reference Fig. 8A , which shows a system diagram of an input device and a host according to a first embodiment of the present invention.

[0101] In the first embodiment where the calibration device 200 is the input device 301, the calibration device 200 may further include a motion detection circuit 206 and an optical sensor 208 coupled to the processor 204. How the calibration device 200 implements the method proposed in the present invention can be referred to the above description and will not be repeated here.

[0102] Reference Figure 8B , which shows a system diagram of an input device and a host according to a second embodiment of the present invention.

[0103] In the second embodiment in which the calibration device 200 is a host 799 connected to the input device 301, the calibration device 200 can implement the method proposed in the present invention based on the information provided by the motion detection circuit 206 and the optical sensor 208 in the input device 301. How the calibration device 200 implements the proposed method can be referred to the above description and will not be repeated here.

[0104] In summary, the embodiments of the present invention provide a solution that realigns the object pose of a virtual object in the virtual world with an input device in the real world according to the direction of the user input. Therefore, the user experience will not be affected by the virtual model and / or ray deviating from the input device in the real world.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for calibrating a virtual object, applied to a calibration device, characterized in that: include: Determining, by the calibration device, a reference direction in the virtual world associated with the first virtual object; The calibration device obtains current motion data of the input device corresponding to the first virtual object from a motion detection circuit of the input device; obtaining, by the calibration device, a predetermined orientation of the motion detection circuit of the input device; determining, by the calibration device, a current posture of the first virtual object relative to the input device based on the predetermined orientation of the motion detection circuit and the current motion data of the input device; determining, by the calibration device, a calibration factor based on the reference direction and the current posture of the first virtual object; as well as The object pose of the first virtual object is calibrated by the calibration device based on the calibration factor. 2 . The method according to claim 1 , wherein the first virtual object has an indicator in the virtual world, and the reference direction corresponds to a specified direction of the indicator.

3. The method according to claim 1, wherein determining, by the calibration device, the reference direction associated with the first virtual object in the virtual world comprises: detecting the sliding direction using an optical sensor of the calibration device; converting the sliding direction into a specified direction based on the relative position between the first coordinate system of the optical sensor and the second coordinate system of the motion detection circuit; and The designated direction is determined as the reference direction.

4. The method according to claim 1, wherein determining, by the calibration device, the reference direction associated with the first virtual object in the virtual world comprises: tracking a gesture of a hand, wherein the hand has a first joint for wearing the input device; In response to determining that the gesture has executed a target gesture, obtaining a joint posture of the first joint of the hand; as well as The reference direction is derived based on the joint posture of the first joint, wherein the reference direction and the joint posture of the first joint have a predetermined relative posture.

5. The method according to claim 1, wherein determining, by the calibration device based on the predetermined orientation of the motion detection circuit and the current motion data of the input device, the current posture of the first virtual object relative to the input device comprises: deriving a transformation matrix based on the current motion data of the input device; The predetermined orientation of the motion detection circuit is converted into the current pose of the first virtual object relative to the input device using the transformation matrix.

6. The method according to claim 5, wherein the current posture of the first virtual object relative to the input device is characterized by: V imu =QV init , where Q is the transformation matrix, V init is the predetermined orientation of the motion detection circuit.

7. The method according to claim 6, wherein the transformation matrix is ​​characterized as: Wherein (Qw, Qx, Qy, Qz) is the current motion data of the input device.

8. The method according to claim 1, wherein determining, by the calibration device, the calibration factor based on the reference direction and the current posture of the first virtual object comprises: determining a first angle based on an inner product result of the reference direction and the current posture of the first virtual object; Determine a normal direction based on an outer product result of the reference direction and the current posture of the first virtual object; as well as The calibration factor is determined based on the first angle and the normal direction.

9. The method of claim 8, wherein the first angle is characterized by: θ=cos -1 (V ref ·V imu ), Where V ref is the reference direction, V imu is the current posture of the first virtual object.

10. The method according to claim 8, wherein the normal direction is characterized by: Where V ref is the reference direction, V imu is the current posture of the first virtual object.

11. The method of claim 10, wherein the calibration factor is characterized by: Q fix =(cos(θ / 2),n x sinθ / 2,n y sinθ / 2,n z sinθ / 2), where θ is the first angle.

12. The method according to claim 1, wherein calibrating, by the calibration device, the object pose of the first virtual object based on the calibration factor comprises: The direction of the first virtual object is calibrated by the calibration factor.

13. The method according to claim 1, further comprising: The reference direction and the calibrated object pose of the first virtual object are provided to a host by the calibration device.

14. The method according to claim 1, further comprising: Displaying, by the calibration device, the first virtual object having the calibrated object posture; An indicator of the first virtual object is displayed by the calibration device, and the indicator indicates the reference direction.

15. A calibration device, characterized in that: include: A storage circuit for storing program codes; as well as A processor, coupled to the storage circuit and accessing the program code to execute: determining a reference direction in the virtual world relative to the first virtual object; obtaining, from a motion detection circuit of the input device, current motion data of the input device corresponding to the first virtual object; obtaining a predetermined orientation of the motion detection circuit of the input device; determining a current posture of the first virtual object relative to the input device based on the predetermined orientation of the motion detection circuit and the current motion data of the input device; determining a calibration factor based on the reference direction and the current pose of the first virtual object; as well as An object pose of the first virtual object is calibrated based on the calibration factor.

Images