Virtual image display system and control method thereof

Abstract

The invention provides a virtual image display system and a control method thereof. A virtual image display system includes a virtual image display apparatus and an accessory. The accessory is worn on a hand of a user and is coupled to the virtual image display device. The virtual image display device is configured to: perform a gesture tracking action of a hand of a user according to position information of the accessory; establishing a virtual pen and a virtual desktop according to the gesture tracking information; and determining whether to generate the virtual handwriting according to the position relationship between the pen point part of the virtual pen and the virtual desktop.

Description

Technical Field

[0001] The present invention relates to a virtual image display system and a control method thereof, and more particularly to a virtual image display system and a control method thereof that can perform a writing action in virtual reality. Background Art

[0002] In order to perform a writing action in a virtual reality environment, products currently on the market all use a virtual reality dedicated pen with specific specifications. This type of virtual reality dedicated pen usually has a specific shape and complex components. It provides identification for the virtual image display device through its special shape, and performs pen positioning actions by setting pressure sensors.

[0003] In practical applications, when a part of the body of a virtual reality dedicated pen with a special shape is blocked, it often cannot be recognized. Moreover, this type of special-shaped dedicated pen is not suitable for the user's hand to hold, and the comfort during use is poor. In addition, the pressure sensor is prone to damage or reduced accuracy after long-term use, affecting the overall performance of the virtual image display system. Summary of the Invention

[0004] The present invention is directed to a virtual image display system and a control method thereof, which can establish a virtual pen to perform a writing action in a virtual image.

[0005] According to an embodiment of the present invention, a virtual image display system includes a virtual image display device and a accessory. The accessory is worn on the user's hand and is coupled to the virtual image display device. The virtual image display device is configured to: perform a gesture tracking action of the user's hand according to the position information of the accessory; establish a virtual pen and a virtual desktop according to the gesture tracking information; and determine whether to generate a virtual handwriting according to the positional relationship between the tip of the virtual pen and the virtual desktop.

[0006] According to an embodiment of the present invention, a control method of a virtual image display system includes: coupling an accessory worn on the user's hand to the virtual image display device; causing the virtual image display device to perform a gesture tracking action of the user's hand according to the position information of the accessory; causing the virtual image display device to establish a virtual pen and a virtual desktop according to the gesture tracking information; and causing the virtual image display device to determine whether to generate a virtual handwriting according to the positional relationship between the tip of the virtual pen and the virtual desktop.

[0007] Based on the above, the virtual image display device of the present invention establishes a virtual pen and a virtual desktop according to the position information of the accessory. Without the need for a complex virtual reality dedicated pen, a writing action in a virtual image can be achieved. Brief Description of the Drawings

[0008] Figure 1 Schematic diagram of a virtual image display system showing an embodiment of the present invention.

[0009] Figure 2 Block diagram of accessories in a virtual image display system showing an embodiment of the present invention.

[0010] Figure 3 Schematic diagram of the appearance of accessories showing an embodiment of the present invention.

[0011] Figure 4 Schematic diagram of a function menu of a virtual image display system showing an embodiment of the present invention.

[0012] Figure 5 Block diagram of a virtual image display device in a virtual image display system showing an embodiment of the present invention.

[0013] Figure 6 Schematic diagram of an operation mode of a virtual image display system showing an embodiment of the present invention.

[0014] Figure 7 Flowchart of a writing action of a virtual image display system showing an embodiment of the present invention.

[0015] Figure 8 Flowchart of a control method of a virtual image display system showing an embodiment of the present invention. Detailed Description of the Invention

[0016] Reference will now be made in detail to exemplary embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals are used in the drawings and the description to refer to the same or like parts.

[0017] Please refer to Figure 1 , Figure 1 Schematic diagram of a virtual image display system showing an embodiment of the present invention. The virtual image display system 100 includes a virtual image display device 110 and an accessory 120. The accessory 120 is for wearing on the user's hand HD and is coupled to the virtual image display device 110. In this embodiment, the accessory 120 can be coupled to the virtual image display device 110 by means of a wireless connection. The virtual image display device 110 is, for example, a head-mounted display device that can be worn on the user's head. The virtual image display device 110 is further configured to: perform a gesture tracking action of the user's hand HD according to the position information of the accessory; establish a virtual pen VP and a virtual desktop VTB according to the gesture tracking information generated by the gesture tracking action; and determine whether to generate a virtual handwriting according to the positional relationship between the tip portion PP of the virtual pen VP and the virtual desktop VTB.

[0018] In terms of details, the accessory 120 can detect its own inertial state to obtain inertial information. Moreover, the virtual image display device 110 can perform a shooting operation on the user's hand HD and the image of the accessory 120, and obtain the image information of the user's hand HD. The accessory 120 can transmit its inertial information to the virtual image display device 110 through a wireless communication method. The virtual image display device 110 can then perform a gesture tracking operation on the user's hand HD according to the image information of the hand HD and the inertial information of the accessory 120. Further, the virtual image display device 110 can establish a virtual pen VP according to the gesture tracking information generated by the gesture tracking operation.

[0019] On the other hand, the virtual image display device 110 can establish a virtual desktop VTB and display the images of the virtual pen VP and the virtual desktop VTB through the display on the virtual image display device 110. Through the images of the virtual pen VP and the virtual desktop VTB displayed by the virtual image display device 110, the user can manipulate the virtual pen VP to contact the virtual desktop VTB. Correspondingly, when the virtual image display device 110 detects that the tip part PP of the virtual pen VP contacts the virtual desktop VTB, it can generate an image of virtual handwriting corresponding to the movement track of the tip part PP.

[0020] The gesture tracking operation in this embodiment can be performed by using a gesture tracking mechanism well-known to those of ordinary skill in the art, without specific limitations.

[0021] Please refer to Figure 2 , Figure 2 FIG. shows a block diagram of an accessory in a virtual image display system according to an embodiment of the present invention. The accessory 200 includes a controller 210, an inertial measurement unit 220, a signal transceiver interface 230, and an optical input interface 240. The controller 210 is coupled to the inertial measurement unit 220, the signal transceiver interface 230, and the optical input interface 240. The inertial measurement unit 220 is used to measure the three-axis attitude angle (or angular rate) and acceleration of the accessory 200, and thereby generate inertial information. The signal transceiver interface 230 is a wireless communication interface used to establish a wireless connection with the virtual image display device and perform a wireless signal transmission operation with the virtual image display device. In this embodiment, the signal transceiver interface 230 can be a Bluetooth (BT) transmission interface, or any other form of signal transmission interface implementation, without specific limitations.

[0022] Incidentally, an optical input interface 240 can be provided on the accessory 200. The optical input interface 240 is used to perform the receiving operation of input commands. Please refer to Figure 2 and Figure 3 , Figure 3A schematic diagram of the appearance of the accessory according to an embodiment of the present invention is shown. Among them, the accessory 200 can be in the shape of a ring and can be worn on the finger of a user. The optical input interface 240 can be disposed on the surface of the accessory 200. The optical input interface 240 performs the receiving operation of the input command ICMD according to whether the received external light source is blocked. In this embodiment, an arrow symbol can be provided on the optical input interface 240 to prompt the user to use a finger to touch the optical input interface 240 according to the arrow direction to generate a valid input command ICMD.

[0023] Incidentally, the signal transceiver interface 230 can be used to transmit the input command ICMD and the inertial information to the corresponding virtual image display device.

[0024] Please refer to Figure 4 , Figure 4 A schematic diagram of the function menu of the virtual image display system according to an embodiment of the present invention is shown. Among them, when the user transmits an input command through the optical input interface, the accessory can transmit the input command to the virtual image display device, and the virtual image display device can display the function menu 400 through the display. The function menu 400 can include multiple function options 410, 420, 430,... for the user to select, and thereby switch the functions to be performed by the virtual pen. For example, the function option 410 is used to switch the color of the handwriting; the function option 420 is used to switch the thickness of the handwriting; and the function option 430 is used to change the tip part to an eraser.

[0025] In terms of operation details, for example, the user can use a finger to quickly slide across the optical input interface to send a first input command, and thereby perform an operation like a mouse wheel to move the cursor among the multiple function options 410 to 430. The user can use a finger to press on the optical input interface for a relatively long time to send a second input command, and select the function option to be selected through the second input command. Taking the function option 410 as the selected function option as an example, the virtual image display device can further display a menu of selectable handwriting colors for the user to further select.

[0026] Please refer to Figure 5 , Figure 5 A block diagram of the virtual image display device in the virtual image display system according to an embodiment of the present invention is shown. The virtual image display device 500 includes a controller 510, a display 520, an image extractor 530, and a signal transceiver 540. The controller 510 is coupled to the display 520, the image extractor 530, and the signal transceiver 540. The display 520 can be a display of any form. The image extractor 530 can be a camera or a video camera. The signal transceiver 540 can be a wireless signal transmission interface.

[0027] In this embodiment, the signal transceiver 540 can receive the inertial information transmitted by the accessory. The image extractor 530 is used to extract the image information of the user's hand. The controller 510 can perform the user's gesture tracking action according to the inertial information and the image information of the user's hand. The controller 510 can further establish a virtual pen and a virtual desktop according to the gesture tracking information generated by the gesture tracking action. The virtual image display device 500 can display the virtual pen and the virtual desktop through the display 520.

[0028] On the other hand, the display 520 of the virtual image display device 500 can synchronously display the user's hand image. The user can perform a manipulation action on the virtual pen according to the hand image in the display 520, and make the tip of the virtual pen touch or leave the virtual desktop.

[0029] Furthermore, the controller 510 can determine whether to perform the virtual handwriting display action according to whether the tip of the virtual pen touches the virtual desktop. When the tip of the virtual pen touches the virtual desktop, the display 520 can display virtual handwriting at the position corresponding to the tip of the virtual pen; correspondingly, when the tip of the virtual pen leaves the virtual desktop, the display 520 can stop the virtual handwriting display action.

[0030] Please note here that in Figure 2 and Figure 5 's embodiments, the controllers 210 and 510 can be processors with computing capabilities. Alternatively, the controllers 210 and 510 can be designed by means of a hardware description language (Hardware Description Language, HDL) or any other design method of digital circuits well-known to those of ordinary skill in the art, and implemented in the form of a field programmable gate array (Field Programmable Gate Array, FPGA), a complex programmable logic device (Complex Programmable Logic Device, CPLD), or an application-specific integrated circuit (Application-specific Integrated Circuit, ASIC).

[0031] Please refer to Figure 6 , Figure 6A schematic diagram showing the operation mode of the virtual image display system according to an embodiment of the present invention. In the initialization operation, the accessory 610 in the virtual image display system can perform a wireless signal pairing operation with the virtual image display device. Then, the display of the virtual image display device can display the virtual desktop VTB and the virtual pen VP. In the initial state, the extending direction of the virtual pen VP can be perpendicular to the desktop direction of the virtual desktop VTB.

[0032] Next, taking the example that the user holds a physical pen in hand. The display can synchronously display the image of the user's hand HD and the physical pen. Thereby, the user can align the tip of the physical pen with the tip PP of the virtual pen VP. And after the tip of the physical pen and the tip PP of the virtual pen VP are aligned, an input command can be sent to the virtual image display device through the optical input interface on the accessory. The virtual image display device can further lock the position of the virtual pen VP according to the received input command.

[0033] After the locking operation of the position of the virtual pen VP is completed, the user can drag (raise or lower) the desktop position of the virtual desktop VTB by manipulating the tip PP of the virtual pen VP, and thereby set the virtual desktop VTB to a suitable position.

[0034] In addition, taking the example that the user does not hold a physical pen in hand. The display can synchronously display the image of the user's hand HD. Thereby, the user can hold the body part of the virtual pen VP according to the image of the hand HD. And an input command can be sent to the virtual image display device through the optical input interface on the accessory. The virtual image display device can further lock the position of the virtual pen VP according to the received input command.

[0035] Similarly, after the locking operation of the position of the virtual pen VP is completed, the user can drag (raise or lower) the desktop position of the virtual desktop VTB by manipulating the tip PP of the virtual pen VP, and thereby set the virtual desktop VTB to a suitable position.

[0036] It is worth mentioning that in this embodiment, after the locking operation of the position of the virtual pen VP is completed, the display can display an image of the virtual pen VP being held by the user's hand HD (as Figure 6 shown).

[0037] After the initialization operation is completed, the user can manipulate the tip of the virtual pen VP to contact or leave the virtual desktop VTB, and thereby perform a virtual handwriting writing operation.

[0038] Please refer to the following Figure 7 , Figure 7Flowchart showing the writing operation of the virtual image display system according to an embodiment of the present invention. In step S710, the user starts writing. In step S720, three sub-steps can be executed. In sub-step S722, the accessory provides its inertial information; in sub-step S723, the virtual image display device can generate the position information of the accessory; in sub-step S721, the virtual image display device can calculate the position of the tip of the virtual pen.

[0039] In step S730, the virtual image display device can determine whether the tip of the virtual pen touches the virtual desktop. When the determination result is yes, step S740 is executed to output the position of the tip to the application program of the virtual image display device, and the corresponding virtual handwriting is generated by executing the application program. When the determination result is no, sub-step S721 is re-executed.

[0040] Please refer to the following Figure 8 , Figure 8 Flowchart showing the control method of the virtual image display system according to an embodiment of the present invention. In step S810, the accessory worn on the user's hand can be coupled to the virtual image display device through wireless pairing. In step S820, the virtual image display device performs the gesture tracking action of the user's hand according to the position information of the accessory. In step S830, the virtual image display device establishes a virtual pen and a virtual desktop according to the gesture tracking information. In step S840, the virtual image display device determines whether to generate virtual handwriting according to the positional relationship between the tip of the virtual pen and the virtual desktop.

[0041] Details of the implementation of the above steps have been described in detail in the foregoing embodiments, and will not be elaborated here.

[0042] In summary, the virtual image display device of the present invention performs the gesture tracking action of the user according to the inertial information of the accessory. Further, according to the gesture tracking information, the virtual image display device of the present invention can establish a virtual pen and a virtual desktop, and determine whether to generate virtual handwriting by detecting the contact state between the virtual pen and the virtual desktop. In this way, in the virtual image display system of the present invention, there is no need to set a pen dedicated to virtual reality with complex functions. The user can complete the writing action to be performed by holding an ordinary physical pen or without holding a physical pen, which can effectively reduce the cost and effectively improve the comfort of use.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A virtual image display system, comprising: a virtual image display device; and a fitting that is worn on a user's hand and is coupled to the virtual image display device; wherein the virtual image display device is configured to: perform a gesture tracking action of the user's hand according to the position information of the fitting; establish a virtual pen and a virtual desktop according to the gesture tracking information; and determine whether to generate a virtual handwriting according to the judged positional relationship between the tip of the virtual pen and the virtual desktop.

2. The virtual image display system according to claim 1, wherein the fitting comprises: an inertial measurement unit for measuring the inertial information of the fitting; a first controller coupled to the inertial measurement unit; and a signal transceiver interface coupled to the controller, wherein the signal transceiver interface is configured to transmit the inertial information to the virtual image display device.

3. The virtual image display system according to claim 2, wherein the fitting further comprises: an optical input interface coupled to the first controller, and the optical input interface is configured to receive an input command and transmit the input command to the controller.

4. The virtual image display system according to claim 3, wherein the signal transceiver interface sends the input command to the virtual image display device, and the virtual image display device displays a function menu according to the input command.

5. The virtual image display system according to claim 2, wherein the virtual image display device comprises: a display for displaying the virtual pen, the virtual desktop, and the virtual handwriting; an image extractor for extracting the image information of the user's hand; and a second controller coupled to the display, and the second controller is configured to: receive the inertial information, obtain the position information of the fitting according to the inertial information and the image information, and thereby perform the gesture tracking action of the user's hand; and when the tip of the virtual pen contacts the virtual desktop, cause the display to display the virtual handwriting according to the position of the tip.

6. The virtual image display system according to claim 4, wherein the virtual image display device, in an initialization operation, is configured to: perform a pairing operation with the fitting; cause the display to display the virtual desktop in a virtual space; and adjust the position of the virtual desktop according to the user's gesture through the gesture tracking action.

7. The virtual image display system according to claim 5, wherein the virtual image display device, in the initialization operation, is further configured to: cause the display to display the virtual pen in the virtual space; instruct the user to align the tip of a physical pen held by the user with the tip of the virtual pen; and receive the input command transmitted by the user through the fitting to lock the position of the virtual pen.

8. The virtual image display system according to claim 5, wherein the virtual image display device, in the initialization operation, is further configured to: cause the display to display the virtual pen in the virtual space; Cause the user to hold the body part of the virtual pen; And Receive an input command transmitted by the user through the accessory to lock the position of the virtual pen.

9. The virtual image display system according to claim 1, wherein the virtual image display device is a head-mounted display.

10. A control method for a virtual image display system, Comprising: Couple an accessory worn on the user's hand to the virtual image display device; Cause the virtual image display device to perform a gesture tracking action of the user's hand according to the position information of the accessory; Cause the virtual image display device to establish a virtual pen and a virtual desktop according to the gesture tracking information; And Cause the virtual image display device to determine whether to generate virtual handwriting according to the positional relationship between the tip part of the virtual pen and the virtual desktop.

11. The control method according to claim 10, wherein the step of causing the virtual image display device to perform the gesture tracking action of the user's hand according to the position information of the accessory Comprises: Measure the inertial information of the accessory through an inertial measurement unit; Extract the image information of the user's hand through an image extraction unit; And Obtain the position information of the accessory according to the inertial information and the image information, and thereby perform the gesture tracking action of the user's hand.

12. The control method according to claim 10, wherein the step of determining whether to generate the virtual handwriting according to the positional relationship between the tip part of the virtual pen and the virtual desktop Comprises: When the tip part of the virtual pen contacts the virtual desktop, cause the display to display the virtual handwriting according to the position of the tip part.

13. The control method according to claim 10, wherein in the initialization operation, it further Comprises: Cause the virtual image display device to perform a pairing operation with the accessory; Cause the display of the virtual image display device to display the virtual desktop in the virtual space; And Cause the virtual image display device to adjust the position of the virtual desktop according to the user's gesture through the gesture tracking action.

14. The control method according to claim 13, wherein in the initialization operation, it further Comprises: Cause the display to display the virtual pen in the virtual space, wherein the extending direction of the virtual pen is perpendicular to the physical ground; Instruct the user to align the tip part of a physical pen held by the user with the tip part of the virtual pen; And Receive an input command transmitted by the user through the accessory to lock the position of the virtual pen.

15. The control method according to claim 13, wherein in the initialization operation, it further Comprises: Cause the display to display the virtual pen in the virtual space, wherein the extending direction of the virtual pen is perpendicular to the physical ground; Cause the user to hold the body part of the virtual pen; And Receive the input command transmitted by the user through the accessory to lock the position of the virtual pen.

Images