System and method for interacting with augmented reality environment
By using the touch sensor and inertial measurement unit of the finger ring controller in HMD, detecting thumb movements and generating interactive commands, the fatigue, inaccuracy and safety of existing HMD interaction methods are solved, achieving a more efficient and safe XR Environmental interaction.
Patent Information
- Application Number
- CN202410468128.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-03-13
- Filing Date
- 2024-04-18
- Publication Date
- 2024-11-12
AI Technical Summary
Existing head-mounted display (HMD) interaction methods, such as handheld controllers, hand tracking and voice recognition, have fatigue, inaccuracy and security issues, resulting in low interaction efficiency and potential safety risks.
It adopts a finger ring controller, equipped with a touch sensor and an inertia measurement unit, communicates with the finger ring controller through a head-mounted display, detects thumb movements and generates corresponding commands, moves the cursor in the XR scene or selects a virtual object.
It provides a convenient and intuitive interaction method, which reduces user fatigue, improves interaction accuracy and efficiency, and reduces security risks.
Smart Images

Figure CN118939150A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to extended reality (XR) technology, and more particularly to a system and method for interacting with an XR environment. Background Art
[0002] As technology improves, head-mounted displays (HMDs) with XR functions are becoming more and more popular. HMDs can create an XR environment for users, so that users can interact with virtual objects displayed in the XR scene provided by the XR environment. Users of HMDs can interact with the XR environment based on, for example, handheld controllers, hand tracking, or voice recognition. However, the above-mentioned methods have some disadvantages. For example, interacting with the XR environment using a handheld controller can cause fatigue or discomfort in the case of long-term use, and the input is inefficient due to the need to physically press buttons or move controllers. Interacting with the XR environment using hand tracking can be less accurate or less reliable than using a physical controller, and can be inefficient due to the need to perform precise hand gestures. Interacting with the XR environment using voice recognition does not always accurately interpret verbal commands, and can cause potential security issues, such as unexpected activation or eavesdropping. Summary of the invention
[0003] The present disclosure relates to a system and method for interacting with an XR environment.
[0004] The present invention relates to a system for interacting with an extended reality environment, comprising a finger ring type controller and a head mounted display. The finger ring type controller comprises a touch sensor having a detection area, wherein the touch sensor generates a touch signal. The head mounted display is connected to the finger ring type controller in a communication manner and is configured to: provide an extended reality scene; determine whether an object is in the detection area according to the touch signal; in response to determining that the object is in the detection area, generate a first command according to the movement of the finger ring type controller; and move a cursor in the extended reality scene according to the first command.
[0005] The present invention relates to a method for interacting with an extended reality environment, comprising: generating a touch signal through a touch sensor having a detection area, wherein the touch sensor is included in a finger-ring type controller; providing an extended reality scene by a head-mounted display; judging through the head-mounted display whether an object is in the detection area according to the touch signal; in response to judging that the object is in the detection area, generating a first command through the head-mounted display according to movement of the finger-ring type controller; and moving a cursor in the extended reality scene according to the first command through the head-mounted display.
[0006] The present invention relates to a system for interacting with an extended reality environment, comprising a ring-type controller and a head-mounted display. The ring-type controller comprises an inertial measurement unit. The head-mounted display is connected to the ring-type controller in a communication manner and is configured to: provide an extended reality scene; receive a measurement result from the inertial measurement unit; determine a yaw angle according to the measurement result and the position of the wrist-metacarpal joint of the thumb of the user wearing the ring-type controller; and select one of a first virtual object and a second virtual object in the extended reality scene according to the yaw angle.
[0007] Based on the above description, the present invention provides a convenient and intuitive way for a user of an HMD to interact with an XR environment.
[0008] In order to make the foregoing content more easily understood, several embodiments are described in detail below with accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] The accompanying drawings are included to provide a further understanding of the present disclosure, and are incorporated in and constitute a part of this specification. The drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure.
[0010] Figure 1 A schematic diagram showing a system for interacting with an XR environment according to one embodiment of the present invention.
[0011] Figure 2 A schematic diagram showing a finger-ring type controller according to an embodiment of the present invention is shown.
[0012] Figure 3 A flow chart showing a method for interacting with an XR environment according to one embodiment of the present invention.
[0013] Figure 4 A flow chart of generating a first command based on a touch signal according to an embodiment of the present invention is shown.
[0014] Figure 5 A flowchart of generating a second command according to an embodiment of the present invention is shown.
[0015] Figure 6 A flow chart showing a method for interacting with an XR environment according to one embodiment of the present invention.
[0016] Figure 7 A flow chart illustrating another method for interacting with an XR environment according to one embodiment of the present invention.
[0017] Description of Figure Numbers
[0018] 10: System;
[0019] 30: virtual circular plane;
[0020] 31: Normal;
[0021] 32: yaw axis;
[0022] 41: Joints;
[0023] 100: Head-mounted display;
[0024] 110, 210: processor;
[0025] 120, 220: storage medium;
[0026] 130, 230: transceiver;
[0027] 140: Camera;
[0028] 150: Display;
[0029] 160, 260: battery;
[0030] 200: Ring type controller;
[0031] 240: touch sensor;
[0032] 250: Inertial measurement unit;
[0033] S301, S302, S303, S304, S305, S306, S307, S401, S402, S403, S501, S502, S503, S504, S505, S506, S601, S602, S603, S604, S605, S701, S702, S703, S704: steps;
[0034] θ1: threshold;
[0035] θ yaw : yaw angle. DETAILED DESCRIPTION
[0036] Figure 1A schematic diagram of a system 10 for interacting with an XR environment according to one embodiment of the present invention is shown, wherein the system 10 may include an HMD 100 and a finger-ring type controller 200 communicatively connected to the HMD 100. The HMD 100 may be worn on the user's head and may be used to provide the user with an XR environment (or XR scene), such as a virtual reality (VR) environment, an augmented reality (AR) environment, or a mixed reality (MR) environment. The finger-ring type controller 200 is suitable for being worn on a finger (e.g., a thumb) of the user. When the finger-ring type controller 200 is worn by the user, the user may interact with the XR scene provided by the HMD 100 through gestures performed by the hand wearing the finger-ring type controller 200.
[0037] The HMD 100 may include a processor 110, a storage medium 120, a transceiver 130, a camera 140, a display 150, and a battery 160. The processor 110 may be, for example, a central processing unit (CPU) or other programmable general or special micro control unit (MCU), a microprocessor, a digital signal processor (DSP), a programmable controller, an application specific integrated circuit (ASIC), a graphics unit (GPU), an arithmetic logic unit (ALU), a complex programmable logic device (CPLD), a field programmable gate array (FPGA), or other similar devices or combinations thereof. The processor 110 may be coupled to the storage medium 120, the transceiver 130, the camera 140, the display 150, and the battery 160.
[0038] The storage medium 120 may be, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid state drive (SSD), or similar element or combination thereof. The storage medium 120 may be a non-transitory computer-readable storage medium configured to record a plurality of executable computer programs, modules, or applications to be loaded by the processor 110 to perform the functions of the HMD 100.
[0039] The transceiver 130 may be configured to transmit or receive wired / wireless signals. The transceiver 130 may also perform operations such as low noise amplification, impedance matching, frequency mixing, up or down frequency conversion, filtering and amplification, etc. The processor 110 may communicate with other devices (e.g., the ring-type controller 200) via the transceiver 130.
[0040] The camera 140 may be a camera device for capturing images and may include a complementary metal oxide semiconductor (CMOS) sensor or a charge-coupled device (CCD) sensor.
[0041] The display 150 may be used to display video data or image data, such as an XR scene of an XR environment for a user wearing the HMD 100. The display 150 may include a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display. In one embodiment, the display 150 may provide an image beam to the user's eyes to form an image on the user's retina so that the user can see the XR scene created by the HMD 100.
[0042] The battery 160 may provide power required by the HMD 100. The battery 160 may include a carbon-zinc battery, an alkaline battery, a lithium battery, a lead-acid battery, a nickel metal hydride battery, a nickel cadmium battery, or a lithium ion battery.
[0043] The finger-ring type controller 200 may include a processor 210, a storage medium 220, a transceiver 230, and a battery 260. In one embodiment, the finger-ring type controller 200 may further include a touch sensor 240 or an inertial measurement unit (IMU) 250. The processor 210 may be, for example, a CPU or other programmable general or dedicated MCU, a microprocessor, a DSP, a programmable controller, an ASIC, a graphics unit GPU, an ALU, a CPLD, an FPGA, or other similar devices or a combination of the above devices. The processor 210 may be coupled to the storage medium 220, the transceiver 230, the touch sensor 240, the IMU 250, and the battery 260.
[0044] The storage medium 220 may be, for example, any type of fixed or removable RAM, ROM, flash memory, HDD, SSD or similar element or a combination thereof. The storage medium 220 may be a non-transitory computer-readable storage medium configured to record a plurality of executable computer programs, modules or applications to be loaded by the processor 210 to perform the functions of the ring-type controller 200.
[0045] The transceiver 230 may be configured to transmit or receive wired / wireless signals. The transceiver 230 may also perform operations such as low noise amplification, impedance matching, frequency mixing, up or down frequency conversion, filtering and amplification. The processor 210 may communicate with other devices (e.g., HMD 100) via the transceiver 230. The processor 210 may transmit sensed data to the HMD 100 via the transceiver 230, wherein the sensed data may include, for example, a measurement result generated by the IMU 250 or a touch signal generated by the touch sensor 240.
[0046] The touch sensor 240 may include an optical sensing component, such as a CMOS sensor or a CCD sensor. The touch sensor 240 may have a detection area for detecting whether an object is in the detection area and may generate a touch signal accordingly. It is assumed that the finger-ring type controller 200 has been worn on the user's hand or a finger of the hand (e.g., thumb). In one embodiment, the detection area of the touch sensor 240 may point to another finger of the hand (e.g., index finger or middle finger). In one embodiment, when the finger-ring type controller 200 is worn on a hand resting on a plane of the object, the detection area of the touch sensor 240 may point to the plane of the object (e.g., table). The touch sensor 240 may include an optical sensor, a proximity sensor, a capacitive sensor, or a resistive sensor that may generate a light signal.
[0047] The IMU 250 may include an accelerometer or a gyroscope. The IMU 250 may detect the acceleration or angular acceleration of the ring-type controller 200 and may generate measurement results accordingly.
[0048] The battery 260 can provide power required by the ring controller 200. The battery 260 can include a carbon-zinc battery, an alkaline battery, a lithium battery, a lead-acid battery, a nickel-metal hydride battery, a nickel-cadmium battery, or a lithium-ion battery.
[0049] The processor 110 may measure the yaw angle of the ring-type controller 200 (or the position of the carpometacarpal (CRC) joint of the user's thumb wearing the ring-type controller 200 ). Figure 2 FIG. 2 is a schematic diagram showing a finger ring type controller 200 according to an embodiment of the present invention. Assuming that the shape of the finger ring type controller 200 can be formed as follows Figure 2 . The yaw axis 32 of the ring-type controller 200 may be perpendicular to the normal line 31 of the virtual circular plane 30, wherein the normal line 31 may be a normal line passing through the center of the virtual circular plane 30. The yaw axis 32 may intersect the normal line 31 at a joint 41, wherein the joint 41 may represent the position of the wrist-metacarpal joint of the thumb of the user wearing the ring-type controller 200. If the ring-type controller 200 rotates around the yaw axis 32, the processor 110 may measure the yaw angle θ of the ring-type controller 200. yaw .
[0050] In one embodiment, the processor 110 may receive the measurement results generated by the IMU 250 and may determine the yaw angle θ of the ring-type controller 200 according to the measurement results. yaw In one embodiment, the processor 110 may capture an image of the user's hand wearing the ring-type controller 200 through the camera 140, and may determine the yaw angle θ of the ring-type controller 200 according to the image. yaw In one embodiment, the processor 110 may determine the yaw angle θ of the ring-type controller 200 by jointly considering the measurement result of the IMU 250 and the image of the ring-type controller 200. yaw .
[0051] Figure 3 A flowchart of a method for interacting with an XR environment according to an embodiment of the present invention is shown, wherein the method may be performed as follows: Figure 1 The system 10 shown in FIG.
[0052] In step S301, the HMD 100 may obtain sensed data, wherein the sensed data may include a measurement result generated by the IMU 250 of the ring-type controller 200, a touch signal generated by the touch sensor 240 of the ring-type controller 200, or an image captured by the camera 140 of the HMD 100. Specifically, the HMD 100 may start a timer and obtain the sensed data while the timer is running, wherein the timer may be implemented by a circuit coupled to the processor 110 or software stored in the storage medium 120.
[0053] In step S302, the HMD 100 may determine whether the finger (e.g., the thumb of the user) wearing the ring-type controller 200 touches an object, such as another finger (e.g., the index finger or the middle finger) of the user or a flat surface of the table. If the HMD 100 determines that the finger wearing the ring-type controller 200 touches the object, the system 10 may execute step S305. If the HMD 100 determines that the finger wearing the ring-type controller 200 does not touch the object, the system 10 may execute step S303.
[0054] Specifically, the HMD 100 may receive a touch signal from the ring-type controller 200, and may determine whether an object is in the detection area of the touch sensor 240 according to the touch signal. If the object is in the detection area of the touch sensor 240 for a period of time, the HMD 100 may determine that the user performs a touch operation. On the other hand, if no object is in the detection area of the touch sensor 240 for a period of time, the HMD 100 may determine that the user does not perform a touch operation.
[0055] In step S303, the HMD 100 may determine whether the timer has expired. If the timer has expired, the system 10 may execute step S304. If the timer has not expired, the system 10 may execute step S302 again.
[0056] In order to prevent a malfunction caused by the accumulation of errors of the sensed data, in step S304, the HMD 100 may reset the sensed data. For example, the HMD 100 may obtain the coordinates of the ring-type controller 200 according to the touch signal generated by the touch sensor 240. When the data of the touch signal is reset, the coordinates corresponding to the current touch signal may be mapped to the origin of the coordinate system. For example, the HMD 100 may obtain the coordinates or yaw angle of the ring-type controller 200 according to the image captured by the camera 140. When the data of the image is reset, the coordinates corresponding to the current image may be mapped to the origin of the coordinate system or the yaw angle corresponding to the current image may be mapped to zero degrees. For example, the HMD 100 may obtain the yaw angle of the ring-type controller 200 (or the position of the CRC joint of the user's thumb wearing the ring-type controller 200) according to the measurement result generated by the IMU 250. When the data of the measurement result is reset, the yaw angle (or the position of the CRC joint) corresponding to the current measurement result may be mapped to zero degrees.
[0057] In step S305, the HMD 100 may determine whether the user wearing the ring-type controller 200 performs a gesture corresponding to a click operation. If the user performs a click operation, the system 10 may perform step S307. The HMD 100 may output a second command in step S307. If the user does not perform a click operation, the system 10 may perform step S306. The HMD 100 may output a first command in step S306.
[0058] In one embodiment, the HMD 100 may determine whether the user wearing the ring-type controller 200 performs a click operation according to the touch signal. Specifically, after the HMD 100 determines that an object (e.g., an index finger or a finger) is in the detection area of the touch sensor 240 according to the touch signal, the HMD 100 may determine whether a change in the touch signal during a time period is greater than a threshold. If the change in the touch signal is greater than the threshold, the HMD 100 may determine that the user performs a click operation (or the touch sensor 240 is touched by the object). The HMD 100 may output a second command accordingly. If the change in the touch signal is less than or equal to the threshold, the HMD 100 may determine that the user does not perform a click operation. The HMD 100 may output a second command accordingly.
[0059] The first command may be generated by the HMD 100 according to the movement of the ring-type controller 200. The HMD 100 may move a cursor in the XR scene provided by the HMD 100 (e.g., a cursor visible or invisible to a user wearing the HMD 100) according to the first command. In one embodiment, the HMD 100 may capture one or more images of the ring-type controller 200, and may perform object tracking according to the captured images in order to obtain the movement of the ring-type controller 200. In one embodiment, the HMD 100 may receive a touch signal generated by a touch sensor, and may obtain the movement of the ring-type controller 200 according to the touch signal.
[0060] Figure 4 1 shows a flowchart of generating a first command based on a touch signal according to an embodiment of the present invention. The HMD 100 may obtain the movement of the ring-type controller 200 according to the touch signal from the touch sensor 240. Specifically, in step S401, the HMD 100 may calculate the coordinates (e.g., two-dimensional coordinates) of the ring-type controller 200 according to the touch signal. In step S402, the HMD 100 may calculate the direction of the ring-type controller 200 based on the change in the coordinates of the ring-type controller 200. In step S403, the HMD 100 may generate a first command according to the coordinates and direction of the ring-type controller 200. The HMD 100 may move the cursor in the XR scene provided by the HMD 100 according to the coordinates and direction of the ring-type controller 200.
[0061] The HMD 100 can be controlled according to the yaw angle θ of the ring-type controller 200. yaw (or the position of the wrist-metacarpal joint of the thumb of the user wearing the ring-type controller 200) to generate a second command. The HMD 100 may select a virtual object (e.g., a virtual button) from a plurality of virtual objects (e.g., from N virtual objects, where N may be a positive integer greater than 1) in the XR scene according to the second command. The user wearing the ring-type controller 200 may move, for example, the wrist-metacarpal joint of his thumb to apply the second command.
[0062] For example, the HMD 100 may select one of three virtual objects (ie, virtual object #0, virtual object #1, and virtual object #2) according to the second command. Figure 5 A flowchart of generating a second command according to an embodiment of the present invention is shown.
[0063] In step S501, the HMD 100 may calculate the yaw angle θ of the ring-type controller 200 according to the measurement result of the IMU 250. yaw .
[0064] In step S502, the HMD 100 may determine the yaw angle θ yaw Is it greater than the threshold θ1? If the yaw angle θyaw is greater than the threshold θ1, the HMD 100 may select virtual object #2 in step S503. yaw is less than or equal to the threshold θ1, the HMD 100 may execute step S504.
[0065] In step S504, the HMD 100 may determine the yaw angle θ yaw Is it less than or equal to the threshold value θ2, where the threshold value θ2 may be less than or equal to the threshold value θ1. yaw is less than or equal to the threshold θ2, the HMD 100 may select the virtual object #0 in step S505. yaw is greater than the threshold θ2, the HMD 100 may select the virtual object #1 in step S506.
[0066] Figure 6 A flowchart of a method for interacting with an XR environment according to an embodiment of the present invention is shown, wherein the method may be performed as follows: Figure 1 The system 10 shown in FIG. 1 is implemented. In step S601, a touch signal is generated by a touch sensor having a detection area, wherein the touch sensor is included in a finger ring type controller. In step S602, an extended reality scene is provided by a head mounted display. In step S603, the head mounted display determines whether an object is in the detection area according to the touch signal. In step S604, in response to determining that the object is in the detection area, the head mounted display generates a first command according to the movement of the finger ring type controller. In step S605, the cursor in the extended reality scene is moved according to the first command by the head mounted display.
[0067] Figure 7 A flowchart showing another method for interacting with an XR environment according to one embodiment of the present invention is shown, wherein the method may be performed as follows: Figure 1 The system 10 shown in FIG. 1 is implemented. In step S701, an augmented reality scene is provided by a head-mounted display. In step S702, a measurement result is received from an inertial measurement unit of a ring-type controller through the head-mounted display. In step S703, the yaw angle of the ring-type controller is determined through the head-mounted display according to the measurement result and the position of the wrist-metacarpal joint of the thumb of the user wearing the ring-type controller. In step S704, one of a first virtual object and a second virtual object in the augmented reality scene is selected through the head-mounted display according to the yaw angle.
[0068] In summary, the system of the present invention provides a finger-ring type controller for a user of an HMD. When the finger-ring type controller is worn on the thumb of the user, the finger-ring type controller can detect gestures of the user of the HMD, where the gestures can include, for example, moving the thumb or performing a click operation. The user can, for example, move a cursor in the XR scene or select an object in the XR scene by hand gestures. Therefore, the present invention provides a convenient and intuitive way for a user of an HMD to interact with an XR environment provided by the HMD.
[0069] It will be apparent to those skilled in the art that various modifications and variations may be made to the disclosed embodiments without departing from the scope or spirit of the present disclosure. In view of the foregoing, it is intended that the present disclosure encompass modifications and variations as long as they fall within the scope of the appended claims and their equivalents.
Claims
1. A system for interacting with an extended reality environment, It is characterized in that include: A finger ring type controller including a touch sensor having a detection area, wherein the touch sensor generates a touch signal; as well as A head mounted display is communicatively connected to the ring-type controller and is configured to: Provide extended reality scenarios; determining whether the object is in the detection area according to the touch signal; In response to determining that the object is in the detection area, generating a first command according to movement of the ring-type controller; as well as The cursor in the extended reality scene is moved according to the first command.
2. The system for interacting with an extended reality environment of claim 1, wherein the head mounted display is further configured to: In response to determining that the object is in the detection area, generating a second command according to the yaw angle of the ring-type controller; and One of the first virtual object and the second virtual object in the augmented reality scene is selected according to the second command.
3. The system for interacting with an extended reality environment of claim 2, wherein the head mounted display is further configured to: In response to the yaw angle being greater than a first threshold, selecting the first virtual object in the augmented reality scene; and In response to the yaw angle being less than or equal to a second threshold, selecting the second virtual object in the augmented reality scene.
4. The system for interacting with an extended reality environment of claim 2, wherein the ring-type controller further comprises an inertial measurement unit, and the head mounted display is further configured to: receiving measurements from the inertial measurement unit; and The yaw angle is determined based on the measurement result and the position of the wrist-metacarpal joint of the user's thumb wearing the ring-type controller.
5. The system for interacting with an extended reality environment of claim 2, wherein the head mounted display comprises a camera and is further configured to: capturing an image of a user's hand wearing the ring-type controller; and The yaw angle of the ring-type controller is determined according to the image. 6 . The system for interacting with an extended reality environment according to claim 2 , wherein the finger-ring type controller forms a virtual circular plane, and a yaw axis of the finger-ring type controller is perpendicular to a normal of the virtual circular plane.
7. The system for interacting with an extended reality environment of claim 2, wherein the head mounted display is further configured to: starting a timer and obtaining sensed data while the timer is running, wherein the sensed data is used to calculate the yaw angle; In response to determining that the object is not in the detection area, determining whether the timer has expired; as well as In response to determining that the timer expires, the sensed data is reset.
8. The system for interacting with an extended reality environment of claim 2, wherein the head mounted display is further configured to: In response to determining that the object is in the detection area, determining whether a change in the touch signal during a time period is greater than a threshold; and In response to determining that the change is greater than the threshold, the second command is generated.
9. The system for interacting with an extended reality environment of claim 1, wherein the head mounted display is further configured to: In response to determining that the object is in the detection area, determining whether a change in the touch signal during a time period is greater than a threshold; and In response to determining that the change is less than or equal to the threshold, the first command is generated.
10. The system for interacting with an extended reality environment of claim 1, wherein the head mounted display further comprises a camera and is further configured to: capturing an image of the ring controller; and The movement of the ring-type controller is obtained according to the image.
11. A system for interacting with an extended reality environment according to claim 1, wherein the ring-type controller is used to be worn on a first finger of a user's hand, and when the ring-type controller is worn on the user's hand, the detection area of the touch sensor points to a second finger of the user's hand.
12. A method for interacting with an extended reality environment, It is characterized in that include: generating a touch signal by a touch sensor having a detection area, wherein the touch sensor is included in a finger ring type controller; The extended reality scene is provided by a head-mounted display; Determining whether an object is in the detection area according to the touch signal by the head mounted display; In response to determining that the object is in the detection area, generating a first command according to movement of the ring-type controller through the head mounted display; as well as A cursor in the augmented reality scene is moved according to the first command through the head mounted display.
13. A system for interacting with an extended reality environment, It is characterized in that include: A ring-type controller including an inertial measurement unit; as well as A head mounted display is communicatively connected to the ring-type controller and is configured to: Provide extended reality scenarios; receiving measurements from the inertial measurement unit; Determine the yaw angle according to the measurement result and the position of the wrist-metacarpal joint of the thumb of the user wearing the ring-type controller; as well as One of a first virtual object and a second virtual object in the augmented reality scene is selected according to the yaw angle.
