Augmented reality system and method of operating augmented reality system

Abstract

An augmented reality system includes a display device, a camera, and a processor. The display device is used for displaying augmented reality with an application window. The camera captures a plurality of images. The processor is used for detecting a first hand and a second hand in the images; according to the position of the finger of the first hand on the palm of the second hand in the images, controlling a cursor in an application program window; and according to the gesture of the first hand on the palm of the second hand, judging to execute the operation at the position where the vernier is located. The interactive design of the gestures provides feedback on operation of the user, and the experience of the user is improved.

Description

Technical Field

[0001] The present disclosure relates to an extended reality system and a method for operating the extended reality system, and more particularly to an extended reality system using a two-dimensional application and a method for operating the extended reality system. Background Art

[0002] Extended reality can realize and display various types of visual content, such as movies, applications, games, etc. Some devices, such as head-mounted devices, allow users to browse visual content and interact with the extended environment. These devices usually use ray casting control methods or hand tracking methods to achieve interaction with the extended environment. However, the ray casting control method may require an additional controller, which will cause inconvenience. The hand tracking method does not provide pressure feedback when pressing, resulting in an unnatural user experience. The relevant industry needs an extended reality system and method that can solve the above problems. Summary of the Invention

[0003] According to some embodiments of the present disclosure, an extended reality system is provided. The extended reality system includes a display device, a camera, and a processor. The display device is configured to display an extended reality image having an application window. The camera captures multiple images. The processor is configured to: detect a first hand and a second hand in the images; control a cursor in the application window based on the position of a finger of the first hand on the palm of the second hand in the images; and determine an operation to be performed at the location of the cursor based on a gesture of the first hand on the palm of the second hand.

[0004] In some embodiments, the processor recognizes text input into a user interface within an application window based on a trajectory of a finger of the first hand on a palm of the second hand.

[0005] In some embodiments, the processor performs a click operation on the location of the cursor in the application window in response to a finger of the first hand clicking the palm of the second hand.

[0006] In some embodiments, the processor is further configured to perform a zoom operation on content displayed in the application window in response to two fingers of the first hand pinching together on a palm of the second hand.

[0007] In some embodiments, the processor is further configured to execute a page change operation on content displayed in the application window in response to a first duration for which two fingers of the first hand move on a palm of the second hand being less than a predetermined duration.

[0008] In some embodiments, the processor performs a pan operation on the content displayed in the application window in response to two fingers of the first hand moving on the palm of the second hand for a second time period greater than a predetermined time period.

[0009] In some embodiments, the processor is further configured to: detect a first position where a finger of the first hand contacts the palm of the second hand and a second position where the finger of the first hand leaves the palm of the second hand; and move the cursor according to a vector between the first position and the second position.

[0010] In some embodiments, the extended reality system further includes: a sensor for sensing when the eyes are closed, wherein the processor is further configured to perform a click operation on the position of a cursor in the application window in response to the sensor sensing when the eyes are closed.

[0011] According to some embodiments of the present disclosure, an augmented reality system is provided, comprising a first camera, a display device, and a processor. The first camera captures multiple images. The display device displays an application window. The processor is configured to: detect a first hand and a second hand in the images; and determine whether a gesture of the first hand on the palm of the second hand is a touchpad operation for controlling the application window.

[0012] In some embodiments, the processor moves a cursor in the application window in response to a finger of the first hand moving on a palm of the second hand.

[0013] In some embodiments, the augmented reality system further includes a second camera that performs eye tracking. In response to the first hand and the second hand being a predetermined distance apart, the processor moves a cursor in the application window according to the eye tracking.

[0014] According to some embodiments of the present disclosure, a method for operating an extended reality system is provided, comprising: displaying extended reality, the extended reality including an application window; capturing multiple images of a first hand and a second hand; controlling a cursor in the application window according to a finger of the first hand on the palm of the second hand; determining a touchpad operation for controlling the application window according to a gesture of the first hand on the palm of the second hand; and adjusting content in the application window according to the touchpad operation.

[0015] In some embodiments, the method further includes: in response to the first duration of the second hand making a fist being less than a preset duration, determining that the touchpad operation on the application window is a short press operation of clicking the cursor position.

[0016] In some embodiments, the method further includes: in response to the second duration of the second hand making a fist being greater than a preset duration, determining that the touchpad operation on the application window is a long press operation of clicking the cursor position.

[0017] In some embodiments, the method further includes: displaying a cursor in a first color in response to the first hand contacting the second hand; displaying the cursor in a second color in response to the first distance between the first hand and the second hand being separated from each other being less than a preset distance; and hiding the cursor in response to the second distance between the first hand and the second hand being separated from each other being greater than a preset distance.

[0018] In some embodiments, the method further includes: in response to the second distance between the first hand and the second hand being greater than a preset distance, judging that the touchpad operation on the application window is a short press operation based on the first eye being closed and the second eye being opened.

[0019] In some embodiments, the method further includes: in response to a second distance between the first hand and the second hand being greater than a preset distance, judging that the touchpad operation on the application window is a long press operation based on the first eye being open and the second eye being closed.

[0020] In some embodiments, the method further includes: in response to the first hand and the second hand being separated from each other by a first distance greater than a predetermined distance, moving the cursor according to the position of the first hand.

[0021] In some embodiments, the method further includes: identifying text used to input the application window based on multiple traces of fingers of the first hand on the palm of the second hand.

[0022] In some embodiments, the method further includes: in response to two fingers of the first hand contacting the palm of the second hand, displaying two arrows, the two arrows starting from the cursor in the application window, wherein the lengths of the two arrows are based on the vectors of movement of the two fingers in the palm of the second hand. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Aspects of one embodiment of the present invention will be best understood from the following detailed description when read in conjunction with the accompanying drawings. It should be noted that, in accordance with standard practice in the industry, various features are not drawn to scale. In fact, the dimensions of various features may be arbitrarily increased or reduced for clarity of discussion.

[0024] Figure 1 is a schematic diagram of a system according to some embodiments of the present disclosure;

[0025] Figure 2 According to some embodiments of the present disclosure Figure 1 A flow chart of the method of the system shown;

[0026] Figures 3 to 5 According to some embodiments of the present disclosure, Figure 1 system, Figure 2 A schematic diagram of an example operation of a cursor in an application window according to a hand gesture according to the method;

[0027] Figures 6 to 9 According to some embodiments of the present disclosure, Figure 1 system, Figure 2 A schematic diagram of example gestures corresponding to different touchpad operations according to the method; and

[0028] Figure 10According to some embodiments of the present disclosure, Figure 1 system, Figure 2 A schematic diagram of the content displayed in the application window corresponding to the touchpad operation of the method.

[0029] Explanation of symbols:

[0030] 100: System

[0031] 110: Processor

[0032] 120: Storage device

[0033] 130: Display device

[0034] 140: Camera

[0035] 200:Method

[0036] 201-205: Operation

[0037] a1:arrow

[0038] a2:arrow

[0039] d1: distance

[0040] d2: length

[0041] d2': length

[0042] p1: location

[0043] p1': position

[0044] p2: location

[0045] p2': position

[0046] v1: vector

[0047] v1': vector

[0048] v2: vector

[0049] v3: vector DETAILED DESCRIPTION

[0050] In the following description, some specific details are presented to provide a comprehensive understanding of the embodiments of this invention. However, those skilled in the relevant art will recognize that the present invention can be implemented without one or more of these specific details, or in combination with other components. To avoid obscuring the various embodiments of the present invention, well-known implementation methods or operations of this invention are not shown or described in detail herein.

[0051] Unless otherwise noted, the terms used herein generally have their ordinary meanings as used in the art, within the context of this disclosure, and within the specific context. The examples used herein (including all examples of terms) are provided for illustrative purposes only and are not intended to limit the scope or meaning of this disclosure or any illustrative terms used herein. Similarly, this disclosure is not limited to the various embodiments disclosed herein.

[0052] The terms "first," "second," etc., as used herein, do not specifically refer to an order or sequence, nor are they intended to limit the present invention. They are simply used to distinguish between elements or operations described by the same technical terminology. For example, a first element could be referred to as a second element, and similarly, a second element could be referred to as a first element. Such variations do not depart from the scope of the present invention.

[0053] The words "include", "including", "have", "contain", etc. used in this article are open-ended terms, which mean including but not limited to.

[0054] Reference will now be made in detail to the embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals will be used throughout the drawings to refer to the same or like parts.

[0055] refer to Figure 1 . Figure 1 FIG1 is a schematic diagram of system 100 according to some embodiments of the present disclosure. In some embodiments, system 100 is an extended reality (XR) system, such as a virtual reality (VR) system, an augmented reality (AR) system, or a mixed reality (MR) system. According to some embodiments, system 100 includes a head mounted device (HMD), a computer, and / or a smart device, such as a smartphone.

[0056] like Figure 1 As shown, in some embodiments, system 100 includes a processor 110, a storage device 120, a display device 130, and a camera 140. In at least one embodiment, processor 110, storage device 120, display device 130, and camera 140 are included in a head-mounted device.

[0057] For illustration, the processor 110 is electrically connected to the storage device 120, the display device 130, and the camera 140. It should be understood that in the embodiments, the term "electrically connected" may generally refer to one element being indirectly electrically coupled to another element through another element, or one element being directly electrically connected to another element without going through other elements.

[0058] In some embodiments, Figure 1 The electrical connection shown may be replaced by a wireless communication connection. For example, the processor 110 may be connected to the camera 140 via a wireless network.

[0059] According to some embodiments, the processor 110 is a central processing unit (CPU), or other programmable general-purpose or special-purpose micro control unit (MCU), microprocessor (microprocessor), digital signal processor (DSP), programmable controller, application specific integrated circuit (ASIC), graphics processing unit (GPU), arithmetic logic unit (ALU), complex programmable logic device (CPLD), field programmable gate array (FPGA), or other similar components or combinations of the above components.

[0060] In various embodiments, storage device 120 is a fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, a hard disk, or any other similar device, or a combination thereof. In some embodiments, storage device 120 is used to store program code executable by processor 110.

[0061] In some embodiments, the display device 130 is a display device embedded in a head-mounted device. The display device 130 is used to provide visual content to the user. For example, the display device 130 in the head-mounted device displays an augmented reality environment to the wearer of the head-mounted device.

[0062] In some embodiments, camera 140 is a camera embedded in the head-mounted display device. Camera 140 is used to capture images of the user's hands. For example, in some embodiments, camera 140 in the head-mounted device captures images of the hands of the wearer of the head-mounted device.

[0063] System 100 is configured to provide an extended reality environment (AR) having an application window therein. In some embodiments, the application window in the AR environment is a two-dimensional projection of the application window within the AR environment. Display device 130 displays the AR environment with the application window. Processor 110 executes the application displayed in the application window based on user input. For example, processor 110 controls the content / components displayed in the application window based on hand gestures in images captured by camera 140.

[0064] Figure 1 The configuration is given for illustrative purposes. Figure 1 Various implementations are contemplated within the scope of one embodiment of the present invention. According to some embodiments, system 100 includes additional devices to detect user input. For example, in some embodiments, system 100 further includes an eye-tracking device to detect eye movements. The eye-tracking device can be a camera, sensor, etc. In some embodiments, system 100 further includes a motion-sensing device to detect hand movements. The motion-sensing device can be a wearable sensor or a handheld controller.

[0065] refer to Figure 1 and Figure 2 . Figure 2 According to some embodiments of the present disclosure Figure 1 Flowchart of method 200 of system 100 shown in FIG. Figure 2 As shown, the method 200 includes the following reference Figures 1 to 10 It should be understood that in Figure 2 Additional operations may be added before, between, and after the operations shown, and for other embodiments of method 200, some operations described below may be replaced or removed. Unless otherwise specified, the order of these operations may be adjusted based on actual needs. In certain circumstances, some or all of the operations may be performed simultaneously.

[0066] In operation 201 , the processor 110 executes the code stored in the storage device 120 to run an application. Then, according to a control signal from the processor 110 , the display device 130 displays an augmented reality environment and an application window of the application in the augmented reality environment.

[0067] In some embodiments, the application is a two-dimensional application (e.g., a mobile device application or a desktop application designed for display on a touch screen or desktop screen). The application window of the application is used to display the content / components of the application executed by processor 110. In some embodiments, the application window is a plane displayed in a three-dimensional augmented reality environment.

[0068] In operation 202, camera 140 captures images of both hands of a user. Processor 110 determines one of the two hands in the image as the dominant hand and the other as the non-dominant hand. In some embodiments, processor 110 treats the non-dominant hand as a virtual touchpad. Specifically, processor 110 detects manipulations of the dominant hand on the non-dominant hand and treats these manipulations as touchpad manipulations on a touchpad. Processor 110 then executes the application based on these touchpad manipulations.

[0069] In some embodiments, based on user settings, the processor 110 determines the hand on the right side of the image as the dominant hand and the other hand on the left side of the image as the non-dominant hand. Conversely, based on different user settings, the processor 110 determines the hand on the left side of the image as the dominant hand and the other hand on the right side of the image as the non-dominant hand.

[0070] In some embodiments, the processor 110 determines whether a hand in each of the multiple images is a dominant hand or a non-dominant hand based on the hand posture. In some embodiments, the processor 110 determines the open hand as the non-dominant hand and the closed hand as the dominant hand.

[0071] In operation 203, processor 110 controls a cursor in the application window based on the hand gesture detected in the image. For example, processor 110 controls the cursor based on the movement of a finger of the dominant hand on the non-dominant hand. Specifically, in some embodiments, when processor 110 detects that only one finger of the dominant hand is in contact with the palm of the non-dominant hand, processor 110 controls the movement of the cursor in the application window based on the movement of the finger on the palm.

[0072] refer to Figures 1 to 2 and Figures 3 to 5 . Figures 3 to 5 According to some embodiments of the present disclosure, Figure 1 System 100, Figure 2 FIG. 2 is a diagram of an example operation of a cursor in an application window according to a hand gesture according to method 200 .

[0073] like Figure 3 As shown, processor 110 determines whether to display the cursor in the application window based on the distance d1 between the dominant hand and the non-dominant hand. In some embodiments, distance d1 is the shortest distance between the dominant hand and the non-dominant hand determined by processor 110. In some embodiments, distance d1 is the distance between a finger of the dominant hand and the palm of the non-dominant hand determined by processor 110.

[0074] For example, when the processor 110 determines that the value of the distance d1 is less than (or less than or equal to) a preset value (e.g., three centimeters) and greater than zero based on the image captured by the camera, the processor 110 generates a control signal to the display device 130, and the display device 130 displays the cursor in response to the control signal. Specifically, in some embodiments, such as Figure 3 As shown, the display device 130 displays a cursor having a first color (eg, gray) in response to a control signal indicating that the distance d1 is between a predetermined value and zero.

[0075] Conversely, when processor 110 determines that distance d1 is greater than the preset value based on the image captured by camera 140, processor 110 generates another control signal to display device 130, and display device 130 stops displaying the cursor in response to the control signal. Specifically, according to some embodiments, display device 130 hides (stops displaying) the cursor or makes it transparent in response to the control signal indicating that distance d1 is greater than the preset value.

[0076] Similarly, if Figure 4 As shown, when the processor 110 determines that the value of the distance d1 is equal to zero (i.e., the fingers of the dominant hand are in contact with the palm of the non-dominant hand) based on the image captured by the camera 140, the processor 110 generates another control signal to the display device 130, and the display device 130 displays a cursor with a second color in response to the control signal. Specifically, Figure 4 As shown, in some embodiments, the display device 130 displays the cursor in a second color (e.g., blue) different from the first color (e.g., gray) in response to a control signal indicating that the distance d1 is equal to zero. The first color is the color of the cursor when the distance d1 is between a predetermined value and zero. The display device 130 displays the cursor in different colors to indicate the distance between the two hands and to inform the user whether the processor 110 has determined that the user's finger is in contact with the palm.

[0077] In some embodiments, the cursor's displayed position in the application window is based on the cursor's position during the last cursor operation. In an example where processor 110 detects a finger controlling the cursor, when the finger leaves the palm of the non-dominant hand and moves from contact to separation exceeding distance d1, the cursor stops moving and display device 130 stops displaying the cursor. Subsequently, processor 110 detects the finger approaching the palm until the distance between them is less than distance d1. In response to this detection, display device 130 displays the cursor, which is now displayed at the position where the cursor stopped when the finger left the palm.

[0078] like Figure 5As shown, in some embodiments, when a finger continuously contacts the palm, the processor 110 detects / determines a vector v1 between position p1 and position p2 on the palm. Position p1 is the initial position where the finger contacts the palm (i.e., the position where contact begins). Position p2 is the current position of the finger on the palm. Specifically, the direction of vector v1 is the direction from position p1 to position p2. The length d2 of vector v1 is equal to the distance between position p1 and position p2. In some embodiments, the processor 110 transmits vector v1 to the storage device 120. The storage device 120 records vector v1 (e.g., stores the length d2 and the direction of vector v1).

[0079] In operation, processor 110 controls / moves the cursor based on vector v1. Specifically, processor 110 determines the current position p2' of the cursor within the application window. To determine position p2', processor 110 first generates vector v1' based on vector v1. In some embodiments, the direction of vector v1' is the same as the direction of vector v1. In some embodiments, the length d2' of vector v1' is n times the length d2 (d2' = n*d2). In some embodiments, the number n is proportional to the size of the application window.

[0080] like Figure 5 As shown, position p2' is the sum of position p1' and vector v1'. Position p1' is the position where the cursor is displayed on display device 130 when the finger begins to contact the palm at position p1. In some embodiments, as described in the preceding paragraph, position p1' is also the position of the cursor after the previous cursor operation. After processor 110 determines position p2', display device 130 displays the cursor at position p2'. In other words, processor 110 moves the cursor from position p1' to position p2' based on vector v1.

[0081] In operation 204, processor 110 determines a touchpad operation on an application window based on a gesture performed by the dominant hand on the palm of the non-dominant hand. Specifically, processor 110 detects a gesture performed by the dominant hand on the palm of the non-dominant hand based on images captured by camera 140. Processor 110 then determines a touchpad operation based on the detected gesture. In practice, a touchpad operation is one of multiple preset operations used to control an application.

[0082] In operation 205, the processor 110 generates a control signal to the display device 130 according to the touchpad operation. The display device 130 adjusts the displayed content in response to the control signal. Generally speaking, the display device 130 adjusts the content / components displayed in the application window according to the touchpad operation.

[0083] refer to Figures 1 to 2 and Figures 6 to 9 . Figures 6 to 9 According to some embodiments of the present disclosure, Figure 1 System 100, Figure 2 FIG. 2 is a schematic diagram of example gestures corresponding to different touchpad operations according to method 200 .

[0084] In terms of explanation, Figure 6 The figure shows a gesture in which only one finger of the dominant hand touches the palm of the non-dominant hand. According to different embodiments, the gesture in which only one finger of the dominant hand touches the palm of the non-dominant hand corresponds to touchpad operations such as writing, cursor movement, short press (click), and long press (click).

[0085] In some embodiments, when a handwriting operation of an application is executed, the processor 110 determines that the touchpad operation is a writing operation based on the movement of a finger on the palm. When the processor 110 determines that the touchpad operation is a writing operation, the processor 110 detects and records the trajectory of the finger moving on the palm. Then, the symbol or word input by the user is determined based on the detected trajectory. In some embodiments, the processor 110 enters the symbol or word into a user interface element (e.g., a text block) in the application window. In some embodiments, the display device 130 displays the trajectory and / or the symbol / word in the application window.

[0086] When the handwriting operation of the application is not executed, the processor 110 determines that the touch panel operation is a cursor movement operation according to the movement of a finger on the palm. Figure 5 For example, refer to Figure 5 In response to detecting that the finger continues to contact the palm from position p1 to position p2, the processor 110 moves the cursor from position p1' to position p2'.

[0087] In some embodiments, the processor 110 performs a click operation on the location of the cursor in the application window in response to a finger of the dominant hand beginning to click the palm of the non-dominant hand. For example, the processor 110 detects that a finger of the dominant hand begins to contact a location of the palm of the non-dominant hand, stays at the same location for a period of time, and then leaves the same location. When it is determined that the period of time is less than a preset time length (e.g., one second), the processor 110 determines that the touchpad operation is a short press operation. Conversely, when it is determined that the period of time is greater than the preset time length, the processor 110 determines that the touchpad operation is a long press operation.

[0088] The short press operation and the long press operation are different operations on the application. For example, the short press operation and the long press operation may correspond to the short press operation and the long press operation of a general (physical) touchpad. In some embodiments, the short press operation and the long press operation correspond to the left click operation and the right click operation of a mouse.

[0089] The short press and long press operations are performed on the cursor at the location of the application window. In some embodiments, the processor 110 selects the element at the location of the cursor according to the short press or long press operation to perform an operation. For example, in response to the short press operation, a button pointed to by the cursor in the application window is clicked.

[0090] In terms of explanation, Figure 7 The figure shows a gesture of a user pressing and holding two fingers on the palm and moving the two fingers in a direction. According to various embodiments, the gesture of pressing and holding two fingers on the palm and moving the two fingers corresponds to a touchpad operation such as a pan operation or a slide operation.

[0091] In some embodiments, the processor 110 detects that two fingers (e.g., the middle finger and the index finger) of the dominant hand contact the palm of the non-dominant hand, and the two fingers move in a certain direction on the palm without leaving the palm for a period of time. If the period of time is determined to be greater than a predetermined time length (e.g., 0.3 seconds), the processor 110 determines that the touchpad operation is a panning operation. Conversely, if the period of time is determined to be less than the predetermined time length, the processor 110 determines that the touchpad operation is a sliding operation.

[0092] In some embodiments, the translation operation may correspond to a translation operation of a general (physical) touchpad. In some embodiments, the translation operation corresponds to a scrolling (scroll wheel) operation of a mouse. In some embodiments, the content displayed in the application window moves according to the translation operation.

[0093] In some embodiments, the sliding operation may correspond to a sliding operation on a typical (physical) touchpad. In some embodiments, a page-changing operation for an application in an application window is performed based on the sliding operation. In other words, processor 110 performs a page-changing operation in response to two fingers of the dominant hand moving on the palm of the non-dominant hand for less than a predetermined time.

[0094] In terms of explanation, Figure 8 A gesture of pinching two fingers of the dominant hand on the palm of the non-dominant hand is shown, which corresponds to a touchpad operation such as a pinch operation.

[0095] In some embodiments, the processor 110 determines that the touchpad operation is a pinch operation based on detecting that two fingers (eg, the index finger and the thumb) press and hold the palm, and that the distance between the two fingers increases or decreases.

[0096] In some embodiments, the pinching operation may correspond to a pinching operation of a normal (physical) touchpad. In some embodiments, the zooming operation on the content of the application in the application window is performed according to the pinching operation.

[0097] In terms of explanation, Figure 9 A fist gesture is shown, which corresponds to touchpad operations such as a short press operation and a long press operation.

[0098] In some embodiments, the processor 110 detects that the non-dominant hand is clenched into a fist and maintained for a period of time. When it is determined that the period of time is less than a preset time length (e.g., one second), the processor 110 determines that the touchpad operation is the reference Figure 6 On the contrary, when it is determined that the time period is greater than the preset time period, the processor 110 determines that the touch panel operation is the reference operation. Figure 6 The long press operation described above.

[0099] refer to Figure 1 、 Figure 2 、 Figure 7 、 Figure 8 and Figure 10 . Figure 10 According to some embodiments of the present disclosure, Figure 1 System 100, Figure 2 FIG. 2 is a schematic diagram of content displayed in an application window corresponding to a touchpad operation in method 200 .

[0100] For illustration, in some embodiments, in response to detecting two fingers contacting the palm of the non-dominant hand, the processor 110 detects and records vectors v2 and v3 corresponding to the two fingers, respectively. Figure 5 In the above content, each of vectors v2 and v3 corresponds to the vector between the position where the finger first touches the palm and the current position of the finger on the palm.

[0101] like Figure 10 As shown, processor 110 controls display device 130 to display arrows a1 and a2 in the application window based on vectors v2 and v3. In some embodiments, the arrows a1 and a2 indicate the directions of vectors v2 and v3, respectively. The lengths of arrows a1 and a2 correspond to the lengths of vectors v2 and v3, respectively. For example, in some embodiments, the lengths of arrows a1 and a2 are equal to n times the lengths of vectors v2 and v3, respectively.

[0102] In some embodiments, the arrow displayed in the application window starts at the position of the cursor or the boundary of the cursor, wherein the position of the cursor is the position to which the cursor was moved according to the last cursor movement operation. Figure 10 The arrows in the are displayed based on a pinch operation.

[0103] Figures 3 to 10 The configuration is given for illustrative purposes. Figures 3 to 10 Various implementations are within the contemplated scope of the present invention. For example, in some embodiments, the application window and / or cursor may have different shapes, sizes, etc.

[0104] In some embodiments, system 100 controls applications and the content displayed in application windows through eye tracking. Specifically, camera 140 and / or other cameras / sensors of system 100 detect eye position and / or eye movement. Processor 100 determines how to operate the application based on the detected eye position and / or eye movement.

[0105] Similar to reference Figure 6 As described above, in some embodiments, the processor 110 performs a short press or a long press operation using eye tracking. For example, in response to detecting that a first eye (e.g., the left eye) is closed for more than a predetermined time (e.g., one second) while a second eye (e.g., the right eye) remains open, the processor 110 performs a short press operation. In response to the second eye being closed for more than a predetermined time while the first eye remains open, the processor 110 performs a long press operation.

[0106] In some embodiments, in response to detecting that the eyes are closed for a predetermined time (e.g., one second) and then opened, the processor 110 performs an operation other than the short press operation and the long press operation. For example, the processor 110 performs a click operation corresponding to the mouse wheel or function key based on the time the eyes are closed for a predetermined time.

[0107] As reference Figure 3 and Figure 4 As described in the paragraph above, when the two hands are separated by a distance greater than d1, the cursor is hidden. According to some embodiments of the present disclosure, in response to the two hands being separated by a distance greater than d1 and the cursor being hidden, the processor 110 controls the application through eye tracking or raycasting, rather than through touchpad operation. For example, the cursor can be controlled based on eye movements, light from a handheld controller, or the position of the dominant hand in the air.

[0108] In summary, the present disclosure provides an extended reality system and method. Users control applications in the extended reality system using both hands. The extended reality system and method treats touchpad operations performed by the dominant hand on the non-dominant hand as touchpad operations, simulating a touchpad for controlling applications. By providing pressure feedback from pressing or sliding on the palm of the hand, the system and method provided by the present disclosure enable users to intuitively and easily control applications.

[0109] Although the present invention has been disclosed above in terms of embodiments, this is not intended to limit the present invention. Anyone skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. An extended reality system, characterized in that: Include: A display device for displaying an extended reality having an application window; a camera for capturing multiple images; and A processor for: detecting a first hand and a second hand in the plurality of images; controlling a cursor in the application window according to a position of a finger of the first hand on the palm of the second hand in the plurality of images; as well as According to a gesture of the first hand on the palm of the second hand, it is determined to perform an operation at the position where the cursor is located.

2. The extended reality system as claimed in claim 1, wherein the processor is further configured to recognize a text input into a user interface within the application window based on a trajectory of the finger of the first hand on the palm of the second hand.

3. The extended reality system as described in claim 1, wherein the processor is further configured to perform a click operation on the position of the cursor in the application window in response to the finger of the first hand clicking the palm of the second hand.

4. The extended reality system of claim 1, wherein the processor is further configured to perform a zoom operation on content displayed in the application window in response to two fingers of the first hand pinching on the palm of the second hand.

5. An extended reality system as described in claim 1, wherein the processor is further used to perform a page change operation on a content displayed in the application window in response to a first duration for which two fingers of the first hand move on the palm of the second hand is less than a predetermined duration.

6. An extended reality system as described in claim 5, wherein the processor is further used to perform a translation operation on the content displayed in the application window in response to the two fingers of the first hand moving on the palm of the second hand for a second time period greater than the predetermined time period.

7. The extended reality system of claim 1 , wherein the processor is further configured to: detecting a first position where the finger of the first hand contacts the palm of the second hand and a second position where the finger of the first hand leaves the palm of the second hand; and The cursor is moved according to a vector between the first position and the second position.

8. The extended reality system of claim 1, further comprising: A sensor is used to sense the closing of an eye, wherein the processor is further used to perform a click operation on the position of the cursor in the application window in response to the sensor sensing the closing of the eye.

9. An extended reality system, characterized in that: Include: a first camera for capturing a plurality of images; a display device for displaying an application window; as well as A processor for: detecting a first hand and a second hand in the plurality of images; as well as A gesture of the first hand on the palm of the second hand is determined to be a touchpad operation for controlling the application window. 10 . The extended reality system of claim 9 , wherein the processor is further configured to move a cursor in the application window in response to a finger of the first hand moving on the palm of the second hand.

11. The extended reality system of claim 9, further comprising: A second camera is configured to perform eye tracking, wherein the processor is further configured to move a cursor in the application window according to the eye tracking in response to the first hand and the second hand being a predetermined distance apart.

12. A method of operating an extended reality system, characterized in that: Include: Displaying an extended reality, wherein the extended reality includes an application window; Shoot multiple images, one first-hand and one second-hand; controlling a cursor in the application window according to a finger of the first hand on the palm of the second hand; determining a touchpad operation for controlling the application window according to a gesture of the first hand on the palm of the second hand; as well as A content in the application window is adjusted according to the touchpad operation.

13. The method of claim 12, further comprising: In response to a first duration of the second hand making a fist being less than a preset duration, it is determined that the touchpad operation on the application window is a short-click operation of clicking the cursor position.

14. The method of claim 13, further comprising: In response to a second duration of the second hand making a fist being longer than the preset duration, it is determined that the touchpad operation on the application window is a long press operation of clicking the cursor position.

15. The method of claim 12, further comprising: In response to the first hand contacting the second hand, displaying the cursor in a first color; In response to a first distance between the first hand and the second hand being less than a predetermined distance, displaying the cursor in a second color; and In response to a second distance between the first hand and the second hand being greater than the predetermined distance, the cursor is hidden.

16. The method of claim 15, further comprising: In response to the second distance between the first hand and the second hand being separated from each other being greater than the preset distance, the touchpad operation on the application window is determined to be a short press operation based on a first eye being closed and a second eye being opened.

17. The method of claim 16, further comprising: In response to the second distance between the first hand and the second hand being separated from each other being greater than the preset distance, and based on the first eye being open and the second eye being closed, it is determined that the touchpad operation on the application window is a long press operation.

18. The method of claim 12, further comprising: In response to a first distance between the first hand and the second hand being greater than a predetermined distance, the cursor is moved according to the position of the first hand.

19. The method of claim 12, further comprising: According to a plurality of tracks of the fingers of the first hand on the palm of the second hand, text for inputting into the application window is recognized.

20. The method of claim 12, further comprising: In response to two fingers of the first hand contacting the palm of the second hand, two arrows are displayed, the two arrows starting from the cursor in the application window, wherein the lengths of the two arrows are based on the vectors of movement of the two fingers in the palm of the second hand.

Images