Virtual touch method using a 3D cursor, storage medium, and chip

Abstract

To provide a touch operation in a three-dimensional virtual space by forming an interactive operation wire with bare hand.SOLUTION: The present invention is applied to an XR augmented reality wearable device or a head-mounted display device, and relates to a virtual touch method by a three-dimensional cursor, including: defining as an operation target the weighted average position of hand joints, fingertips and a plurality of joints of a human, projected by a preset light source in a virtual space; forming an interactive operation wire by projection through the operation target from the preset light source; and displaying the three-dimensional cursor or a brush at a distal end of the interactive operation wire, and to a storage medium and a chip. The invention adopts a quick space position calculation method and a touch determination method, forms the interactive operation wire having a non-fixed length or a fixed length by bare hand, and realizes a three-dimensional virtual space touch operation, writing or drawing.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to the field of virtual touch, and more specifically to a virtual touch method using a 3D cursor applied to XR augmented reality wearable devices, head-mounted display devices, storage media, and chips. [Background technology]

[0002] Extended Reality (XR) refers to a combination of reality and virtual reality created by computer technology and wearable devices, an environment where human-computer interaction is possible, and is a general term for various forms such as augmented reality (AR), virtual reality (VR), and mixed reality (MR). With the spread and development of XR in all industries and business types, various XR smart glasses have been created, enabling user-system interaction through virtual keyboards and 3D touch input.

[0003] When using XR glasses smart devices, users view a world of two screens with both eyes, a world different from the two-dimensional screens of mobile phones, tablets, and conventional displays. The world of binocular displays is three-dimensional. While conventional two-dimensional screens can be moved and clicked in the (X, Y) direction with a simple cursor, in three-dimensional space, conventional cursors cannot manipulate movement and clicking in the (X, Y, Z) direction, which has depth. Generally, XR glasses devices manipulate the position of the cursor in three-dimensional space by drawing "straight lines" like a laser pen or "curves" like a fishing rod using remote controls, game handles, mobile phones, or other similar sensors. Virtual objects at close range can be touched or manipulated with fingers or gestures. Currently, no published patents or documents that allow commanding a cursor to a distant position with bare hands disclose a method for visually calculating the three-dimensional position of the cursor. [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] The present invention aims to provide a virtual touch method, storage medium, and chip that enable touch operation, writing, and drawing in a three-dimensional virtual space using a three-dimensional cursor by forming an interactive operation wire of non-fixed or fixed length with bare hands. [Means for solving the problem]

[0005] The virtual touch method using a 3D cursor according to the present invention is applied to the system of an XR augmented reality wearable device and a head-mounted display device. In a virtual space, the weighted average position of a human wrist, fingertips, or multiple joints projected by a preset light source is defined as an operation target. An interactive operation wire is formed by projecting from the preset light source through the operation target, and a 3D cursor is displayed at the far end of the interactive operation wire. Specifically, Step 1 defines a trigger area bound to an operation target, a switching finger that initiates a 3D cursor projection, and a clicking finger that touches the trigger area. Step 2 includes the following steps: When a switching finger touches the trigger area, the spatial positions of the operation target and the preset light source are obtained, and an interactive operation wire is formed by projecting from the preset light source through the operation target; if the switching finger is removed and does not touch the trigger area before the trigger area is clicked by the clicking finger, the interactive operation wire disappears, the interactive operation wire is displayed, and the orientation of the operation target is changed so that the movement of the interactive operation wire can be guided; when the interactive operation wire forms an intersection with a virtual object or virtual model in virtual space, a 3D cursor is displayed at the intersection, and if the clicking finger touches the trigger area at this time, a virtual touch is realized on the virtual object or virtual model touched by the 3D cursor.

[0006] Obtaining the spatial positions of the aforementioned operation target and preset light source means that If the preset light source is within the visible range of the glasses and is a certain preset joint on both hands other than the operation target, the operation target and the preset light source adopt the same spatial position calculation method. Taking the connection line of the center points L and R of the left and right cameras as the X-axis, in the field of view of the left camera, the included angle between the connection line of the center point L of the left camera and the target joint point T of the calculated target spatial position and the X-axis is set as TθL. Similarly, in the field of view of the right camera, the included angle between the connection line of the center point R of the right camera and the target joint point T of the calculated target spatial position and the X-axis is set as TθR. If the visual distance difference d between the two center points L and R of the left and right cameras is set, the position (X, Z) of the target joint point T is calculated. Specifically, If the target joint point T is between the two center points L and R of the left and right cameras, it becomes Equation 1,

Equation

Equation

Equation

[0007] Touch the trigger area with the switching finger or click finger as the trigger fingertip P. The specific judgment steps are Let the width of the trigger area bound to the operation target be W. At the left W / 2 and right W / 2 parallel to the X-axis, take the left trigger judgment point WL and the right trigger judgment point WR. That is, the left trigger judgment point WL and the right trigger judgment point WR are points corresponding to the left and right two boundaries of the trigger area. The system acquires N image video streams with binocular disparity, where N is an integer and N ≥ 2. For the N images in the same frame, it tracks and judges whether the position of the trigger fingertip P in all images falls between the corresponding left trigger judgment point WL and right trigger judgment point WR of the trigger area. If so, calculate the position values of three target points in each image. The target points include the left trigger judgment point WL, the trigger fingertip P, and the right trigger judgment point WR. Take the X-axis values (WRX, PX, WLX) among the position values of the three target points, and calculate the ratios (PX - WRX):(WLX - PX) of the differences between WL and P and between P and WR respectively. Only when all the ratios of the N images are the same does it indicate that the trigger fingertip P touches the trigger area.

[0008] The virtual touch means using the fingertip of the thumb as the operation target, using one finger among the other four fingers as the click finger, and using at least one finger among the other three fingers as the switching finger to activate the three-dimensional cursor projection. When multiple switching fingers are provided, the multiple switching fingers are defined to activate different functions.

[0009] The virtual touch means using the fingertip of the thumb as the operation target, using one finger among the other four fingers as the switching finger to activate the projection of the three-dimensional cursor, and defining two fingers among the remaining three fingers as the right mouse button click finger and the left mouse button click finger respectively.

[0010] The interactive operating wire is a linear or parabolic radiation with no fixed length.

[0011] The aforementioned interactive operation wire is a virtual brush of a preset length. The virtual brush is displayed, and when the trigger area is touched by a clicking finger, the tip of the virtual brush displays a dot or stroke in the air, enabling operations such as drawing or writing.

[0012] The head-mounted display device includes at least two cameras for capturing a target image of a target area, and the head-mounted display device further includes a memory for storing a computer program and a processor for executing the computer program to realize the three-dimensional cursor virtual touch method described in any one of the above paragraphs.

[0013] A computer-readable storage medium storing a computer program that, when executed by a processor, implements a virtual touch method using a three-dimensional cursor as described in any one of the above paragraphs.

[0014] A chip for executing commands, comprising an internally sealed integrated circuit board, wherein the integrated circuit board performs a virtual touch method using a three-dimensional cursor as described in any one of the above items.

[0015] This invention enables touch operation, writing, and drawing in a 3D virtual space by using the weighted average position of the fingertip, wrist, or multiple joints as the operation target for a 3D cursor, pre-setting a light source whose position is converted from the center point of the glasses to the wrist or body part, and forming a remote cursor or virtual brush that can be operated with bare hands by radiation of a non-fixed or fixed length projected from the light source position through the operation target, thereby achieving the following technical effects.

[0016] (1) In the present invention, the XR glasses smart terminal can quickly calculate the three-dimensional spatial positions of the operation target and the preset light source by converting the three-dimensional spatial position (X, Y, Z) of the finger joints using multiple cameras and the parallax and physical distance between the cameras, and furthermore, it can project from the preset light source through the operation target to form an interactive operation wire.

[0017] (2) The present invention defines a switching finger that binds a trigger area to an operation target and activates 3D cursor projection, and a clicking finger that touches the trigger area. When the switching finger touches the trigger area, it obtains the spatial positions of the operation target and a preset light source, and projects from the preset light source through the operation target to form an interactive operation wire. If the switching finger leaves the trigger area before the click finger clicks and touches the trigger area, the interactive operation wire disappears, and the interactive operation wire is displayed. The orientation of the operation target is also changed to guide the movement of the interactive operation wire. When the interactive operation wire forms an intersection with a virtual object or virtual model in the virtual space, a 3D cursor is displayed at the intersection. If the clicking finger touches the trigger area at this time, a virtual touch is realized on the virtual object or virtual model touched by the 3D cursor. According to the above technical proposal, the present invention makes it possible to form an interactive operation wire of non-fixed or fixed length with bare hands and realize touch operation, writing, and drawing in a 3D virtual space. [Brief explanation of the drawing]

[0018] [Figure 1] These are the 21 identifiable joint points of the human hand and their names, as shown on the official Mediapipe website. [Figure 2] This is a schematic diagram illustrating how the left camera of the smart glasses in this invention calculates the spatial position of the target joint point T. [Figure 3] This is a schematic diagram illustrating how the right camera of the smart glasses in this invention calculates the spatial position of the target joint point T. [Figure 4] This is a schematic diagram illustrating how the two joint points of a single finger in the present invention form a three-dimensional direction by corresponding Y positions. [Figure 5] This is a schematic diagram illustrating how a preset light source is converted at the center of the glasses according to the present invention, and an interactive operation wire is formed by projection from the tip of the thumb. [Figure 6] In the present invention, when implementing left and right button operation using a mouse, the ring finger is schematically represented as the switching finger. [Figure 7] In this invention, when operating the left and right buttons with a mouse, the index finger is schematically represented as the right-button clicking finger. [Figure 8] This diagram schematically shows the middle finger as the left button clicking finger when performing left and right button operations using a mouse according to the present invention. [Figure 9] This is a schematic diagram illustrating the proportional relationship between the trigger fingertip and the two trigger judgment points during the merging of left and right images, in the present invention where the trigger fingertip is not touching the trigger area. [Figure 10] This is a schematic diagram of the proportional relationship between the trigger fingertip and two trigger judgment points during the merging of left and right images, when the trigger fingertip touches the trigger area in the present invention. [Figure 11] This diagram shows the proportional relationship between the trigger fingertip and the two trigger judgment points in the left and right images when the trigger fingertip is not touching (top) and touching (bottom) the trigger area in the present invention. [Figure 12] This is a block diagram of the functional configuration of the head-mounted display device according to the present invention. [Modes for carrying out the invention]

[0019] The following describes the technical proposal in the embodiments of this application clearly and completely with reference to the drawings of the embodiments, and it is clear that the embodiments described below are not all embodiments, but only a part of the embodiments of this application. According to the embodiments of this application, all other embodiments obtained by those skilled in the art without ingenuity are included in the scope of the present invention.

[0020] Furthermore, the terms “includes” and “equipment,” and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or server comprising a set of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or specific to those processes, methods, products, or apparatus. In the embodiments of this application, the use of words such as “exemplary” or “for example” is intended to represent the relevant concepts in a specific form.

[0021] Explanation of the principle of the technical realization of the present invention (1) Regarding the recognition model used to acquire palm position information, the present invention describes Mediapipe as an example of commercially available open-source software for pre-trained human wrist joint detection models that can acquire the planar position of human wrist joints. Mediapipe is an open-source item from Google, a tool library of machine learning and mainly visual algorithms, which integrates a large number of models such as face detection, face keypoint, gesture recognition, avatar segmentation, and posture recognition, and can output time-series position information of 21 joint points (also called keypoints) in a video screen of a human hand, as shown in Figure 1. In general, human wrist joint detection models output joint position information with (X, Y) pixels of the image screen as the X and Y axes. The present invention may also use a human wrist joint detection model that has been trained on itself. The present invention further includes performing learning recognition using an artificial intelligence chip such as a GPU graphics processor or an NPU neural network processor, with a convolutional KNN, RNN, Transformer or other learning model plus reinforced or any reinforced pre-training method.

[0022] (2) Calculation of joint point spatial position As shown in Figure 2, the line connecting the center points L / R of the left and right cameras is defined as the X-axis. In the field of view of the left camera, the angle between the line connecting the center point L of the left camera and the target joint point T of the spatial position to be calculated, and the X-axis is defined as TθL. Similarly, as shown in Figure 3, in the field of view of the right camera, the angle between the line connecting the center point R of the right camera and the target joint point T of the spatial position to be calculated, and the X-axis is defined as TθR.

[0023] If we let d be the parallax distance between the two center points L and R of the left and right cameras, we can calculate the position (X, Z) of the target joint point T, and specifically, If the target joint point T is located between the two center points L and R of the left and right cameras, then equation 4 is obtained.

number

number

number

[0024] In this invention, the X-axis is defined as the parallel lines of the left and right eyes of the eyeglasses, so parallax occurs only on the X-axis. Consequently, there is no parallax on the Y-axis. The Y values ​​visible to the left and right eyes are always uniform. The origin of Y may be defined as the bottom of the display video frame or another definable position. Thus, how high the Y pixels are positioned (or the distance unit for their conversion) is determined by the Y-axis value. As shown in Figure 4, when the Y value is added to the position of the target joint point T(X, Z), the position of the target joint point T(X, Y, Z) is formed.

[0025] (3) Obtain the spatial position of the preset light source. For a 3D cursor to appear in virtual space, radiation must be generated, and emitting radiation requires one radiation source or a preset light source. The weighted average position of a preset joint, fingertip, or multiple joints projected by the light source is called the operation target. A radiation direction is formed from the preset light source through the operation target, and an interactive operation wire is projected and displayed. When the far end of the interactive operation wire is on the surface of a virtual object, a "shadow" or 3D cursor is displayed.

[0026] If the preset light source is within the visible range of the glasses and is a preset joint on both hands other than the target of operation, the spatial position of the preset light source is obtained using the calculation method for the spatial position of the joint point in (2).

[0027] If the preset light source is not within the visible range of the glasses, the present invention uses the glasses or a position opposite the glasses (Offset) as the light source position. Generally, if the light source is located in the middle of both eyes on the glasses, the 3D cursor projected onto an object as radiation generated when the light source is projected onto the fingertip of the thumb and manipulated will be blocked by the thumb, and the user will not be able to see the 3D cursor through the display of the glasses. Disclosed articles and patents often use the shoulder or groin as the light source, but they do not disclose how the 3D spatial position of the shoulder or groin is calculated. The present invention uses the center point of the glasses (X) in the middle of both eyes. 中心 ,Y 中心 If the relative position offset of ) is adopted as the light source, and the fingertip of the right thumb is used as the operation target, then at the relative position (X light source, Y light source) of the center point of the glasses, X 光源 =X 中心 +βx, Y 光源 =Y 中心 -βy, and if the fingertip of the left thumb is used as the operating target, the relative position of the center point of the glasses (X 光源, Y 光源 ) in X 光源 =X 中心 -βx, Y 光源 =Y 中心-βy, and the offset values ​​βx and βy can be preset as needed, for example, 20 centimeters and 30 centimeters, as shown in Figure 5. In this case, if the relative position of the light source is not in the center of the glasses but on one side below, the 3D cursor projected onto a virtual object by the fingertip of the thumb as the target of operation will be clearly visible without being obstructed by your hand.

[0028] (4) Touch detection between trigger fingertip P and trigger area A trigger region with width W is provided, and at left W / 2 and right W / 2 parallel to the X-axis, the left trigger decision point WL and the right trigger decision point WR are taken, that is, the left trigger decision point WL and the right trigger decision point WR are points corresponding to the two boundaries on the left and right of the trigger region. The system acquires N image video streams with parallax distance, where N is an integer and N≧2, and for N images of the same frame, the position of the trigger fingertip P in all images is the corresponding left trigger decision point WL and right trigger decision point of the trigger region. The system tracks whether the target falls between points WR and determines if it does. If so, it calculates the position values ​​of three target points within each image. These target points include the left trigger judgment point WL, the trigger fingertip P, and the right trigger judgment point WR. The X-axis values ​​(WRX, PX, WLX) of the three target point position values ​​are taken, and the ratios of the difference between WL and P and the difference between P and WR (PX-WRX):(WLX-PX) are calculated. As shown in Figures 9 to 11, if all ratios for all N images are the same, it indicates that the trigger fingertip touches the thumbtip. Figure 4 shows the joint positions within the visible range of the light source. As long as both the light source and the operation target are within the visible range of the glasses camera, the function of projecting a shadow cursor can be realized.

[0029] (5) Regarding the virtual touch method using a 3D cursor, The system includes a trigger area bound to the operation target, a clicking finger, and a switching finger. In this embodiment, the fingertip of the thumb is the operation target, the index finger is the clicking finger, and the middle, ring, and little fingers (excluding the thumb and index finger) are the switching fingers that activate the projection of a 3D cursor. When a switching finger touches the trigger area, the spatial positions of the operation target and the preset light source are acquired, and an interactive operation wire is formed that is connected in a straight line or ray, and runs along the line or ray in a direction outside the operation target, such as a laser pen (straight line) or a fishing rod (parabola). If the switching finger leaves the trigger area before the trigger area is clicked / touched by the clicking finger, the interactive operation wire disappears, the interactive operation wire is displayed, and the orientation of the operation target is changed to guide the movement of the interactive operation wire. When the interactive operation wire forms an intersection with a virtual object (or a virtual model of a real object) in the virtual space, a 3D cursor is displayed at the intersection. If the clicking finger touches the trigger area at this time, various mouse button-like operations such as clicking, dragging, selecting, and drawing are performed on the virtual object touched by the 3D cursor. To implement left and right mouse button operation, two different clicking fingers can be defined. For example, the tip of the thumb can be designated as the operation target, and as shown in Figures 6-8, the index finger can be designated as the right-button click finger, the middle finger as the left-button click finger, and the ring finger as the switching finger that activates the projection of the 3D cursor. When the tip of the thumb touches the ring finger to display the interactive operation wire, if the index finger touches the trigger area, it is a right-button click operation of the mouse, and if the middle finger touches the trigger area, it is a left-button click operation of the mouse.

[0030] (6) 3D brushes enable virtual operations such as virtual brushes. The interactive operation wire is a virtual brush of a preset length, and the far end of the interactive operation wire is the position of the pen tip. When the clicking finger touches the trigger area, the pen tip of the virtual brush can display a dot or stroke in the air, enabling spatial writing or drawing operations such as drawing pictures or writing letters.

[0031] Example 1 Embodiment 1 of the present invention relates to a virtual touch method using a 3D cursor, and is applied to a system of an XR augmented reality wearable device and a head-mounted display device. In a virtual space, a preset weighted average position of a human wrist joint, fingertips, or multiple joints projected by a preset light source is defined as an operation target. An interactive operation wire is formed by projecting from the preset light source through the operation target, and a 3D cursor is displayed at the far end of the interactive operation wire. Specifically, Step 1 involves binding a trigger area to an operation target, defining a switching finger that activates the projection of a 3D cursor, and a clicking finger that touches the trigger area. In this embodiment, the fingertip of the thumb is the operation target, the index finger is the clicking finger, and either the middle finger, ring finger, or little finger (excluding the thumb and index finger) is the switching finger that activates the projection of the 3D cursor, or each of these fingers is a different switching finger, and different switching fingers can activate different functions. Step 2 includes the following steps: When a switching finger touches the trigger area, the spatial positions of the operation target and the preset light source are acquired, and an interactive operation wire is formed and displayed by projecting from the preset light source through the operation target; if the switching finger is removed and does not touch the trigger area before the trigger area is clicked by the clicking finger, the interactive operation wire disappears, the interactive operation wire is displayed, and the orientation of the operation target is changed so that the movement of the interactive operation wire can be guided; when the interactive operation wire forms an intersection with a virtual object or virtual model of a real object in virtual space, a 3D cursor is displayed at the intersection, and if the clicking finger touches the trigger area at this time, a virtual touch is realized on the virtual object or virtual model touched by the 3D cursor.

[0032] Obtaining the spatial positions of the aforementioned operation target and preset light source means, specifically, If the preset light source is within the visible range of the glasses and is a preset joint on both hands other than the target of operation, the same spatial position calculation method is used for the target of operation and the preset light source, and the connecting line between the center points L / R of the left and right cameras is defined as the X-axis, and in the field of view of the left camera, the angle between the connecting line between the center point L of the left camera and the target joint point T of the target spatial position and the X-axis is defined as TθL, and similarly, as shown in Figure 3, in the field of view of the right camera, the angle between the connecting line between the center point R of the right camera and the target joint point T of the target spatial position and the X-axis is defined as TθR, and this includes the step of doing the same. If we let d be the parallax distance between the two center points L and R of the left and right cameras, then calculating the position (X, Z) of the target joint point T can be done as follows: If the target joint point T is located between the two center points L and R of the left and right cameras, then equation 7 is obtained.

number

number

number

[0033] The aforementioned virtual touch is defined as using the thumbtack as the target of operation, one of the other four fingers as a click finger like the index finger, and at least one of the other three fingers as a switching finger that activates a 3D cursor projection. If multiple switching fingers are provided, it is defined that multiple switching fingers activate different functions. For example, the middle finger corresponds to the 3D cursor, and the ring finger corresponds to the virtual brush.

[0034] The aforementioned virtual touch involves defining the fingertip of the thumb as the operation target, one of the other four fingers as a switching finger, such as the ring finger, to activate the projection of a 3D cursor, and two of the remaining three fingers as the right-click finger and left-click finger of the mouse, such as the index finger and middle finger.

[0035] The interactive operating wire is either a linear beam, like a laser pen, or a parabolic beam, like a fishing rod, with no fixed length.

[0036] The interactive operation wire is a virtual brush of a preset length, and the far end of the interactive operation wire is the position of the pen tip, which displays the virtual brush. When the clicking finger touches the trigger area, the pen tip displays a dot or stroke in the air, enabling operations such as drawing or writing.

[0037] Those skilled in the art will understand that the units and algorithmic steps of each example disclosed in the present invention can be combined and implemented in electronic hardware, computer software, or a combination of both. It should be further understood that, in order to clearly illustrate the compatibility between hardware and software, the configurations and steps of each example have been generally described in the above description according to their function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical proposal. Those skilled in the art may implement the described functions using different methods for each specific application, but such implementations should not be considered beyond the scope of the present invention.

[0038] Specifically, each step of the method embodiment in the embodiments of the present application may be completed by hardware integrated logic circuits and / or software-based instructions in a processor, and the steps to be combined with the method disclosed in the embodiments of the present application may be directly embodied as the completion of execution by a hardware decoding processor, or may be completed by a combination of hardware and software modules in a decoding processor. Preferably, the software modules are stored in a storage medium mature in the art, such as random memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in memory, and the processor reads information in memory and combines it with its hardware to complete the steps in the embodiments of the method.

[0039] Example 2 Embodiment 2 of the present invention, as shown in Figure 12, includes a memory 710 and a processor 720, wherein the memory 710 provides a head-mounted display device 700 that stores a computer program and transmits the program code to the processor 720. In other words, the processor 720 can call and execute a computer program from the memory 710 to implement the method in the embodiment of the present application. For example, the processor 720 is for executing the processing steps described in the method of Embodiment 1 according to instructions in the computer program.

[0040] In some embodiments of the present invention, the computer program may be divided into one or more modules, which are stored in memory 710 and executed by a processor 720, in order to complete the method of Embodiment 1 according to the present application. The one or more modules may be a series of computer program instruction segments that can complete a particular function, and the instruction segments are for describing the execution process of the computer program on the head-mounted display device 700.

[0041] The head-mounted display device may further include a transceiver 730 connected to a processor 720 or memory 710, as shown in Figure 12. Here, the processor 720 can control the transceiver 730 to communicate with another device, specifically, to send information or data to another device or to receive information or data sent from another device. The transceiver 730 may include at least two cameras for capturing target images of at least a target area.

[0042] Each component of the head-mounted display device 700 is connected via a bus system, which is understood to include a power bus, a control bus, and a status signal bus in addition to a data bus.

[0043] Example 3 Embodiment 3 of the present invention further provides a computer storage medium that stores a computer program that, when executed by a computer, enables the computer to perform the processing steps described in Embodiment 1.

[0044] Example 4 Embodiment 4 of the present invention further provides a chip for running commands, which includes an internally sealed integrated circuit substrate, for performing the processing steps described in Embodiment 1 above.

[0045] The above specific embodiments have further explained the objectives, technical proposals, and beneficial effects of the present invention. However, these are merely specific embodiments of the present invention and are not intended to limit the scope of the invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be understood to be included within the scope of the present invention. [Explanation of Symbols]

[0046] 700 Head-mounted display device, 710 Memory, 720 Processor, 730 Transmitter / Receiver, L Center point, P Trigger fingertip, R Center point, T Target joint point, Trigger fingertip, WL Left trigger decision point, WR Right trigger decision point, Y Pixel position

Claims

1. A method for performing virtual touch operations using a three-dimensional cursor, applicable to a system of an XR augmented reality wearable device or head-mounted display device, comprising the following steps: In a virtual space, the weighted average position of a human wrist, fingertips, or multiple joints that a light ray projected by a preset light source passes through before being projected to a distant location is defined as the manipulation target. A ray of light projected from a preset light source and projected to the remote position via the operation target is defined as an interactive operation wire. Set the interactive control wire so that a three-dimensional cursor is displayed at its far end. The above method further includes the following steps: A trigger area is bound to the aforementioned operation target, and a switching finger is assigned to initiate the 3D cursor projection, and a clicking finger is assigned to touch the trigger area. When it is detected that the switching finger has touched the trigger area, the spatial positions of the operating target and the preset light source are obtained, and a light ray is projected from the preset light source through the operating target to form an interactive operating wire. The interactive operation wire can be made to disappear when it is detected that the switching finger has left the trigger area and is no longer touching it, before the trigger area is clicked by the clicking finger. Based on the change in direction indicated by the aforementioned target, when an interactive control wire is present, the system guides the interactive control wire to move. When the interactive operation wire intersects with a virtual object or virtual model in the virtual space, a 3D cursor is displayed at the intersection of the interactive operation wire and the virtual object or virtual model in the virtual space, and If it is detected that the clicking finger is touching the trigger area, a virtual touch is performed on the virtual object or virtual model that the 3D cursor touched. method.

2. The spatial positions of the aforementioned operation target and the preset light source are obtained by the following steps: Assuming the preset light source is within the visible range of the glasses and is located at a preset joint on both hands other than the target of manipulation, the same spatial position calculation method is used for the target of manipulation and the preset light source. The connection line between the center points L and R of the left and right cameras is defined as the X-axis. In the field of view of the left camera, the angle between the connection line between the center point L of the left camera and the target joint point T of the calculated spatial position and the X-axis is defined as TθL. Similarly, in the field of view of the right camera, the angle between the connection line between the center point R of the right camera and the target joint point T of the calculated spatial position and the X-axis is defined as TθR. If the parallax distance between the two center points L and R of the left and right cameras is d, the position (X, Z) of the target joint point T is calculated, and specifically, If the target joint point T is located between the two center points L and R of the left and right cameras, then equation 1 is obtained. [Math 1] If the target joint point T is to the left of the center point L of the left camera, then equation 2 is obtained. [Math 2] If the target joint point T is to the right of the center point R of the right camera, then equation 3 is obtained. [Math 3] Let the origin of Y be one of the lowest points in the display video frame, and let the pixel position Y value of the target joint point T in the video frame be the Y value at the spatial position (X, Y, Z) of the target joint point T. If the preset light source is not within the visible range of the glasses, the preset light source adopts a spatial position calculation method different from that of the operation target, and uses the relative position of the glasses center point (X 中心 , Y 中心 ) in the middle of both eyes as the light source, and uses the joint or fingertip of the right hand as the operation target, then at the relative position (X 光源 , Y 光源 ) of the glasses center point, X 光源 = X 中心 + βx, Y 光源 = Y 中心 - βy. When using the joint or fingertip of the left hand as the operation target, at the relative position (X 光源， Y 光源 ) of the glasses center point, X 光源 = X 中心 - βx, Y 光源 = Y 中心 - βy. The βx and βy are the offset values of the preset step The method according to claim 1, characterized by including the following:

3. The switching finger or clicking finger touches the trigger area as the trigger fingertip P, and the specific decision step is as follows: Let W be the width of the trigger region bound to the target of operation. Take the left trigger decision point WL and the right trigger decision point WR at left W / 2 and right W / 2 parallel to the X-axis, i.e., the left trigger decision point WL and the right trigger decision point WR are points corresponding to the two boundaries of the trigger region. The system acquires N image video streams with parallax distance, where N is an integer and N ≥ 2. For N images in the same frame, the position of the trigger fingertip P in all images corresponds to the left trigger decision point WL and the right trigger decision point W of the trigger region. The method according to claim 1, characterized in that it tracks and determines whether or not the trigger is between R and the target, and if so, calculates the position values ​​of three target points in each image, the target points include the left trigger judgment point WL, the trigger fingertip P and the right trigger judgment point WR, takes the X-axis value (WRX, PX, WLX) of the position values ​​of the three target points, calculates the ratio of the difference between WL and P and the difference between T and WR (PX-WRX):(WLX-PX), and indicates that the trigger fingertip P is touching the trigger area only if all the ratios for N images are the same.

4. The method according to claim 1, characterized in that the virtual touch is defined as having the fingertip of the thumb as the operation target, one of the other four fingers as the click finger, and at least one of the other three fingers as the switching finger that activates a three-dimensional cursor projection, and if multiple switching fingers are provided, the multiple switching fingers activate different functions.

5. The method according to claim 1, characterized in that the virtual touch is defined as the fingertip of the thumb as the operation target, one of the other four fingers as the switching finger that activates the projection of a three-dimensional cursor, and two of the remaining three fingers as the right-button click finger and the left-button click finger of the mouse, respectively.

6. The method according to claim 1, characterized in that the interactive operating wire is a linear or parabolic radiation without a fixed length.

7. The method according to claim 1, characterized in that the interactive operation wire is a virtual brush of a preset length, and when the trigger area is touched by a clicking finger, the pen tip of the virtual brush displays a dot or stroke in the air, thereby enabling operations such as drawing or writing.

8. The head-mounted display device further comprises at least two cameras for capturing a target image of a target area, and the head-mounted display device further comprises a memory for storing a computer program and a processor for executing the computer program to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium characterized in that it stores a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 7.

10. A chip for executing commands, which includes an internally sealed integrated circuit board, The chip for executing a command is characterized in that the integrated circuit substrate is for performing the method according to any one of claims 1 to 7.

Images