Virtual touch method by three-dimensional cursor, storage medium and chip

Abstract

To provide a touch operation in a three-dimensional virtual space by forming an interactive operation wire with bare hand.SOLUTION: The present invention is applied to an XR augmented reality wearable device or a head-mounted display device, and relates to a virtual touch method by a three-dimensional cursor, including: defining as an operation target the weighted average position of hand joints, fingertips and a plurality of joints of a human, projected by a preset light source in a virtual space; forming an interactive operation wire by projection through the operation target from the preset light source; and displaying the three-dimensional cursor or a brush at a distal end of the interactive operation wire, and to a storage medium and a chip. The invention adopts a quick space position calculation method and a touch determination method, forms the interactive operation wire having a non-fixed length or a fixed length by bare hand, and realizes a three-dimensional virtual space touch operation, writing or drawing.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to the technical field of virtual touch, and particularly to a virtual touch method using a three-dimensional cursor applied to an XR augmented reality wearable device, a head-mounted display device, a storage medium and a chip. [Background technology]

[0002] Extended Reality (XR) refers to the combination of reality and virtuality through computer technology and wearable devices, creating an environment where human-computer interaction is possible, and is a general term for various forms such as augmented reality (AR), virtual reality (VR), and mixed reality (MR). As augmented reality (XR) becomes more widespread and develops in all industries and business types, various XR smart glasses have been developed, enabling user-system interaction through virtual keyboards and 3D touch input.

[0003] When using an XR glasses smart device, users view a world on two screens with both eyes, a world that differs from the 2D world seen on mobile phones, tablets, or conventional displays. The world on a binocular display screen is three-dimensional. While a conventional two-dimensional screen allows users to move and click in the (X, Y) directions on the screen with a simple cursor, in three-dimensional space, a conventional cursor cannot control movement and clicking in the (X, Y, Z) directions with depth. XR glasses devices typically use a remote control, game console, mobile phone, or other similar sensor to control the cursor's position in three-dimensional space by drawing a "straight line" like a laser pen or a "curve" like a fishing rod. Close virtual objects can be touched or manipulated with fingers or gestures. Currently, none of the published patents or documents that allow users to control a cursor over a long distance with bare hands disclose a method for visually calculating the cursor's three-dimensional position. Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention aims to provide a virtual touch method, storage medium, and chip using a three-dimensional cursor that allows for touch operations, writing, and drawing in a three-dimensional virtual space by forming an interactive operation wire of a non-fixed or fixed length with bare hands. [Means for solving the problem]

[0005] The virtual touch method using a three-dimensional cursor according to the present invention is applied to an XR augmented reality wearable device and a head-mounted display system. In the virtual space, the weighted average position of a human wrist, fingertip, or multiple joints projected by a preset light source is defined as an operation target. An interactive operation wire is formed by projecting from the preset light source through the operation target, and a three-dimensional cursor is displayed at the far end of the interactive operation wire. Step 1, in which a trigger area is bound to an operation target, and a switching finger is defined to activate a 3D cursor projection and a clicking finger is defined to touch the trigger area; and step 2, when the switching finger touches the trigger area, the spatial positions of the operation target and the preset light source are obtained, and an interactive operation wire is formed by projecting from the preset light source through the operation target; when the trigger area is not click-touched by the click finger, once the switching finger is released and no longer touches the trigger area, the interactive operation wire disappears, the interactive operation wire is displayed, and the operation target is changed in direction to guide the movement of the interactive operation wire; when the interactive operation wire forms an intersection with a virtual object or virtual model in the virtual space, a three-dimensional cursor is displayed at the intersection; at this time, when the click finger touches the trigger area, a virtual touch is realized for the virtual object or virtual model touched by the three-dimensional cursor.

[0006] acquiring spatial positions of the operation target and the preset light source; If the preset light source is within the visible range of the glasses and is a preset joint on both hands other than the operation target, the operation target and the preset light source adopt the same spatial position calculation method. The connecting line between the center points L and R of the left and right cameras is the X axis. In the left camera's field of view, the included angle between the connecting line between the center point L of the left camera and the target joint point T of the calculation target space position and the X axis is TθL. Similarly, in the right camera's field of view, the included angle between the connecting line between the center point R of the right camera and the target joint point T of the calculation target space position and the X axis is TθR. The parallax distance between the two center points L and R of the left and right cameras is d. The position (X, Z) of the target joint point T is calculated. Specifically, If the target joint point T is between the two center points L and R of the left and right cameras, then Equation 1 is obtained:

number

number

number

[0007] The switching finger or the clicking finger is used as the trigger fingertip P to touch the trigger area, and the specific judgment steps are as follows: The width of the trigger area bound to the operation target is W, and the left trigger decision point WL and the right trigger decision point WR are taken at W / 2 and W / 2 parallel to the X-axis, respectively. That is, the left trigger decision point WL and the right trigger decision point WR are points corresponding to the left and right boundaries of the trigger area. The system acquires N image video streams with parallax distance, where N is an integer and N≧2. For the N images of the same frame, it tracks and determines whether the position of the trigger fingertip P in all images is between the corresponding left trigger decision point WL and right trigger decision point WR of the trigger area. If so, it calculates the position values ​​of three target points in each image, including the left trigger decision point WL, the trigger fingertip P, and the right trigger decision point WR. The X-axis values ​​(WRX, PX, WLX) of the position values ​​of the three target points are taken, and the ratios (PX-WRX):(WLX-PX) of the difference between WL and P and the difference between P and WR are calculated respectively. Only if the ratios of all N images are the same, it indicates that the trigger fingertip P touches the trigger area.

[0008] The virtual touch defines that the tip of the thumb is the operation target, one of the other four fingers is a click finger, and at least one of the other three fingers is a switching finger that activates 3D cursor projection, and if multiple switching fingers are provided, the multiple switching fingers activate different functions.

[0009] The virtual touch is defined as follows: the tip of the thumb is the operation target, one of the other four fingers is a switching finger that activates the projection of a 3D cursor, and two of the remaining three fingers are defined as the right mouse button click finger and the left mouse button click finger, respectively.

[0010] The interactive manipulation wire is a linear or parabolic ray with no fixed length.

[0011] The interactive operation wire is a virtual brush of a preset length. When the virtual brush is displayed and the click finger touches the trigger area, the pen tip of the virtual brush displays a dot or stroke in the air, realizing operations such as drawing pictures or writing letters.

[0012] The head-mounted display device includes at least two cameras for capturing target images of a target area, and the head-mounted display device further includes a memory for storing a computer program and a processor for executing the computer program to realize the virtual touch method using a three-dimensional cursor described in any one of the above items.

[0013] A computer-readable storage medium has stored thereon a computer program that, when executed by a processor, implements the virtual touch method using a three-dimensional cursor according to any one of the above claims.

[0014] A chip for running instructions, including an integrated circuit board sealed therein, the integrated circuit board implementing the method for virtual touch with a three-dimensional cursor according to any one of the above claims.

[0015] The present invention uses the weighted average position of a fingertip, wrist joint, or multiple joints as the operation target of a 3D cursor, and presets a light source whose position is converted to the wrist joint or body part as the center point of glasses. A non-fixed or fixed length ray projected from the light source position through the operation target forms a far-end cursor or virtual brush that can be operated with bare hands, thereby realizing touch operation, writing, and drawing in a 3D virtual space, and has the following technical effects:

[0016] (1) In the present invention, the XR glasses smart terminal converts the three-dimensional spatial position (X, Y, Z) of the finger joints based on multiple cameras and the parallax and physical distance between the cameras, thereby quickly calculating and obtaining the three-dimensional spatial positions of the operation target and the preset light source, and then projects from the preset light source through the operation target to form an interactive operation wire.

[0017] (2) The present invention defines a switching finger that binds a trigger area to an operation target and activates 3D cursor projection, and a click finger that touches the trigger area. When the switching finger touches the trigger area, the spatial positions of the operation target and a preset light source are obtained, and an interactive operation wire is formed by projecting from the preset light source through the operation target. If the switching finger is released and no longer touches the trigger area before the click finger clicks and touches the trigger area, the interactive operation wire disappears, the interactive operation wire is displayed, and the operation target changes direction to guide the movement of the interactive operation wire. When the interactive operation wire intersects with a virtual object or virtual model in a virtual space, a 3D cursor is displayed at the intersection. At this time, when the click finger touches the trigger area, a virtual touch is realized for the virtual object or virtual model touched by the 3D cursor. According to the above technical solution, the present invention can form an interactive operation wire of a non-fixed or fixed length with bare hands, thereby realizing touch operations, writing, and drawing in a 3D virtual space. [Brief explanation of the drawings]

[0018] [Figure 1] These are the 21 identifiable joint points and their names for the human hand, as shown on the Mediapipe official website. [Figure 2] FIG. 10 is a schematic diagram illustrating the calculation of the spatial position of the target joint point T using the left camera of the smart glasses according to the present invention. [Figure 3] FIG. 10 is a schematic diagram illustrating how the spatial position of a target joint point T is calculated using the right camera of the smart glasses of the present invention. [Figure 4] FIG. 2 is a schematic diagram showing how two joint points of a single finger form a three-dimensional direction according to the corresponding Y positions in the present invention. [Figure 5] 10 is a schematic diagram illustrating a preset light source at the center of the glasses according to the present invention, and an interactive operation wire is formed by projecting the thumb tip. FIG. [Figure 6] 1 is a schematic diagram of the ring finger as the switching finger when realizing left and right button operation with a mouse in the present invention; [Figure 7] 1 is a schematic diagram of the index finger used to click the right button when operating the left and right buttons of the mouse in the present invention; [Figure 8] 1 is a schematic diagram of the middle finger used to click the left button when operating the left and right buttons of the mouse according to the present invention; [Figure 9] 10 is a schematic diagram of the proportional relationship between the trigger fingertip and two trigger decision points when the trigger fingertip is not touching the trigger area and the left and right images are being merged in the present invention; FIG. [Figure 10] 1 is a schematic diagram showing the proportional relationship between the trigger fingertip and two trigger decision points when the trigger fingertip touches the trigger area and the left and right images are merging in the present invention; FIG. [Figure 11] 10A and 10B are schematic diagrams illustrating the proportional relationship between the trigger fingertip and two trigger decision points in the left and right images when the trigger fingertip is not touching (above) and touching (below) the trigger area in the present invention. [Figure 12] 1 is a functional block diagram of a head-mounted display device according to the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0019] The technical solutions in the embodiments of the present application will be described below clearly and completely with reference to the drawings in the embodiments of the present application, and it is clear that the following embodiments are only some of the embodiments of the present application, rather than all of the embodiments, and all other embodiments that a person skilled in the art can obtain without inventive ideas are within the scope of the present invention.

[0020] Additionally, the terms "comprises" and "comprises" and any variations thereof are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or server comprising a series of steps or units is not necessarily limited to those steps or units expressly recited, but may include other steps or units not expressly recited or inherent to the process, method, product or apparatus. In the examples herein, the use of words such as "exemplary" or "for example" is intended to present the relevant concept in a concrete manner.

[0021] Explanation of the principles of technical realization of the present invention (1) Regarding the recognition model used to acquire palm position information, the present invention uses Mediapipe as an example of a commercially available pre-trained human hand joint detection model open-source software that can acquire the planar position of a human hand joint. Mediapipe is an open-source project from Google, a tool library for machine learning and primarily vision algorithms. It integrates a large number of models, such as face detection, face keypoints, gesture recognition, avatar segmentation, and pose recognition, and can output time-sequenced position information of 21 joint points (also called keypoints) in a video image of a human hand, as shown in Figure 1. Generally, a human hand joint detection model outputs joint position information with the (X, Y) pixels of the captured image as the X and Y axes. The present invention may also use a self-trained human hand joint detection model. The present invention further includes using an artificial intelligence chip, such as a GPU graphics processor or an NPU neural network processor, to perform learning and recognition using a tag convolutional KNN, RNN, Transformer, or other learning model plus reinforced or any other reinforced pre-training method.

[0022] (2) Calculation of joint point spatial position As shown in FIG. 2, the connecting line between the center points L / R of the left and right cameras is defined as the X-axis, and in the field of view of the left camera, the angle formed by the connecting line between the center point L of the left camera and the target joint point T of the space position to be calculated is defined as TθL. Similarly, as shown in FIG. 3, in the field of view of the right camera, the angle formed by the connecting line between the center point R of the right camera and the target joint point T of the space position to be calculated is defined as TθR.

[0023] If the parallax distance between the two center points L and R of the left and right cameras is d, the target joint point T position (X, Z) is calculated as follows: If the target joint point T is between the two center points L and R of the left and right cameras, then Equation 4 is obtained.

number

number

number

[0024] In the present invention, the X axis is defined as a parallel line between the left and right eyes of the glasses, so parallax occurs only on the X axis. Therefore, there is no parallax on the Y axis. The Y seen by the left and right eyes is always uniform. The origin of Y may be defined as the bottom of the display video frame or another definable position. In this way, the number on the Y axis determines how far up the Y pixel is (or the distance unit equivalent thereto). As shown in FIG. 4, adding the Y value to the position of the target joint point T(X, Z) forms the position (X, Y, Z) of the target joint point T.

[0025] (3) Obtain the spatial position of the preset light source. To display a 3D cursor in virtual space, a ray must be formed, and a single radiation source or preset light source is required to emit the ray. The weighted average position of a preset joint, fingertip, or multiple joints projected by the light source is called the operation target. A ray direction is formed from the preset light source through the operation target, and an interactive operation wire is projected and displayed. When the distal end of the interactive operation wire is on the surface of a virtual object, a "shadow" or 3D cursor is displayed.

[0026] If the preset light source is within the visible range of the glasses and is a preset joint in both hands other than the operation target, the spatial position of the preset light source is obtained using the calculation method for the spatial position of the joint point in (2).

[0027] If the preset light source is not within the visible range of the glasses, the present invention uses the glasses or a position (Offset) facing the glasses as the light source position. Generally, if the light source is located at the center of both eyes of the glasses, the light source will be projected onto the tip of the thumb, and the 3D cursor will be projected onto an object as a radiation generated at the operation target. The 3D cursor will be blocked by the user's thumb, and the user will not be able to see it through the display of the glasses. The disclosed articles and patents often use the shoulders or crotch as the light source, but do not disclose how to calculate the 3D spatial position of the shoulders or crotch. The present invention uses the center point (X) of the glasses in the center of both eyes. 中心 ,Y 中心 ) is used as the light source, and the tip of the right thumb is used as the operation target. In the relative position (X light source, Y light source) of the center point of the glasses, 光源 =X 中心 +βx, Y 光源 =Y 中心 -βy, and if the tip of the left thumb is the operation target, the relative position of the center of the glasses (X 光源, Y 光源 ) in X 光源 =X 中心 -βx, Y 光源 =Y 中心-βy, where the offset values ​​βx and βy can be preset as needed. For example, take 20 centimeters and 30 centimeters as shown in Figure 5. In this case, if the relative position of the light source is not in the center of the glasses but is located below one side, the 3D cursor projected onto a virtual object by the tip of the thumb as the operation target will be clearly visible without being obstructed by the user's hand.

[0028] (4) Touch determination between trigger fingertip P and trigger area A trigger area with a width W is provided, and the left trigger decision point WL and the right trigger decision point WR are taken at left W / 2 and right W / 2 parallel to the X axis, respectively. That is, the left trigger decision point WL and the right trigger decision point WR correspond to the two boundaries of the trigger area, respectively. The system acquires N image video streams with a parallax distance, where N is an integer and N≧2. For the N images in the same frame, the positions of the trigger fingertip P in all images are determined by the corresponding left trigger decision point WL and right trigger decision point WR of the trigger area. The target point is tracked to determine whether it is between point WR. If so, the position values ​​of three target points in each image are calculated. The target points include the left trigger decision point WL, the trigger fingertip P, and the right trigger decision point WR. The X-axis values ​​(WRX, PX, WLX) of the three target point position values ​​are taken, and the ratios (PX - WRX):(WLX - PX) of the difference between WL and P and the difference between P and WR are calculated. As shown in Figures 9 to 11, only when all ratios are the same for all N images does the trigger fingertip touch the thumb tip. Figure 4 shows the joint positions within the visible range of the light source. Since both the light source and the operation target are within the visible range of the glasses camera, the function of projecting a shadow cursor can be realized.

[0029] (5) Regarding the virtual touch method using the 3D cursor, The touch panel includes a trigger area bound to the operation target, a click finger, and a switching finger. In this embodiment, the tip of the thumb is the operation target, the index finger is the click finger, and the middle finger, ring finger, and little finger other than the thumb and index finger are the switching fingers that trigger the projection of a 3D cursor. When the switching finger touches the trigger area, it obtains the spatial positions of the operation target and the preset light source, and forms an interactive operation wire that is connected in a straight line or a radial direction and resembles a laser pen (straight line) or a fishing rod (parabola) along the straight line or radial direction toward the outside of the operation target. Before the click finger clicks and touches the trigger area, if the switching finger is released and no longer touches the trigger area, the interactive operation wire disappears, and the interactive operation wire is displayed. At the same time, the direction of the operation target is changed to guide the movement of the interactive operation wire. When the interactive operation wire intersects with a virtual object (or a virtual model of the real thing) in the virtual space, a 3D cursor is displayed at the intersection. At this time, if the click finger touches the trigger area, various operations similar to those of a mouse button, such as clicking, dragging, selecting, and drawing, can be performed on the virtual object touched by the 3D cursor. To realize left and right button operation of a mouse, two different click fingers are defined. For example, the tip of the thumb is used as the operation target. As shown in Figures 6 to 8, the index finger is used as the right button click finger, the middle finger is used as the left button click finger, and the ring finger is used as the switching finger that activates the projection of a 3D cursor. When the tip of the thumb touches the ring finger to display the interactive operation wire, when the index finger touches the trigger area, it is a right button click operation of the mouse, and when the middle finger touches the trigger area, it is a left button click operation of the mouse.

[0030] (6) 3D brushes realize virtual operations such as virtual brushes. The interactive operation wire is a virtual brush with a preset length, and the far end of the interactive operation wire is the position of the pen tip. When the click finger touches the trigger area, the pen tip of the virtual brush can display a dot or stroke in the air, thereby realizing spatial writing or drawing operations such as drawing pictures or writing letters.

[0031] Example 1 The first embodiment of the present invention relates to a virtual touch method using a three-dimensional cursor, which is applied to an XR augmented reality wearable device and a head-mounted display system. In the virtual space, a preset position of a human wrist joint, fingertip, or weighted average position of multiple joints projected by a preset light source is defined as an operation target. An interactive operation wire is formed by projecting from the preset light source through the operation target, and a three-dimensional cursor is displayed at the far end of the interactive operation wire. Step 1: A trigger area is bound to the operation target, and a switching finger for triggering the projection of the 3D cursor and a click finger for touching the trigger area are defined. In this embodiment, the tip of the thumb is the operation target, the index finger is the click finger, and any of the middle finger, ring finger, and little finger excluding the thumb and index finger is the switching finger for triggering the projection of the 3D cursor, or each is a different switching finger, and different switching fingers can activate different functions; and step 2, when the switching finger touches the trigger area, the spatial positions of the operation target and the preset light source are obtained, and an interactive operation wire is formed and displayed by projecting it from the preset light source through the operation target; when the trigger area is not click-touched by the click finger, once the switching finger is released and no longer touches the trigger area, the interactive operation wire disappears and the interactive operation wire is displayed, and the operation target is changed in direction to guide the movement of the interactive operation wire; when the interactive operation wire forms an intersection with a virtual object or a virtual model of a real object in the virtual space, a three-dimensional cursor is displayed at the intersection; at this time, when the click finger touches the trigger area, a virtual touch is realized for the virtual object or virtual model touched by the three-dimensional cursor.

[0032] Specifically, acquiring the spatial positions of the operation target and the preset light source includes: If the preset light source is within the visible range of the glasses and is a preset joint in both hands other than the operation target, the operation target and the preset light source adopt the same spatial position calculation method, and the connecting line between the center points L / R of the left and right cameras is set as the X axis, and in the field of view of the left camera, the connecting line between the center point L of the left camera and the target joint point T of the calculation target space position and the X axis is set as TθL; similarly, in the field of view of the right camera, as shown in Figure 3, the connecting line between the center point R of the right camera and the target joint point T of the calculation target space position and the X axis is set as TθR; If the parallax distance between the two center points L and R of the left and right cameras is d, the position (X, Z) of the target joint point T can be calculated as follows: If the target joint point T is between the two center points L and R of the left and right cameras, then Equation 7 is obtained.

number

number

number

[0033] The virtual touch is defined as the operation target of the thumb, one of the other four fingers as a click finger such as the index finger, and at least one of the other three fingers as a switching finger for activating a 3D cursor projection, and when multiple switching fingers are provided, the multiple switching fingers activate different functions. For example, the middle finger corresponds to the 3D cursor, and the ring finger corresponds to the virtual brush.

[0034] The virtual touch is defined as follows: the tip of the thumb is the operation target, one of the other four fingers is a switching finger such as the ring finger that activates the projection of a 3D cursor, and two of the remaining three fingers are defined as the right mouse button click finger and the left mouse button click finger, such as the index finger and middle finger, respectively.

[0035] The interactive manipulation wire is a linear radiation beam like a laser pen or a parabolic radiation beam like a fishing rod without a fixed length.

[0036] The interactive operation wire is a virtual brush with a preset length, and the far end of the interactive operation wire is the position of the pen tip, which displays the virtual brush. When the click finger touches the trigger area, the pen tip displays a dot or stroke in the air, realizing operations such as drawing pictures or writing letters.

[0037] It should be further understood by those skilled in the art that the units and algorithm steps of each example according to the embodiments disclosed in the present invention can be combined and realized in electronic hardware, computer software, or a combination of both. In order to clearly explain the compatibility between hardware and software, the configurations and steps of each example have been generally described according to their functions in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may realize the described functions using different methods for each specific application, but such realization should not be considered beyond the scope of the present invention.

[0038] Specifically, each step of the method embodiment in the embodiments of the present application may be completed by an integrated logic circuit of hardware and / or instructions in the form of software in a processor, and the steps combined with the method disclosed in the embodiments of the present application may be directly embodied as the execution completion of a hardware decoding processor, or may be performed and completed by a combination of hardware and software modules in the decoding processor. Preferably, the software modules are stored in a storage medium mature in the field, such as a random memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads information in the memory and completes the steps of the method embodiment in combination with the hardware.

[0039] Example 2 12, a head-mounted display device 700 is provided, which includes a memory 710 and a processor 720. The memory 710 stores a computer program and transmits the program code to the processor 720. In other words, the processor 720 can call and execute the computer program from the memory 710 to implement the method in the embodiment of the present application. For example, the processor 720 is configured to execute the processing steps described in the method in the first embodiment according to instructions in the computer program.

[0040] In some embodiments of the present invention, the computer program may be divided into one or more modules that are stored in the memory 710 and executed by the processor 720 to complete the method of embodiment 1 of the present application. The one or more modules may be a series of computer program instruction segments that can complete a specific function, and the instruction segments are for describing the execution process of the computer program in the head-mounted display device 700.

[0041] 12, the head-mounted display device may further include a transceiver 730 connected to the processor 720 or the memory 710. Here, the processor 720 can control the transceiver 730 to communicate with another device, specifically, to transmit information or data to another device or receive information or data transmitted from another device. The transceiver 730 may include at least two cameras for capturing target images of at least a target area.

[0042] It should be understood that the components of the head-mounted display device 700 are connected via a bus system, which in addition to a data bus further includes a power bus, a control bus, and a status signal bus.

[0043] Example 3 A third embodiment of the present invention further provides a computer storage medium having stored thereon a computer program which, when executed by a computer, enables the computer to perform the processing steps recited in the method of the first embodiment above.

[0044] Example 4 Example 4 of the present invention further provides a chip for running instructions, including an integrated circuit substrate encapsulated therein, for performing the processing steps described in the method of Example 1 above.

[0045] The above specific embodiments have further explained the objectives, technical solutions and beneficial effects of the present invention, but it should be understood that the above are only specific embodiments of the present invention and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should all be included within the scope of the present invention. [Explanation of symbols]

[0046] 700 head-mounted display device, 710 memory, 720 processor, 730 transceiver, L center point, P trigger fingertip, R center point, T target joint point, trigger fingertip, WL left trigger decision point, WR right trigger decision point, Y pixel position

Claims

1. A virtual touch method using a three-dimensional cursor applied to an XR augmented reality wearable device and a head-mounted display system, In the virtual space, the weighted average position of a human wrist joint, fingertip or multiple joints projected by a preset light source is defined as an operation target, and an interactive operation wire is formed by projecting from the preset light source through the operation target, and a three-dimensional cursor is displayed at the far end of the interactive operation wire. Step 1, in which a trigger area is bound to an operation target, and a switching finger is defined to activate a 3D cursor projection and a clicking finger is defined to touch the trigger area; Step 2: when the switching finger touches the trigger area, obtain the spatial positions of the operation target and the preset light source, and project from the preset light source through the operation target to form an interactive operation wire; when the switching finger is released and no longer touches the trigger area before the click finger clicks and touches the trigger area, the interactive operation wire disappears, the interactive operation wire is displayed, and the operation target is changed in direction to guide the movement of the interactive operation wire; when the interactive operation wire forms an intersection with a virtual object or virtual model in the virtual space, a three-dimensional cursor is displayed at the intersection; and when the click finger touches the trigger area at this time, a virtual touch is realized for the virtual object or virtual model touched by the three-dimensional cursor; A virtual touch method using a three-dimensional cursor, comprising:

2. acquiring spatial positions of the operation target and the preset light source; If the preset light source is within the visible range of the glasses and is a preset joint on both hands other than the operation target, the operation target and the preset light source adopt the same spatial position calculation method, and the connecting line between the center points L and R of the left and right cameras is set to the X axis. In the field of view of the left camera, the angle between the connecting line between the center point L of the left camera and the target joint point T of the calculation target space position and the X axis is set to TθL. Similarly, in the field of view of the right camera, the angle between the connecting line between the center point R of the right camera and the target joint point T of the calculation target space position and the X axis is set to TθR. The parallax distance between the two center points L and R of the left and right cameras is set to d. The position (X, Z) of the target joint point T is calculated, specifically, as follows: If the target joint point T is between the two center points L and R of the left and right cameras, then Equation 1 is obtained. [Equation 1] If the target joint point T is on the left side of the center point L of the left camera, then Equation 2 is given: [Equation 2] If the target joint point T is located to the right of the center point R of the right camera, then Equation 3 is given: [Equation 3] The origin of Y is set to any point at the bottom of the display video frame, and the Y value of the pixel position in the video frame of the target joint point T is set to the Y value at the spatial position (X, Y, Z) of the target joint point T, If the preset light source is not within the visible range of the glasses, the preset light source will use a different spatial position calculation method from the operation target, and will be located at the center point of the glasses (X 中心 , Y 中心 ) is used as the light source, and the joints or fingertips of the right hand are used as the operation targets. 光源 , Y 光源 ) in which X 光源 =X 中心 +βx, Y 光源 = Y 中心 -βy, and if the joints or fingertips of the left hand are used as the operation target, the relative position of the center point of the glasses (X 光源， Y 光源 ) in which X 光源 =X 中心 -βx, Y 光源 = Y 中心 -βy, and the sum of βx and βy is a preset offset value.

2. The virtual touch method using a three-dimensional cursor according to claim 1, further comprising:

3. The switching finger or the clicking finger is used as the trigger fingertip P to touch the trigger area, and the specific determination steps are as follows: The width of the trigger area bound to the operation target is W, and the left trigger decision point WL and the right trigger decision point WR are taken at W / 2 on the left and W / 2 on the right parallel to the X axis, i.e., the left trigger decision point WL and the right trigger decision point WR are points corresponding to the two boundaries of the trigger area on the left and right. The system acquires N image video streams with a parallax distance, where N is an integer and N≧2. For the N images in the same frame, it is determined whether the positions of the trigger fingertip P in all images are between the corresponding left trigger decision point WL and right trigger decision point WR of the trigger area.

2. The virtual touch method using a 3D cursor according to claim 1, wherein the first and second images are tracked to determine whether the trigger fingertip P touches the trigger area. If so, the position values ​​of three target points in each image are calculated, the target points including a left trigger decision point WL, a trigger fingertip P, and a right trigger decision point WR. The method takes the X-axis values ​​(WRX, PX, WLX) of the position values ​​of the three target points and calculates the ratios (PX-WRX):(WLX-PX) of the difference between WL and P and the difference between T and WR. Only when all the ratios are the same for all N images, it indicates that the trigger fingertip P touches the trigger area.

4. 2. The virtual touch method using a three-dimensional cursor according to claim 1, wherein the virtual touch is defined as follows: the tip of the thumb is an operation target, one of the other four fingers is a click finger, and at least one of the other three fingers is a switching finger that activates three-dimensional cursor projection; and when multiple switching fingers are provided, the multiple switching fingers activate different functions.

5. 2. The virtual touch method using a three-dimensional cursor according to claim 1, wherein the virtual touch is defined as an operation target at the tip of the thumb, one of the other four fingers as a switching finger that activates the projection of the three-dimensional cursor, and two of the remaining three fingers as a right mouse button click finger and a left mouse button click finger, respectively.

6. The virtual touch method using a three-dimensional cursor according to claim 1 , wherein the interactive operation wire is a linear ray or a parabolic ray with no fixed length.

7. The method for virtual touch using a three-dimensional cursor according to claim 1, characterized in that the interactive operation wire is a virtual brush of a preset length, and the virtual brush is displayed. When the click finger touches the trigger area, the pen tip of the virtual brush displays a dot or a stroke in the air, realizing operations such as drawing pictures or writing letters.

8. The head-mounted display device includes at least two cameras for capturing a target image of a target area, and further includes a memory for storing a computer program and a processor for executing the computer program to realize the virtual touch method using a three-dimensional cursor described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program that, when executed by a processor, realizes the virtual touch method using a three-dimensional cursor according to any one of claims 1 to 7.

10. A chip for executing instructions, comprising an integrated circuit substrate encapsulated therein, The integrated circuit board is a chip for running instructions, characterized in that it is for executing the virtual touch method using a three-dimensional cursor according to any one of claims 1 to 7.

Images