A system for setting colliders and raycast input for hands

Abstract

The present invention relates to the field of spatial computing technology, and in particular to a system for using hands to input in a three-dimensional space. With the hand position as a reference, multiple interactive areas are mapped in the three-dimensional space, and different input events are defined for the multiple interactive areas respectively; the interactive area includes a passive sensing area and an active operation area, and the passive sensing area is triggered by the active operation area to realize the interactive input of information in the three-dimensional space. The method for the active operation area to trigger the passive sensing area is a spatial position calculation method, a collider method, or a collider plus ray method. The system for using hands to input in a three-dimensional space relies on the human body's own perception system, can bidirectionally perceive the position and rhythm of the input operation, is suitable for blind typing and fast input, can accurately input when the user moves, has the advantages of simple operation, not easy to get tired, and the virtual keyboard will not block the field of vision.

Description

[0001] This invention is a divisional application of application number: 2023113326361; application date: October 16, 2023; invention name: "System for using hands for input in three-dimensional space". Technical Field

[0002] The present invention relates to the field of spatial computing technology, and in particular to a system for performing hand input in a three-dimensional space. Background Art

[0003] With the continuous development of spatial interaction technologies such as VR, AR, and MR, it is a difficult problem to enable users to conduct human-computer interaction more conveniently in space, improve user experience, and increase input efficiency.

[0004] Commonly used input methods in the prior art include the following:

[0005] (1) Input is performed through hardware devices such as traditional keyboards, mice, and drawing tablets. It is mainly used for computer input. Some VR, AR, and MR devices can also be connected to such devices for input. Disadvantages of traditional hardware: Current hardware input devices are mainly generated in the interaction with the screen 2D space, and the user's interaction range is limited to the screen range. Among them, the use of computers has a fixed desktop, which is suitable for placing hardware devices such as keyboards, mice, and touchpads, and can be operated naturally. Mobile phones and tablets do not need a fixed desktop. The viewing angle is on the screen, and the virtual input buttons are also on the screen, which can also be operated naturally.

[0006] However, the interactive perspective in three-dimensional space is not limited to the 2D space of the screen. The head has greater freedom of turning, a wider perspective, and increased interactivity with the 3D environment around people. The interactive area has been increased by one dimension, which is the purpose of using three-dimensional space interaction. If hardware devices are used for input, it will affect the flexible and free interaction method in three-dimensional space, and lose the meaning of using three-dimensional space interaction.

[0007] (2) Display a virtual keyboard and a virtual joystick on the touch screen of the device, and then slide or tap the corresponding position on the screen with your finger for interactive input. Mainly used for mobile phones and tablets. Since most devices for 3D space interaction are head-mounted devices, they do not have touch screens. Even if a touch screen is added, the shortcomings of traditional hardware input devices still exist.

[0008] (3) Input is performed by using a virtual keyboard suspended in space imaged in the device and clicking the position of the virtual keyboard in space with a hand or a handle. Most existing VR, AR, and MR devices use this method. Since the virtual keyboard is imaged in space, there is no physical contact with the virtual keyboard whether clicking with a finger or a handle, resulting in the following defects:

[0009] ① The user's arm will be very tired during the operation. Since there is no proper physical contact as support, the suspended operation of the hand will cause the user's arm to fatigue quickly. ② Blind typing is not possible. The virtual keyboard in the space only gives the user visual instructions, and there is no tactile position mark like the "F" and "J" keys in the keyboard, and unlike the keyboard, the arm can be supported on the table as a perceptual reference. The user must look at the virtual keyboard to accurately click the corresponding key position for input. ③ It is not suitable for input while moving. As long as the user's position changes, even if the virtual keyboard in the space moves with the user, it is difficult to accurately click the key position above. ④ It is impossible to input quickly and in large quantities, because relying on visual perception of spatial distance, each input will be very slow, resulting in the current three-dimensional space input can only slowly input a small amount of content. ⑤ Displaying a virtual keyboard in space will block the user's field of vision to a certain extent.

[0010] (4) By establishing the correspondence between specific gestures and characters, and then using the hand to make the corresponding gesture image in front of the image acquisition device, the corresponding character is input. Common methods include training models through artificial intelligence or directly using specific algorithms to recognize gestures. This input method has the following defects: a large number of gestures are required to participate in the interaction, and it is difficult for users to remember the correspondence between these gestures and characters. Some gestures require hand flexibility and are not suitable for all users. The input efficiency is low. Switching gesture input for each character will cause hand fatigue and the speed is also very slow. It can only simulate character input, and cannot simulate mouse, joystick, or touchpad input. If artificial intelligence is used to train the correspondence between images and characters, the development cost will increase and the trained model will take up a large amount of space. If images are used to detect gestures, the gesture and the device that collects the image need to be at a specific angle. If the angle deviation is too large, the recognition rate will be very low.

[0011] (5) Input is performed through bioelectricity. Wearing a bracelet, ring or other bioelectricity sensing device, posing in different postures or touching a specific position of the device for feedback input. Since the technology for bioelectric posture detection is not yet mature, the recognition accuracy is very low.

[0012] (6) Projecting a virtual keyboard or touchpad to a certain location in real space, and then clicking the projected location with your hand to trigger the corresponding key input through sensing mechanisms such as blocking light. Although projection allows users to click and swipe on physical locations such as tables, it is impossible to perform blind typing or fast input due to the lack of feature point touch, and it cannot meet the needs of users to input while moving. Summary of the invention

[0013] The technical problem that the present invention intends to solve is to provide a system for using hands for input in three-dimensional space in response to the above shortcomings. The system relies on the human body's own perception system, can bidirectionally perceive the position and rhythm of input operations, is suitable for blind typing and fast input, and can accurately input while the user moves. The system has the advantages of simple operation, low fatigue, and the virtual keyboard will not block the field of vision.

[0014] To solve the above problems, the technical solution adopted by the present invention is as follows:

[0015] A system that uses hands for input in three-dimensional space sets multiple interactive areas on the hands and defines different input events for the multiple interactive areas respectively; the interactive areas include passive perception areas and active operation areas, and the active operation areas are used to trigger the passive perception areas to realize interactive input of information in three-dimensional space.

[0016] As an improvement, the method for the active operation area to trigger the passive sensing area is a spatial position calculation method. A coordinate system is established at the position of the passive sensing area, and a trigger range is set. When the active operation area enters the trigger range, interactive input of information in three-dimensional space can be performed.

[0017] As an improvement, when the passive sensing area is a plane, three tracking points that are not on the same straight line are selected on the passive sensing area to establish the coordinate system, and the coordinate system is a two-dimensional coordinate system; a spatial passive sensing area corresponding to the position of the passive sensing area is established in the coordinate system; an operating point is selected on the active operation area; when the projection of the operating point on the coordinate system is located in the spatial passive sensing area, and the vertical distance from the operating point to the coordinate system is less than a set value, it is determined that the active operation area enters the trigger range, and interactive input of information in three-dimensional space can be performed.

[0018] As an improvement, when the passive sensing area is a three-dimensional space, the coordinate system is established at the position of the passive sensing area, and the coordinate system is a three-dimensional coordinate system; an operation point is selected on the active operation area; when the distance from the operation point to the passive sensing area is less than a set value, the active operation area is determined to enter the trigger range, and interactive input of information in the three-dimensional space can be performed.

[0019] As an improvement, the method for the active operation area to trigger the passive sensing area is a collider method, and colliders are respectively set up at the position of the passive sensing area and the position of the active operation area, and the position of the collider is synchronized with the position of its corresponding passive sensing area or active operation area in real time; when the collider corresponding to the active operation area collides or intersects with the collider corresponding to the passive sensing area, interactive input of information in three-dimensional space can be performed.

[0020] As an improvement, the method for the active operation area to trigger the passive perception area is a collider plus ray method, a collider is set up at the position of the passive perception area, and the position of the collider is synchronized with the position of its corresponding passive perception area in real time; rays are set in the 3D engine, and an operation point is selected on the active operation area as the endpoint of the ray; rays are emitted from the active operation area to the passive perception area, and interactive input of information in the three-dimensional space is performed according to the collision point of the ray on the collider.

[0021] As an improvement, when mapping multiple interactive areas in three-dimensional space with the hand position as a reference, a combination of one or more of the medical anatomical structure, hand texture structure, tactile structure and spatial position structure is used; the multiple interactive areas are numbered and classified, and the first-level distinction marking is performed based on the hand structure, the second-level distinction marking is performed based on the association of the commonly used parts of the hand, and the third-level distinction marking is performed based on the specific position of the hand; an input layout is established: different input events are defined for multiple interactive areas, the functional area layout is performed, and an interactive input mode of information in space is established.

[0022] As an improvement, the passive sensing areas and active operation areas in multiple interactive areas are combined and used to set up multiple groups of functional area layouts. Different functional area layouts can be switched through specific instructions or gestures.

[0023] As an improvement, the input layout includes a combination of one or more of a keyboard mode input layout, a mouse mode input layout, a touchpad mode input layout, a joystick mode input layout, and a floating operation mode input layout.

[0024] The present invention adopts the above technical solution, and has the following advantages compared with the prior art:

[0025] When using this system for input, in addition to the spatial interaction device itself, the method no longer requires carrying additional input devices, thereby solving the problem in the prior art that hardware input devices are inconvenient to carry.

[0026] The operation of this system does not require complex postures or movements, but only simple hand movements. When using this system for input, users can input accurately while moving. This system relies on the human body's own perception system, which can bidirectionally perceive the position and rhythm of input operations, and is suitable for blind typing and fast input.

[0027] This system relies on the contact between the hand feature points for judgment, and does not require the recognition of specific angle graphics. This system does not require the arm to be raised significantly to click, but only requires a small movement of the hand. The operation range and posture are similar to the current operation of a smartphone, and will not cause fatigue. This system can not only simulate keyboard input, but also simulate input methods such as mouse, touch screen and joystick.

[0028] The system does not need to display corresponding prompts of key positions during input, and the corresponding key position area can be accurately found through the perception of the hand, thus solving the problem of virtual keyboard blocking the field of vision in the prior art.

[0029] The present invention is described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Schematic diagram of the medical anatomical structure of the hand in Example 1 Figure 1 ;

[0031] Figure 2 Schematic diagram of the medical anatomical structure of the hand in Example 1 Figure 2 ;

[0032] Figure 3 is a schematic diagram of the hand texture structure in Example 1;

[0033] Figure 4 Schematic diagram of the interactive area of ​​the hand marked by bones in Example 1 Figure 1 ;

[0034] Figure 5 Schematic diagram of the interactive area of ​​the hand marked by bones in Example 1 Figure 2 ;

[0035] Figure 6 Schematic diagram of the interactive area of ​​the hand marked by the handprint in Example 1 Figure 1 ;

[0036] Figure 7 Schematic diagram of the interactive area of ​​the hand marked by the handprint in Example 1 Figure 2 ;

[0037] Figure 8 Schematic diagram of the interactive area marked by touch of the hand in Example 1 Figure 1 ;

[0038] Fig. 9 Schematic diagram of the interactive area marked by touch of the hand in Example 1 Figure 2 ;

[0039] Fig.10 Schematic diagram of the interactive area of ​​the palm marked by touch in Example 1 Figure 1;

[0040] Fig.11 Schematic diagram of the interactive area of ​​the palm marked by touch in Example 1 Figure 2 ;

[0041] Fig.12 Schematic diagram of the interactive area of ​​the palm marked by touch in Example 1 Figure 3 ;

[0042] Fig.13 This is a schematic diagram of the interactive area of ​​the palm marked by the spatial position in Example 1;

[0043] Fig.14 The horizontal palm nine-square grid input layout in Example 1 Figure 1 ;

[0044] Fig.15 The horizontal palm nine-square grid input layout in Example 1 Figure 2 ;

[0045] Fig.16 The horizontal palm nine-square grid input layout in Example 1 Figure 3 ;

[0046] Fig.17 The horizontal palm nine-square grid input layout in Example 1 Figure 4 ;

[0047] Fig.18 The horizontal palm nine-square grid input layout in Example 1 Figure 5 ;

[0048] Fig.19 The usage status reference of the horizontal palm nine-square grid input layout in Example 1 Figure 1 ;

[0049] Fig. 20 The usage status reference of the horizontal palm nine-square grid input layout in Example 1 Figure 2 ;

[0050] Fig.21 The usage status reference of the horizontal palm nine-square grid input layout in Example 1 Figure 3 ;

[0051] Fig. 22 The usage status reference of the horizontal palm nine-square grid input layout in Example 1 Figure 4 ;

[0052] Fig.23 A reference diagram of the use state of the mouse mode input layout in Example 1;

[0053] Fig.24 A reference diagram of the use status of the dual-stick mode input layout in Example 1;

[0054] Fig.25 A reference diagram of the use status of the single joystick mode input layout in Example 1;

[0055] Fig.26 Inputting a layout for the suspension operation mode in Example 1;

[0056] Fig. 27 The usage status reference of the floating operation mode input layout in Example 1 Figure 1 ;

[0057] Fig.28 The usage status reference of the floating operation mode input layout in Example 1 Figure 2 ;

[0058] Fig.29 This is the standard diagram for hand joint tracking of the OpenXR standard;

[0059] Fig.30 A reference diagram of the interactive area in the input layout for triggering keyboard mode in Example 2;

[0060] Fig.31 is a schematic diagram of the perpendicular distance from a point to a surface in Example 2;

[0061] Fig.32 A reference diagram of the interactive area in the layout for triggering the mouse mode input in Example 2;

[0062] Fig.33 A reference diagram of the interactive area in the trigger board mode input layout in Example 2;

[0063] Fig.34 The usage status reference of Example 1 of the present invention Figure 1 ;

[0064] Fig.35 The usage status reference of Example 1 of the present invention Figure 2 ;

[0065] Fig.36 The usage status reference of Example 1 of the present invention Figure 3 ;

[0066] Fig.37 The usage status reference of Example 1 of the present invention Figure 4 ;

[0067] Among them, 1 - Bone_Index_Tip; 2 - Bone_Index_Distal; 3 - Bone_Index_Intermediate; 4 - Bone_Index_Proximal; 5 - Bone_Index_Metacarpal; 6 - Bone_Thumb_Tip; 7 - Bone_Thumb_Distal; 8 - Bone_Thumb_Proximal; 9 - Bone_Thumb_Metacarpal; 10 - Bone_Pinky_Tip; 11 - Bone_Pinky_Distal; 12 - Bone_Pinky_Intermediate; 13 - Bone_Pinky_Proximal; 14 - Bone_Pinky_Metacarpal; 15 - Bone_Middle_Tip; 16 - Bone_Middle_Distal; 17 - Bone_Middle_Intermediate; 18 - Bone_Middle_Proximal; 19 - Bone_Middle_Metacarpal; 20 - Bone_Ring_Tip; 21 - Bone_Ring_Distal; 22 - Bone_Ring_Intermediate; 23 - Bone_Ring_Proximal; 24 - Bone_Ring_Metacarpal; 25 - Striation_Index_Distal; 26 - Striation_Index_Intermediate; 27 - Striation_Index_Proximal; 28 - Striation_Thumb_Distal; 29 - Striation_Thumb_Proximal; 30 - Striation_Pinky_Distal; 31 - Striation_Pinky_Intermediate; 32 - Striation_Pinky_Proximal; 33 - Striation_Middle_Distal; 34 - Striation_Middle_Intermediate; 35 - Striation_Middle_Proximal; 36 - Striation_Palm_Vertical; 37 - Striation_Wrist_Horizontal; 38 - Striation_Palm_Horizontal; 39 - Striation_Ring_Proximal; 40 - Striation_Ring_Intermediate;41-Striation_Ring_Distal; 42-Tactile_Index_Distal; 43-Tactile_Index_Intermediate; 44-Tactile_Index_Proximal; 45-Tactile_Thumb_Distal; 46-Tactile_Thumb_Proximal; 47-Tac tile_Pinky_Proximal;48-Tactile_Pinky_Intermediate;49-Tactile_Pinky_Distal;50-Tactile_Middle_Distal;51-Tactile_Middle_Intermediate;52-Tactile_Middle_Proximal;53-T actile_Ring_Proximal;54-Tactile_Ring_Intermediate;55-Tactile_Ring_Distal;56-Tip;57-Distal;58-Intermediate;59-Proximal;60-Palm;61-Metacarpal;62-Wrist;63-Tip;64-Di stal; 65-Proximal; 66-Metacarpal; 67-Thumb; 68-Index; 69-Palm; 70-Wrist; 71-Middle; 72-Palm; 73-Ring; 74-Little; 75-Tip; 76-Distal; 77-Intermediate; 78-Proximal; 79-Metacarpal. ; DETAILED DESCRIPTION

[0068] Example 1

[0069] A system for using hands for input in three-dimensional space includes determining an interactive area, establishing an input layout, and triggering the interactive area in the input layout.

[0070] Determine the interactive area: Set multiple interactive areas on the hand, and define different input events for each of the interactive areas. The interactive area includes a passive sensing area and an active operation area. The passive sensing area is triggered by the active operation area to realize interactive input of information in three-dimensional space. In a system that uses hands for input in three-dimensional space, the interactive input modes include: keyboard mode, mouse mode, touchpad mode, joystick mode, and floating operation mode.

[0071] The first step is to identify the interactive areas in the hand structure that can be used for input operations. The structures of the human hand used in this system include: medical anatomical structure, hand texture structure, tactile structure, and spatial position structure. These positions can be clearly perceived by the human body through touch, vision, or spatial sense, which facilitates input operations. Small interactive areas can be used as buttons, large interactive areas can be used as virtual joysticks, touchpads, etc., and bar-shaped interactive areas can be used as sliders.

[0072] like Figure 1 and Figure 2 As shown in the figure, the medical anatomical structure in this system refers to the position of the bones and muscles of the hand. Common positions in the hand bones are: the tip of the distal phalanx, the joint between the distal phalanx and the middle phalanx, the joint between the middle phalanx and the proximal phalanx, the joint between the proximal phalanx and the metacarpal bone, and the finger joints. Common positions in the hand muscles are: thenar muscle and hypothenar muscle. Figure 3 As shown, the hand texture structure in this system refers to the position of the texture when the fingers and palm are bent.

[0073] The tactile structure in this system refers to the area in the hand that has no obvious texture or bone boundaries, but is distinguished by tactile perception and the difficulty of contact, such as: the entire palm area, the palm area, the palm edge area, the five fingertips area, etc.

[0074] The spatial position structure in this system refers to the position that has no direct contact with the hand but can be clearly perceived by the hand, such as: the area above the palm, the edge of the palm, the top of the back of the hand, etc.

[0075] To confirm the interactive area in the hand structure that can be used for input operations, it is necessary to obtain the tracking position of the hand joints, the mapping bones, and the hand mapping mesh. These data are easy to obtain, and 3D engines, hardware manufacturers, etc. will provide interfaces for tracking the position of hand joints. 3D engines that develop three-dimensional software, such as Unity, UE, etc., will provide SDKs for spatial hand capture, such as Unity's XR Interaction Toolkit SDK. Hardware manufacturers such as HoloLens2, Pico, MagicLeap, and Quest will also provide this data in their own development tools; I will not go into details here.

[0076] In this embodiment, the hand joint tracking standard of the OpenXR standard is preferably used as an example. OpenXR is a standard for the development of VR, AR, and XR technologies. Fig.29 As shown, these are the joint points that can be tracked in OpenXR.

[0077] The second step is to identify the interactive areas in the hand structure that can be used for input operations and then number and classify the specific interactive areas according to their functions.

[0078] The first-level distinguishing mark is made based on the structure. In this embodiment, preferably, the position related to the bone is recorded as Bone. The position related to the muscle is recorded as Muscle. The position related to the texture on the hand is recorded as Striation. The position related to the sense of touch is recorded as Tactile. The position related to the spatial position is recorded as Space.

[0079] Secondary differentiation marking is performed based on the association of commonly used parts. In this embodiment, preferably, the palm-related position is recorded as Palm, the wrist-related position is recorded as Wrist, the thumb-related position is recorded as Thumb, the index finger-related structure is recorded as Index, the middle finger-related position is recorded as Middle, the ring finger-related position is recorded as Ring, the little finger-related position is recorded as Pinky, and the complete palm is recorded as Hand.

[0080] The three-level distinguishing mark is made based on the specific location and combined with the first and second levels to form a complete mark.

[0081] like Figure 4 and Figure 5 As shown in the figure, the hand is marked with bones as the interactive area. In this embodiment, preferably, the thumb is marked with bones as the interactive area:

[0082] Bone_Thumb_Tip,Bone_Thumb_Distal,Bone_Thumb_Proximal,Bone_Thumb_Metacarpal

[0083] The interactive area of ​​the index finger is marked by bones:

[0084] Bone_Index_Tip,Bone_Index_Distal,Bone_Index_Intermediate,Bone_Index_Proximal, Bone_Index_Metacarpal

[0085] Interactive areas marked by bones in the pinky finger: Bone_Pinky_Tip, Bone_Pinky _Distal, Bone_Pinky_Intermediate, Bone_Pinky_Proximal, Bone_Pinky_Metacarpal

[0086] Interactive areas of the middle finger marked by bones: Bone_Middle_Tip, Bone_Pinky _Distal, Bone_ Middle _Intermediate, Bone_ Middle _ Proximal, Bone_ Middle_Metacarpal

[0087] Interactive areas marked by bones in the ring finger: Bone_Ring_Tip, Bone_Pinky _Distal, Bone_ Ring _Intermediate, Bone_ Ring_Proximal, Bone_Ring _Metacarpal

[0088] like Figure 6 and Figure 7 As shown in the figure, the thumb area is marked by hand lines:

[0089] Striation_Thumb_Distal, Striation_Thumb_Proximal

[0090] The interactive area on the index finger is marked by the handprint:

[0091] Striation_Index_Distal, Striation_Index_Intermediate, Striation_Index_Proximal

[0092] Interactive areas on the pinky finger marked by hand lines: Striation_Pinky_Distal, Striation_Pinky_Intermediate, Striation_Pinky_Proximal

[0093] Interactive areas on the middle finger marked by handprints: Striation_Pinky_Distal, Striation_Middle_Intermediate, Striation_Middle_Proximal

[0094] The interactive area on the ring finger marked by the handprint: Striation_Ring_Distal, Striation_Ring_Intermediate, Striation_Ring_Proximal

[0095] like Figure 8 and Fig. 9As shown in the figure, the thumb area is marked with touch as the interactive area:

[0096] Tactile_Thumb_Distal, Tactile_Thumb_Proximal

[0097] The interactive area on the index finger is marked by tactile sensation:

[0098] Tactile_Index_Distal, Tactile_Index_Intermediate, Tactile_Index_Proximal

[0099] Interactive areas on the pinky finger marked by tactile sensations: Tactile_Pinky_Distal, Tactile_Pinky_Intermediate, Tactile_Pinky_ Proximal

[0100] Interactive areas of the middle finger marked by tactile sensations: Tactile_Middle_Distal, Tactile_Middle_Intermediate, Tactile_Middle_Proximal

[0101] Interactive areas on the ring finger marked by tactile sensations: Tactile_Ring_Distal, Tactile_Ring_Intermediate, Tactile_Ring_Proximal

[0102] like Fig.10 , Fig.11 and Fig.12 Shown together are the interactive areas of the palm marked by touch. Fig.10 The interactive area in the game is labeled: Tactile_Hand_Plane. Fig.11 In the figure, the interactive area corresponding to the finger is marked as: Tactile_Finger_Plane; the interactive area corresponding to the palm is marked as: Tactile_Palem_Center. Fig.12 In the figure, the interactive area corresponding to the finger is marked as: Tactile_FingerMain; the interactive area corresponding to the thenar is marked as: Tactile_Hand_Thenar; the interactive area corresponding to the hypothenar is marked as: Tactile_Hand_Hypothenar.

[0103] like Fig.13As shown, the palm’s spatial position is used as the interactive area: Space_Hand_Suspend.

[0104] The third step is to establish the input layout. Select the interactive input mode according to the actual functional needs, including keyboard mode, mouse mode, touchpad mode, joystick mode, and floating operation mode. Different interactive input modes are matched and combined with different interactive areas, and different input events are assigned to the interactive areas.

[0105] Create a keyboard mode input layout: Place the characters to be input in different interactive areas, and place the auxiliary functions required for inputting characters in nearby interactive areas. Input is performed by clicking, sliding, etc. Fig.14 , Fig.15 , Fig.16 , Fig.17 and Fig.18 As shown together, in this embodiment, the preferred keyboard mode input layout is a horizontal palm nine-square input layout. Of course, a vertical palm nine-square input layout or other layouts can be used according to the needs of actual applications.

[0106] like Fig.14 As shown, the Chinese input method can be used with the existing commonly used nine-square grid input layout, and the corresponding relationship is as follows:

[0107] like Fig.15 As shown, in Fig.14 Switching the digital panel based on the Chinese input method only changes the functions of some interactive areas. The corresponding changes are:

[0108] like Fig.16 As shown in the figure, switching to the English panel based on the Chinese input method only changes the functions of some interactive areas. The corresponding relationships are as follows:

[0109] Make a fist and then release it to switch to the second page of the interactive area of ​​the English letters. Fig.17 As shown, the corresponding relationship of the change is:

[0110] Make a fist and then release it to switch to the third page of the interactive area of ​​the English letters. Fig.18 As shown, the corresponding relationship of the change is:

[0111] This arrangement is to use one hand as a keyboard, while the other hand performs single-finger click, double-finger click, sliding and other operations. According to the hand habits of most operators, the interactive area set on the palm of the left hand is preferably a passive sensing area, and the interactive area on the palm of the right hand is an active operation area.

[0112] like Fig.19As shown, single-finger click to input the corresponding character. Click the Bone_Index_Tip area on the right hand and touch the Tactile_Index_Proximal area on the left hand to input the character "U".

[0113] like Fig. 20 As shown, after changing from the non-contact state to the contact state, the slider operation area is slid back and forth in the longer direction to drag the corresponding slider in the virtual space. Fig. 20 In the figure, the right hand Bone_Index_Tip area touches the left hand Tactile_Hand_Hypothenar area to slide, causing the associated horizontal slider to slide accordingly. The non-contact state here refers to the state where the interactive area in the input layout is not triggered; the contact state refers to the state where the interactive area in the input layout is triggered. As for how to trigger the interactive area in the input layout, it will be explained later.

[0114] From the non-contact state to the contact state, slide in the circular touch area in four operation directions, and distinguish the operation mode by double fingers and single finger. Single finger sliding controls the position of the cursor at the input character position, such as Fig.21 As shown, the right hand Bone_Index_Tip area touches the center of the left hand Tactile_Palm_Center area and then slides, so that the cursor continues to move in the sliding direction. If the right hand Bone_Index_Tip area directly touches the left, right, top, and bottom positions around the left hand Tactile_Palm_Center area, the cursor moves one grid each time in the corresponding direction, and keeps pressing to continue moving in the corresponding direction.

[0115] like Fig. 22 As shown, two fingers slide to control the input character cursor to select characters. Fig. 22 In the middle, the right hand Bone_Middle_Tip area touches the center of the left hand Tactile_Palm_Center area and then slides, so that the cursor continues to select characters in the sliding direction. If the right hand Bone_Middle_Tip area directly touches the left, right, top, and bottom positions around the left hand Tactile_Palm_Center area, the cursor will select one more grid each time in the corresponding direction. If you keep pressing, it will continue to select in the corresponding direction.

[0116] In the selection state, the interactive area of ​​the right hand leaves the interactive area of ​​the left hand, and the selection function options such as "Copy", "Cut", and "Paste" appear in the virtual input box. At this time, whether you click the Tactile_Palm_Center area representing the direction with one or two fingers, it is the selection of the selection function option. After indicating the desired selection function option, click the middle confirmation to complete the selection function operation.

[0117] The above is just an example of a keyboard input method that can be implemented by the system. As for how to combine functions and how to set the corresponding relationship between interactive areas and characters, it can be defined by the developer or user. For example, the current horizontal palm layout can be changed to a vertical palm layout.

[0118] Establish a mouse mode input layout: use a larger functional area in one hand as the movable area of ​​the mouse. Use the other hand as a mouse, select a smaller functional area as the sliding perception point position of the mouse, and select other functional areas as the left and right mouse button areas. According to the hand habits of most operators, it is preferred in this embodiment to use a larger functional area on the palm of the left hand as the movable area of ​​the mouse, and the interactive area on the left hand is the passive perception area. Use the right hand as a mouse, select a smaller functional area as the sliding perception point position of the mouse, select other functional areas as the left and right mouse button areas, and the interactive area on the right hand is the active operation area.

[0119] Specifically, Fig.23 As shown, the Tactile_Hand_Plane area of ​​the left hand is used as the movable area of ​​the mouse. The right hand is used as the mouse, where the Bone_Middle_Tip area of ​​the right hand is used as the mouse sliding perception point, the Bone_Index_Tip area of ​​the right hand is used as the left button of the mouse when the left hand is clicked, and the Bone_Index_Tip area of ​​the right hand is used as the right button of the mouse when the Bone_Thumb_Tip area of ​​the right hand is clicked.

[0120] Establishing the touchpad mode input layout: The touchpad mode input layout is consistent with the mouse mode input layout, but the calculation method is different. Analogous to the mouse and touchpad of the hardware device, after the mouse is lifted up and the position is changed and put down, the position of the mouse pointer remains unchanged, and the position of the mouse pointer changes only after the contact is moved. Regardless of whether the touchpad moves after the contact point is lifted, the position input into the virtual panel when it is re-contacted is the position of the contact point in the sliding area with the same horizontal and vertical proportions, mapped in the virtual panel. According to the hand habits of most operators, in this embodiment, preferably, a larger functional area on the palm of the left hand is used as the movable area of ​​the mouse, and the interactive area on the left hand is the passive perception area. Use the right hand as a mouse, select one of the smaller functional areas as the sliding perception point position of the mouse, select other functional areas as the left and right mouse button areas, and the interactive area on the right hand is the active operation area.

[0121] Specifically, use Fig.24In the layout shown in the figure, in the MR device, there is a 2D virtual drawing board in space, the width and height of the drawing board are 1 meter, and the Tactile_Hand_Plane area of ​​the user's left hand is 18cm*8cm. No matter where the user's Bone_Middle_Tip touched the left hand before, if the position of the left palm is 1.8cm horizontally and 4cm vertically from the lower left, then the position of the pen stroke in the virtual drawing board is 0.1m horizontally and 0.5m vertically at the lower left corner of the virtual drawing board.

[0122] Create a joystick mode input layout. The joystick is a common function in current touch screen devices, such as mobile phones and tablets, in virtual simulations and 3D games. It is often used to control movement and steering in 3D space. You can use both hands to create a joystick mode input layout, or you can use one hand to create a joystick mode input layout.

[0123] Establish a dual joystick mode input layout. In one hand, use two areas as the joystick area, and use two areas of the other hand as the joystick control area. According to the hand habits of most operators, in this embodiment, it is preferred that the interactive area on the left hand is a passive sensing area; the interactive area on the right hand is an active operation area.

[0124] Specifically, Fig.24 As shown, in the game, use the Tactile_Palm_Center area of ​​the left hand as the mobile joystick, and use the Tactile_Finger_Main area of ​​the left hand as the perspective joystick. Use the Bone_Thumb_Tip area of ​​the right hand to touch the mobile joystick area for mobile control, use the Bone_Middle_Tip area of ​​the right hand to touch the perspective joystick area for perspective control, and use the Bone_Index_Tip area of ​​the right hand to click on actions or touch the left hand to trigger special functions.

[0125] Create a single joystick mode input layout. In the same hand, either the left or right hand can be used. Use a larger interactive area as the joystick area and another smaller interactive area as the joystick control area. The joystick area is the passive sensing area; the joystick control area is the active operation area.

[0126] Specifically, Fig.25 As shown, taking the left hand as an example, the Tactile_Finger_Main area is used as the joystick area, and the Bone_Thumb_Tip area is used as the joystick control area.

[0127] A floating operation mode input layout is established, with one hand as the fixed hand, the spatial interactive area of ​​the fixed hand as the operating table, and the smaller interactive area near the operating table as the function button. Both the operating table and the function button are passive sensing areas. The other hand is used as the mobile hand, the smaller interactive area in the mobile hand is used for operation, and the interactive area in the mobile hand is the active operation area. According to the hand-using habits of most operators, it is preferred in this embodiment that the left hand is the fixed hand and the right hand is the mobile hand.

[0128] Specifically, Fig.26 As shown, adjusting the position of a model in space requires a suspended operation mode input layout. The left hand is used as the fixed hand, and the Space_Hand_Suspend area is used as the suspended interactive area. Tactile_Index_Distal represents redoing the previous operation, Tactile_Middle_Distal represents undoing the previous operation, Tactile_Ring_Distal represents displacement, and Tactile_Pinky_Distal represents rotation.

[0129] like Fig. 27 As shown, the right hand is used as the moving hand. In the rotating state, the Bone_Index_Tip of the right hand is rotated in the operating console to control the rotation of the target object.

[0130] like Fig.28 As shown, in the displacement state, the Bone_Index_Tip and the Bone_Thumb_Tip of the right hand are closed after entering the operating table area, and the displacement of the object begins to be controlled.

[0131] Trigger the interactive area in the input layout. The triggering methods include click, slide, surround, and drag.

[0132] The trigger needs to be a passive sensing area as a reference system and an active operation area used to dominate the movement. In the layout example, except for the single joystick mode input layout, all the interactive areas of the left hand in other input layouts are passive sensing areas, and all the interactive areas of the right hand are active operation areas. In the single joystick mode input layout, Tactile_Finger_Main is the passive sensing area, and Bone_Thumb_Tip is the active operation area.

[0133] The passive sensing area is divided into a flat area and a three-dimensional area. Except for the input layout of the hover operation mode, which is a three-dimensional area, other passive sensing areas are flat.

[0134] Trigger the interactive area in the input layout in a way of collider. Colliders are set up at the positions of the passive sensing area and the active operation area respectively, and the positions of the colliders are synchronized with the positions of their corresponding passive sensing areas or active operation areas in real time; when the collider corresponding to the active operation area collides with or penetrates the collider corresponding to the passive sensing area, interactive input of information in three-dimensional space can be performed.

[0135] Specifically, create a collider similar in size to the interactive area, and synchronize the position of the collider with the corresponding position in real time. When the triggerable collider combinations in the input layout collide or penetrate, the corresponding effects are triggered.

[0136] The collider method is implemented through real-time coordinate synchronization, that is, the position where the corresponding collider should be located is calculated based on the obtained hand joint positions and angles, and the collider is placed at this position and angle. Taking the input layout in keyboard mode as an example, the Unity engine and the MRTK2 development package are used, and the hardware device is HoloLens2.

[0137] Set up the collider: Create an empty object, attach a collider component to the empty object, and adjust the collider component to make its size consistent with the size of the sensing area.

[0138] The "Chinese / English" passive sensing area: The position for "Chinese / English" switching in the left palm is exactly the position of the Proximal joint of the ring finger. The collider is corresponding to the number of this trigger area, and the position is synchronized with the point tracking this joint in real time.

[0139] The "K" character passive sensing area: The position of the character "K" is not at the joint point, but in the middle of the two joints of the middle finger. The position of the midpoint in space is obtained through the Distal joint position and the Intermediate joint position of the middle finger.

[0140] The "joystick" passive sensing area: The collider corresponding to the joystick at the palm follows the position of the Palm joint of the left hand.

[0141] The "slider" passive sensing area: The position of the collider corresponding to the slider at the hypothenar follows the center point of the Proximal joint and the Metacarpal of the little finger.

[0142] The position of the tip of the right index finger follows a collider A, which is the active operation area. When A touches the collider at the "Chinese / English" place, the Chinese / English switching can be triggered. When A touches the collider at the "K" place, the input of the character "K" can be triggered.

[0143] When A hits the collider at the joystick, the joystick event starts to be triggered. As long as A does not leave the collider at the joystick, the current direction can be obtained through the aspect ratio of the collision point in the joystick collider, and the corresponding direction value can be entered.

[0144] When A hits the collider at the hypothenar eminence, the slider event starts to be triggered. As long as A does not leave the collider at the slider, the current slider ratio can be obtained through the ratio of the position of their collision point in the slider collider, and the corresponding value can be entered.

[0145] This is done by making it a child of the bound joints in the mapped hand mesh. Find the prefab model used by the hand mesh mapping for tracking hands in the SDK. Place the collider on the corresponding position of the prefab model.

[0146] The hand mesh mapping is driven by the joint hierarchy to which it is bound. The collider moves according to the joint to which it is bound, and the collider is made a child of the joint hierarchy.

[0147] In this way, each collider will be translated and rotated synchronously with the hand mapping.

[0148] The keyboard mode input layout is used as an example to illustrate the use of Unity engine, MRTK2 development kit, and HoloLens2 as the hardware device.

[0149] Find the mesh mapping prefab used to track the hand, place the prefab in the scene, and place the collider at the corresponding position of the prefab hand.

[0150] "Chinese / English" passive sensing area: The position of the "Chinese / English" switch in the left palm is exactly the position of the proximal point of the ring finger joint. The collider corresponds to the number of the trigger area, and the position is consistent with the point tracked to the joint in real time.

[0151] Passive perception area of ​​the character “K”: The position of the character “K” is not at the joint point, but between the two joint points of the middle finger. The position of their spatial midpoint is calculated through the Distal joint position and the Intermediate joint position of the middle finger.

[0152] "Joystick" passive sensing area: The collider corresponding to the joystick at the palm of the hand can follow the position of the left hand joint point Palm.

[0153] "Slider" passive sensing area: The collider position corresponding to the slider at the hypothenar follows the center point of the proximal and metacarpal of the little finger.

[0154] The position of the fingertip of the right index finger follows a collider A, which is the active operation area. When A touches the collider at the "Chinese / English" position, the Chinese-English switch can be triggered. When A touches the collider at the "K" position, the input of the character "K" can be triggered.

[0155] When A touches the collider at the joystick, the joystick event starts to be triggered. As long as A does not leave the collider at the joystick, the current represented orientation can be known from the aspect ratio of the horizontal and vertical positions of their collision point in the joystick collider, and the corresponding direction value can be input.

[0156] When A touches the collider at the hypothenar, the slider event starts to be triggered. As long as A does not leave the collider at the slider, the current represented slider ratio can be known from the ratio of the position of their collision point in the slider collider, and the corresponding value can be input.

[0157] Such as Fig.34 、 Fig.36 、 Fig.36 and Fig.37 As shown jointly by

[0158] When the system for hand input in three-dimensional space of the present invention is used, when using this system for input, except for the spatial interaction device itself, this method no longer requires an additional input device to be carried; it solves the problem in the prior art that the hardware input device is not convenient to carry.

[0159] The operation mode of this system does not require complex postures or movements, and can be a standing operation or a sitting operation. Only simple hand movements are required to complete. When using this system for input, the user can also input accurately when moving. This system relies on the human body's own perception system, can sense the position and rhythm of the input operation bidirectionally, and is suitable for touch typing and fast input.

[0160] This system relies on the contact between the hand feature points for judgment, and does not require the recognition of specific angle graphics. This system does not require the operation of lifting the arm significantly to click, only requires small movements of the hand, and the operation amplitude and posture are similar to the current operation of a smart phone, and will not cause fatigue. This system can not only simulate keyboard input, but also simulate input methods such as mouse, touch screen and joystick.

[0161] Embodiment 2

[0162] The difference between this embodiment and Embodiment 1 is that when triggering the interactive area in the input layout, the triggering method is a spatial position calculation method. That is, a coordinate system is set up at the position of the passive sensing area, and the triggering range is set. When the active operation area enters the triggering range, information interaction input in three-dimensional space can be carried out.

[0163] When the passive sensing area is a plane, three tracking points that are not on the same straight line are selected on the passive sensing area to establish the coordinate system, which is a two-dimensional coordinate system; a spatial passive sensing area corresponding to the position of the passive sensing area is established in the coordinate system; an operating point is selected on the active operation area. When the projection of the operating point on the coordinate system is within the spatial passive sensing area, and the vertical distance from the operating point to the coordinate system is less than the set value, the active operation area is determined to enter the trigger range, and interactive input of information in three-dimensional space can be performed.

[0164] When the passive sensing area is a three-dimensional space, the coordinate system is established at the position of the passive sensing area, and the coordinate system is a three-dimensional coordinate system. A spatial passive sensing area corresponding to the position of the passive sensing area is established in the coordinate system, and the spatial passive sensing area is a three-dimensional spatial area; an operation point is selected on the active operation area. When the distance from the operation point to the spatial passive sensing area is less than the set value, the active operation area is determined to enter the trigger range, and interactive input of information in the three-dimensional space can be performed.

[0165] The spatial position calculation method triggers the interactive area in the input layout. The relative relationship between the joint points tracked by the passive area and the joint points tracked by the active operation area is judged and input.

[0166] The input of the plane interactive area, the virtual joystick, mouse, and touchpad are all input into the plane area. Only three tracking points in the plane area are needed to determine a plane, and this plane is the passive perception area.

[0167] Depending on the input method, you can select different shapes of areas in the two-dimensional space where the three-point plane is located. The plane used for mouse input is an infinite extension of the two-dimensional space formed by the three-point plane. The plane used for touchpad input is a rectangular area within the hand range in the two-dimensional space formed by the three-point plane. The plane used for virtual joystick input is a circular area within the hand range in the two-dimensional space formed by the three-point plane.

[0168] The active operation area is a point. When the distance from the point to the plane reaches the set standard, the position calculation is triggered to calculate the ratio or displacement of the vertical point from the point to the plane in the plane, and enter the horizontal and vertical ratio value or displacement value.

[0169] like Fig.30As shown in the figure, when the plane interactive area is triggered, the system uses the Unity engine, the MRTK2 development kit, and the hardware device is the HoloLens2. The Distal joint point of the left index finger, the Tip joint point of the left little finger, and the Metacarpal joint point of the left index finger corresponding to the metacarpal bone are tracked. The plane formed by these three points is used as the passively sensed plane interactive area. This plane is recorded as PlaneMDT.

[0170] The Bone_Index_Tip on the right hand side is used as the active operation area. Because the position obtained through the SDK is itself a three-dimensional coordinate point, this point is considered to be the operation point of the active operation area and is recorded as OptPoint.

[0171] like Fig.31 As shown, the vertical point TarPoint from OptPoint to the plane MDT and the distance DisTar from OptPoint to TarPoint are obtained by mathematical methods. The calculation method is a common application question type in the geometry part of junior high school and high school mathematics, and the detailed calculation process is not listed here. Set a distance DisActive, and when DisTar is less than DisActive, start triggering the plane input.

[0172] like Fig.32 As shown, when the mouse mode is triggered to input the interactive area in the layout, the interactive area is an infinite extension of the two-dimensional space formed by the PlaneMDT plane as the plane where the mouse moves. Make a two-dimensional coordinate system on the plane PlaneMDT and determine the position of the origin of the two-dimensional coordinate system. In this example, point M is used as the origin of the coordinate system and MD is used as the X-axis. The line in the plane that points to the thumb, passes through point M and is perpendicular to MD is used as the Y-axis.

[0173] Starting from the trigger input, the change of the vertical point on the plane is recorded, and this change value is input, and the mouse position in the virtual interface in the space is controlled by the input. Fig.32 As shown in the figure, each black dot represents the position of the vertical point at each time point of the input detection frequency, and the line connecting the black dots represents the trajectory of the vertical foot moving in the plane, which corresponds to the trajectory of the mouse in the spatial virtual interface.

[0174] like Fig.33 As shown, when the interactive area in the touchpad mode input layout is triggered, the interactive area is a two-dimensional space formed by PlaneMDT, a rectangular area within the hand range.

[0175] like Fig.33As shown, a 10cm line segment is extended from point M in the PlaneMDT plane toward the index finger, and the other point of the line segment is Q. A 6cm line segment is extended from point M in the PlaneMDT plane in the opposite direction of the thumb and perpendicular to MQ, and the other point of the line segment is W. According to the general definition of screen coordinates, point W is used as the origin of the rectangular input area, and a line segment WP is made through point W parallel to MQ. Then the input range of the touchpad is the rectangle WMQP.

[0176] Starting from the trigger input, the ratio of the vertical point position on the plane to the horizontal and vertical coordinates of the rectangle is input. Then the input point in the touch panel in the virtual interface in the space is controlled by the input.

[0177] If the vertical point in the rectangle is 5cm in the x direction and 3cm in the y direction, then the input is (0.5, 0.5). This is a proportional coordinate, that is, no matter how large the touchpad input range is in space, the input position is the center position. If the vertical point in the rectangle is 8cm in the x direction and 0.6cm in the y direction, then the input is (0.8, 0.1).

[0178] When triggering the interactive area in the keyboard mode input layout, the passive sensing area is a point, and the active operation area is also a point. Set a trigger distance and trigger by judging the distance. Take the Unity engine, MRTK2 development kit, and the hardware device as an example to illustrate.

[0179] The passive sensing area of ​​the number 2 on the left hand is tracked to the midpoint between the left index finger joint Dsital and Intermediate. The coordinates of Dsital and Intermediate can be obtained through the SDK, and the coordinates of their center point, i.e. the coordinates of the passive sensing area of ​​the number 2, can be obtained.

[0180] The operation sensing area Bone_Index_Tip of the right index finger is the coordinate point of the index finger Tip obtained in the SDK. Set a trigger distance DisClick. When the distance between the passive sensing area and the active operation area is less than DisClick, it triggers once to input character 2. After triggering once, you can close the distance to trigger again when the distance is greater than DisClick.

[0181] When the slider interactive area is triggered, it can be analogous to triggering the plane interactive area, which will not be described here.

[0182] When the interactive area in the input layout of the floating operation mode is triggered, a three-dimensional coordinate system is established at the location of the passive sensing area, and the trigger range of the interaction is determined. When the active operation area enters the trigger range, the corresponding input is performed according to the displacement or rotation of the active operation area.

[0183] Example: Use Unity engine, MRTK2 development kit, and the hardware device is HoloLens2.

[0184] When the interactive area in the touchpad mode input layout is triggered, the interactive area is a two-dimensional space formed by PlaneMDT. When the interactive area in the floating operation mode input layout is triggered, based on this two-dimensional space, a line perpendicular to the plane is made at point W in the rectangle WMQP as the Z axis, and the direction of the palm is the positive direction of the Z axis, so a three-dimensional coordinate system is established.

[0185] Let the right hand Bone_Index_Tip and Bone_Thumb be the active operation area. Set a value DisInSpace to enter the operation area and a value DisPinch to trigger input.

[0186] When the distance from the right hand Bone_Index_Tip to the plane is less than DisInSpace, trigger input is allowed. When the trigger input is allowed and the distance between the right hand Bone_Index_Tip and Bone_Thumb_Tip is less than DisPinch, the change value of Bone_Thumb_Tip in the three-dimensional coordinate is input.

[0187] If the associated object is a spaceship model in space, the input three-dimensional coordinate change value can be multiplied by a multiple to change the displacement of the spaceship.

[0188] Example 3

[0189] The difference between this embodiment and embodiment 1 is that when the interactive area in the trigger input layout is triggered, the triggering method is the collider plus ray input method. In the 3D engine, rays are a commonly used method. The engine interface can be called to set the ray, and the collision point where the ray collides with the virtual content and the distance of the ray can be read. For example, Unreal and Unity have this function. A collider is set up at the position of the passive sensing area, and the position of the collider is synchronized with the position of the corresponding passive sensing area in real time; a ray is set up in the 3D engine, and an operation point is selected as the endpoint of the ray on the active operation area; a ray is emitted from the active operation area to the passive sensing area, and the interactive input of information in the three-dimensional space is performed according to the collision point of the ray on the collider. When the passive sensing area is a button, the ray interface in the engine is used to obtain which button the collider that the ray collides with is, and the distance from the origin of the ray emission to the collision point can be easily obtained, and whether the button is triggered is determined by whether the distance reaches a certain value. When the passive area is a sliding area, the moving trajectory of the operation point or the collision point is recorded if the distance from the origin of the ray to the touch point reaches the standard.

[0190] In summary, the system of using hands for input in the three-dimensional space relies on the human body's own perception system, can bidirectionally perceive the position and rhythm of input operations, is suitable for blind typing and fast input, can accurately input while the user moves, and has the advantages of simple operation, low fatigue, and the virtual keyboard will not block the field of vision.

[0191] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A system for setting colliders and rays for hand input, characterized in that: Based on the hand position as a reference, multiple interactive areas are mapped in the three-dimensional space, and different input events are defined for the multiple interactive areas. The interactive areas include passive sensing areas and active operation areas. The active operation areas are used to trigger the passive sensing areas to realize the interactive input of information in the three-dimensional space. The method for the active operation area to trigger the passive sensing area is a collider plus ray method, a collider is set at the position of the passive sensing area, and the position of the collider is synchronized with the position of the corresponding passive sensing area in real time; a ray is set in the 3D engine, and an operation point is selected on the active operation area as the end point of the ray; a ray is emitted from the active operation area to the passive sensing area, and interactive input of information in the three-dimensional space is performed according to the collision point of the ray on the collider; When mapping multiple interactive areas in three-dimensional space with the hand position as a reference, a combination of one or more of a medical anatomical structure, a hand texture structure, a tactile structure, and a spatial position structure is used; To identify the interactive areas in the hand structure that can be used for input operations, it is necessary to obtain the tracking positions of the hand joints, the mapping bones, and the hand mapping mesh; to number and classify multiple interactive areas, to make primary distinction marks based on the hand structure, to make secondary distinction marks based on the association of the commonly used parts of the hand, and to make tertiary distinction marks based on the specific position of the hand; Establish input layout: define different input events for multiple interactive areas, layout functional areas, and establish interactive input mode for information in the space.

2. The system for setting colliders and rays for hand input as claimed in claim 1, characterized in that: The passive sensing areas and active operation areas in multiple interactive areas are combined and used to set up multiple groups of functional area layouts. Different functional area layouts can be switched through specific instructions or gestures.

3. The system for setting a collider and a ray for hand input as claimed in claim 1 or 2, characterized in that: The input layout includes a combination of one or more of a keyboard mode input layout, a mouse mode input layout, a touchpad mode input layout, a joystick mode input layout, and a floating operation mode input layout.

4. The system for setting colliders and rays for hand input as claimed in claim 3, characterized in that: The keyboard mode input layout is to use one of the two hands as a keyboard, and the interactive areas on it are all set as passive sensing areas; the interactive areas on the other hand are all set as active operation areas, which can be clicked with one finger, clicked with two fingers, or slid; the characters or functions that need to be input are set in different interactive areas respectively.

5. The system for setting colliders and rays for hand input as claimed in claim 3, characterized in that: The mouse mode input layout is to set a passive sensing area on one hand as the movable area of ​​the fixed mouse; and select three active operation areas in the other hand as the sliding sensing point position of the mouse, the left mouse button, and the right mouse button.

6. The system for setting colliders and rays for hand input as claimed in claim 3, characterized in that: Establishing the input layout of the touchpad mode: The input layout of the touchpad mode is consistent with the input layout of the mouse mode, but the calculation method is different. Analogous to the mouse and touchpad of the hardware device, after the mouse is lifted up and then put down, the position of the mouse pointer remains unchanged, and the position of the mouse pointer changes only when it is moved after contact. For the touchpad, regardless of whether the contact point moves after being lifted up, the position input into the virtual panel when it is re-contacted is the position of the contact point in the sliding area with the same horizontal and vertical ratios, which is mapped in the virtual panel. According to the hand habits of most operators, a functional area on the palm of the left hand is used as the movable area of ​​the mouse, and the interactive area on the left hand is the passive sensing area. Use your right hand as a mouse, select one of the functional areas as the mouse sliding perception point, select other functional areas as the left and right mouse button areas, and the interactive area on the right hand is the active operation area.

7. The system for setting colliders and rays for hand input as claimed in claim 3, characterized in that: When the joystick mode input layout is the dual joystick mode input layout, two areas are set in one hand of the operator as joystick areas, and two areas are selected in the other hand as joystick control areas; the joystick area is the passive sensing area; the joystick control area is the active operation area; When the joystick mode input layout is a single joystick mode input layout, in one hand of the operator, the interactive area is set as the joystick area, and the other interactive area is set as the joystick control area; the stick area is the passive perception area; and the joystick control area is the active operation area.

8. The system for setting colliders and rays for hand input as claimed in claim 3, characterized in that: The input layout of the floating operation mode is to use one hand of the operator as the fixed hand, set up a spatial interactive area on the fixed hand as the operation table; and set up an interactive area near the operation table as function keys; the operation table and function keys are both passive sensing areas; The other hand of the operator is used as the moving hand, and an interactive area is set on the moving hand for operation. The interactive area in the moving hand is the active operation area.

Images