Mixed reality window management interaction method and system based on palm interface

Abstract

The invention discloses a mixed reality window management interaction method and system based on a palm interface, and the method comprises the steps: collecting different contact modes of a non-habitual hand and a habitual hand of a user to an operation panel, comparing the different contact modes, and obtaining an optimal interaction mode of a palm and the operation panel; recognizing a gesture action based on the contact state of the virtual hand and the operation panel; and selecting an interaction target, and completing mixed reality desktop gesture interaction according to the recognized desktop gesture. According to the method, the 2D interface is converted into the three-dimensional mixed reality space, the participation degree of the user can be improved, the interaction speed can be increased, other gesture interaction technologies are not interfered, and the mixed reality scene requirements needing frequent interaction and long-term interaction are met.

Description

Technical Field

[0001] The present invention belongs to the technical field of mixed reality, and relates to a mixed reality window management interaction method and system based on a palm interface. Background Art

[0002] Mixed reality merges the physical and digital worlds, creating an environment where digital and physical objects coexist and interact in real time. In traditional computing environments, content is usually managed using 2D graphical user interfaces with windows, icons, menus, and pointers. However, these interfaces optimized for desktops and other displays do not translate well into three-dimensional mixed reality spaces. Current window management methods in mixed reality are mainly derived from traditional 2D interfaces, which can lead to poor user experience, reduce user engagement, and affect the development of virtual spaces. Summary of the invention

[0003] The purpose of the present invention is to solve the problem in the prior art that a 2D interface cannot be converted into a three-dimensional mixed reality space, resulting in a poor user experience for users, reducing user participation and affecting the development of virtual space, and to provide a mixed reality window management interaction method and system based on a palm interface.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] The present invention proposes a mixed reality window management interaction method based on a palm interface, comprising:

[0006] Adjusting the optimal angle of the display device based on the mixed reality head-mounted display worn by the user;

[0007] Collect different contact methods of the user's non-dominant hand and dominant hand on the operation panel, compare different contact methods, and obtain the optimal interaction method between the palm and the operation panel;

[0008] Under the premise that the interaction between the palm and the operation panel is optimal, the gesture action is recognized based on the contact state between the virtual hand and the operation panel;

[0009] Select the interaction target and complete the mixed reality desktop gesture interaction based on the recognized desktop gesture.

[0010] Preferably, the display device has two layers, each layer has several display screens; the upper and lower layers of the display screens are set to be flat, horizontally curved, vertically curved, and horizontally and vertically curved; an expert evaluation method is used to select the display device setting state that the user likes.

[0011] Preferably, the different ways in which the user's non-dominant hand and dominant hand contact the operation panel include: single-palm holding, nested-palm holding and combined holding; the single-palm holding is that the user's non-dominant hand holds the operation panel, and the dominant hand operates the operation panel; the nested-palm holding is that the fingers of the non-dominant hand are placed together on the fingers of the dominant hand, the operation panel is placed on the palms of both hands, and the thumbs of the non-dominant hand and the dominant hand operate the operation panel; the combined holding is that the fingers of the non-dominant hand and the dominant hand are curled up, the operation panel is placed on the index fingers of the non-dominant hand and the dominant hand, and the thumbs of the non-dominant hand and the dominant hand operate the operation panel.

[0012] Preferably, the comparing of different contact modes to obtain the optimal interaction mode between the palm and the operation panel is specifically: comparing the information richness and interactive operation space presented by the operation panels of single-hand holding, nested hand holding and combined holding, and obtaining the interaction mode between the palm and the operation panel that is the optimal one of single-hand holding.

[0013] Preferably, before comparing different contact modes, the method further includes: collecting the shapes of the user's non-dominant hand and dominant hand, specifically: collecting the shapes of the user's non-dominant hand and dominant hand based on the MegaTrack gesture tracking system, including:

[0014] collecting images of the non-dominant hand and the dominant hand, and preprocessing the collected images;

[0015] Detect the preprocessed image based on a lightweight object detection model to obtain a 2D square bounding box of the hand area, where the 2D square bounding box is represented by a center position and a radius;

[0016] Based on the 2D square bounding box, the hand area is cropped from the original image, and the cropped image is input into KeyNet to obtain the 2D coordinates of the hand key points;

[0017] Track the detected hand based on the tracking algorithm. If the hand is successfully tracked, use the optimized 3D gesture model to estimate the 3D gesture pose and extrapolate the 3D gesture pose to the future frame; project the 3D pose back to the 2D plane of each camera, and calculate the minimum circular or square range of the 2D key points based on the projection results;

[0018] If tracking fails, in the next frame, we force DeNet to run hand detection on only one camera, and then infer the 2D square bounding box in the remaining cameras based on the detection results;

[0019] Based on the extrapolation of 3D hand gestures to future frames and the expansion of the 2D square bounding box by 20%, the 2.5D positions of 21 key points are calculated; the 2.5D key point positions are converted into heat maps; the heat maps are normalized and used as the input of KeyNet; if hand tracking fails, the heat map of the channel is set to 0.

[0020] Preferably, after performing 2.5D skeleton key points, the optimization process of motion gestures is also included, specifically:

[0021] Collect key point data of the hand; initialize the hand model and adjust the size of the 3D hand template according to the actual hand size of the user;

[0022] Optimize the hand model parameters based on the least squares method to minimize the reprojection error and distance constraints; and apply temporal constraints to reduce jitter and improve continuity;

[0023] The key point positions of the hand model are adjusted according to the prediction results of KeyNet; the adjusted key points are mapped to the skin of the hand model using LBS technology.

[0024] Preferably, the user's management of the window includes: clicking, closing, repositioning and scaling.

[0025] Preferably, the contact state between the virtual hand and the operation panel is specifically as follows: the front end of the operating finger of the dominant hand model is used as an indicator in the 2D operation plane. If the operation panel receives a collision between the front end of the finger and the UI on the operation panel, a judgment is made. If the user directly raises his hand, it is judged as a click, and the screen at the corresponding position in the virtual reality environment is exchanged with the center screen. If the front end of the user's finger continues to collide with the operation panel and a certain displacement occurs, the operation panel transmits the position information of the front end of the finger to Unity in real time, and the UI on the operation panel is periodically updated to the position of the front end of the finger. If the user raises his hand at this time, Unity Calculate the distance between the position of the operated UI and the rest of the UIs and compare them, pass the UI position corresponding to the smallest distance to Unity, and exchange the position of the operated UI with the corresponding UI; if the difference between the position of the operated UI and the vertical coordinate of the initial position exceeds a certain distance, then when the user raises his hand, Unity will hide the corresponding UI; if the front end of the thumb of the user's dominant hand also collides with the operation panel, Unity will adjust the Boolean value of the two-finger collision to true, and Unity will calculate the distance between the front ends of the two fingers in real time. If the distance increases, the corresponding size of the UI increases, and if the distance decreases, the corresponding size of the UI decreases.

[0026] The present invention proposes a mixed reality window management interaction system based on a palm interface, comprising:

[0027] An adjustment module, wherein the adjustment module adjusts an optimal angle of a display device based on a mixed reality head mounted display worn by a user;

[0028] A comparison module, wherein the comparison module collects different contact modes of the user's non-dominant hand and dominant hand on the operation panel, compares the different contact modes, and obtains an optimal interaction mode between the palm and the operation panel;

[0029] A recognition module, which recognizes gestures based on the contact state between the virtual hand and the operation panel under the premise that the interaction between the palm and the operation panel is optimal;

[0030] A completion module selects an interaction target and completes the mixed reality desktop gesture interaction according to the recognized desktop gesture.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] The present invention proposes a mixed reality window management interaction method based on a palm interface, which acquires the optimal interaction mode between the palm and the operation panel by collecting different contact modes of the user's non-dominant hand and dominant hand on the operation panel, and comparing the different contact modes; identifies gesture actions based on the contact state of the virtual hand and the operation panel; selects the interaction target, and completes the mixed reality desktop gesture interaction according to the identified desktop gesture. The present invention converts a 2D interface into a three-dimensional mixed reality space, which can improve user participation and interaction speed, and does not interfere with other gesture interaction technologies, meeting the needs of mixed reality scenarios that require frequent and long-term interactions.

[0033] The present invention proposes a mixed reality window management and interaction system based on a palm interface, which divides the system into an adjustment module, a comparison module, a recognition module and a completion module, and finally completes the mixed reality desktop gesture interaction according to the recognized desktop gesture. The modularization concept is adopted to make each module independent of each other, which is convenient for unified management of each module. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 It is a flowchart of the mixed reality window management interaction method based on the palm interface of the present invention.

[0036] Figure 2This is a diagram of the mixed reality window management interaction system based on the palm interface of the present invention.

[0037] Figure 3 A diagram showing the interactive function of mixed reality window management based on the palm interface.

[0038] Figure 4 Window layout diagram ((a) is a schematic diagram of a flat window layout; (b) is a schematic diagram of a horizontally curved window layout; (c) is a schematic diagram of a vertically curved window layout; (d) is a schematic diagram of a horizontal and vertically curved window layout).

[0039] Figure 5 Different perspectives of the default layout ((a) is the side view of the default layout; (b) is the top view of the default layout).

[0040] Figure 6 Schematic diagram of multiple modes ((a) is a schematic diagram of single-hand holding; (b) is a schematic diagram of nested-hand holding; (c) is a schematic diagram of combined holding).

[0041] Figure 7 Demonstration diagram for user operation methods and corresponding functions. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0043] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0045] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0046] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0047] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0048] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0049] See also Figure 1 The present invention discloses a mixed reality window management interaction method based on a palm interface, comprising:

[0050] S1, adjusting the optimal angle of the display device based on the mixed reality head-mounted display worn by the user;

[0051] The display device consists of two layers, each layer consists of several display screens; the display screens of the upper and lower layers are set to be flat, horizontally curved, vertically curved, and horizontally and vertically curved; an expert evaluation method is used to select the user's favorite display device setting state.

[0052] S2, collecting different contact modes of the user's non-dominant hand and dominant hand on the operation panel, comparing the different contact modes, and obtaining the optimal interaction mode between the palm and the operation panel;

[0053] Different contact modes of the user's non-dominant hand and dominant hand with the operation panel include: single-palm holding, nested-palm holding and combined holding; the single-palm holding is that the user's non-dominant hand holds the operation panel, and the dominant hand operates the operation panel; the nested-palm holding is that the fingers of the non-dominant hand are placed together on the fingers of the dominant hand, the operation panel is placed on the palms of both hands, and the thumbs of the non-dominant hand and the dominant hand operate the operation panel; the combined holding is that the fingers of the non-dominant hand and the dominant hand are curled up, the operation panel is placed on the index fingers of the non-dominant hand and the dominant hand, and the thumbs of the non-dominant hand and the dominant hand operate the operation panel.

[0054] By comparing different contact methods, the optimal interaction method between the palm and the operation panel is obtained. Specifically, the information richness and interactive operation space presented by the operation panels of single-handed holding, nested hand holding and combined holding are compared, and the interaction method between the palm and the operation panel is obtained as the optimal one when held with one hand.

[0055] Before comparing different contact methods, it also includes: collecting the shapes of the user's non-dominant hand and dominant hand. Specifically, the MegaTrack gesture tracking system will collect the shapes of the user's non-dominant hand and dominant hand, including:

[0056] collecting images of the non-dominant hand and the dominant hand, and preprocessing the collected images;

[0057] Detect the preprocessed image based on a lightweight object detection model to obtain a 2D square bounding box of the hand area, where the 2D square bounding box is represented by a center position and a radius;

[0058] Based on the 2D square bounding box, the hand area is cropped from the original image, and the cropped image is input into KeyNet to obtain the 2D coordinates of the hand key points;

[0059] Track the detected hand based on the tracking algorithm. If the hand is successfully tracked, use the optimized 3D gesture model to estimate the 3D gesture pose and extrapolate the 3D gesture pose to the future frame; project the 3D pose back to the 2D plane of each camera, and calculate the minimum circular or square range of the 2D key points based on the projection results;

[0060] If tracking fails, in the next frame, we force DeNet to run hand detection on only one camera, and then infer the 2D square bounding box in the remaining cameras based on the detection results.

[0061] Based on the extrapolation of 3D hand gestures to future frames and the expansion of the 2D square bounding box by 20%, the 2.5D positions of 21 key points are calculated; the 2.5D key point positions are converted into heat maps; the heat maps are normalized and used as the input of KeyNet; if hand tracking fails, the heat map of the channel is set to 0.

[0062] After the 2.5D skeleton key points are processed, the optimization of motion gestures is also included, specifically:

[0063] Collect key point data of the hand; initialize the hand model and adjust the size of the 3D hand template according to the actual hand size of the user;

[0064] Optimize the hand model parameters based on the least squares method to minimize the reprojection error and distance constraints; and apply temporal constraints to reduce jitter and improve continuity;

[0065] The key point positions of the hand model are adjusted according to the prediction results of KeyNet; the adjusted key points are mapped to the skin of the hand model using LBS technology.

[0066] S3, under the premise that the interaction mode between the palm and the operation panel is optimal, the gesture action is recognized based on the contact state between the virtual hand and the operation panel;

[0067] The user's window management methods include: clicking, closing, repositioning, and scaling.

[0068] The contact state between the virtual hand and the operation panel is as follows: the front end of the operating finger of the dominant hand model is used as an indicator in the 2D operation plane. If the operation panel receives a collision between the front end of the finger and the UI on the operation panel, it makes a judgment. If the user directly raises his hand, it is judged as a click, and the screen at the corresponding position in the virtual reality environment is exchanged with the center screen. If the front end of the user's finger continues to collide with the operation panel and a certain displacement occurs, the operation panel transmits the position information of the front end of the finger to Unity in real time, and the UI on the operation panel is regularly updated to the position of the front end of the finger. If the user raises his hand at this time, Unity calculates the operation. The distance between the position of the operated UI and the rest of the UIs is compared, and the UI position corresponding to the smallest distance is passed to Unity, and the position of the operated UI is exchanged with the corresponding UI; if the difference between the vertical coordinate of the position of the operated UI and the initial position exceeds a certain distance, then when the user raises his hand, Unity will hide the corresponding UI; if the front end of the thumb of the user's dominant hand also collides with the operation panel, Unity will adjust the Boolean value of the two-finger collision to true, and Unity will calculate the distance between the front ends of the two fingers in real time. If the distance becomes larger, the corresponding size of the UI becomes larger, and if the distance becomes smaller, the corresponding size of the UI becomes smaller.

[0069] S4, selecting an interaction target and completing the mixed reality desktop gesture interaction according to the recognized desktop gesture.

[0070] See also Figure 2 The present invention discloses a mixed reality window management interaction system based on a palm interface, comprising:

[0071] An adjustment module, wherein the adjustment module adjusts an optimal angle of a display device based on a mixed reality head mounted display worn by a user;

[0072] A comparison module, wherein the comparison module collects different contact modes of the user's non-dominant hand and dominant hand on the operation panel, compares the different contact modes, and obtains an optimal interaction mode between the palm and the operation panel;

[0073] A recognition module, which recognizes gestures based on the contact state between the virtual hand and the operation panel under the premise that the interaction between the palm and the operation panel is optimal;

[0074] A completion module selects an interaction target and completes the mixed reality desktop gesture interaction according to the recognized desktop gesture.

[0075] Example:

[0076] Figure 3 This is a functional display diagram of a mixed reality interaction method implemented by the present invention. These functions are applicable to the case where an application window is opened in a mixed reality environment and both hands are used as interaction tools. The present invention mainly operates and manages application windows in mixed reality. The specific steps are as follows:

[0077] Step 1: Open the application window and set the layout of multiple application windows.

[0078] Figure 4 (a), (b), (c) and (d) are studies on the placement of multiple application windows in the present invention. Compared with a single application window, users prefer the layout of multiple application windows to cope with multi-tasking and provide more immersive operations. Considering the user experience of using multiple application windows, this study mainly explores how to reduce the distance that the user's eyes and head move when using the application and the optimal distance between the window and the user to reduce the user's eye fatigue problem.

[0079] This study finally designed four window layout curvatures, such as Figure 4 As shown in (a), (b), (c) and (d) in the figure: flat, horizontally curved, vertically curved, and horizontally and vertically curved, the Barrett method was used to conduct an experiment on eight participants and it was finally found that the combination of horizontal and vertical curved layouts can effectively reduce user fatigue and enhance the user's sense of immersion when using the application window.

[0080] Participants counted objects of specific shapes and colors on the screen around the central display, completing 20 tasks for each layout. After completing the task, participants ranked their preferred layout, and the order in which the layouts were presented was counterbalanced to avoid order effects. Results showed that participants preferred a combination of horizontal and vertical curvatures, as detailed in Figure 5 All screens are placed within the area where the user's head can rotate no more than 45 degrees to the left or right and 10 degrees upward. The main screen is located in the optimal area, and the size of each screen is 19 inches. Figure 5 (a) and (b) in the figure are the research on the window placement and angle of the present invention, and the combination of horizontal and vertical curved layouts is regarded as the default layout; in order to further optimize user interaction, this study mainly studies the maximization of the visibility of the main window, the most comfortable angle range and the optimal viewing distance for window placement.

[0081] The study found that placing the top of the central window flush with the field of vision, with the outer sides of the three horizontal windows forming a 120-degree angle with the center of the field of vision and a horizontal distance of 58 cm, and placing the peripheral windows above the central window to prevent overlap with physical desktop elements, can effectively enhance comfort and practicality and improve application functionality in a mixed reality environment.

[0082] Step 2: Research on user operation gestures

[0083] Figure 6 (a), (b) and (c) in the present invention are studies on whether the panels with different operation gestures can present richer information and provide sufficient space for interactive operations. Figure 6 In (a), (b) and (c), the user's non-dominant hand and dominant hand hold the operation panel with one palm, nested palm and combined palm, respectively; the single palm holding means that the user's non-dominant hand holds the operation panel, and the dominant hand operates the operation panel; the nested palm holding means that the fingers of the non-dominant hand are placed together on the fingers of the dominant hand, the operation panel is placed on the palms of both hands, and the thumbs of the non-dominant hand and dominant hand operate the operation panel; the combined holding means that the fingers of the non-dominant hand and dominant hand are curled up, the operation panel is placed on the index fingers of the non-dominant hand and dominant hand, and the thumbs of the non-dominant hand and dominant hand operate the operation panel.

[0084] The study found that maintaining a natural gesture similar to holding a smartphone, providing a passive tactile feedback surface that mimics a smartphone screen, and presenting windows in a minimized format for interaction can create a window management experience with high usability and ergonomics. Therefore, the present invention chooses to use the palm of the non-dominant hand as a control panel to display window thumbnails, which can be interacted with using the dominant hand in a natural mobile phone holding posture.

[0085] In this experiment, the way to obtain hand information is through the MegaTrack gesture tracking system, which is divided into three parts:

[0086] Hand perception: accept 640x480 images, detect the hand bbox one by one, and input the left and right hands into KeyNet after cropping. Use a lightweight detector for simultaneous positioning and classification. For any input, it is assumed that there are only two hands, so for each hand, only the center position and radius of the 2D square bbox exceeding a certain confidence level are predicted, thereby further reducing the number of parameters. DeNet is not run in every frame of the four cameras. Instead, if the hand is tracked, the optimized 3D gesture is extrapolated and then projected to each camera, and the minimum circular or square range including these 2D key points is calculated; if the hand is not tracked, DeNet is run in the next frame and forced to run DeNet only on one camera, and then the 2D bbox in the remaining cameras is calculated in the next frame.

[0087] Calculate 2.5D skeleton key points: accept the result of gesture extrapolation and expand the bbox of the previous step by 20%, and output a 2.5D heat map of 21 key points. After extrapolating the optimized 3D gesture, calculate the 2.5D gesture of the 21 key points in reverse, normalize them and give them as input to KeyNet. If the hand is not tracked, then this channel is set to 0. This especially solves the problem of discontinuity in estimation results when the hand switches between the visible and common view ranges of different cameras, as well as the flickering of some hands when they are blocked; special treatment for fisheye cameras: Unlike the previous deep network monocular depth method that often estimates a relative depth, the estimated distance here is a relative distance, which is actually the distance from the key point to the lens, which effectively improves the impact of fisheye distortion.

[0088] Optimization of motion gestures: This step combines a 3D hand template and kinematics to perform least squares optimization of 3D gestures. Multiple temporary constraints are used to improve jitter, and temporary error equation constraints are added to improve the consistency of previous and next frames. The first two errors are a common reprojection error and a distance constraint. For hand scale estimation, this step is performed when the Oculus glasses are initialized, which can be applied to different hand sizes of different users. The key points predicted by KeyNet are used to fit a hand template similar to an embodied hand, and the hand model is implemented using classic LBS.

[0089] Step 3: Research on user operation methods

[0090] Figure 7This study is aimed at exploring the functions and corresponding implementation methods applicable to the present invention, as well as the comparison between the present invention and other existing interaction methods. Figure 7 As shown, when the user's non-dominant hand is facing upward, the system activates the window management system. Users can manage windows in four ways: click, close, transpose, and zoom. When the user clicks on the corresponding window, the hand model contacts the hand panel to select the corresponding window. After releasing the hand, the hand panel and the window in the mixed reality environment are moved to the center position; the user clicks on the window and slides up a certain distance, and releases the finger to close the hand panel and the corresponding window in the mixed reality environment; the user clicks on the corresponding window, moves it to the corresponding window position, and releases the finger to achieve the transposition of the hand panel and the corresponding window in the mixed reality environment; the user clicks on the corresponding window and uses two fingers to pinch and open to achieve the scaling of the hand panel and the corresponding window size in the mixed reality.

[0091] The position touched by the user, the mixed reality glasses use image processing technology to analyze whether the user touches and the touch position, the screen touch information can at least include the touch time and the touch position, so that each touch position is displayed in the virtual input interface according to the touch time.

[0092] According to the relative position relationship between the dominant hand and the non-dominant hand in reality, a virtual hand area is generated in the virtual scene, and an operation mapping relationship between the real hand area and the virtual hand area is established, so that the user can perform input operations in the virtual scene according to the input mapping relationship and the operation mapping relationship.

[0093] Instance, Figure 7 The steps of operating a virtual hand in a virtual reality environment are shown. Figure 7 As shown, the mixed reality glasses track the dominant hand and the non-dominant hand, and combine the collision position, time and other information obtained in Unity to track the virtual hand's click, slide, leave and other operations on the virtual panel, so as to realize the functions of clicking, swapping, closing and zooming the UI on the virtual panel.

[0094] Operations on the 2D operation plane: In the 2D operation panel, the user can use a pointer similar to a mouse in a WIMP operation interface to click on the pages in the panel and perform corresponding operations. The operation panel processes the information transmitted when receiving the pointer click and movement.

[0095] Contact determination between the virtual hand and the operation panel: The front end of the operating finger of the dominant hand model is used as an indicator in the 2D operation plane. If the operation panel receives a collision between the front end of the finger and the UI on the operation panel, it will make a determination. If the user directly raises his hand, it will be judged as a click, and the screen at the corresponding position in the virtual reality environment will be exchanged with the center screen; if the front end of the user's finger continues to collide with the operation panel and a certain displacement occurs, the operation panel will transmit the position information of the front end of the finger to Unity in real time, and the UI on the operation panel will be updated to the position of the front end of the finger every 0.02 seconds. If the user raises his hand at this time, Unity will calculate The distance between the position of the operated UI and the rest of the UI is compared, and the UI position corresponding to the smallest distance is passed to Unity, and the position of the operated UI is exchanged with the corresponding UI; if the difference between the vertical coordinate of the position of the operated UI and the initial position exceeds a certain distance, when the user raises his hand, Unity will hide the corresponding UI; if the front end of the thumb of the user's dominant hand also collides with the operation panel, Unity will adjust the Boolean value of the two-finger collision to true, and Unity will calculate the distance between the front ends of the two fingers in real time. If the distance increases, the corresponding size of the UI increases, and if the distance decreases, the corresponding size of the UI decreases.

[0096] The present invention provides certain tactile feedback for window management by touching the palm of the non-dominant hand with the fingers of the dominant hand to improve the user's operation accuracy and usage experience. By comparing with two other interaction methods: using a thumb rocker and using mid-air gesture recognition, the study found that the present invention can effectively reduce the movement distance of the user's hand and head, reduce user fatigue, and at the same time reduce the operation time of different operations to improve operation efficiency.

[0097] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A mixed reality window management interaction method based on a palm interface, characterized in that: include: Adjusting the optimal angle of the display device based on the mixed reality head-mounted display worn by the user; Collect different contact methods of the user's non-dominant hand and dominant hand on the operation panel, compare different contact methods, and obtain the optimal interaction method between the palm and the operation panel; Under the premise that the interaction between the palm and the operation panel is optimal, the gesture action is recognized based on the contact state between the virtual hand and the operation panel; Select the interaction target and complete the mixed reality desktop gesture interaction based on the recognized desktop gesture.

2. The palm interface-based mixed reality window management interaction method according to claim 1, characterized in that: The display device comprises two layers, each layer comprising a plurality of display screens; the display screens of the upper and lower layers are set to be flat, horizontally curved, vertically curved, and horizontally and vertically curved; and an expert evaluation method is used to select the display device setting state that the user likes.

3. The palm interface-based mixed reality window management interaction method according to claim 1, characterized in that: The different contact modes of the user's non-dominant hand and dominant hand with the operation panel include: single-palm holding, nested-hand holding and combined holding; the single-palm holding is that the user's non-dominant hand holds the operation panel, and the dominant hand operates the operation panel; the nested-hand holding is that the fingers of the non-dominant hand are placed together on the fingers of the dominant hand, the operation panel is placed on the palms of both hands, and the thumbs of the non-dominant hand and the dominant hand operate the operation panel; the combined holding is that the fingers of the non-dominant hand and the dominant hand are curled up, the operation panel is placed on the index fingers of the non-dominant hand and the dominant hand, and the thumbs of the non-dominant hand and the dominant hand operate the operation panel.

4. The palm interface-based mixed reality window management interaction method according to claim 1, characterized in that: The comparing of different contact modes to obtain the optimal interaction mode between the palm and the operation panel specifically comprises: comparing the information richness and interactive operation space presented by the operation panels of single-hand holding, nested-hand holding and combined holding, and obtaining the interaction mode between the palm and the operation panel as the optimal one of single-hand holding.

5. The palm interface-based mixed reality window management interaction method according to claim 4, characterized in that: Before comparing different contact methods, it also includes: collecting the shapes of the user's non-dominant hand and dominant hand. Specifically, the MegaTrack gesture tracking system will collect the shapes of the user's non-dominant hand and dominant hand, including: collecting images of the non-dominant hand and the dominant hand, and preprocessing the collected images; Detect the preprocessed image based on a lightweight object detection model to obtain a 2D square bounding box of the hand area, where the 2D square bounding box is represented by a center position and a radius; Based on the 2D square bounding box, the hand area is cropped from the original image, and the cropped image is input into KeyNet to obtain the 2D coordinates of the hand key points; Track the detected hand based on the tracking algorithm. If the hand is successfully tracked, use the optimized 3D gesture model to estimate the 3D gesture pose and extrapolate the 3D gesture pose to the future frame; project the 3D pose back to the 2D plane of each camera, and calculate the minimum circular or square range of the 2D key points based on the projection results; If tracking fails, in the next frame, we force DeNet to run hand detection on only one camera, and then infer the 2D square bounding box in the remaining cameras based on the detection results; Based on the extrapolation of 3D hand gestures to future frames and the expansion of the 2D square bounding box by 20%, the 2.5D positions of 21 key points are calculated; the 2.5D key point positions are converted into heat maps; the heat maps are normalized and used as the input of KeyNet; if hand tracking fails, the heat map of the channel is set to 0.

6. The palm interface-based mixed reality window management interaction method according to claim 5, characterized in that: After the 2.5D skeleton key points are processed, the optimization of motion gestures is also included, specifically: Collect key point data of the hand; initialize the hand model and adjust the size of the 3D hand template according to the actual hand size of the user; Optimize the hand model parameters based on the least squares method to minimize the reprojection error and distance constraints; and apply temporal constraints to reduce jitter and improve continuity; The key point positions of the hand model are adjusted according to the prediction results of KeyNet; the adjusted key points are mapped to the skin of the hand model using LBS technology.

7. The palm interface-based mixed reality window management interaction method according to claim 6, characterized in that: The user's management of the window includes: clicking, closing, repositioning and scaling.

8. The palm interface-based mixed reality window management interaction method according to claim 1, characterized in that: The contact state between the virtual hand and the operation panel is specifically as follows: the front end of the operating finger of the dominant hand model is used as an indicator in the 2D operation plane. If the operation panel receives a collision between the front end of the finger and the UI on the operation panel, a judgment is made. If the user directly raises his hand, it is judged as a click, and the screen at the corresponding position in the virtual reality environment is exchanged with the center screen; if the front end of the user's finger continues to collide with the operation panel and a certain displacement occurs, the operation panel transmits the position information of the front end of the finger to Unity in real time, and the UI on the operation panel is regularly updated to the position of the front end of the finger. If the user raises his hand at this time, Unity calculates The distance between the position of the operated UI and the rest of the UIs is compared, and the UI position corresponding to the smallest distance is passed to Unity, and the position of the operated UI is exchanged with the corresponding UI; if the difference between the vertical coordinate of the position of the operated UI and the initial position exceeds a certain distance, when the user raises his hand, Unity will hide the corresponding UI; if the front end of the thumb of the user's dominant hand also collides with the operation panel, Unity will adjust the Boolean value of the two-finger collision to true, and Unity will calculate the distance between the front ends of the two fingers in real time. If the distance increases, the corresponding size of the UI increases, and if the distance decreases, the corresponding size of the UI decreases.

9. A mixed reality window management interaction system based on a palm interface, characterized in that: include: An adjustment module, wherein the adjustment module adjusts an optimal angle of a display device based on a mixed reality head mounted display worn by a user; A comparison module, wherein the comparison module collects different contact modes of the user's non-dominant hand and dominant hand on the operation panel, compares the different contact modes, and obtains an optimal interaction mode between the palm and the operation panel; A recognition module, which recognizes gestures based on the contact state between the virtual hand and the operation panel under the premise that the interaction between the palm and the operation panel is optimal; A completion module selects an interaction target and completes the mixed reality desktop gesture interaction according to the recognized desktop gesture.

Images