A mixed reality window management interaction method and system based on a palm interface

Abstract

The application discloses a mixed reality window management interaction method and system based on a palm interface, and comprises the following steps: collecting different contact modes of a non-dominant hand and a dominant hand of a user on an operation panel, comparing the different contact modes, and obtaining an optimal palm and operation panel interaction mode; based on the contact state of a virtual hand and the operation panel, a gesture action is recognized; an interaction target is selected, and mixed reality desktop gesture interaction is completed according to the recognized desktop gesture. The application converts a 2D interface into a three-dimensional mixed reality space, can improve the participation of users, can improve the interaction speed, does not interfere with other gesture interaction technologies, and meets the needs of mixed reality scene requirements that need frequent interaction and long-term interaction.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of mixed reality, and relates to a mixed reality window management interaction method and system based on a palm interface. BACKGROUND

[0002] Mixed reality combines the physical world and the digital world to create an environment in which digital and physical objects coexist and interact in real time. In traditional computing environments, windows, icons, menus and pointers are usually used to manage content in a 2D graphical user interface. However, these interfaces optimized for desktops and other displays cannot be well translated into the three-dimensional space of mixed reality. The current window management method in mixed reality is mainly derived from traditional 2D interfaces, which can lead to poor user experience, reduce user engagement and affect the development of virtual space. SUMMARY

[0003] The application aims to solve the problem that the 2D interface in the prior art cannot be translated into the three-dimensional space of mixed reality, leading to poor user experience, reducing user engagement and affecting the development of virtual space, and provides a mixed reality window management interaction method and system based on a palm interface.

[0004] To achieve the above-mentioned purpose, the application adopts the following technical solutions:

[0005] The application provides a mixed reality window management interaction method based on a palm interface, which comprises:

[0006] Adjusting the optimal angle of the display device based on the mixed reality head-mounted display worn by the user;

[0007] Collecting different contact modes of the non-dominant hand and the dominant hand of the user on the operation panel, comparing the different contact modes, and obtaining the optimal palm and operation panel interaction mode;

[0008] On the premise that the palm and operation panel interaction mode is optimal, recognizing the gesture action based on the contact state of the virtual hand and the operation panel;

[0009] Selecting the interaction target and completing the mixed reality desktop gesture interaction according to the recognized desktop gesture.

[0010] Preferably, the display device has two layers, each layer has a plurality of display screens; the display screens of the two layers are set to be flat, horizontally curved, vertically curved, and horizontally and vertically curved; and an expert evaluation method is used to select the display device setting state preferred by the user.

[0011] Preferably, the different contact modes of the user's non-dominant hand and dominant hand with the operation panel include: single-palm holding, nested-hand holding and combined holding; the single-palm holding is for the user's non-dominant hand to hold the operation panel, and the dominant hand to operate the operation panel; the nested-hand holding is for the fingers of the non-dominant hand to be placed together on the fingers of the dominant hand, the operation panel is placed on the palms of both hands, and the thumbs of the non-dominant hand and dominant hand to operate the operation panel; the combined holding is for the fingers of the non-dominant hand and dominant hand to be curled up, the operation panel is placed on the index fingers of the non-dominant hand and dominant hand, and the thumbs of the non-dominant hand and dominant hand to operate the operation panel.

[0012] Preferably, the comparison of different contact methods to obtain the optimal interaction method between the palm and the operation panel is specifically: comparing the information richness and interactive operation space presented by the operation panels of single-hand holding, nested hand holding and merged holding, and obtaining the interaction method between the palm and the operation panel that is optimal when single-hand holding is used.

[0013] Preferably, before comparing different contact modes, the method further includes: collecting the shapes of the user's non-dominant hand and dominant hand. Specifically, collecting the shapes of the user's non-dominant hand and dominant hand based on the MegaTrack gesture tracking system includes:

[0014] collecting images of the non-dominant hand and the dominant hand, and preprocessing the collected images;

[0015] Detect the preprocessed image based on a lightweight object detection model to obtain a 2D square bounding box of the hand area, where the 2D square bounding box is represented by a center position and a radius;

[0016] Based on the 2D square bounding box, the hand area is cropped from the original image and the cropped image is input into KeyNet to obtain the 2D coordinates of the hand key points;

[0017] Track the detected hand based on the tracking algorithm. If the hand is successfully tracked, use the optimized 3D gesture model to estimate the 3D gesture pose and extrapolate the 3D gesture pose to future frames. Project the 3D pose back to the 2D plane of each camera and calculate the minimum circular or square range of the 2D key points based on the projection results.

[0018] If tracking fails, in the next frame, we force DeNet to run hand detection on only one camera, and then infer the 2D square bounding box in the remaining cameras based on the detection results;

[0019] Based on the extrapolation of 3D hand gestures to future frames and the expansion of the 2D square bounding box by 20%, the 2.5D positions of 21 key points are calculated; the 2.5D key point positions are converted into a heat map; the heat map is normalized and used as the input of KeyNet; if hand tracking fails, the heat map of that channel is set to 0.

[0020] Preferably, after performing 2.5D skeleton key points, the optimization process of motion gestures is also included, specifically:

[0021] Collect key point data of the hand; initialize the hand model and adjust the size of the 3D hand template according to the user's actual hand size;

[0022] Optimize hand model parameters based on least squares to minimize reprojection error and distance constraints; and apply temporal constraints to reduce jitter and improve continuity;

[0023] Adjust the key point positions of the hand model according to the prediction results of KeyNet; use LBS technology to map the adjusted key points to the skin of the hand model.

[0024] Preferably, the user's management of the window includes: clicking, closing, repositioning and scaling.

[0025] Preferably, the contact state between the virtual hand and the operation panel is specifically as follows: the front end of the operating finger of the dominant hand model is used as an indicator in the 2D operation plane. If the operation panel receives a collision between the front end of the finger and the UI on the operation panel, it makes a judgment. If the user directly raises his hand, it is judged as a click, and the screen at the corresponding position in the virtual reality environment is exchanged with the center screen; if the front end of the user's finger continues to collide with the operation panel and a certain displacement occurs, the operation panel transmits the position information of the front end of the finger to Unity in real time, and the UI on the operation panel is regularly updated to the position of the front end of the finger. If the user raises his hand at this time, Unity Calculate the distance between the position of the operated UI and the rest of the UI and compare them, pass the UI position corresponding to the smallest distance to Unity, and exchange the position of the operated UI with the corresponding UI; if the difference between the vertical coordinate of the position of the operated UI and the initial position exceeds a certain distance, when the user raises his hand, Unity will hide the corresponding UI; if the front end of the thumb of the user's dominant hand also collides with the operation panel, Unity will adjust the Boolean value of the two-finger collision to true, and Unity will calculate the distance between the front ends of the two fingers in real time. If the distance increases, the corresponding size of the UI will increase, and if the distance decreases, the corresponding size of the UI will decrease.

[0026] The present invention proposes a mixed reality window management and interaction system based on a palm interface, comprising:

[0027] An adjusting module adjusts an optimal angle of the display device based on a mixed reality head-mounted display worn by the user;

[0028] A comparison module collects different contact modes of the non-dominant hand and the dominant hand of the user on the operation panel, compares the different contact modes, and obtains an optimal palm and operation panel interaction mode;

[0029] An identification module identifies a gesture action based on a virtual hand and the operation panel contact state under the premise that the palm and operation panel interaction mode is optimal;

[0030] A completion module selects an interaction target and completes the mixed reality desktop gesture interaction according to the identified desktop gesture.

[0031] Compared with the prior art, the present application has the following beneficial effects:

[0032] The mixed reality window management interaction method based on the palm interface provided by the present application collects different contact modes of the non-dominant hand and the dominant hand of the user on the operation panel, compares the different contact modes, and obtains an optimal palm and operation panel interaction mode; a gesture action is identified based on a virtual hand and the operation panel contact state; an interaction target is selected, and the mixed reality desktop gesture interaction is completed according to the identified desktop gesture. The present application converts a 2D interface into a three-dimensional mixed reality space, can improve the participation of the user, can improve the interaction speed, does not interfere with other gesture interaction technologies, and meets the needs of mixed reality scenes that require frequent interaction and long-term interaction.

[0033] The mixed reality window management interaction system based on the palm interface provided by the present application is finally completed according to the identified desktop gesture, and the mixed reality desktop gesture interaction is completed. The modularization idea is adopted to make each module independent of each other, and the unified management of each module is facilitated. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation to the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the premise of the drawings.

[0035] Figure 1 The flowchart of the mixed reality window management interaction method based on the palm interface of the present application.

[0036] Figure 2This is a diagram of the mixed reality window management interaction system based on the palm interface of the present invention.

[0037] Figure 3 A diagram showing the interactive functionality of mixed reality window management based on the palm interface.

[0038] Figure 4 Window layout diagrams ((a) is a schematic diagram of a flat window layout; (b) is a schematic diagram of a horizontally curved window layout; (c) is a schematic diagram of a vertically curved window layout; (d) is a schematic diagram of a horizontal and vertically curved window layout).

[0039] Figure 5 Different perspectives of the default layout ((a) is the side view of the default layout; (b) is the top view of the default layout).

[0040] Figure 6 Schematic diagram of multiple modes ((a) is a schematic diagram of single-hand holding; (b) is a schematic diagram of nested hand holding; (c) is a schematic diagram of combined holding).

[0041] Figure 7 Demonstration diagram for user operation methods and corresponding functions. DETAILED DESCRIPTION

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0043] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0044] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0045] In the description of the embodiments of the present application, it should be noted that if the terms "upper", "lower", "horizontal", "inner" and the like indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, or the orientation or position relationship in which the product of the present application is usually placed, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" and the like are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0046] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly inclined. For example, "horizontal" only means that its direction is relatively more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined.

[0047] In the description of the embodiments of the present application, it should be noted that unless otherwise explicitly specified and limited, if the terms "arrangement", "installation", "connection", "connection" appear, they should be understood in a broad sense. For example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0048] The present application will be described in further detail below in conjunction with the accompanying drawings:

[0049] Referring to Figure 1 The present application discloses a mixed reality window management interaction method based on a palm interface, comprising:

[0050] S1, based on the mixed reality head-mounted display worn by the user, adjusting the optimal angle of the display device;

[0051] The display device has two layers, each layer has a plurality of display screens; the display screens of the two layers are set to be flat, horizontally curved, vertically curved, and horizontally and vertically curved; an expert evaluation method is used to select the display device setting state preferred by the user.

[0052] S2, collecting different contact modes of the non-dominant hand and the dominant hand of the user on the operation panel, comparing different contact modes, and obtaining the optimal palm and operation panel interaction mode;

[0053] Different contact methods of the user's non-dominant hand and dominant hand with the operation panel include: single-palm holding, nested-hand holding and combined holding; the single-palm holding is for the user's non-dominant hand to hold the operation panel, and the dominant hand to operate the operation panel; the nested-hand holding is for the fingers of the non-dominant hand to be placed together on the fingers of the dominant hand, the operation panel is placed on the palms of both hands, and the thumbs of the non-dominant hand and dominant hand to operate the operation panel; the combined holding is for the fingers of both the non-dominant hand and dominant hand to be curled up, the operation panel is placed on the index fingers of the non-dominant hand and dominant hand, and the thumbs of the non-dominant hand and dominant hand to operate the operation panel.

[0054] Compare different contact methods to obtain the optimal interaction method between the palm and the operation panel. Specifically, compare the information richness and interactive operation space presented by the operation panel when holding with one hand, nested hands, and combined hands, and obtain the optimal interaction method between the palm and the operation panel when holding with one hand.

[0055] Before comparing different contact methods, the following also needs to be done: the shape of the user's non-dominant hand and dominant hand is collected. Specifically, the MegaTrack gesture tracking system will collect the shape of the user's non-dominant hand and dominant hand, including:

[0056] collecting images of the non-dominant hand and the dominant hand, and preprocessing the collected images;

[0057] Detect the preprocessed image based on a lightweight object detection model to obtain a 2D square bounding box of the hand area, where the 2D square bounding box is represented by a center position and a radius;

[0058] Based on the 2D square bounding box, the hand area is cropped from the original image and the cropped image is input into KeyNet to obtain the 2D coordinates of the hand key points;

[0059] Track the detected hand based on the tracking algorithm. If the hand is successfully tracked, use the optimized 3D gesture model to estimate the 3D gesture pose and extrapolate the 3D gesture pose to future frames. Project the 3D pose back to the 2D plane of each camera and calculate the minimum circular or square range of the 2D key points based on the projection results.

[0060] If tracking fails, in the next frame, we force DeNet to run hand detection on only one camera, and then infer the 2D square bounding box in the remaining cameras based on the detection results.

[0061] Based on 3D gesture posture extrapolation to future frames and 2D square bounding box expansion by 20%, the 2.5D positions of 21 key points are calculated; and the 2.5D key point positions are converted into a hotspot map; the normalized hotspot map is used as the input of KeyNet; if hand tracking fails, the hotspot map of this channel is set to 0.

[0062] After performing 2.5D skeleton key points, the optimization processing of the motion gesture is also included, specifically:

[0063] Collecting hand key point data; initializing the hand model, adjusting the size of the three-dimensional hand template according to the actual hand size of the user;

[0064] Optimizing the hand model parameters based on the least square method to minimize the re-projection error and distance constraint; and applying temporary constraints to reduce jitter and improve continuity;

[0065] Adjusting the key point positions of the hand model according to the prediction results of KeyNet; using LBS technology to map the adjusted key points to the skin of the hand model.

[0066] S3, under the premise that the interaction mode of the palm and the operation panel is optimal, based on the contact state of the virtual hand and the operation panel, the gesture action is identified;

[0067] The user's management mode of the window includes: point selection, closing, transposition and scaling.

[0068] The contact state of the virtual hand and the operation panel is as follows: the front end of the operating finger of the habitual hand model is taken as the indicator in the 2D operation plane, if the operation panel receives the collision of the front end of the finger and the UI on the operation panel, then it is determined, if the user directly raises his hand, it is judged as point selection, and the screen corresponding to the position in the virtual reality environment is exchanged with the center screen; if the front end of the user's finger continuously collides with the operation panel and a certain displacement occurs, the operation panel will transmit the position information of the front end of the finger to Unity in real time, and the UI on the operation panel is updated to the position of the front end of the finger at regular intervals, if the user raises his hand at this time, Unity calculates the distance between the position of the operating UI and the positions of the remaining UIs and compares them, and the position of the UI corresponding to the smallest distance is transmitted to Unity, and the position of the operating UI and the corresponding UI is exchanged; if the difference between the vertical coordinates of the position of the operating UI and the initial position exceeds a certain distance, when the user raises his hand, Unity will hide the corresponding UI; if the front end of the user's habitual hand thumb also collides with the operation panel, Unity will adjust the Boolean value of the double-finger collision to true, and Unity will calculate the distance between the front ends of the two fingers in real time, if the distance becomes larger, the corresponding size of the UI becomes larger, if the distance becomes smaller, the corresponding size of the UI becomes smaller.

[0069] S4, select the interaction target and complete the mixed reality desktop gesture interaction based on the recognized desktop gesture.

[0070] See also Figure 2 The present invention discloses a mixed reality window management and interaction system based on a palm interface, comprising:

[0071] an adjustment module, configured to adjust an optimal angle of a display device based on a mixed reality head-mounted display worn by a user;

[0072] a comparison module, wherein the comparison module collects different contact modes of the user's non-dominant hand and dominant hand on the operation panel, compares the different contact modes, and obtains an optimal interaction mode between the palm and the operation panel;

[0073] a recognition module that recognizes gestures based on the contact state between the virtual hand and the operation panel, under the premise that the interaction between the palm and the operation panel is optimal;

[0074] A completion module selects an interaction target and completes the mixed reality desktop gesture interaction according to the recognized desktop gesture.

[0075] Example:

[0076] Figure 3 This is a functional diagram of a mixed reality interaction method implemented by the present invention. These functions are applicable to opening application windows in a mixed reality environment and using both hands as interaction tools. The present invention mainly operates and manages application windows in mixed reality. The specific steps are as follows:

[0077] Step 1. Open the application window and set the layout of multiple application windows.

[0078] Figure 4 (a), (b), (c), and (d) represent the present invention's study of multi-application window placement. Compared to a single application window, users prefer a multi-application window layout for multitasking and a more immersive experience. Considering the user experience of using multiple application windows, this study primarily explores how to reduce the distance users move their eyes and head during use, as well as the optimal distance between the window and the user, to reduce eye fatigue.

[0079] This study finally designed four window layout curvatures, such as Figure 4 As shown in (a), (b), (c) and (d) in the figure: flat, horizontally curved, vertically curved, and horizontally and vertically curved, an experiment was conducted on eight participants using the Barrett method and it was finally found that the combination of horizontal and vertical curved layouts can effectively reduce user fatigue during use and enhance the user's immersion when using the application window.

[0080] Participants counted objects of specific shapes and colors on the screen surrounding the central display, completing 20 tasks for each layout. After completing the tasks, participants ranked their preferred layout, and the order of layout presentation was counterbalanced to avoid order effects. The results showed that participants preferred a combination of horizontal and vertical curvatures, as detailed in [1]. Figure 5 As shown in (a) and (b) of Figure 1, all screens are placed within an area where the user's head can rotate no more than 45 degrees to the left or right, or 10 degrees upward. The main screen is located in the optimal area and each screen is 19 inches in size. Figure 5 (a) and (b) in the figure are the research on window placement and angle in the present invention, and the combination of horizontal and vertical curved layouts is regarded as the default layout; in order to further optimize user interaction, this study mainly studies the maximization of the visibility of the main window, the most comfortable angle range of window placement and the optimal viewing distance.

[0081] The study found that aligning the top of the central window with the field of view, with the outer sides of the three horizontal windows forming a 120-degree angle with the center of the field of view and a horizontal distance of 58 cm, while also placing the peripheral windows above the central window to prevent overlap with physical desktop elements, can effectively enhance comfort and practicality and improve application functionality in a mixed reality environment.

[0082] Step 2: Research on user gestures

[0083] Figure 6 (a), (b) and (c) in the present invention are studies on whether the panels with different operation gestures can present richer information and provide sufficient space for interactive operations. Figure 6 In (a), (b) and (c), the user's non-dominant hand and dominant hand hold the operation panel with one palm, nested hand hold and combined hold respectively; the single palm hold means that the user's non-dominant hand holds the operation panel, and the dominant hand operates the operation panel; the nested hand hold means that the fingers of the non-dominant hand are placed together on the fingers of the dominant hand, the operation panel is placed on the palms of both hands, and the thumbs of the non-dominant hand and dominant hand operate the operation panel; the combined hold means that the fingers of the non-dominant hand and dominant hand are curled up, the operation panel is placed on the index fingers of the non-dominant hand and dominant hand, and the thumbs of the non-dominant hand and dominant hand operate the operation panel.

[0084] This study found that maintaining a natural gesture similar to holding a smartphone, providing a passive tactile feedback surface that mimics a smartphone screen, and presenting windows in a minimized format for interaction can create a highly usable and ergonomic window management experience. Therefore, the present invention chooses to use the palm of the non-dominant hand as a control panel to display window thumbnails, which can be interacted with using the dominant hand in a natural phone-holding posture.

[0085] In this experiment, hand information is obtained through the MegaTrack gesture tracking system, which is divided into three parts:

[0086] Hand perception: Accepts a 640x480 image, detects the hand bboxes hand by hand, and then crops and feeds both the left and right hand parts into KeyNet. A lightweight detector is used for simultaneous localization and classification. For any input, it is assumed that there are only two hands, so for each hand, only the center position and radius of the 2D square bbox exceeding a certain confidence level are predicted, further reducing the number of parameters. DeNet is not run on every frame of the four cameras. Instead, if a hand is tracked, the optimized 3D gesture is extrapolated and then projected to each camera, and the minimum circular or square range that includes these 2D key points is calculated. If the hand is not tracked, DeNet is run on the next frame, and DeNet is forced to run on only one camera. Then, in the next frame, the 2D bboxes of the remaining cameras are calculated.

[0087] Calculate 2.5D skeletal key points: accept the result of gesture extrapolation and expand the bbox of the previous step by 20%, and output a 2.5D heat map of 21 key points. After extrapolating the optimized 3D gesture, calculate the 2.5D gesture of the 21 key points in reverse, normalize them and give them together as input to KeyNet. If the hand is not tracked, then this channel is set to 0. This especially solves the problem of discontinuous estimation results when the hand switches between the visible and common view ranges of different cameras, as well as the flickering of parts of the hand when they are blocked; special treatment for fisheye cameras: Unlike previous deep network monocular depth estimation methods that often estimate a relative depth, the estimated distance here is actually the distance from the key point to the lens, which effectively improves the impact of fisheye distortion.

[0088] Optimization of motion gestures: This step combines a 3D hand template and kinematics to perform least squares optimization of 3D gestures. Multiple temporary constraints are used to improve jitter, and temporary error equation constraints are added to improve the consistency of previous and next frames. The first two errors are the common reprojection error and the distance constraint. For hand scale estimation, this step is performed when the Oculus glasses are initialized, which can be adapted to the different hand sizes of different users. The key points predicted by KeyNet are used to fit a hand template similar to the embodiment, and the hand model is implemented using the classic LBS.

[0089] Step 3: Research on user operation methods

[0090] Figure 7This study aims to explore the functions and corresponding implementation methods applicable to the present invention, as well as the comparison between the present invention and other existing interaction methods. Figure 7 As shown, when the user's non-dominant hand is facing upwards, the system activates the window management system. Users can manage windows in four ways: click, close, swap, and zoom. When the user clicks on the corresponding window, the hand model contacts the hand panel to select the corresponding window. After releasing the hand, the hand panel and the window in the mixed reality environment are moved to the center position; the user clicks on the window and slides upward a certain distance, then releases the finger to close the hand panel and the corresponding window in the mixed reality environment; the user clicks on the corresponding window, moves it to the corresponding window position, and releases the finger to swap the hand panel and the corresponding window in the mixed reality environment; the user clicks on the corresponding window and pinches and opens two fingers to scale the hand panel and the corresponding window in the mixed reality.

[0091] The mixed reality glasses use image processing technology to analyze the position touched by the user and determine whether the user has touched the screen and the touch position. The screen touch information may include at least the touch time and touch position, so that each touch position is displayed in the virtual input interface according to the touch time.

[0092] Based on the relative positional relationship between the dominant hand and the non-dominant hand in reality, a virtual hand region is generated in the virtual scene. An operation mapping relationship is established between the real hand region and the virtual hand region, so that the user can perform input operations in the virtual scene according to the input mapping relationship and the operation mapping relationship.

[0093] Instance, Figure 7 The steps of how a virtual hand operates in a virtual reality environment are shown. Figure 7 As shown, the mixed reality glasses track the dominant and non-dominant hands, combined with the collision position, time and other information obtained in Unity, to track the virtual hand's click, slide, and release operations on the virtual panel, thereby realizing the functions of clicking, swapping, closing and zooming the UI on the virtual panel.

[0094] Operations on the 2D operation plane: In the 2D operation panel, users can use a pointer similar to a mouse in a WIMP operation interface to click on pages in the panel and perform corresponding operations. The operation panel processes the information transmitted when receiving the pointer click and movement.

[0095] Contact determination between the virtual hand and the operation panel: The front end of the operating finger of the dominant hand model is used as an indicator in the 2D operation plane. If the operation panel detects a collision between the front end of the finger and the UI on the operation panel, it will make a judgment. If the user directly raises his hand, it will be judged as a click, and the screen at the corresponding position in the virtual reality environment will be swapped with the center screen; if the front end of the user's finger continues to collide with the operation panel and a certain displacement occurs, the operation panel will transmit the position information of the front end of the finger to Unity in real time, and the UI on the operation panel will be updated to the position of the front end of the finger every 0.02 seconds. If the user raises his hand at this time, Unity will calculate The distance between the position of the operated UI and the rest of the UI is compared, and the UI position corresponding to the smallest distance is passed to Unity, and the position of the operated UI is exchanged with the corresponding UI; if the difference between the vertical coordinate of the position of the operated UI and the initial position exceeds a certain distance, when the user raises his hand, Unity will hide the corresponding UI; if the front end of the thumb of the user's dominant hand also collides with the operation panel, Unity will adjust the Boolean value of the two-finger collision to true, and Unity will calculate the distance between the front ends of the two fingers in real time. If the distance increases, the corresponding size of the UI will increase, and if the distance decreases, the corresponding size of the UI will decrease.

[0096] This invention provides tactile feedback for window management by using the dominant hand's fingers to touch the palm of the non-dominant hand, improving user accuracy and user experience. Comparing this with two other interaction methods: using a thumb joystick and using mid-air gesture recognition, the study found that this invention can effectively reduce the distance users move their hands and head, reducing user fatigue. It also reduces the time required for different operations, improving efficiency.

[0097] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A mixed reality window management interaction method based on a palm interface, characterized in that: include: Adjusting the optimal angle of the display device based on the mixed reality head-mounted display worn by the user; Collect different contact methods of the user's non-dominant hand and dominant hand on the operation panel, compare different contact methods, and obtain the optimal interaction method between the palm and the operation panel; Under the premise that the interaction between the palm and the operation panel is optimal, the gesture action is recognized based on the contact state between the virtual hand and the operation panel; Select the interaction target and complete the mixed reality desktop gesture interaction based on the recognized desktop gesture; Before comparing different contact methods, the following also needs to be done: the shape of the user's non-dominant hand and dominant hand is collected. Specifically, the MegaTrack gesture tracking system will collect the shape of the user's non-dominant hand and dominant hand, including: collecting images of the non-dominant hand and the dominant hand, and preprocessing the collected images; Detect the preprocessed image based on a lightweight object detection model to obtain a 2D square bounding box of the hand area, where the 2D square bounding box is represented by a center position and a radius; Based on the 2D square bounding box, the hand area is cropped from the original image and the cropped image is input into KeyNet to obtain the 2D coordinates of the hand key points; Track the detected hand based on the tracking algorithm. If the hand is successfully tracked, use the optimized 3D gesture model to estimate the 3D gesture pose and extrapolate the 3D gesture pose to future frames. Project the 3D pose back to the 2D plane of each camera and calculate the minimum circular or square range of the 2D key points based on the projection results. If tracking fails, in the next frame, we force DeNet to run hand detection on only one camera, and then infer the 2D square bounding box in the remaining cameras based on the detection results; Based on the extrapolation of 3D hand gestures to future frames and the expansion of the 2D square bounding box by 20%, the 2.5D positions of 21 key points are calculated; the 2.5D key point positions are converted into a heat map; the heat map is normalized and used as the input of KeyNet; if hand tracking fails, the heat map of that channel is set to 0.

2. The palm interface-based mixed reality window management interaction method according to claim 1, characterized in that: The display device comprises two layers, each layer comprising a plurality of display screens; the display screens of the upper and lower layers are set to be flat, horizontally curved, vertically curved, and horizontally and vertically curved; and an expert evaluation method is used to select the display device setting state preferred by the user.

3. The palm interface-based mixed reality window management interaction method according to claim 1, characterized in that: The different contact modes of the user's non-dominant hand and dominant hand with the operation panel include: single-palm holding, nested-hand holding and combined holding; the single-palm holding is for the user's non-dominant hand to hold the operation panel, and the dominant hand to operate the operation panel; the nested-hand holding is for the fingers of the non-dominant hand to be placed together on the fingers of the dominant hand, the operation panel is placed on the palms of both hands, and the thumbs of the non-dominant hand and dominant hand to operate the operation panel; the combined holding is for the fingers of both the non-dominant hand and dominant hand to be curled up, the operation panel is placed on the index fingers of the non-dominant hand and dominant hand, and the thumbs of the non-dominant hand and dominant hand to operate the operation panel.

4. The palm interface-based mixed reality window management interaction method according to claim 1, characterized in that: The comparison of different contact methods to obtain the optimal interaction method between the palm and the operation panel is specifically: comparing the information richness and interactive operation space presented by the operation panels held with one hand, nested hands, and merged hands, and obtaining the interaction method between the palm and the operation panel that is optimal when held with one hand.

5. The palm interface-based mixed reality window management interaction method according to claim 1, characterized in that: After the 2.5D skeleton key points are processed, the optimization of motion gestures is also included, specifically: Collect key point data of the hand; initialize the hand model and adjust the size of the 3D hand template according to the user's actual hand size; Optimize hand model parameters based on least squares to minimize reprojection error and distance constraints; and apply temporal constraints to reduce jitter and improve continuity; Adjust the key point positions of the hand model according to the prediction results of KeyNet; use LBS technology to map the adjusted key points to the skin of the hand model.

6. The palm interface-based mixed reality window management interaction method according to claim 5, characterized in that: The user's management of the window includes: clicking, closing, repositioning and scaling.

7. The palm interface-based mixed reality window management interaction method according to claim 1, characterized in that: The contact state between the virtual hand and the operation panel is specifically as follows: the front end of the operating finger of the dominant hand model is used as an indicator in the 2D operation plane. If the operation panel receives a collision between the front end of the finger and the UI on the operation panel, it will make a judgment. If the user directly raises his hand, it will be judged as a click, and the screen at the corresponding position in the virtual reality environment will be exchanged with the center screen; if the front end of the user's finger continues to collide with the operation panel and a certain displacement occurs, the operation panel will transmit the position information of the front end of the finger to Unity in real time, and the UI on the operation panel will be updated to the position of the front end of the finger at regular intervals. If the user raises his hand at this time, Unity will calculate The distance between the position of the operated UI and the rest of the UI is compared, and the UI position corresponding to the smallest distance is passed to Unity, and the position of the operated UI is exchanged with the corresponding UI; if the difference between the vertical coordinate of the position of the operated UI and the initial position exceeds a certain distance, when the user raises his hand, Unity will hide the corresponding UI; if the front end of the thumb of the user's dominant hand also collides with the operation panel, Unity will adjust the Boolean value of the two-finger collision to true, and Unity will calculate the distance between the front ends of the two fingers in real time. If the distance increases, the corresponding size of the UI will increase, and if the distance decreases, the corresponding size of the UI will decrease.

8. A mixed reality window management and interaction system based on a palm interface, characterized in that: The method for managing and interacting with a mixed reality window based on a palm interface according to any one of claims 1 to 7 comprises: an adjustment module, configured to adjust an optimal angle of a display device based on a mixed reality head-mounted display worn by a user; a comparison module, wherein the comparison module collects different contact modes of the user's non-dominant hand and dominant hand on the operation panel, compares the different contact modes, and obtains an optimal interaction mode between the palm and the operation panel; a recognition module that recognizes gestures based on the contact state between the virtual hand and the operation panel, under the premise that the interaction between the palm and the operation panel is optimal; A completion module selects an interaction target and completes the mixed reality desktop gesture interaction according to the recognized desktop gesture.

Images