Suspended input method and device of head-mounted equipment, equipment and storage medium

Abstract

The invention provides a suspended input method and device for head-mounted equipment, equipment and a storage medium, and the method comprises the steps: recognizing a first fingertip coordinate of a current user through a fingertip tracking algorithm, and when a first coordinate distance between the first fingertip coordinate and an anchoring coordinate region of a suspended keyboard is smaller than or equal to a preset distance, inputting the suspended keyboard into the suspended keyboard; the suspended keyboard is automatically popped up and displayed, the operation of manually activating the keyboard can be reduced, and the input efficiency is improved. The target character is recognized and input by recognizing the target key pointed by the second fingertip coordinate and the character input instruction corresponding to the first operation gesture, the specific input intention of the current user can be understood, the gesture and the character input instruction are combined, rapid and accurate character input is achieved, the input process is simplified, and the input efficiency is improved. And the learning cost of the user is reduced, so that the text input of the head-mounted equipment becomes more visual and convenient.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment and storage medium for suspended input of a head-mounted device. Background Art

[0002] Text input is one of the basic interaction tasks in head-mounted devices. With the development of technology, users' demand for more natural and intuitive interaction methods is growing.

[0003] Existing head-mounted devices mostly rely on external devices for text input operations, such as physical keyboards, handles, touch screens or touchpads, and wearable interactive sensing devices (such as gloves and wristbands).

[0004] However, the traditional physical keyboard input method is not flexible and convenient enough in some scenarios. Physical keyboards require a flat surface for support and are bulky to carry; traditional handles such as TV remote controls and game controllers can only realize simple text input, and multiple discrete key presses will result in very slow input speed; text input on a touch screen or touchpad requires an additional large-size touch screen or touchpad as an input device; wearable interactive sensor devices, although they can realize text input, will affect users' daily interactions and have poor flexibility and comfort.

[0005] Therefore, how to improve the convenience of text input interaction operations of head-mounted devices has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The present application provides a method, apparatus, device and storage medium for suspended input of a head-mounted device, aiming to improve the convenience of text input interactive operations of the head-mounted device.

[0007] In a first aspect, the present application provides a method for mid-air input of a head mounted device, the method comprising:

[0008] Based on the fingertip tracking algorithm, identify the first fingertip coordinates of the current user;

[0009] When a first coordinate distance between the first fingertip coordinate and the anchor coordinate area of ​​the suspended keyboard is less than or equal to a preset distance, displaying the suspended keyboard;

[0010] Based on the fingertip tracking algorithm, identifying the second fingertip coordinates and the first operation gesture of the current user;

[0011] Based on the second fingertip coordinate pointing to the target key of the suspended keyboard and the character input instruction corresponding to the first operation gesture, the target character is input.

[0012] In a second aspect, the present application further provides a suspended input device for a head-mounted device, the suspended input device for the head-mounted device comprising:

[0013] A fingertip tracking module, used for identifying the first fingertip coordinates of the current user based on a fingertip tracking algorithm;

[0014] A keyboard display determination module, configured to display the suspended keyboard when a first coordinate distance between the first fingertip coordinate and an anchor coordinate region of the suspended keyboard is less than or equal to a preset distance;

[0015] An operation gesture recognition module, used for recognizing the second fingertip coordinates and the first operation gesture of the current user based on the fingertip tracking algorithm;

[0016] The input character determination module is used to input the target character based on the target key of the suspended keyboard pointed to by the second fingertip coordinate and the character input instruction corresponding to the first operation gesture.

[0017] In a third aspect, the present application also provides a head-mounted device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the suspended input method of the head-mounted device as described above are implemented.

[0018] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the suspended input method of the head-mounted device as described above are implemented.

[0019] The present application provides a method, device, head-mounted device and storage medium for suspended input of a head-mounted device. The method of the present application can accurately capture the user's finger movements by identifying the first fingertip coordinates of the user through a fingertip tracking algorithm, thereby reducing the dependence on the physical keyboard, so that the head-mounted device can also perform text input without a physical keyboard. When the first coordinate distance between the first fingertip coordinate and the anchor coordinate area of ​​the suspended keyboard is less than or equal to the preset distance, the suspended keyboard is automatically popped up and displayed, which can reduce the operation of manually activating the keyboard and improve the naturalness of interaction and input efficiency. By identifying the second fingertip coordinates and the first operation gesture, the user's input intention can be accurately captured, and the gesture can be directly converted into character input, reducing the need for physical contact and improving the flexibility and speed of input. Based on the target key pointed to by the second fingertip coordinate and the character input instruction corresponding to the first operation gesture, the target character is identified and input, and the specific input intention of the current user can be understood, and the gesture is combined with the character input instruction to achieve fast and accurate character input, which not only simplifies the input process, but also reduces the user's learning cost, making the text input interaction operation of the head-mounted device more intuitive and convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A schematic diagram of a flow chart of a first embodiment of a method for mid-air input of a head mounted device provided in an embodiment of the present application;

[0022] Figure 2 A schematic flow chart of a second embodiment of a method for mid-air input of a head mounted device provided in an embodiment of the present application;

[0023] Figure 3 It is a structural schematic diagram of a first embodiment of a suspended input device for a head mounted device provided by the present application;

[0024] Figure 4 It is a schematic block diagram of the structure of a head-mounted device provided in an embodiment of the present application.

[0025] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0028] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0029] Please refer to Figure 1 , Figure 1 A schematic flow chart of a first embodiment of a hovering input method for a head mounted device provided in an embodiment of the present application.

[0030] like Figure 1 As shown, the hovering input method of the head mounted device includes steps S101 to S104.

[0031] S101, identifying the first fingertip coordinates of the current user based on a fingertip tracking algorithm;

[0032] In one embodiment, when the current user wears a head-mounted device (such as AR glasses, smart helmets, etc.), the image acquisition device (such as a camera) of the head-mounted device captures the user's hand features within the field of view. The hand contour and key points can be detected by a key point tracking algorithm for hand recognition. The key point tracking algorithm can identify the key point information of the hand, such as fingertips, joints, etc., which can be used to identify hand features, improve the accuracy of fingertip coordinate recognition, and can also be used for fingertip coordinate recognition and tracking.

[0033] In one embodiment, when the hand of the current user is identified to be displayed within the field of view, the first fingertip coordinates of the current user are identified through a fingertip tracking algorithm. The first fingertip coordinates may be the coordinates of a specific fingertip, such as an index finger, and are used to determine whether the user needs to use the suspended keyboard.

[0034] Exemplarily, the fingertip tracking algorithm can be a fingertip tracking and trajectory recognition algorithm based on depth image information, a real-time hand key point detection and tracking algorithm, gesture recognition based on MediaPipe (machine learning application framework), and a 3D gesture recognition algorithm based on a dynamic template matching algorithm, etc.

[0035] Among them, the fingertip tracking and trajectory recognition algorithm based on depth image information uses depth image information to segment the hand and detect the fingertips. Through the depth image information captured by Kinect (body sensing technology), the hand is segmented and the fingertips are detected. Then, the compressed sensing method is used to extract the features of the tracked object, and the naive Bayes (NB) algorithm is used to classify the tracked object and the background. Finally, the support vector machine (SVM) method is used to identify the tracking trajectory.

[0036] The real-time hand key point detection and tracking algorithm uses the HandDetector class in the cvzone library for hand detection and key point tracking. It can process video frames in real time, detect and draw hand contours and key points. By obtaining the list of key point coordinates of the hand, the index and middle fingertips can be located and the distance between the fingertips can be calculated.

[0037] The gesture recognition algorithm based on MediaPipe provides information about 21 key points of the hand, which can be used to identify the bending or straightening state of the fingers. By calculating the angles between the key points, the state of the fingers can be determined and the gesture can be recognized accordingly.

[0038] The 3D gesture recognition algorithm based on the dynamic template matching algorithm uses the head-mounted device's sensor to read the 3D gesture data and uses an optimized dynamic time warping algorithm to recognize gestures. By pre-storing the distortion threshold, the calculation cost of template matching and gesture recognition is reduced.

[0039] It can be understood that in actual application, one or more of the above-mentioned fingertip tracking algorithms can be adopted according to actual needs, or other fingertip tracking algorithms or gesture recognition algorithms that can realize the functions described in the embodiments of the present application can be adopted to identify the fingertip coordinates and gesture movements of the current user.

[0040] This embodiment tracks the first fingertip coordinates of the current user through a fingertip tracking algorithm, achieves accurate tracking of the user's fingertip coordinates, and provides accurate fingertip position information for subsequent text input recognition, thereby improving input accuracy and efficiency.

[0041] S102, when a first coordinate distance between the first fingertip coordinate and an anchor coordinate area of ​​the floating keyboard is less than or equal to a preset distance, displaying the floating keyboard;

[0042] In one embodiment, before using the floating keyboard, the current user can set the anchor coordinate area of ​​the floating keyboard according to his / her own needs and usage habits, so that the floating keyboard is fixed and displayed at a certain coordinate area of ​​the field of view.

[0043] Among them, the floating keyboard in the head-mounted display device is a virtual keyboard that allows users to input text without a physical keyboard. This technology displays a virtual keyboard image in the user's field of view through a head-mounted display device (such as AR glasses or a headset), and the user can interact with the virtual keyboard through gesture recognition or eye tracking to achieve text input.

[0044] When not in use, the floating keyboard can be hidden to avoid interfering with the visual field of the head-mounted device. The head-mounted device determines whether the current user intends to use the floating keyboard for text input by calculating the first coordinate distance between the first fingertip coordinate and the anchor coordinate area of ​​the floating keyboard in real time.

[0045] In one embodiment, the first fingertip coordinates may be the coordinates of a specified single finger, which are used to determine whether the user intends to use the suspended keyboard. For example, when the user needs to use the suspended keyboard, the user may extend a specified single finger (such as an index finger). The first fingertip coordinates of the finger are tracked in real time by a fingertip tracking algorithm. When the first fingertip coordinates are close to the area where the suspended keyboard is located, the shortest distance between the first fingertip coordinates and the boundary line of the anchor coordinate area may be calculated as the first coordinate spacing. When the first coordinate spacing between the first fingertip coordinates and the boundary line of the anchor coordinate area is less than or equal to a preset spacing, it is determined that the current user intends to use the suspended keyboard for text input.

[0046] Generally, one or more key points in the suspended keyboard can also be specified as the basis for calculating the first coordinate spacing, such as the center point, corner point, etc. of the suspended keyboard, or a number of coordinate points in the anchor coordinate area of ​​the suspended keyboard can be evenly selected according to parameters such as the shape and size of the suspended keyboard, and the distance between the first fingertip coordinate is calculated to obtain the spacing between each coordinate point and the first fingertip coordinate. When the spacing between any coordinate point and the first fingertip coordinate is less than or equal to the preset spacing, it can be determined that the current user intends to use the suspended keyboard for text input.

[0047] Exemplarily, based on the anchor coordinate area of ​​the suspended keyboard set by the current user, when the fingertip tracking algorithm tracks the first fingertip coordinate of the current user, it can be determined whether the current user has the intention to use the suspended keyboard by the first coordinate distance between the first fingertip coordinate and the anchor coordinate area. When the user's fingertip approaches the key of the suspended keyboard, the first coordinate distance between the fingertip and the anchor coordinate area is monitored in real time. If the first coordinate distance is less than the preset distance (such as 5 cm), a weak vibration is triggered, and the suspended keyboard immediately pops up and displays at the anchor coordinate area.

[0048] During the pop-up display of the suspended keyboard, you can set accompanying animation effects and / or sound feedback to prompt the current user to prepare to confirm the key press. Animation effects and sound feedback can be set according to the current user's usage needs and preferences. For example, animation effects can be gradual appearance, ripple effects, halo effects, 3D flipping, color changes, dynamic background lights, and sound feedback can be prompt sounds, theme sound effects, or voice prompts. Prompts can also be made through screen jitter, device vibration, and other methods.

[0049] Animation effects and sound feedback can be set at the same time. For example, you can set the floating keyboard to appear gradually and provide sound feedback with the rhythm of music. You can also set one separately, for example, only set the floating keyboard to flip in 3D, or the floating keyboard to pop up directly with a prompt sound. You can also turn off animation effects and sound feedback.

[0050] This embodiment realizes the function of intelligently displaying the suspended keyboard by real-time tracking the coordinates of the user's fingertips and calculating the distance with the anchor coordinate area of ​​the suspended keyboard, thereby improving the usability of the virtual keyboard. When the user's fingertips approach the preset area, the keyboard will automatically display, accompanied by animation effects and sound feedback, which enhances the intuitiveness and fun of the interaction and makes text input more convenient and natural.

[0051] Furthermore, based on the keyboard layout requirements of the suspended keyboard, a keyboard layout instruction is generated; based on the keyboard layout instruction, the layout of the suspended keyboard is switched so that the suspended keyboard is displayed based on the keyboard layout requirements; wherein the keyboard layout requirements include key layout requirements and keyboard display layout requirements.

[0052] In one embodiment, the key layout requirement may be a keyboard key layout, such as which keys are required, key positions, key sizes, key shapes, etc. The keyboard display layout requirement may include layout parameters such as keyboard display position, keyboard shape, keyboard display effects, keyboard pop-up and hiding animation effects, etc.

[0053] In one embodiment, the keyboard layout requirement may include keyboard shape, function keys, character key layout, etc., such as whether a QWERTY layout, a Dvorak layout, or other specific language or customized layout is required. The user can quickly switch between different keyboard layout options as needed, such as "QWERTY", "Dvorak" or "numeric keypad", etc. The user's switching habits can be recorded, and quick switching can be provided when the current user needs it.

[0054] Exemplarily, in the suspended keyboard, the function keys may include shortcut keys (F1-F12), Esc key (Escape: used to cancel the current operation or exit), Ctrl key (Control: used to perform shortcut operations in combination with other keys), arrow keys, etc. The character keys may include letter keys (uppercase letters AZ, lowercase letters az), number keys (0-9), symbol keys (punctuation marks and special symbols, such as!, @, #, etc.), etc.

[0055] In one embodiment, the current user can manually adjust the shape of the keyboard and add custom function keys, such as multimedia control, shortcut commands, etc., to achieve personalized settings, making the suspended keyboard more flexible and comfortable when typing in the air.

[0056] Specifically, according to the keyboard layout requirements of the current user, a keyboard layout configuration file and a keyboard layout instruction can be generated. The keyboard layout instruction will contain key mapping information, such as which key corresponds to which character or function. The processor of the head-mounted device can parse the keyboard layout instruction, adjust the layout of the suspended keyboard keys according to the instruction, and apply the parsed keyboard layout to the suspended keyboard.

[0057] By allowing users to customize the layout of the suspended keyboard according to personal preferences and usage scenarios, the personalization and flexibility of the keyboard layout is achieved, making the suspended keyboard more adaptable to the user's input habits and improving input efficiency and comfort.

[0058] In one embodiment, when the current user inputs text through the suspended keyboard, the behavior pattern of the current user can be learned, and then the layout of the suspended keyboard can be adaptively and automatically adjusted.

[0059] Generally, after learning the behavior pattern of the current user and calculating the adaptive layout parameters of the suspended keyboard, an information pop-up window can be popped up to remind the current user to adjust the layout of the suspended keyboard. The current user can choose to adjust immediately, not adjust, or adjust with a delay, etc. The background adjustment can also be performed after the current text input is completed and before the current user uses the suspended keyboard next time.

[0060] Specifically, when the current user inputs text characters, the input behavior data of the current user can be collected through the depth camera and motion sensor, and then the collected input behavior data can be analyzed through the machine learning algorithm to analyze the current user's behavior pattern, including finger movement trajectory, input frequency, key strength and key duration, etc. According to the user's behavior analysis results, the keyboard layout is dynamically adjusted.

[0061] For example, if the current user frequently uses certain specific characters when inputting, the head-mounted device can automatically increase the key area of ​​these characters to make them easier to reach and improve input efficiency. Automatically adjust the shape and layout of the suspended keyboard. When the user inputs quickly, the keyboard size can be automatically reduced and the key spacing can be reduced to improve operation accuracy and reduce the risk of accidental touches. When precise input is required, the keyboard can be enlarged to provide a more relaxed operating space. Automatically adjust the size of the key area according to the user's input frequency and habits. Frequently used keys can be enlarged, while infrequently used keys can be reduced or rearranged.

[0062] This embodiment learns the user's behavior patterns during the use of the suspended keyboard, and adaptively and automatically adjusts the keyboard layout according to the user's behavior patterns, so that the keyboard layout of the suspended keyboard is more suitable for the user's input habits and needs of using the suspended keyboard, thereby improving input efficiency and user experience.

[0063] S103, identifying the second fingertip coordinates and the first operation gesture of the current user based on the fingertip tracking algorithm;

[0064] In one embodiment, a fingertip tracking algorithm is used to identify the second fingertip coordinates of the current user, and the character key or function key pointed to by the current user on the floating keyboard is identified based on the second fingertip coordinates. Then, the input operation performed by the current user on the pointed character key or function key is identified based on the current user's first operation gesture.

[0065] Specifically, the fingertip tracking algorithm can be used to extract the hand contour area of ​​the current user, perform feature extraction on the hand contour area, and identify the position of the fingertip. For example, the second fingertip coordinate can be identified by calculating the key points of the hand, including the position of the wrist, palm, and each fingertip.

[0066] Specifically, the first operation gesture is identified by analyzing the bending and straightening states of the fingers and the relative positions between the fingertips. For example, the bending state of the fingers can be determined by calculating the angle between the fingers, and then the gesture can be identified. The click state of the fingers can also be analyzed, such as performing a single click, double click, or long press gesture on a certain character key.

[0067] When identifying the first operation gesture, in order to accurately identify the gesture action made by the current user, the gesture state of multiple consecutive frames can be detected, and if the gesture state is stable, the current gesture can be output as the first operation gesture. For example, if the gesture state of 30 consecutive frames is the same, the gesture is considered to be stable and is output as the first operation gesture.

[0068] For example, when the floating keyboard pops up and is displayed in the field of view, the current user can touch the keys of the floating keyboard with his fingertips in the air, and the fingertip tracking algorithm captures the fingertip position and movement in real time to identify the corresponding character input. The current user can input ordinary characters by "light touch", "deep touch" to input a letter key to input the uppercase letter or related symbols, for example, long pressing "A" can input "A" or "@", and the "slide" gesture is used to quickly select multiple characters.

[0069] In one embodiment, based on the second fingertip coordinates and the anchor coordinate area of ​​the suspended keyboard, the second coordinate distance between the current user's fingertip and the suspended keyboard is calculated; when the second coordinate distance is greater than the preset distance, operation reminder information is generated to remind the current user of the operation effective area of ​​the first operation gesture.

[0070] In one embodiment, based on the second fingertip coordinates of the current user collected by the fingertip tracking algorithm, the front, back, left, and right distances between the fingertip of the current user and the anchor coordinate area where the suspended keyboard is located, that is, the second coordinate spacing, are calculated in real time.

[0071] In one embodiment, because the current user can use one or more fingers to tap the keyboard keys when using the suspended keyboard for text input, the fingertip tracking algorithm may collect one or more second fingertip coordinates. At this time, it is necessary to compare the second coordinate distance between each collected second fingertip coordinate and the anchor coordinate area where the suspended keyboard is located, and determine the size of the second coordinate distance between the second fingertip coordinate corresponding to each finger and the preset distance. When the distance between the second fingertip coordinate corresponding to any finger and the second coordinate of the anchor coordinate area is less than or equal to the preset distance, the first operation gesture of the current user is considered valid; when the distance between the second fingertip coordinates of all tracked fingers and the second coordinate of the anchor coordinate area is greater than the preset distance, the first operation gesture of the current user is considered invalid.

[0072] When the second coordinate spacing is greater than the preset spacing (such as 5cm), the floating keyboard cannot be operated, the current user's first operation gesture is invalid, and valid character text cannot be entered. At this time, a virtual pop-up box can pop up to prompt how much distance the fingertip needs to move forward / backward / left / right from the floating keyboard to enable floating typing, so that the input character text is effective.

[0073] In one embodiment, the processor of the head mounted device can analyze the user's input information in real time and provide word association suggestions, and the current user can select the recommended word by sliding a finger.

[0074] This embodiment uses a fingertip tracking algorithm to accurately identify the user's second fingertip coordinates and first operation gesture, and realizes efficient operation of character keys or function keys on the suspended keyboard. By analyzing the bending, straightening state and clicking action of the finger, the user's input intention is identified, and operation feedback is provided according to the relative position of the fingertip and the keyboard, which not only improves the accuracy and efficiency of input, but also enhances the intuitiveness and naturalness of the user's interaction with the virtual keyboard. In addition, word association suggestions are provided according to the user's input habits, further improving the convenience and input efficiency of text input.

[0075] S104: Input a target character based on the target key of the suspended keyboard pointed to by the second fingertip coordinate and the character input instruction corresponding to the first operation gesture.

[0076] In one embodiment, the target key pointed to by the second fingertip coordinates can be determined by calculating the matching degree between the second fingertip coordinates and the coordinates of each key in the suspended keyboard. For example, if the user points to the "T" key on the keyboard, the target key pointed to by the current user can be identified by matching the second fingertip coordinates with the key position coordinates on the suspended keyboard layout. The target key can be a character key or a function key.

[0077] In one embodiment, the coordinate area of ​​each key in the suspended keyboard can be pre-recorded, for example, the coordinate area surrounded by the key edge line, or the coordinates of the key center point can be used as the basis for key recognition. The matching degree is calculated by calculating the coordinate distance between the second fingertip coordinate and the coordinate area or center point coordinate corresponding to each key. The smaller the coordinate distance, the higher the matching degree, and the larger the coordinate distance, the lower the matching degree. Therefore, the key with the smallest coordinate distance can be determined as the target key.

[0078] In one embodiment, the coordinate distance between the second fingertip coordinates and the coordinate area can be calculated separately as the matching degree, or the coordinate distance between the second fingertip coordinates and the center point coordinates of the key can be calculated separately as the matching degree. If the coordinate distance between the center point coordinates (or coordinate area) and the second fingertip coordinates is calculated separately, and multiple keys with the same matching degree are screened out, the coordinate distance between the coordinate area (or center point coordinates) of the multiple screened keys and the second fingertip coordinates is calculated again, and the key with the smallest coordinate distance is determined as the target key.

[0079] Furthermore, based on the key position attributes of the target key, at least one preset operation gesture corresponding to the target key is determined; the first operation gesture is matched with each of the preset operation gestures for similarity to determine the character input instruction corresponding to the preset operation gesture with the highest similarity; based on the character input instruction, the target character is input.

[0080] In one embodiment, the key attributes may include the position coordinates, size, shape, and special functions associated with the key (such as shortcut key combinations). The specific operation gesture associated with each key can be defined according to the key attributes. For the same key, each specific operation gesture corresponds to a character input instruction. For example, for an alphabetic character key, a single click corresponds to a lowercase letter input instruction, and a long press corresponds to an uppercase letter input instruction. Different keys can use the same preset operation gesture. For example, for alphabetic character keys AZ, it can be defined that a single click corresponds to a lowercase letter input instruction, and a long press corresponds to an uppercase letter input instruction.

[0081] In one embodiment, each button may be pre-set with one or more preset operation gestures, such as tap, click, long press or slide.

[0082] In one embodiment, the similarity matching algorithm can be used to compare the similarity between the first operation gesture performed by the current user and the preset operation gesture. The similarity matching algorithm can perform similarity calculation based on parameters such as gesture shape, speed, direction, and duration, identify the preset operation gesture with the highest similarity to the first operation gesture, identify the character input instruction corresponding to the preset operation gesture with the highest similarity as the character input instruction corresponding to the first operation gesture, and perform the corresponding character input operation. The input target character can be a single letter, number, symbol, or a special character generated by a key combination.

[0083] Specifically, the features of the first operation gesture, including parameters such as gesture shape, speed, direction and duration, are identified through a fingertip tracking algorithm or a gesture recognition algorithm. For example, a gesture tracking algorithm can be used to track dynamic gestures and extract features such as key points of the hand (such as fingertips, joints and other key points). The feature vector of the first operation gesture is matched with the feature vector of each preset operation gesture through a template matching algorithm, and the similarity between the two is calculated to realize gesture recognition. For example, in three-dimensional gesture recognition, the matrix singular value decomposition method can be used to calculate the optimal rotation matrix between the normalized gesture trajectory of the first operation gesture and the preset operation gesture, thereby realizing efficient template matching. Among them, a variety of algorithms can be used for similarity calculation, such as cosine similarity, Euclidean distance, etc.

[0084] Exemplarily, after determining the target key, the current user's operation intention on the target key is identified by identifying the first operation gesture performed by the current user on the target key. For example, one or more gesture actions for performing a typing operation on the character keys of the suspended keyboard can be pre-defined, such as for letter keys, a single click can be used to input lowercase letters, and a long press can be used to input uppercase letters, etc. Then, by identifying the current user's first operation gesture on the target key, and matching the first operation gesture with the pre-defined gesture action by similarity, the character input instruction corresponding to the first operation gesture is identified.

[0085] For example, after identifying the target key pointed to by the user and the first operation gesture performed on the target key, this information can be converted into a character input instruction. For example, if the current user points to the "T" key with the index finger and makes a "tap" gesture, this action can be recognized as an instruction to input "T".

[0086] This embodiment realizes accurate recognition of user gestures and intelligent matching of character input instructions by combining the key attributes of the target key and the preset operation gesture. By analyzing the first gesture feature of the user and matching the similarity with the second gesture feature of the preset operation gesture, the user's input intention is accurately recognized and the corresponding character input operation is performed, thereby improving the accuracy and response speed of input.

[0087] Furthermore, based on the gesture recognition algorithm, the gesture actions of the current user are identified; when the floating keyboard is in a displayed state and the gesture action is a second operation gesture, the floating keyboard is hidden and the current field of view of the head-mounted device is restored to display; when the floating keyboard is in a hidden state and the gesture action is a third operation gesture, the floating keyboard is popped up and displayed.

[0088] In one embodiment, the operation gestures for popping up and retracting (or displaying and hiding) the floating keyboard can be preset. When the floating keyboard is in the display state, the floating keyboard is hidden through the second operation gesture; when the floating keyboard is in the hidden state, the floating keyboard is displayed through the third operation gesture.

[0089] For example, if the user wants to hide the floating keyboard after finishing the use, he only needs to simply spread his palm gesture (first operation gesture) to fold or hide the floating keyboard and restore the display screen of the head mounted device. In this process, visual feedback can be provided at the same time, such as the keyboard gradually fading out, accompanied by slight sound effects, to enhance the user's interactive experience.

[0090] Exemplarily, if the user wants to use the floating keyboard again, he or she can make a fist gesture (the third operation gesture). When the head-mounted device recognizes the gesture, the floating keyboard will pop up and display; or the user can move the fingertips away from the floating keyboard beyond a preset distance (such as 5 cm). At this time, the head-mounted device recognizes that the distance between the fingertip coordinates and the coordinates of the fingertips in the anchor coordinate area of ​​the floating keyboard is greater than the preset distance, then it will generate and execute an instruction to hide the floating keyboard. After the floating keyboard is hidden, if the user puts the fingertips back into the preset distance, that is, the head-mounted device recognizes that the distance between the fingertip coordinates and the coordinates of the fingertip in the anchor coordinate area of ​​the floating keyboard is less than or equal to the preset distance, then the floating keyboard pop-up and display operation will be executed again.

[0091] This embodiment provides a method for floating input of a head-mounted device. The method can accurately capture the user's finger movements by identifying the first fingertip coordinates of the user through a fingertip tracking algorithm, thereby reducing the dependence on the physical keyboard, so that the head-mounted device can also perform text input without a physical keyboard. When the first coordinate distance between the first fingertip coordinate and the anchor coordinate area of ​​the floating keyboard is less than or equal to the preset distance, the floating keyboard is automatically popped up and displayed, which can reduce the operation of manually activating the keyboard and improve the naturalness of interaction and input efficiency. By identifying the second fingertip coordinates and the first operation gesture, the user's input intention can be accurately captured, and the gesture can be directly converted into character input, which reduces the need for physical contact and improves the flexibility and speed of input. Based on the target key pointed to by the second fingertip coordinate and the character input instruction corresponding to the first operation gesture, the target character is identified and input, and the specific input intention of the current user can be understood, and the gesture is combined with the character input instruction to achieve fast and accurate character input, which not only simplifies the input process, but also reduces the user's learning cost, making the text input of the head-mounted device more intuitive and convenient.

[0092] Please refer to Figure 2 , Figure 2 A schematic flow chart of a second embodiment of a hovering input method for a head mounted device provided in an embodiment of the present application.

[0093] like Figure 2 As shown, based on the above Figure 1 In the illustrated embodiment, before step S102, the following steps are further included:

[0094] S201, determining the anchoring coordinates and display parameters of the suspended keyboard based on the keyboard anchoring distance of the suspended keyboard, wherein the display parameters include a magnification ratio;

[0095] In one embodiment, the current user can manually configure the keyboard anchor distance when wearing the head mounted device, and the keyboard anchor distance can be a fixed distance relative to the user's eyes. When using the floating keyboard, the depth camera can be used to obtain the real-time position data of the user's eyes, and the anchor coordinates corresponding to the center point of the floating keyboard can be calculated in combination with the anchor distance input by the user.

[0096] In one embodiment, the magnification ratio of the keyboard can be adjusted according to the anchor distance and the visual habits of the current user, or the correspondence between the magnification ratio and the anchor distance can be solidified to form a linear correspondence or a nonlinear correspondence, so that the anchor distance and the magnification ratio form a one-to-one correspondence. After the current user inputs the keyboard anchor distance, the magnification ratio corresponding to the keyboard anchor distance can be determined according to the correspondence.

[0097] In one embodiment, the display parameters may further include clarity. The clarity may be adjusted according to the anchoring distance and the user's vision. A higher clarity may be configured for a longer distance to ensure readability of the key.

[0098] In one embodiment, the current user can set the magnification ratio and clarity according to his or her own preferences, and the system can remember and apply these parameters according to the settings of the current user.

[0099] Exemplarily, when the current user is wearing a head-mounted device, the anchoring distance of the suspended keyboard can be customized through the setting interface. A slider or distance input box can be provided in the display screen of the head-mounted device, and the current user can intuitively adjust the distance between the keyboard and the line of sight, and adjust it within a certain range, such as the range can range from 30 cm to 90 cm. This setting can provide real-time feedback. When the current user adjusts the anchoring distance, the size and transparency of the suspended keyboard can dynamically change according to the adjustment of the anchoring distance, providing visual comfort. For example, when the user adjusts the suspended keyboard closer, the keyboard can be enlarged and become clearer to ensure that the user can input conveniently.

[0100] At the same time, the relative position between the display direction of the floating keyboard and the current user's perspective can be solidified. When the floating keyboard is displayed, the current user's perspective can be monitored in real time through the depth camera. No matter how the current user moves or changes his posture, the floating keyboard is fixedly displayed at a certain coordinate position of the current perspective, ensuring that the floating keyboard is always in the best position.

[0101] S202: Determine an anchor coordinate area of ​​the suspended keyboard based on the anchor coordinates and the display parameters.

[0102] In one embodiment, the anchor coordinate area refers to a specific display area of ​​the floating keyboard in three-dimensional space, which is defined by the anchor coordinates and their related size parameters (such as width, height, and depth). The center point coordinates of the floating keyboard are calculated using the user's eye position data obtained by the depth camera and the anchor distance input by the user. Based on the center point coordinates and the size parameters of the keyboard (which may be determined by user preferences or default settings), the boundaries of the anchor coordinate area are calculated.

[0103] In one embodiment, the display direction and position of the floating keyboard are consistent with the current user's perspective. By solidifying the relative posture between the floating keyboard and the current user, the current user's perspective is monitored in real time, and the anchor coordinate area is adjusted according to the change in perspective. Even if the current user moves or changes posture, the floating keyboard remains within the current user's optimal field of view to ensure the visibility and usability of the floating keyboard.

[0104] This embodiment dynamically adjusts the anchoring coordinate area and display parameters of the suspended keyboard according to the anchoring distance of the suspended keyboard and the current user's visual habits, thereby improving the readability, accessibility and user input comfort of the suspended keyboard. It not only enhances the user experience, but also improves the input efficiency and adaptability of the suspended keyboard through personalized settings and real-time feedback mechanisms.

[0105] See also Figure 3 , Figure 3 It is a structural schematic diagram of a first embodiment of a suspended input device for a head-mounted device provided in the present application. The suspended input device for the head-mounted device is used to execute the aforementioned suspended input method for the head-mounted device.

[0106] like Figure 3 As shown, the suspended input device 300 of the head mounted device includes: a fingertip tracking module 301, a keyboard display determination module 302, an operation gesture recognition module 303 and an input character determination module 304.

[0107] A fingertip tracking module 301 is used to identify the first fingertip coordinates of the current user based on a fingertip tracking algorithm;

[0108] A keyboard display determination module 302, configured to display the suspended keyboard when a first coordinate distance between the first fingertip coordinate and an anchor coordinate region of the suspended keyboard is less than or equal to a preset distance;

[0109] An operation gesture recognition module 303 is used to recognize the second fingertip coordinates and the first operation gesture of the current user based on the fingertip tracking algorithm;

[0110] The input character determination module 304 is used to input a target character based on the target key of the suspended keyboard pointed to by the second fingertip coordinates and the character input instruction corresponding to the first operation gesture.

[0111] In one embodiment, the suspended input device 300 of the head mounted device further includes an anchor coordinate area determination module, including:

[0112] A keyboard anchoring distance determining unit, used to determine the anchoring coordinates and display parameters of the suspended keyboard based on the keyboard anchoring distance of the suspended keyboard, wherein the display parameters include a magnification ratio;

[0113] An anchor coordinate region determining unit is used to determine an anchor coordinate region of the suspended keyboard based on the anchor coordinates and the display parameters.

[0114] In one embodiment, the suspended input device 300 of the head mounted device further includes a keyboard layout module, including:

[0115] A keyboard layout instruction generating unit, used for generating a keyboard layout instruction based on the keyboard layout requirement of the suspended keyboard;

[0116] A layout switching unit, configured to switch the layout of the suspended keyboard based on the keyboard layout instruction, so that the suspended keyboard is displayed in a layout based on the keyboard layout requirement;

[0117] Among them, the keyboard layout requirements include key layout requirements and keyboard display layout requirements.

[0118] In one embodiment, the suspended input device 300 of the head mounted device further includes a quick gesture operation module, including:

[0119] A gesture recognition unit, used to recognize the gestures of the current user based on a gesture recognition algorithm;

[0120] The keyboard hiding unit is used to hide the floating keyboard and restore the current field of view of the head mounted device when the floating keyboard is in a displayed state and the gesture action is a second operation gesture.

[0121] In one embodiment, the shortcut gesture operation module further includes:

[0122] The keyboard display unit is used to pop up and display the suspended keyboard when the suspended keyboard is in a hidden state and the gesture action is a third operation gesture.

[0123] In one embodiment, the suspended input device 300 of the head mounted device further includes an operation reminder module, including:

[0124] A second coordinate distance calculation unit, used for calculating a second coordinate distance between the fingertip of the current user and the suspended keyboard based on the second fingertip coordinate and the anchor coordinate area of ​​the suspended keyboard;

[0125] The operation reminder information generating unit is used to generate operation reminder information when the second coordinate interval is greater than a preset interval, so as to remind the current user of the operation effective area of ​​the first operation gesture.

[0126] In one embodiment, the input character determination module 304 includes:

[0127] A preset operation gesture determining unit, used to determine at least one preset operation gesture corresponding to the target key based on the key position attribute of the target key;

[0128] a character input instruction determining unit, configured to perform similarity matching between the first operation gesture and each of the preset operation gestures, and determine the character input instruction corresponding to the preset operation gesture having the highest similarity;

[0129] The target character generating unit is used to input the target character based on the character input instruction.

[0130] It should be noted that technicians in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned embodiment of the suspended input method of the head-mounted device, and will not be repeated here.

[0131] The apparatus provided in the above embodiment may be implemented in the form of a computer program. The computer program may be Figure 4 Running on the headset shown.

[0132] See also Figure 4 , Figure 4 1 is a schematic block diagram of a head mounted device provided in an embodiment of the present application. The head mounted device may be a server.

[0133] See also Figure 4 The head mounted device includes a processor, a memory and a network interface connected via a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0134] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any one of the suspended input methods of the head mounted device.

[0135] The processor is used to provide computing and control capabilities to support the operation of the entire head-mounted device.

[0136] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any one of the suspended input methods of the head-mounted device.

[0137] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 4 The structure shown in the figure is merely a block diagram of a portion of the structure related to the scheme of the present application, and does not constitute a limitation on the head-mounted device to which the scheme of the present application is applied. The specific head-mounted device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0138] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0139] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0140] Based on the fingertip tracking algorithm, identify the first fingertip coordinates of the current user;

[0141] When a first coordinate distance between the first fingertip coordinate and the anchor coordinate area of ​​the suspended keyboard is less than or equal to a preset distance, displaying the suspended keyboard;

[0142] Based on the fingertip tracking algorithm, identifying the second fingertip coordinates and the first operation gesture of the current user;

[0143] Based on the second fingertip coordinate pointing to the target key of the suspended keyboard and the character input instruction corresponding to the first operation gesture, the target character is input.

[0144] In one embodiment, the processor is further configured to implement, before displaying the suspended keyboard, when the first coordinate distance between the first fingertip coordinate and the anchor coordinate area of ​​the suspended keyboard is less than or equal to a preset distance:

[0145] Determining the anchoring coordinates and display parameters of the suspended keyboard based on the keyboard anchoring distance of the suspended keyboard, wherein the display parameters include a magnification ratio;

[0146] Based on the anchor coordinates and the display parameters, an anchor coordinate area of ​​the floating keyboard is determined.

[0147] In one embodiment, before implementing the identifying the second fingertip coordinates and the first operation gesture of the current user based on the fingertip tracking algorithm, the processor is further configured to implement:

[0148] Generate keyboard layout instructions based on keyboard layout requirements of the suspended keyboard;

[0149] Based on the keyboard layout instruction, the layout of the suspended keyboard is switched, so that the suspended keyboard is displayed in a layout based on the keyboard layout requirement;

[0150] Among them, the keyboard layout requirements include key layout requirements and keyboard display layout requirements.

[0151] In one embodiment, after implementing the character input instruction corresponding to the first operation gesture pointing to the target key of the suspended keyboard based on the second fingertip coordinates and inputting the target character, the processor is further configured to implement:

[0152] Based on the gesture recognition algorithm, identify the current user's gesture actions;

[0153] When the suspended keyboard is in a displayed state and the gesture action is a second operation gesture, the suspended keyboard is hidden and the current field of view of the head mounted device is restored to display.

[0154] In one embodiment, after implementing the gesture recognition algorithm to identify the gesture action of the current user, the processor is further configured to implement:

[0155] When the suspended keyboard is in a hidden state and the gesture action is a third operation gesture, the suspended keyboard is popped up and displayed.

[0156] In one embodiment, after implementing the identification of the second fingertip coordinates and the first operation gesture of the current user based on the fingertip tracking algorithm, the processor is further configured to implement:

[0157] Calculate a second coordinate distance between the fingertip of the current user and the suspended keyboard based on the second fingertip coordinate and the anchor coordinate area of ​​the suspended keyboard;

[0158] When the second coordinate interval is greater than a preset interval, operation reminder information is generated to remind the current user of the operation effective area of ​​the first operation gesture.

[0159] In one embodiment, when the processor implements the character input instruction corresponding to the first operation gesture and points to the target key of the suspended keyboard based on the second fingertip coordinates to input the target character, it is used to implement:

[0160] Determining at least one preset operation gesture corresponding to the target key based on the key position attribute of the target key;

[0161] Performing similarity matching between the first operation gesture and each of the preset operation gestures, and determining the character input instruction corresponding to the preset operation gesture with the highest similarity;

[0162] Based on the character input instruction, the target character is input.

[0163] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any of the suspended input methods for a head-mounted device provided in the embodiments of the present application.

[0164] The computer-readable storage medium may be an internal storage unit of the head-mounted device described in the foregoing embodiment, such as a hard disk or memory of the head-mounted device. The computer-readable storage medium may also be an external storage device of the head-mounted device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., provided on the head-mounted device.

[0165] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for mid-air input of a head mounted device, characterized in that: The method comprises: Based on the fingertip tracking algorithm, identify the first fingertip coordinates of the current user; When a first coordinate distance between the first fingertip coordinate and the anchor coordinate area of ​​the suspended keyboard is less than or equal to a preset distance, displaying the suspended keyboard; Based on the fingertip tracking algorithm, identifying the second fingertip coordinates and the first operation gesture of the current user; Based on the second fingertip coordinate pointing to the target key of the suspended keyboard and the character input instruction corresponding to the first operation gesture, the target character is input.

2. The method for mid-air input of a head mounted device according to claim 1, characterized in that: When the first coordinate distance between the first fingertip coordinate and the anchor coordinate area of ​​the suspended keyboard is less than or equal to the preset distance, before displaying the suspended keyboard, the method further includes: Determining the anchoring coordinates and display parameters of the suspended keyboard based on the keyboard anchoring distance of the suspended keyboard, wherein the display parameters include a magnification ratio; Based on the anchor coordinates and the display parameters, an anchor coordinate area of ​​the floating keyboard is determined.

3. The method for mid-air input of a head mounted device according to claim 1, characterized in that: Before identifying the second fingertip coordinates and the first operation gesture of the current user based on the fingertip tracking algorithm, the method further includes: Generate keyboard layout instructions based on keyboard layout requirements of the suspended keyboard; Based on the keyboard layout instruction, the layout of the suspended keyboard is switched, so that the suspended keyboard is displayed in a layout based on the keyboard layout requirement; Among them, the keyboard layout requirements include key layout requirements and keyboard display layout requirements.

4. The method for mid-air input of a head mounted device according to claim 1, wherein: After inputting the target character based on the target key pointed to by the second fingertip coordinates and the character input instruction corresponding to the first operation gesture, the method further includes: Based on the gesture recognition algorithm, identify the current user's gesture actions; When the suspended keyboard is in a displayed state and the gesture action is a second operation gesture, the suspended keyboard is hidden and the current field of view of the head mounted device is restored to display.

5. The method for mid-air input of a head mounted device according to claim 4, characterized in that: After the gesture recognition algorithm is used to recognize the gesture action of the current user, the method further includes: When the suspended keyboard is in a hidden state and the gesture action is a third operation gesture, the suspended keyboard is popped up and displayed.

6. The method for mid-air input of a head mounted device according to claim 1, characterized in that: After identifying the second fingertip coordinates and the first operation gesture of the current user based on the fingertip tracking algorithm, the method further includes: Calculate a second coordinate distance between the fingertip of the current user and the suspended keyboard based on the second fingertip coordinate and the anchor coordinate area of ​​the suspended keyboard; When the second coordinate interval is greater than a preset interval, operation reminder information is generated to remind the current user of the operation effective area of ​​the first operation gesture.

7. The method for mid-air input of a head mounted device according to claim 1, characterized in that: The inputting of the target character based on the target key pointed to by the second fingertip coordinates and the character input instruction corresponding to the first operation gesture includes: Determining at least one preset operation gesture corresponding to the target key based on the key position attribute of the target key; Performing similarity matching between the first operation gesture and each of the preset operation gestures, and determining the character input instruction corresponding to the preset operation gesture with the highest similarity; Based on the character input instruction, the target character is input.

8. A suspended input device for a head mounted device, characterized in that: The suspended input device of the head mounted device comprises: A fingertip tracking module, used for identifying the first fingertip coordinates of the current user based on a fingertip tracking algorithm; A keyboard display determination module, configured to display the suspended keyboard when a first coordinate distance between the first fingertip coordinate and an anchor coordinate region of the suspended keyboard is less than or equal to a preset distance; An operation gesture recognition module, used for recognizing the second fingertip coordinates and the first operation gesture of the current user based on the fingertip tracking algorithm; The input character determination module is used to input the target character based on the target key of the suspended keyboard pointed to by the second fingertip coordinate and the character input instruction corresponding to the first operation gesture.

9. A head mounted device, characterized in that: The head-mounted device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the hovering input method of the head-mounted device as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the hovering input method for a head mounted device as claimed in any one of claims 1 to 7 are implemented.

Images