Implementation method of augmented reality-based input method, and application and implementation system thereof

Abstract

This invention belongs to the technical field of virtual keyboard input, and discloses a method for implementing an augmented reality (AR) input method, its application, and an implementation system. The method includes: S1: capturing real-world images including the user's hand in real time; S2: constructing a virtual hand model based on the user's hand joint data collected from the real-world image, and using joint motion data to drive the virtual hand model to recreate the user's movements within the AR application; S3: superimposing the virtual keyboard onto the fingers in the real-world image except for the thumb, triggering the virtual keyboard with the thumb, acquiring the joint motion driving the virtual hand model during the triggering process, and detecting the thumb trigger area in the virtual model to input input method commands. This application can combine physical touch and a virtual keyboard, solving the problem of text input in an AR environment in existing technologies, greatly improving the user experience and accuracy.

Description

Technical Field

[0001] This invention belongs to the technical field of virtual keyboard input, and more specifically, relates to an implementation method of an augmented reality-based input method, its application, and an implementation system. Background Technology

[0002] In current augmented reality environments, virtual keyboard input, as a text input method, allows users to input text through a virtual keyboard generated in a virtual environment, similar to traditional computer or mobile phone input methods. This method is familiar to users and easy to learn. However, virtual keyboard input requires users to type on a virtual keyboard in a virtual scene. Due to the lack of a fixed physical reference frame and the tactile feedback of a physical keyboard, the experience differs significantly from typing on a real keyboard, which can easily cause user discomfort.

[0003] The accuracy and stability of current virtual keyboard input are affected by various factors, including keyboard layout design, feedback mechanism, ambient lighting, and background clutter. Furthermore, AR virtual keyboards require a certain amount of space to project, which may limit their effectiveness in confined environments. In some virtual keyboard implementations, the mixed reality system uses one or more holograms to represent the mixed reality environment. To prevent accidental interaction between fingers and virtual keys, the input state of each finger needs to be switched between active and inactive states. When a finger's input is active, the input is mapped to the active state. Conversely, when a finger is inactive, user input is prevented from being provided to the input elements (virtual keys) of the input receiver, which may lead to input delays or accidental touches.

[0004] Virtual gesture input is an intuitive, natural, and flexible input method that can accurately express the user's intent and avoid the discomfort and operational difficulties associated with other input methods such as virtual keyboards. However, this input method requires high gesture recognition accuracy and a certain period of training and adaptation for users, demanding a high level of expertise from the operator. Therefore, virtual gesture input may suffer from issues such as accidental touches, requiring optimization and improvement.

[0005] Therefore, there is an urgent need to design a method that can provide both physical tactile feedback and virtual input, thereby improving user experience and input accuracy. Summary of the Invention

[0006] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides an implementation method and application system of an augmented reality-based input method, which can combine physical touch and virtual keyboard, greatly improving user experience and accuracy.

[0007] To achieve the above objectives, according to one aspect of the present invention, a method for implementing an augmented reality-based input method is provided, comprising: S1: capturing a real-world scene including the user's hand in real time; S2: constructing a virtual hand model based on the user's hand joint data collected in the real-world scene, and using joint motion data to drive the virtual hand model to recreate the user's actions within the AR application; S3: superimposing the virtual keyboard onto the fingers in the real-world scene except for the thumb, triggering the virtual keyboard with the thumb, acquiring the joint motion driving the virtual hand model during the triggering process, and detecting the thumb trigger area in the virtual model to realize the input method command input.

[0008] Preferably, overlaying the virtual keyboard with the fingers of the hand in the real-world scene, excluding the thumb, specifically involves: recognizing a virtual hand model, using each knuckle of the fingers (excluding the thumb) as key partitions, and achieving matching between the virtual keyboard and the fingers.

[0009] Preferably, recognizing the virtual hand model further includes recognizing finger length and palm size, the size of the virtual keyboard matching the finger length, and the position, angle, color, and transparency of the virtual keyboard having a manual adjustment function.

[0010] Preferably, step S3, which overlays the virtual keyboard with the fingers in the real-world scene except for the thumb, specifically involves: constructing a local coordinate system centered on a fixed position in the virtual hand model, obtaining the joint coordinates of the finger joints in the local coordinate system, aligning the center of the virtual keyboard with the center of the virtual coordinate system, and matching the key coordinates on the virtual keyboard with the joint coordinates of the finger joints.

[0011] The second aspect of this application provides an application of the above-mentioned augmented reality-based input method, which is used for Chinese nine-key input, and the fingers other than the thumb are mapped to the nine-key keyboard.

[0012] Preferably, the implementation method is used for Chinese nine-key input, where the knuckles of the fingers other than the thumb correspond to the nine-key keyboard, part of the finger roots of the fingers other than the thumb are used as the space key, and another part of the finger roots of the fingers other than the thumb are used as the symbol key.

[0013] Preferably, the base of the little finger is used as the symbol key, and the base of the index finger and / or middle finger and / or ring finger is used as the space key.

[0014] A third aspect of this application provides an augmented reality-based input method implementation system, comprising: a real-scene acquisition module for capturing real-scene images including the user's hand in real time; a gesture mapping module for constructing a virtual hand model based on the user's hand joint data collected in the real-scene image, and using joint motion data to drive the virtual hand model to recreate the user's actions within the AR application; and a matching module for overlaying the virtual keyboard with the fingers in the real-scene image except for the thumb, using the thumb to trigger the virtual keyboard, acquiring the joint motion driving the virtual hand model during the triggering process, and detecting the thumb trigger area in the virtual model to realize input method commands.

[0015] The fourth aspect of this application provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to perform the above-described implementation method.

[0016] In summary, compared with the prior art, the implementation method, application, and implementation system of the augmented reality-based input method provided by this invention have the following beneficial effects:

[0017] 1. This invention uses AR technology to generate a virtual keyboard based on fixed parts of the human body. In actual use, it has a fixed physical reference system and a realistic tactile feel, which greatly enhances the user experience compared to existing virtual keyboards.

[0018] 2. This application uses human hand joint clustering and regression technology to construct a virtual hand model based on the hand features in the real scene. It maps human joints to the virtual hand model and provides an effective text input method for AR scenes. This method does not require voice input or expensive motion capture equipment, and solves the problem of text input in AR environment in existing technologies.

[0019] 3. In terms of recognizing keystrokes, this application generates a virtual keyboard based on the hand. The keys are bound to the relationships between the knuckles. The software can use a virtual hand model to reproduce the user's typing actions, thereby determining the user's keystrokes by the touch area of ​​the thumb of the hand model. Compared with the existing method of generating virtual keyboards directly in free space, the positioning accuracy of this application is greatly improved.

[0020] 4. In the implementation method provided by this invention, the smart terminal captures a user's palm in the real world. After recognition and processing by the AR application installed on the smart terminal, the virtual keyboard function is activated. A virtual nine-key input method keyboard will be displayed on the smart terminal screen, superimposed on the real-world palm and fingers. When the user moves their fingers, the corresponding keys will also move. The user simulates typing on the virtual keyboard by touching the corresponding keys with the pad of their thumb. Finally, by capturing the user's typing position and typing posture on the virtual keyboard, the key value command that the user wants to input on the virtual keyboard is identified. After processing, the input character is obtained, resulting in a better and more accurate user experience. Attached Figure Description

[0021] Figure 1 This is a schematic diagram illustrating the steps of an implementation method for an augmented reality-based input method according to an embodiment of this application;

[0022] Figure 2 This is a schematic diagram illustrating an application scenario of the implementation method of the augmented reality-based input method according to an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0024] This invention provides a method for implementing an augmented reality-based input method, such as... Figure 1 As shown, the main steps include S1 to S3.

[0025] S1: Real-time capture of live footage including the user's hand.

[0026] A terminal device with camera capabilities (such as Kinect) can be used to capture real-time images including the user's hand.

[0027] S2: Construct a virtual hand model based on user hand joint data collected from real-world images, and use the joint motion data to drive the virtual hand model to recreate the user's movements within the AR application.

[0028] A virtual hand model can be constructed based on the user's hand features in real-world images using human hand joint clustering and regression technology. This technology, based on clustering and regression algorithms, infers the positions of hand bones and joints from depth images acquired from a camera.

[0029] The software uses AR to process real-world images, obtains user hand joint data, generates a virtual hand model within the software, and synchronizes the user's hand movements.

[0030] Subsequently, when the terminal device recognizes the hand and the AR application receives the input command, it activates the hand virtual keyboard function and displays the hand virtual keyboard in the virtual screen.

[0031] S3: Overlay the virtual keyboard with the fingers in the real scene except for the thumb, use the thumb to trigger the virtual keyboard, obtain the joint movement driving the virtual hand model during the triggering process, and detect the thumb triggering area in the virtual model to realize the input method command input.

[0032] The virtual keyboard is superimposed on the user's fingers in the real-world view to mark the key values ​​of each key. When the user issues a keystroke command, the virtual hand inside the software synchronizes the action. At the same time, the software detects the key value area triggered by the virtual hand's thumb and outputs the corresponding character.

[0033] In a further preferred embodiment, this step also includes the AR application constructing a virtual hand model based on the user's hand features. Specifically, the AR application acquires the user's hand bone and joint information and generates a virtual hand model internally. Based on the user's tapping images captured by the camera, a clustering and regression algorithm is used to drive the virtual hand model to complete synchronized movements, ultimately identifying the key areas tapped by the model's hand as the user's key value commands. Based on the identified key value commands, the AR application searches for corresponding characters from its built-in input method, combines them in the form of Chinese Pinyin or English words, displays candidate words in a pop-up window for the user to select and confirm before outputting the final word.

[0034] Then, based on pre-set mapping rules between keys and finger joints, the key areas of the virtual keyboard are mapped one-to-one to the finger joints. The purpose of this is to bind the virtual keyboard keys to the skeletal and joint features of the hand. The system can then recreate the user's input actions on the virtual hand model based on the user's typing movements. As a result, the final output command is accurately reproduced by the thumb striking area of ​​the virtual hand model, enabling the smart terminal to more clearly recognize the keys pressed by the user and output the corresponding characters.

[0035] In a further preferred embodiment, superimposing the virtual keyboard onto the four fingers of the hand in the real-world image (excluding the thumb) specifically involves: recognizing a virtual hand model, using each knuckle of the fingers (excluding the thumb) as key partitions, and thus matching the virtual keyboard with the fingers.

[0036] Specifically, identifying the virtual hand model also includes identifying finger length and palm size, the size of the virtual keyboard matching the finger length, and the position, angle, color, and transparency of the virtual keyboard having manual adjustment capabilities.

[0037] The specific implementation method is as follows: A local coordinate system is constructed with a fixed position in the virtual hand model as the center. The joint coordinates of the finger joints in the local coordinate system are obtained. The center of the virtual keyboard is aligned with the center of the virtual coordinate system. The key coordinates on the virtual keyboard are matched with the joint coordinates of the finger joints. This ensures that the keys always remain in a fixed position on the knuckles and do not detach or change their relative position as the fingers move.

[0038] This application also provides an application of an augmented reality-based input method implementation method, such as... Figure 2 As shown, the implementation method is used for Chinese nine-key input, and the fingers other than the thumb are matched with the nine-key keyboard.

[0039] Applying the aforementioned method to Chinese 9-key input allows for the mapping of knuckles of fingers other than the thumb to the 9-key keyboard. This means that any number of knuckles of fingers other than the thumb can be mapped to the 9-key keyboard. For example, the knuckles of four fingers other than the thumb can be mapped one-to-one to the 9-key keyboard, or the knuckles of three fingers other than the thumb can be mapped to the 9-key keyboard. In this case, the two sides of the knuckle of a certain finger can be mapped to two columns in the 9-key keyboard.

[0040] In a further preferred embodiment, part of the base of the fingers other than the thumb is used as the space key, and another part of the base of the fingers other than the thumb is used as the symbol key.

[0041] For example, the base of the index and / or middle and / or ring fingers can be used as the space bar, and the base of the little finger as the symbol key. Based on the palm information recognized by the camera, the knuckles of the fingers other than the thumb are used as key zones. The key layout is set by referencing the key distribution of the Chinese nine-key input method on smartphones. The pad of the thumb can be used as the selection recognition area, and virtual keys can also be added to other areas of the palm, such as the side of the index finger near the thumb and the part of the palm near the fourth finger.

[0042] During the application, the user simulates typing by touching the virtual key areas on the knuckles of other fingers with their thumb, using the virtual keyboard and their own finger movements. During the typing process, the AR application captures the user's hand posture and internally recreates the user's actual movements onto the virtual hand model. Based on the thumb's tapping area in the virtual hand model, the AR application determines the key value command typed by the user. Based on the recognized key value command, the AR application searches for the corresponding character from the built-in input method and combines it in the form of Chinese Pinyin or English words. The candidate words are displayed in a pop-up window for the user to select and confirm before being output.

[0043] The camera on the smart terminal can capture real-world images in real time. When it detects the user's hand, the AR software receives the instruction to generate a virtual keyboard. Based on the user's hand features, it generates virtual keys at the knuckles, forming a virtual keyboard similar to the nine-key input method on a mobile phone, and overlays it onto the real hand and fingers. The generated keys have an effect of always "fitting" the surface of the fingers. The user can simulate the typing process by touching the corresponding keys with the pad of their thumb. The terminal captures the typing position and posture in real time to identify the key value command the user wants to input on the virtual keyboard. Then, it searches for the corresponding character in the input method built into the AR application and outputs it in the form of a pop-up window in the virtual screen after the user selects and confirms it.

[0044] This application also provides an augmented reality-based input method implementation system, which includes a real-scene acquisition module, a gesture mapping module, and a matching module, specifically:

[0045] Real-scene acquisition module: used to capture real-scene images including the user's hand in real time;

[0046] Gesture mapping module: used to build a virtual hand model based on the user's hand joint data collected from the real scene, and use the joint motion data to drive the virtual hand model to restore the user's movements within the AR application;

[0047] Matching module: used to overlay the virtual keyboard with the fingers in the real scene except for the thumb, use the thumb to trigger the virtual keyboard, obtain joint motion data during the triggering process to drive the virtual hand model, and detect the thumb triggering area in the virtual model to realize the input method command input.

[0048] This embodiment further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described method. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this embodiment are the same as those of the network testing method provided by the above-described technical solutions, and will not be elaborated upon here.

[0049] Those skilled in the art will understand that all or part of the steps in the above-described method can be implemented by a program guiding related hardware. The program can be stored in a computer-readable storage medium. When the program is executed, it includes the steps of the methods in the above embodiments. The storage medium can be ROM / RAM, magnetic disk, optical disk, memory card, etc.

[0050] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for implementing an augmented reality-based input method, characterized in that, include: S1: Real-time capture of live footage including the user's hand; S2: Construct a virtual hand model based on user hand joint data collected from real-world images, and use joint motion data to drive the virtual hand model to reproduce user actions within the AR application; S3: Overlay the virtual keyboard onto the fingers in the real-world scene except for the thumb, use the thumb to trigger the virtual keyboard, obtain the joint movement driving the virtual hand model during the triggering process, and detect the thumb triggering area in the virtual model to realize the input method command input; Superimposing the virtual keyboard onto the fingers of a hand (excluding the thumb) in the real-world image specifically involves: The system identifies virtual hand models and uses the knuckles of each finger (excluding the thumb) as key zones to match the virtual keyboard with the fingers. Step S3, which overlays the virtual keyboard onto the fingers in the real-world image except for the thumb, specifically involves: A local coordinate system is constructed with a fixed position in the virtual hand model as the center. The joint coordinates of the finger joints in the local coordinate system are obtained. The center of the virtual keyboard is aligned with the center of the virtual coordinate system. The key coordinates on the virtual keyboard are matched with the joint coordinates of the finger joints. The aforementioned implementation method is applied to Chinese nine-key input, where fingers other than the thumb are mapped to the nine-key keyboard. Use the base of your little finger as the symbol key, and the base of your index and / or middle and / or ring fingers as the space key.

2. The implementation method according to claim 1, characterized in that, The identification of the virtual hand model also includes identifying finger length and palm size, the size of the virtual keyboard being matched with the finger length, and the position, angle, color, and transparency of the virtual keyboard being manually adjustable.

3. A system for implementing an input method based on augmented reality, characterized in that, include: Real-scene acquisition module: used to capture real-scene images including the user's hand in real time; Gesture mapping module: used to build a virtual hand model based on the user's hand joint data collected from the real scene, and use the joint motion data to drive the virtual hand model to restore the user's movements within the AR application; Matching module: used to overlay the virtual keyboard with the four fingers (excluding the thumb) in the real scene, using the thumb to trigger the virtual keyboard, acquiring the joint movement driving the virtual hand model during the triggering process, and detecting the thumb triggering area in the virtual model to realize input method commands; Superimposing the virtual keyboard onto the fingers of a hand (excluding the thumb) in the real-world image specifically involves: The system identifies virtual hand models and uses the knuckles of each finger (excluding the thumb) as key zones to match the virtual keyboard with the fingers. Superimposing the virtual keyboard onto the fingers in the real-world image, excluding the thumb, specifically involves: A local coordinate system is constructed with a fixed position in the virtual hand model as the center. The joint coordinates of the finger joints in the local coordinate system are obtained. The center of the virtual keyboard is aligned with the center of the virtual coordinate system. The key coordinates on the virtual keyboard are matched with the joint coordinates of the finger joints. The system is used for Chinese 9-key input, with fingers other than the thumb corresponding to the 9-key keyboard. Use the base of your little finger as the symbol key, and the base of your index and / or middle and / or ring fingers as the space key.

4. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program is executed by the processor to perform the implementation method described in claim 1 or 2 above.

Images