KEY FUNCTION EXECUTION METHOD, KEY FUNCTION EXECUTION SYSTEM, KEY FUNCTION EXECUTION DEVICE, ELECTRONIC DEVICE, AND COMPUTER PROGRAM

Abstract

The present application discloses a method, an apparatus, a device, and a storage medium for performing a key function, which belong to the technical field of computers. The method includes the steps of: displaying a virtual keyboard (601); displaying a cursor at a first position based on a biometric feature of the target gesture in response to detecting that the hand is in a target gesture (602); displaying the cursor moving to follow the movement of the hand in response to the hand moving while holding the target gesture (603); and executing a function of a key corresponding to a second position of the virtual keyboard in response to a change in the hand gesture, the second position being a position where the cursor is located when the hand gesture changes (604). In the embodiment of the present application, the operation is very simple and convenient, and the selection and triggering of one key can also be performed precisely, thus providing a flexible and easy operation manner that improves the granularity and accuracy of the triggering of the key.

Description

[Technical field]

[0001] This application claims priority to a Chinese patent application filed on June 22, 2021, bearing application number 202110693794.4 and titled "Key function execution method, device, equipment, and storage medium," the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the technical field of computers, and in particular to a key function execution method, device, apparatus, and storage medium. [Background technology]

[0003] With the development of computer technology, keys can be displayed on electronic devices, and when the user operates the keys, the electronic device can be triggered to automatically perform the corresponding function, greatly reducing the labor cost and realizing smartness.

[0004] Currently, a key function execution method typically involves displaying a virtual keyboard on a screen, and in response to detecting a touch operation on the screen by a user, executing the function of a key corresponding to the contact point of the touch operation. Summary of the Invention [Problem to be solved by the invention]

[0005] The embodiments of the present application provide a method, device, apparatus, and storage medium for executing a key function with improved granularity and accuracy of triggering to a key and good applicability. [Means for solving the problem]

[0006] In one aspect, a method for performing a key function is provided, the method comprising: A step of the key function execution system displaying a virtual keyboard, the virtual keyboard including a first area and a second area, The aboveeach second region including at least two keys, said first region corresponding to a left hand and said second region corresponding to a right hand; in response to detecting a hand in a target gesture, the system displaying a cursor at a first location based on a biometric feature of the target gesture, the first location being within an area of ​​the virtual keyboard corresponding to the biometric feature and corresponding to the biometric feature; in response to the hand moving while holding the target gesture, the system displays the cursor moving to track the hand's movement; and in response to the change in the hand gesture, the system executes a function of a key corresponding to a second position of the virtual keyboard, the second position being a position where the cursor is located when the hand gesture changes.

[0007] In one aspect, a key function execution system is provided, the system including an electronic device, a gesture tracking sensor, and a display device, the gesture tracking sensor and the display device being each connected to the electronic device, The display device displays a virtual keyboard, the virtual keyboard including a first area and a second area, The above Each of the second regions includes at least two keys, the first region corresponding to a left hand and the second region corresponding to a right hand; the gesture tracking sensor detects that a hand is in a target gesture; The display device further, in response to detecting the hand being in the target gesture, based on a biometric characteristic of the target gesture: No. displaying a cursor at a first position, the first position being within an area of ​​the virtual keyboard corresponding to the biometric feature and corresponding to the biometric feature; The gesture tracking sensor further detects that the hand is moving while holding the target gesture; the display device further displays, in response to the hand moving while holding the target gesture, that the cursor moves in accordance with the movement of the hand; The electronic device performs a function of a key corresponding to a second position of the virtual keyboard in response to the change in the hand gesture, the second position being a position where the cursor is located when the hand gesture is changed.

[0008] In one aspect, a key function execution device is provided, the device including: a display module; and an execution module; The display module displays a virtual keyboard, the virtual keyboard including a first area and a second area, The above Each of the second regions includes at least two keys, the first region corresponding to a left hand and the second region corresponding to a right hand; The display module further, in response to detecting the hand being in a target gesture, based on a biometric characteristic of the target gesture: No. displaying a cursor at a first position, the first position being within an area of ​​the virtual keyboard corresponding to the biometric feature and corresponding to the biometric feature; The display module further displays, in response to the hand moving while holding the target gesture, that the cursor moves in accordance with the movement of the hand; The execution module executes a function of a key corresponding to a second position of the virtual keyboard in response to the change in the hand gesture, the second position being a position where the cursor is located when the hand gesture changes.

[0009] In some embodiments, the display module comprises: displaying a cursor at a target location within the first region of the virtual keyboard in response to the biometric characteristic indicating that the hand is a left hand; In response to the biometric characteristic indicating that the hand is a right hand, a cursor is displayed at a target location within the second area of ​​the virtual keyboard.

[0010] In some embodiments, the display module comprises: determining, in response to the biometric feature indicating that the hand is a left hand and that the target gesture is formed with a first finger and thumb of the hand, to display a cursor at a target location within a sub-area of ​​the first region of the virtual keyboard corresponding to the first finger; In response to the biometric feature indicating that the hand is a right hand and that the target gesture is formed with a first finger and thumb of the hand, it is determined to display a cursor at a target location within a sub-area of ​​the second area of ​​the virtual keyboard corresponding to the first finger.

[0011] In some embodiments, the target location is a center location of the area.

[0012] In some embodiments, the display module comprises: acquiring a displacement of the hand in response to the hand moving while holding the target gesture; determining a third position based on the first position and the displacement of the hand; The cursor is displayed moving from the first position to the third position.

[0013] In some embodiments, the execution module: inputting, in response to the key corresponding to the second position being a character key, a character indicated by the key corresponding to the second position into an input box or a character display area of ​​the virtual keyboard; deleting a last character in an input box or a character display area of ​​the virtual keyboard in response to the key corresponding to the second position being a delete key; In response to the key corresponding to the second position being a line feed key, displaying an input cursor in an input box of the virtual keyboard to feed to the next line; and in response to the key corresponding to the second position being a confirmation key, canceling display of the virtual keyboard with the input content in the input box of the virtual keyboard as the target content.

[0014] In some embodiments, the display module further displays, in response to a presence of a candidate Chinese character corresponding to the input character combination in a character display area of ​​the virtual keyboard, a candidate Chinese character corresponding to the input character combination within a target range of the character display area.

[0015] In some embodiments, the virtual keyboard further comprises at least one virtual control; The display module further displays, in response to detecting that a hand is not in the target gesture and that the hand is moving, that the cursor moves to track the movement of the hand; The execution module further executes a function corresponding to the target virtual control in response to detecting that the hand makes a target gesture and the cursor is located on a target virtual control of the at least one virtual control.

[0016] In some embodiments, the execution module: in response to the target virtual control being an uppercase input / lowercase input toggle control, toggle characters displayed in the virtual keyboard between uppercase input characters and lowercase input characters; in response to the target virtual control being a symbol input / alphabetic input switch control, switching characters displayed in the virtual keyboard between alphabetic and symbolic; In response to the target virtual control being a switching control between Chinese input and English input, a character input mode of the virtual keyboard is switched between Chinese input and English input.

[0017] In some embodiments, the execution module further updates a display content of the target virtual control, and the updated display content corresponds to a character switch or a character input mode switch in the virtual keyboard. 。

[0018] In one aspect, an electronic device is provided, comprising one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program, when loaded and executed by the one or more processors, realizes various optional realizations of the key function execution method.

[0019] In one aspect, a computer-readable storage medium is provided having stored thereon at least one computer program which, when loaded and executed by a processor, provides various optional realizations of the key function execution method described above.

[0020] In another aspect, a computer program product or computer program product is provided, the computer program product or computer program product including one or more program codes stored in a computer readable storage medium, the computer readable storage medium being read by one or more processors of an electronic device, the computer readable storage medium being read by one or more processors ... Effect of the Invention

[0021] The embodiment of the present application provides a flexible and simple operation method for a scenario in which a key on a virtual keyboard is triggered by combining two operations, gesture and movement. In this method, a user can trigger the display of a cursor by simply making a target gesture with his / her hand, and then control the movement of the cursor by simply moving his / her hand while holding the target gesture. Such an operation is very simple and convenient, and the movement of the cursor can also be controlled to precisely select and trigger one key. This improves the granularity and accuracy of triggering a key, and makes it possible to realize character input in any form, so that the applicability is good. In addition, when a target gesture is detected, the display position of the cursor can also be determined based on the biometric characteristics of the target gesture, so that a user can flexibly perform gesture operations using different hands, reduce the cursor movement distance as much as possible, reduce the complexity of the operation, and improve the operation efficiency.

[0022] In order to more clearly describe the configuration of the embodiments of the present application, the following briefly introduces drawings necessary for the description of the embodiments. Obviously, the drawings in the following description only show some embodiments of the present application, and those skilled in the art can obtain other drawings from these drawings without creative labor. [Brief description of the drawings]

[0023] [Figure 1] FIG. 2 is a schematic diagram of the implementation environment of the key function execution method provided in an embodiment of the present application; [Diagram 2] 1 is a schematic diagram of an implementation environment of another key function execution method provided in an embodiment of the present application; [Diagram 3] 1 is a schematic diagram of an implementation environment of another key function execution method provided in an embodiment of the present application; [Figure 4] 1 is a schematic diagram of an implementation environment of another key function execution method provided in an embodiment of the present application; [Diagram 5] 1 is a schematic diagram of an implementation environment of another key function execution method provided in an embodiment of the present application; [Figure 6]4 is a flowchart of a key function executing method provided in an embodiment of the present application; [Figure 7] 4 is a flowchart of a key function executing method provided in an embodiment of the present application; [Figure 8] FIG. 2 is a schematic diagram of a terminal interface provided in an embodiment of the present application. [Figure 9] FIG. 2 is a schematic diagram of a terminal interface provided in an embodiment of the present application. [Figure 10] FIG. 2 is a schematic diagram of a terminal interface provided in an embodiment of the present application. [Figure 11] FIG. 2 is a schematic diagram of a terminal interface provided in an embodiment of the present application. [Figure 12] FIG. 2 is a schematic diagram of a terminal interface provided in an embodiment of the present application. [Figure 13] FIG. 2 is a schematic diagram of a terminal interface provided in an embodiment of the present application. [Figure 14] FIG. 2 is a schematic diagram of a terminal interface provided in an embodiment of the present application. [Figure 15] FIG. 2 is a schematic diagram of a terminal interface provided in an embodiment of the present application. [Figure 16] FIG. 2 is a schematic diagram of a terminal interface provided in an embodiment of the present application. [Figure 17] FIG. 2 is a schematic diagram of a terminal interface provided in an embodiment of the present application. [Figure 18] FIG. 2 is a schematic diagram of a key function execution system provided in an embodiment of the present application. [Figure 19] FIG. 2 is a schematic diagram of the configuration of a key function execution device provided in an embodiment of the present application; [Figure 20] FIG. 1 is a schematic diagram of a configuration of an electronic device provided in an embodiment of the present application. [Figure 21] FIG. 2 is a block diagram of a configuration of a terminal provided in an embodiment of the present application. [Figure 22] FIG. 2 is a schematic diagram of a server configuration provided in an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] In order to more clearly illustrate the object, configuration, and advantages of the present application, the embodiments of the present application will be described in more detail below with reference to the drawings.

[0025] The terms "first", "second", and the like in this application are used to distinguish between identical or similar items having substantially the same roles and functions. It should be understood that there is no logical or chronological dependency between "first", "second", and "nth", and the quantity and execution order are not limited. It should also be understood that in the following description, the terms "first", "second", and the like are used to describe various elements, but these elements should not be limited by the terms. These terms are merely used to distinguish one element from another element. For example, the first image is called the second image, and similarly, the second image is called the first image, without departing from the scope of various examples. The first image and the second image are both images, and in some cases, are separate and different images.

[0026] As used herein, the term "at least one" means one or more, and as used herein, the term "plurality" means more than one, e.g., a plurality of keys means two or more keys.

[0027] The implementation environment of the present application will be described below.

[0028] Figures 1, 2, 3 and 4 are schematic diagrams of an implementation environment of a key function execution method provided in an embodiment of the present application. As shown in Figure 1, the implementation environment includes a terminal 101. As shown in Figure 2, the implementation environment includes a terminal 101 and a gesture tracking sensor 102. As shown in Figure 3, the implementation environment includes a terminal 101, a gesture tracking sensor 102 and a display device 103. As shown in Figure 4, the implementation environment includes a terminal 101 and a display device 103.

[0029] Here, the terminal 101 is at least one of a smartphone, a game machine, a desktop computer, a tablet computer, an e-book reader, a Moving Picture Experts Group Audio Layer III (MP3) player or a Moving Picture Experts Group Audio Layer IV (MP4) player, and a laptop portable computer. An application that supports the execution of a key function is installed and executed on the terminal 101. For example, the application is a system application, a virtual reality (VR) application, an instant messenger application, a news push application, a shopping application, an online video application, and a social application.

[0030] As shown in Fig. 1, the implementation environment includes a terminal 101, which is equipped with a gesture tracking sensor and a screen display. In this way, the terminal can track a user's hand with the gesture tracking sensor, recognize a gesture or movement by the hand, and trigger an instruction according to the gesture or movement. The terminal also has a display function, and can display a virtual keyboard by the screen display, and when recognizing a gesture or movement of the hand, can display a change according to the gesture or movement by the screen display. For example, the gesture tracking sensor is a camera, and the camera can capture a hand to collect the gesture or movement of the hand.

[0031] As shown in Fig. 2, the implementation environment includes a terminal 101 and a gesture tracking sensor 102. The terminal 101 and the gesture tracking sensor 102 are connected via a wired or wireless network, and the terminal 101 has a display function and can display a virtual keyboard on a screen display. The gesture tracking sensor 102 can track a user's hand, recognize gestures and actions made by the hand, and transmit the recognition result to the terminal 101. The terminal 101 can display content according to the recognition result on the virtual keyboard.

[0032] The gesture tracking sensor may be any gesture sensor, such as a Leap Motion sensor, or may be, for example, a finger tracking glove, but the embodiments of the present application are not limited thereto.

[0033] As shown in Fig. 3, the implementation environment includes a terminal 101, a gesture tracking sensor 102, and a display device 103. The terminal 101 is connected to each of the gesture tracking sensor 102 and the display device 103 via a wired or wireless network. The terminal 101 has a data processing function. The terminal 101 controls the display device 103 to display a virtual keyboard. The gesture tracking sensor 102 can track a user's hand, recognize gestures and actions made by the hand, and transmit the recognition result to the terminal 101. The terminal 101 can control the display device 103 to display content according to the recognition result on the virtual keyboard.

[0034] The display device may be any display device, such as a screen display, or may be a VR head-mounted display device (abbreviated as VR head-mounted display), but the embodiments of the present application are not limited thereto.

[0035] As shown in FIG. 4, the implementation environment includes a terminal 101 and a display device 103. The terminal 101 and the display device 103 are connected via a wired or wireless network. The terminal 101 has a data processing function. The terminal 101 controls the display device 103 to display a virtual keyboard. A gesture tracking sensor is attached to the terminal 101. The terminal 101 can track the user's hand by using the attached gesture tracking sensor, and can recognize gestures and actions made by the hand. Next, the terminal 101 determines a change in display content based on the gesture or action, and displays it on the display device 103.

[0036] Although four possible implementation environments are provided above, the embodiments of the present application do not limit which one is specifically adopted. Hereinafter, the implementation environment of the key function execution method will be comprehensively described with reference to FIG. 5. As shown in FIG. 5, the implementation environment includes a gesture tracking sensor 501, a device system 502, and a display device 503. The gesture tracking sensor 501 and the device system 502 are connected.

[0037] The gesture tracking sensor 501 detects and tracks the user's finger operations. Here, the gesture tracking sensor 501 can be a different type of device, such as a Leap Motion sensor, a camera, or a finger tracking glove. The gesture tracking sensor 501 can obtain rotation and coordinate information of each finger joint.

[0038] The device system 502 may execute a program, such as a gaming application or other application, that includes an input method program module within the program.

[0039] The display device 503 can display a screen. Here, the display device 503 is a virtual reality (VR) head-mounted display, a screen display, or the like.

[0040] 6 is a flow chart of a key function execution method provided in an embodiment of the present application. The method is applied to an electronic device. The electronic device is a terminal or a server. Referring to FIG. 6, taking the method applied to a terminal as an example, the method includes the following steps:

[0041] In 601, the terminal displays a virtual keyboard, the virtual keyboard including a first area and a second area, each of the first area and the second area including at least two keys, the first area and the second area corresponding to a left hand and a right hand respectively, i.e., the first area corresponds to the left hand and the second area corresponds to the right hand.

[0042] A virtual keyboard is a virtual keyboard that is displayed on a screen. The virtual keyboard includes at least two keys, where at least two means more than one, i.e., multiple.

[0043] The keys of the virtual keyboard may include various types of keys, for example, the virtual keyboard may display multiple alphabetic keys, numeric keys, or symbol keys.

[0044] In the embodiment of the present application, the virtual keyboard is divided into two areas, a first area and a second area. The first area corresponds to the left hand, and the second area corresponds to the right hand. Each area includes at least two keys, and the operation of any one of the keys can trigger the function of that key. In this way, a user can make target gestures with different hands to control the triggering of keys in different areas of the virtual keyboard.

[0045] At 602, in response to detecting that the hand is in a target gesture, the terminal displays a cursor at a first location based on a biometric feature of the target gesture, the first location being within an area of ​​the virtual keyboard corresponding to the biometric feature and corresponding to the biometric feature.

[0046] In the embodiment of the present application, the user makes a gesture with his / her hand without touching the screen, and the terminal detects the hand gesture, determines the corresponding operation position in the virtual keyboard based on the gesture, and further determines which key the user really wants to trigger, and performs the function of that key.

[0047] For said target gesture, optionally a virtual keyboard is associated with a biometric feature of a hand, and for different biometric features, the location in the virtual keyboard corresponding to said target gesture may be different, where the biometric feature is used to indicate which hand is making the gesture, or also which fingers are making the gesture, respectively.

[0048] In one possible embodiment, when the terminal detects a hand gesture and determines that the hand is in a target gesture, it determines a first position in the area corresponding to the biometric feature according to the biometric feature of the target gesture, the first position being the initial display position of the cursor. The terminal displays the cursor at the first position to indicate that the operating position corresponding to the position where the hand is currently located is the first position. In this way, the hand can only make a target gesture to trigger the terminal to display the cursor at the first position without touching the screen.

[0049] As can be seen, the virtual keyboard is divided into two regions, and when a user makes a target gesture with his / her left hand, the region corresponding to the biometric feature is the first region, and when a user makes a target gesture with his / her right hand, the region corresponding to the biometric feature is the second region.

[0050] At 603, in response to the hand moving while holding the target gesture, the terminal displays that the cursor moves to follow the movement of the hand.

[0051] The embodiment of the present application combines two operations, gesture and movement, and when the user does not actually trigger the key at the first position after making the target gesture, the user moves the hand while holding the target gesture. In this way, the terminal can track the hand and control the cursor to move following the movement of the hand. This allows the user to easily move the cursor and accurately select the key to be triggered simply by moving the hand. Accurate triggering of the key can be achieved with a simple operation, improving the accuracy and flexibility of the execution of the key function.

[0052] At 604, the terminal, in response to the change in the hand gesture, executes a function of a key corresponding to a second position of the virtual keyboard, the second position being the position where the cursor was located when the hand gesture changed.

[0053] By the above step 603, the position of the cursor has already changed, and may have moved to another position, or may have moved back to the first position. After the user moves the cursor to the position of the key they want to trigger, they may change the gesture, i.e., not hold the target gesture. In this way, the terminal can determine that the user wants to trigger the key at the current position of the cursor, and thus can execute the function of the key at the current position of the cursor (i.e., the second position).

[0054] Here, the second position is a position where the cursor is located when the hand moves while holding the target gesture and then no longer holds the target gesture. The second position is the same as the first position, or the second position is a position different from the first position.

[0055] The embodiment of the present application provides a flexible and simple operation method for a scenario in which a key on a virtual keyboard is triggered by combining two operations, gesture and movement. In this method, a user can trigger the display of a cursor by simply making a target gesture with his / her hand, and then control the movement of the cursor by simply moving his / her hand while holding the target gesture. Such an operation is very simple and convenient, and the movement of the cursor can also be controlled to precisely select and trigger one key. This improves the granularity and accuracy of triggering a key, and makes it possible to realize character input in any form, so that the applicability is good. In addition, when a target gesture is detected, the display position of the cursor can also be determined based on the biometric characteristics of the target gesture, so that a user can flexibly perform gesture operations using different hands, reduce the cursor movement distance as much as possible, reduce the complexity of the operation, and improve the operation efficiency.

[0056] 7 is a flowchart of a key function execution method provided in an embodiment of the present application. Referring to FIG. 7, the method includes the following steps:

[0057] In 701, a terminal displays a virtual keyboard, the virtual keyboard including a first area and a second area, each of the first area and the second area including at least two keys, the first area corresponding to a left hand, and the second area corresponding to a right hand.

[0058] Optionally, the virtual keyboard displays a number of alphabetic, numeric or symbolic keys.

[0059] For example, the virtual keyboard is as shown in Figure 8. The virtual keyboard includes a number of alphabet keys, such as Q, A, Z, and W, which will not be listed here. The virtual keyboard also includes symbol keys, such as "empty" representing a space, and "," and ".", which are all symbol keys. The virtual keyboard also includes function keys, such as "←" which is the delete key.

[0060] In some embodiments, the display of the virtual keyboard is triggered by a trigger operation on the input box. For example, the terminal displays a first page, and an input box is displayed on the first page. The user controls the cursor to move into the input box following the movement of the hand by moving the hand, and then performs a pressing or tapping action with the hand. When the pressing or tapping action of the hand is detected and the cursor is located in the input box, the terminal displays the virtual keyboard on the first page.

[0061] In some other embodiments, the display of the virtual keyboard is triggered by a trigger operation on the virtual keyboard control. For example, the terminal displays a first page, and the virtual keyboard control is displayed on the first page. The user controls the cursor to move to the position of the virtual keyboard control by moving his / her hand, and then performs a pressing or tapping action with his / her hand. When the pressing or tapping action of the hand is detected and the cursor is located at the position of the virtual keyboard control, the terminal displays the virtual keyboard on the first page.

[0062] At 702, in response to detecting that the hand is in a target gesture, the terminal displays a cursor at a first location based on a biometric feature of the target gesture, the first location being within an area of ​​the virtual keyboard corresponding to the biometric feature and corresponding to the biometric feature.

[0063] In an embodiment of the present application, a user can select a key in the virtual keyboard and trigger the execution of the key's function by making a target gesture or making a target gesture and moving a hand without touching the display.

[0064] In some embodiments, the terminal collects the user's hand gestures, movements or positions by means of a gesture tracking sensor.

[0065] Optionally, the gesture tracking sensor is a sensor attached to the terminal, for example, a camera attached to the terminal. In response, the terminal takes a picture of the user with a camera, and then analyzes the captured image in the manner of computer vision detection to determine the gesture, movement, or position of the user's hand. For example, when performing computer vision detection, the hand is detected in the image to determine the position where the hand is located, and then the shape of each finger is determined, and further matched with the shape of each finger of the candidate gesture to determine the current gesture of the hand.

[0066] Optionally, the gesture tracking sensor is an external gesture tracking sensor, for example, the gesture tracking sensor is a Leap Motion sensor, a finger tracking glove, or a camera device, where the finger tracking glove can collect the rotation, position, etc. of each phalange of the hand, and further analyze the shape of each finger according to the rotation, position, etc. of each phalange, and further match the shape of each finger of the candidate gesture to obtain the current hand gesture.

[0067] In some embodiments, the target gesture is a pinch gesture, i.e., a gesture in which the thumb of one hand is brought together with the pads of the fingers of another hand. The terminal collects an image of the hand, or Finger Tracking After acquiring the shape parameters of the hand with the glove, the candidate gesture with the highest matching score is determined as the target gesture by gesture recognition, and the cursor display step is performed based on the determined target gesture. Of course, the target gesture may be other gestures. For example, gesture is a gesture of putting five fingers together, or a gesture of making a fist, etc. The target gesture may be set by relevant technical personnel as needed, or may be set by a user according to his or her own operating habits, but the embodiment of the present application is not limited thereto.

[0068] Gesture recognition includes three parts: gesture segmentation, gesture analysis, and recognition. teeth , gesture segmentation using monocular vision, and gesture segmentation using stereo vision.

[0069] In the case of monocular vision, an image acquisition device collects images of the gesture, creates a planar model of the gesture, and matches the planar model with a gesture in a gesture shape database to determine the type of the gesture. The method of creating a gesture shape database makes it easy to match gesture templates by creating a database of all possible gestures, but it is computationally expensive and does not help the system recognize it quickly.

[0070] In the case of stereo vision, different images of a gesture are captured by multiple image acquisition devices, and the different images of the same gesture are converted into a stereo model of the gesture. The stereo matching method is similar to the template matching method of monocular vision, and similarly requires creating a large number of gesture libraries and performing 3D reconstruction of each gesture in the gesture library. 3D reconstruction requires creating a 3D model of the gesture, which increases the amount of calculation, but has a good segmentation effect.

[0071] Gesture analysis is one of the key techniques for realizing gesture recognition. Gesture analysis can obtain the shape features or movement trajectory of the gesture. The shape features and movement trajectory of the gesture are important features in dynamic gesture recognition and are directly related to the meaning expressed by the gesture. Gesture analysis methods include edge contour extraction method, multiple feature combination method such as mass center finger, and finger joint tracking method. For example, edge contour extraction method is a gesture analysis method in which the shape of the hand is distinguished from other objects by its unique shape. Alternatively, a gesture recognition algorithm that combines geometric moments and edge detection is adopted, and the weights of the two features are set to calculate the distance between the images to realize the recognition of alphabet gestures. The multiple feature combination method analyzes the posture or trajectory of the gesture based on the physical characteristics of the hand. Meenakshi Panwar combines the shape of the gesture with the features of the fingertips to realize the recognition of the gesture. The finger joint tracking method mainly builds a two-dimensional or three-dimensional model of the hand and tracks it based on the change in the position of the joint points of the hand, and is mainly applied to dynamic trajectory tracking.

[0072] Gesture recognition is a process of classifying trajectories in a model parameter space into a certain subset in the space, and includes static gesture recognition and dynamic gesture recognition, which can be finally transformed into static gesture recognition. From the technical realization of gesture recognition, gesture recognition methods mainly include template matching method, 、 There are neural network methods and hidden Markov model methods.

[0073] Template matching methods consider a gesture motion as a sequence of static gesture images and recognize gestures by comparing the gesture sequence to be recognized with a sequence of known gesture templates.

[0074] Hidden Markov Model ( HMM (Hidden Markov Model) is a statistical model ModelThe system modeled in has a dual random process including the random processes of state transition and observation output, in which the random process of state transition is hidden and represented by the random process of observation sequence.

[0075] For said target gesture, optionally a virtual keyboard is associated with a biometric feature of a hand, and for different biometric features, the location in the virtual keyboard corresponding to said target gesture is different, where said biometric feature is used to indicate which hand is making the gesture, or also which finger is making the gesture, respectively.

[0076] In some embodiments, in step 702, the terminal first determines a first position and then displays a cursor at the first position. Optionally, in response to detecting that the hand is in a target gesture, the terminal determines the first position within the virtual keyboard based on a biometric feature of the target gesture, in a region corresponding to the biometric feature, and then displays a cursor at the first position of the virtual keyboard.

[0077] When a user uses a different hand, the biometric feature may be different and the corresponding area may be different. For example, a first area corresponds to a left hand and a second area corresponds to a right hand. In step 702, the terminal displays a cursor at a target location in the first area of ​​the virtual keyboard in response to the biometric feature indicating that the hand is a left hand, and displays a cursor at a target location in the second area of ​​the virtual keyboard in response to the biometric feature indicating that the hand is a right hand.

[0078] That is, in response to the biometric feature indicating that the hand is a left hand, the terminal determines a target location within the first region of the virtual keyboard as a first location and then displays a cursor at the first location, and in response to the biometric feature indicating that the hand is a right hand, the terminal determines a target location within the second region of the virtual keyboard as a first location and then displays a cursor at the first location.

[0079] For example, as shown in Fig. 9, when the virtual keyboard is divided equally, a left region 901 and a right region 902 are obtained, where the left region 901 is the first region and the right region 902 is the second region. Thus, when a user makes a target gesture with the left hand, the corresponding first position is in the left region 901. When a user makes a target gesture with the right hand, the corresponding first position is in the right region 902.

[0080] The division allows the user to distinguish between the left and right hands and thus to distinguish between small areas in the virtual keyboard. The user can choose to use the left hand or the right hand depending on the position where the user wants to trigger the key. In this way, the user can make more precise selections within the small area and reduce the range of further selections. Compared with the method of not dividing the virtual keyboard, this method can reduce the user's cursor adjustment operation as much as possible and improve operation efficiency.

[0081] In some embodiments, the first and second regions are further divided, with the first region corresponding to the left hand and the second region corresponding to the right hand. The first and second regions are each divided into a number of sub-regions, with different fingers of the hand corresponding to different sub-regions. Since the target gesture is formed with one finger and the thumb of the hand, the finger other than the thumb corresponds to one sub-region. The user can select different finger By forming a target gesture with the thumb, the cursor can be controlled to appear in different sub-regions.

[0082] Accordingly, the step7 In 02, the terminal determines, in response to the biometric feature indicating that the hand is a left hand and that the target gesture is formed with a first finger and thumb of the hand, to display a cursor at a target position within a sub-area of ​​the first area of ​​the virtual keyboard corresponding to the first finger.

[0083] In response to the biometric feature indicating that the hand is a right hand and that the target gesture is formed with a first finger and thumb of the hand, the terminal determines to display a cursor at a target position within a sub-area of ​​the second region of the virtual keyboard corresponding to the first finger.

[0084] In one example, the first region is divided into two sub-regions, and the second region is also divided into two sub-regions. The first sub-region of the first region corresponds to the second finger of the left hand, and the second sub-region of the first region corresponds to the third finger of the left hand. The third sub-region of the second region corresponds to the fourth finger of the right hand, and the fourth sub-region of the second region corresponds to the fifth finger of the right hand. As can be understood, the hand includes four fingers, including the index finger, middle finger, ring finger, and little finger, in addition to the thumb. Which of the four fingers the second finger and the third finger are, respectively, is set by a related artisan as necessary, or is set by a user according to operating habits. The fourth finger and the fifth finger are similar. The embodiment of the present application is not limited thereto.

[0085] For example, as shown in FIG. 10, the left hand is positioned on the left half of the keyboard (first area 100 2), and the right hand is used for input to the right half keyboard (second area 1001). Take for example a case where the target gesture is a pinch gesture, the second finger is the middle finger of the left hand, the third finger is the index finger of the left hand, the fourth finger is the index finger of the right hand, and the fifth finger is the middle finger of the right hand. When the user pinches with the index finger and thumb of the left hand, the cursor appears in the second sub-area 1003 of the first area 1001 and controls triggers to the E, R, T, D, F, G, C, V, and B keys. When the user pinches with the middle finger and thumb of the left hand, the cursor appears in the first sub-area 1004 of the first area 1001 and controls triggers to the Q, W, A, S, Z, and X keys.

[0086] In some embodiments, the target position is the center position of the region. In this way, by displaying the cursor at the center position, it is possible to ensure that the total distance to other keys is minimized when the user subsequently wants to move the cursor to select a key. This allows the user to greatly reduce the distance the cursor needs to move, making the operation more convenient and efficient.

[0087] 10 as an example, when a user pinches with the index finger and thumb of his left hand, the cursor appears on the F key at the center position in the second sub-region 1003 of the first region 1001. When a user pinches with the middle finger and thumb of his left hand, the cursor appears between the A key and the S key at the center position in the first sub-region 1004 of the first region 1001.

[0088] In another example, the first region is divided into three sub-regions, and the second region is also divided into three sub-regions. The first sub-region of the first region corresponds to the second finger of the left hand, and the second sub-region of the first region corresponds to the third finger of the left hand. The third sub-region of the first region corresponds to the fourth finger of the right hand. The fourth sub-region of the second region corresponds to the fifth finger of the right hand, the fifth sub-region of the second region corresponds to the sixth finger of the right hand, and the sixth sub-region of the second region corresponds to the seventh finger of the right hand. Which of the four fingers the second finger, the third finger, and the fourth finger are, is set by a related engineer as necessary, or is set by a user according to operating habits. The fifth finger, the sixth finger, and the seventh finger are similar. The embodiment of the present application is not limited thereto.

[0089] For example, as shown in FIG. 11, the left hand is used for input to the left half keyboard (first area 1101), and the right hand is used for input to the right half keyboard (second area 1102). Take as an example a case where the target gesture is a pinch gesture, the second finger, the third finger, and the fourth finger are the ring finger, the middle finger, and the index finger of the left hand, respectively, and the fifth finger, the sixth finger, and the seventh finger are the index finger, the middle finger, and the ring finger of the right hand, respectively. When the user pinches with the index finger and thumb of the left hand, the cursor appears in the third sub-area 1103 of the first area 1101 and controls triggers to the R, T, F, G, V, and B keys. When the user pinches with the middle finger and thumb of the left hand, the cursor appears in the second sub-area 1104 of the first area 1101 and controls triggers to the W, E, S, D, X, and C keys. When the user pinches with the ring finger and thumb of the left hand, the cursor appears in the first sub-area 1105 of the first area 1101 and displays the Q, A, Z key The right hand controls the trigger to the left. The right hand is similar, but we will not explain it here.

[0090] In some embodiments, the target position is the center position of the region. Taking the virtual keyboard shown in Fig. 11 as an example, when a user pinches with the index finger and thumb of the left hand, the cursor appears between the F key and the G key at the center position in the third sub-region 1103 of the first region 1101.

[0091] In another example, the first region is divided into four sub-regions and the second region is also divided into four sub-regions. The first sub-region of the first region corresponds to the second finger of the left hand, the second sub-region of the first region corresponds to the third finger of the left hand, and the third sub-region of the first region corresponds to the third finger of the left hand. left The fourth sub-region of the first region corresponds to the fourth finger of the left hand, and the fourth sub-region of the first region corresponds to the fifth finger of the left hand. The fifth sub-region of the second region corresponds to the sixth finger of the right hand, the sixth sub-region of the second region corresponds to the seventh finger of the right hand, the seventh sub-region of the second region corresponds to the eighth finger of the right hand, and the eighth sub-region of the second region corresponds to the ninth finger of the right hand. Which of the four fingers the second finger, third finger, fourth finger, and fifth finger are respectively is set by relevant technical personnel as necessary, or is set by the user according to the operating habits. The sixth finger, seventh finger, eighth finger, and ninth finger are similar. The embodiment of the present application is not limited thereto.

[0092] For example, as shown in FIG. 12, the left hand is used for input to the left half of the keyboard (first area 1201), and the right hand is used for input to the right half of the keyboard (second area 1202). Take the example where the target gesture is a pinch gesture, the second finger, the third finger, the fourth finger, and the fifth finger are the little finger, the ring finger, the middle finger, and the index finger of the left hand, respectively, and the sixth finger, the seventh finger, the eighth finger, and the ninth finger are the index finger, the middle finger, the ring finger, and the little finger of the right hand, respectively. When the user pinches with the index finger and thumb of the left hand, the cursor appears in the fourth sub-area 1203 of the first area 1201 and controls triggers to the R, T, F, G, V, and B keys. When the user pinches with the middle finger and thumb of the left hand, the cursor appears in the third sub-area 1204 of the first area 1201 and controls triggers to the E, D, C, D, E, E, F, F, G, V, and B keys. key When the user pinches with the ring finger and thumb of the left hand, a cursor appears in a second sub-area 1205 of the first area 1201 and controls triggers for the W, S, and X keys. When the user pinches with the little finger and thumb of the left hand, a cursor appears in a first sub-area 1206 of the first area 1201 and controls triggers for the Q, A, Z It controls the triggers to the keys. The right hand is similar, but we will not explain it here.

[0093] In some embodiments, the target position is the center position of the region. Taking the virtual keyboard shown in FIG. 12 as an example, when a user pinches with the index finger and thumb of the left hand, the cursor appears between the F key and the G key at the center position in the fourth sub-region 1203 of the first region 1201. When a user pinches with the middle finger and thumb of the left hand, the cursor appears on the D key at the center position in the third sub-region 1204 of the first region 1201.

[0094] Illustratively, when the terminal detects that the hand is in a target gesture, the terminal may determine a biometric feature of the target gesture, and if the biometric feature indicates that the target gesture is made with a first finger and thumb of the hand, and if there is no first position corresponding to the first finger, ignore the current operation.

[0095] At 703, the terminal acquires a displacement of the hand in response to the hand moving while holding the target gesture.

[0096] The first position is an initial display position of the cursor, which may be on a key or between keys. The user can control the movement of the cursor by moving his / her hand.

[0097] When the hand moves while holding the target gesture, the terminal acquires the displacement of the hand. The process of acquiring the displacement of the hand may adopt the same steps as the gesture recognition described above. The terminal collects two images of the hand and recognizes the position of the hand, thereby determining the change in the position of the hand in the two images, i.e., the displacement.

[0098] Alternatively, the features such as rotation determined by the gesture tracking sensor are used to describe the shape of the fingers and are also used to determine the position of each phalanx of the hand (which position can be expressed in terms of coordinates).The terminal may then continue to obtain the position of each phalanx of the hand as the hand moves and compare it with the positions obtained in step 702 above to determine the displacement of the hand.

[0099] At 704, the terminal determines a third position based on the first position and the displacement of the hand.

[0100] The initial display position (first position) of the cursor is known, and the terminal determines the displacement of the cursor from the position to which the hand has moved, and further determines the third position, i.e., the position to which the cursor should move, from the first position and the displacement of the cursor.

[0101] The terminal may determine the cursor displacement from the hand displacement in various implementations. In some embodiments, the terminal determines the cursor displacement based on the hand displacement. In other embodiments, the terminal determines the cursor displacement based on the hand displacement and sensitivity.

[0102] The sensitivity can be set by relevant engineers as needed, or by users according to their own operating habits, but the embodiments of the present application are not limited thereto.

[0103] At 705, the terminal displays the cursor moving from the first position to the third position.

[0104] When the terminal determines the third position, the terminal can control the cursor to move from the first position to the third position, thereby expressing the effect of the cursor moving in accordance with the movement of the hand.

[0105] The above steps 703 to 705 are one possible implementation form of displaying the cursor moving to follow the movement of the hand in response to the hand moving while holding the target gesture. Although the above description is given taking the case where the cursor position is determined from the hand displacement to control the cursor movement as an example, the process may be implemented in other ways. For example, the terminal directly obtains the cursor displacement without determining the third position to which the cursor moves, and controls the cursor movement according to the cursor displacement. The embodiment of the present application is not limited thereto.

[0106] In one example, it is assumed that the finger displacement and the cursor displacement increase or decrease equally, i.e., the sensitivity is 1. Assume that the first position is represented by InitCursorPos, the third position is represented by CursorPos, and when a hand is detected moving while holding a target gesture, the hand displacement is obtained (the hand displacement is represented by TipMovement), and the first position, the hand displacement, and the third position satisfy the relationship CursorPos=InitCursorPos+TipMovement.

[0107] At 706, the terminal, in response to the change in the hand gesture, executes a function of a key corresponding to a second position of the virtual keyboard, the second position being the position where the cursor was located when the hand gesture changed.

[0108] When the user moves the cursor to a desired location, such as the second location, the target gesture does not need to be held, and the terminal can execute the function of the key in the second location when it detects the change in the hand gesture.

[0109] The virtual keyboard includes various types of keys, such as character keys, function keys, etc. If the key corresponding to the second position belongs to a different type, the function performed by the terminal is different. Accordingly, step 706 includes the following cases:

[0110] Case 1: In response to the key corresponding to the second position being a character key, the character indicated by the key corresponding to the second position is input into an input box or a character display area of ​​the virtual keyboard.

[0111] In case 1, if the terminal is currently in an English input mode, or if the terminal is currently in a Chinese input mode but the character key is not an alphabet key, the terminal may input the character represented by the key corresponding to the second position into the input box of the virtual keyboard. If the terminal is currently in a Chinese input mode and the character key is an alphabet key, the terminal may input the character represented by the key corresponding to the second position into the character display area.

[0112] Case 2: In response to the key corresponding to the second position being a delete key, delete the last character in an input box or a character display area of ​​the virtual keyboard.

[0113] In case 2, the delete key is a function key, and does not input characters but deletes input characters. If the current input mode is the English input mode or the Chinese input mode and there is no input character in the character display area, the terminal may delete the last character in the input box of the virtual keyboard. If the current input mode is the Chinese input mode and there is an input character in the character display area, the terminal may delete the last character in the character display area.

[0114] In case 3, in response to the key corresponding to the second position being a line feed key, an input cursor in an input box of the virtual keyboard indicates that a line feed will be performed on the next line.

[0115] In case 4, in response to the key corresponding to the second position being a confirmation key, the input content in the input box of the virtual keyboard is taken as the target content, and the display of the virtual keyboard is cancelled.

[0116] The above provides only some possible key types, but the virtual keyboard is not limited to only include these types. Those skilled in the art can arbitrarily set the keys in the virtual keyboard as needed. For example, the virtual keyboard also includes an escape key or a return key. In response to the key corresponding to the second position being the escape key or the return key, the terminal cancels the display of the virtual keyboard. For example, the virtual keyboard also includes a parameter adjustment key. In response to the key corresponding to the second position being the parameter adjustment key, the terminal adjusts a parameter of the currently displayed interface. The parameter includes at least one of a display parameter, a volume parameter, and a control parameter. The embodiment of the present application is not limited thereto.

[0117] Of course, there is another possible case: if the second position is not a key position, the terminal may ignore the operation.

[0118] In some embodiments, the virtual keyboard further includes at least one virtual control. In response to detecting that the hand is not in the target gesture and that the hand is moving, the terminal may display that the cursor moves to follow the movement of the hand, and then in response to detecting that the hand is performing a target action and that the cursor is located on a target virtual control among the at least one virtual control, execute a function corresponding to the target virtual control. Here, the process of displaying that the cursor moves to follow the movement of the hand is similar to the above steps 303 to 305, and therefore will not be described here.

[0119] Here, the target action may be set by a related artisan as needed, or may be set by a user according to operating habits. For example, the target action may be a pushing action. For example, the target action may be a tapping action. The embodiments of the present application are not limited thereto.

[0120] In some embodiments, the virtual controls include a switch control between uppercase and lowercase input, a switch control between symbol and alphabet input, and a switch control between Chinese and English input. These virtual controls are described below.

[0121] In response to the target virtual control being an uppercase / lowercase input switching control, the terminal switches characters displayed in the virtual keyboard between uppercase input characters and lowercase input characters.

[0122] In some embodiments, the upper case input may include two types: upper case input and fixed upper case input.

[0123] In some other embodiments, the terminal updates the display content of the target virtual control, and the updated display content is consistent with switching of characters or switching of character input modes in the virtual keyboard.

[0124] For example, as shown in FIG. 13, there are three types of uppercase / lowercase states: lowercase, uppercase, and fixed uppercase. Taking the case where the target action is a pressing action as an example, when the cursor is on the uppercase / lowercase switching key 1301, the user presses it once to enter the next state, and pressing it in the third state to enter the first state, and so on. The alphabets on the keyboard are switched to uppercase or lowercase depending on the state, and the key displays text or an icon indicating the current state. FIG. 13 shows an example where the uppercase / lowercase switching key 1301 displays "uppercase" in the uppercase state. In the uppercase state, the user inputs one alphabet, and the keyboard automatically returns to the lowercase state. In the fixed uppercase state, the user can continuously input uppercase alphabets.

[0125] In some embodiments, the terminal, in response to the target virtual control being a control for switching between symbol input and alphabetic input, switches characters displayed in the virtual keyboard between alphabetic and symbolic.

[0126] For example, as shown in Fig. 14, when the cursor is located on the symbol input / alphabet input switching control 1401, one pressing action switches the symbol input / alphabet input switching control 1401 from the alphabet keys to the symbol keys. In response, the characters displayed on the virtual keyboard change from alphabets to symbols, as shown in Fig. 14. Another pressing action returns to the alphabet keys. In response, the characters displayed on the virtual keyboard change from symbols to alphabets.

[0127] In response to the target virtual control being a switching control between Chinese input and English input, the terminal switches a character input mode of the virtual keyboard between Chinese input and English input.

[0128] For example, as shown in FIG. 15, when the cursor is located on the switching control 1501 for Chinese input and English input, pressing the control switches between two modes, English and Chinese pinyin. Here, in the English input mode, the switching control 1501 displays "EN", and in the Chinese input mode, the switching control 1501 displays "pinyin". In the Chinese input mode, when the user inputs pinyin by the above-mentioned target gesture and movement, the pinyin appears on the switching control 1501. FIG. 15 shows an example in which the input pinyin "fa" is displayed. That is, the character display area is the area where the switching control 1501 is located. Of course, the character display area may be another area other than the area where the switching control 1501 is located.

[0129] In case of Chinese input, in response to the presence of a candidate Chinese character corresponding to the input character combination in the character display area of ​​the virtual keyboard, the terminal displays the candidate Chinese character corresponding to the input character combination within the target range of the character display area, where in a Chinese input mode, when the character display area is a pinyin display area and the characters input into the character display area are pinyin characters for representing the sounds of Chinese characters, the input character combination represents the pinyin of the candidate Chinese character.

[0130] For example, as shown in Fig. 15, when the character display area contains multiple characters and there are candidate kanji characters corresponding to the multiple characters, the candidate kanji characters may be displayed on the right. In Fig. 15, there are multiple candidate kanji characters corresponding to "fa", for example, [Table 1] etc. In the figure, [Table 2] For other candidate kanji characters, if the user performs a slide operation while the cursor is on the candidate kanji character, the displayed candidate kanji characters are scrolled and other candidate kanji characters are displayed.

[0131] For details of the above method, please refer to FIG. 16 and FIG. 17. Wait for a user's gesture operation at each detection period. If there is a gesture operation, determine whether or not the user's hand is detected. If the hand is detected, determine whether or not the user has already made a pinching gesture. If the user has already made a pinching gesture, it indicates that the user is currently holding the pinching gesture. Furthermore, determine whether or not the current pinching gesture has been released. If the pinching gesture has been released, trigger the function of the key where the cursor is located to release the cursor. There are various types of key functions, but the description will be omitted here. If the gesture has not been released, record the spatial coordinate NewTipPos of the pinching finger at this time, and determine the finger displacement TipMovement based on the spatial coordinate NewTipPos of the pinching finger and the previous spatial coordinate InitTipPos of the finger. Next, the cursor is moved, that is, the amount of finger movement (displacement) is added to the position, and a new position CursorPos of the cursor is determined so that the new position CursorPos of the cursor, InitCursorPos, and TipMovement satisfy the relationship of CursorPos=InitCursorPos+TipMovement. If it is determined that a pinch gesture has not been performed before, that is, if a pinch gesture is currently detected, it is determined whether or not it is the left hand. If it is the left hand, the operation is applied to the left part of the keyboard. If it is not the left hand, the operation is applied to the right part of the keyboard. Furthermore, it is determined which finger and thumb are performing the pinch gesture. Next, based on the determined finger, it is determined whether or not there is a corresponding keyboard operation area. If there is a corresponding keyboard operation area, the cursor is displayed at the center of the corresponding keyboard operation area. The position where this cursor is located is the initial cursor position InitCursorPos. Next, a finger identifier Fingerindex (the index finger through the pinky finger are numbers 1 through 4, respectively) may be recorded, and the spatial coordinate InitTipPos of that finger may be recorded.

[0132] In the Chinese input mode, when the input of one alphabet is confirmed by the pinch gesture, an alphabet may be added to the end of the input pinyin. Then, it is determined whether a Chinese character corresponding to the currently input pinyin can be found, and if so, the Chinese characters are listed, and if not, the user's gesture operation is continued to be detected. If the Chinese characters are listed, the user performs a pressing operation to trigger the Chinese character key and further input the Chinese character into the input box. Alternatively, the user can slide the horizontal bar of the Chinese character left and right by a left and right sliding operation, so that the horizontal bar scrolls left and right to display other Chinese characters, and the user can select the Chinese character.

[0133] In this application, the pinch and move gestures trigger the virtual keys in the keyboard without the need for a handle. After the pinch gesture, the user starts from the center of the operation control area, shortening the distance to each key in the operation control area. The left and right hands can select a large area (select the left or right part of the keyboard), and each finger can select a small area (select one or two rows of keyboard keys). After selecting the last key area with the pinch, the gesture displacement makes a precise key selection (precise selection of top, middle, and bottom). The gesture improves the selection efficiency, and the displacement operation supplements the freedom of selecting the last specific key.

[0134] The embodiment of the present application provides a flexible and simple operation method for a scenario in which a key on a virtual keyboard is triggered by combining two operations, gesture and movement. In this method, a user can trigger the display of a cursor by simply making a target gesture with his / her hand, and then control the movement of the cursor by simply moving his / her hand while holding the target gesture. Such an operation is very simple and convenient, and the movement of the cursor can also be controlled to precisely select and trigger one key. This improves the granularity and accuracy of triggering a key, and makes it possible to realize character input in any form, so that the applicability is good. In addition, when a target gesture is detected, the display position of the cursor can also be determined based on the biometric characteristics of the target gesture, so that a user can flexibly perform gesture operations using different hands, reduce the cursor movement distance as much as possible, reduce the complexity of the operation, and improve the operation efficiency.

[0135] All of the above optional configurations, when combined in any way, form optional embodiments of the present application, and so a detailed description thereof will be omitted here.

[0136] 18 is a schematic diagram of a key function execution system provided in an embodiment of the present application. The key function execution system includes an electronic device 1801, a gesture tracking sensor 1802, and a display device 1803. 3 are each connected to the electronic device 1801.

[0137] Applicable display device 180 3 displays a virtual keyboard, the virtual keyboard including a first area and a second area, each of the first area and the second area including at least two keys, the first area corresponding to a left hand and the second area corresponding to a right hand.

[0138] The gesture tracking sensor 1802 detects when the hand is in a target gesture.

[0139] Applicable display device 180 3 further comprising, in response to detecting the hand being in a target gesture, based on a biometric characteristic of the target gesture: No. A cursor is displayed at a first position, the first position being within an area of ​​the virtual keyboard that corresponds to the biometric feature and corresponds to the biometric feature.

[0140] The gesture tracking sensor 1802 further detects that the hand moves while holding the target gesture.

[0141] Applicable display device 180 3 further displays that in response to the hand moving while holding the target gesture, the cursor moves to follow the movement of the hand.

[0142] In response to the change in hand gesture, the electronic device 1801 executes a function of a key corresponding to a second position of the virtual keyboard, the second position being the position where the cursor is located when the hand gesture changes.

[0143] In some embodiments, the display device 180 3 is a virtual reality (VR) display device or screen display.

[0144] In some embodiments, the display device 180 3 teeth, displaying a cursor at a target location within the first region of the virtual keyboard in response to the biometric characteristic indicating that the hand is a left hand; In response to the biometric characteristic indicating that the hand is a right hand, a cursor is displayed at a target location within the second region of the virtual keyboard.

[0145] In some embodiments, the display device 180 3 teeth, in response to the biometric feature indicating that the hand is a left hand and that the target gesture is formed with a first finger and thumb of the hand, determine to display a cursor at a target location within a sub-area of ​​the first region of the virtual keyboard corresponding to the first finger; In response to the biometric feature indicating that the hand is a right hand and that the target gesture is formed with a first finger and thumb of the hand, it is determined to display a cursor at a target location within a sub-area of ​​the second region of the virtual keyboard corresponding to the first finger.

[0146] In some embodiments, the target location is a center location of the region.

[0147] In some embodiments, the electronic device 1801 acquires a displacement of the hand in response to the hand moving while holding the target gesture; and determines a third position based on the first position and the displacement of the hand; Applicable display device 180 3 indicates that the cursor is moving from the first position to the third position.

[0148] In some embodiments, the display device 180 3 teeth, in response to the key corresponding to the second position being a character key, inputting into an input box or a character display area of ​​the virtual keyboard the character indicated by the key corresponding to the second position; deleting a last character in an input box or a character display area of ​​the virtual keyboard in response to the key corresponding to the second position being a delete key; in response to the key corresponding to the second position being a line feed key, displaying an input cursor in an input box of the virtual keyboard to be fed to a next line; in response to the key corresponding to the second position being a confirmation key, canceling display of the virtual keyboard with the input content in the input box of the virtual keyboard as the target content.

[0149] In some embodiments, the display device 180 3 further displays, in response to the presence of a candidate Chinese character corresponding to the input character combination in a character display area of ​​the virtual keyboard, a candidate Chinese character corresponding to the input character combination within a target range of the character display area.

[0150] In some embodiments, the virtual keyboard further comprises at least one virtual control; Applicable display device 180 3 further comprising: in response to a hand being moved out of the target gesture, displaying that the cursor moves to track the movement of the hand; The electronic device 1801 further performs a function corresponding to the target virtual control in response to detecting that the hand has made a target gesture and the cursor is located on a target virtual control of the at least one virtual control.

[0151] In some embodiments, the electronic device 1801 includes: in response to the target virtual control being an uppercase input / lowercase input toggle control, toggle characters displayed within the virtual keyboard between uppercase input characters and lowercase input characters; in response to the target virtual control being a symbol input / alphabetic input switching control, switching characters displayed in the virtual keyboard between alphabetic and symbolic; In response to the target virtual control being a Chinese input / English input switching control, a character input mode of the virtual keyboard is switched between Chinese input and English input.

[0152] In some embodiments, the display device 180 3 further updates the display content of the target virtual control, so that the updated display content is consistent with the switching of characters or the switching of character input modes in the virtual keyboard.

[0153] 19 is a schematic diagram of the configuration of a key function execution device provided in an embodiment of the present application. As shown in FIG. 19, the device includes: a display module 1901 and an execution module 1902.

[0154] The display module 1901 displays a virtual keyboard, the virtual keyboard including a first area and a second area, each of the first area and the second area including at least two keys, the first area corresponding to a left hand, and the second area corresponding to a right hand.

[0155] The display module 1901 further, in response to detecting the hand being in the target gesture, before A cursor is displayed at a first location based on the biometric feature of the target gesture, the first location being within an area of ​​the virtual keyboard corresponding to the biometric feature and corresponding to the biometric feature.

[0156] The display module 1901 further displays, in response to the hand moving while holding the target gesture, that the cursor moves to follow the movement of the hand.

[0157] An execution module 1902, in response to the change in the hand gesture, executes a function of a key corresponding to a second position of the virtual keyboard, the second position being the position where the cursor is located when the hand gesture changes.

[0158] In some embodiments, the display module 1901 includes: displaying a cursor at a target location within the first region of the virtual keyboard in response to the biometric characteristic indicating that the hand is a left hand; In response to the biometric characteristic indicating that the hand is a right hand, a cursor is displayed at a target location within the second region of the virtual keyboard.

[0159] In some embodiments, the display module 1901 includes: in response to the biometric feature indicating that the hand is a left hand and that the target gesture is formed with a first finger and thumb of the hand, determine to display a cursor at a target location within a sub-area of ​​the first region of the virtual keyboard corresponding to the first finger; In response to the biometric feature indicating that the hand is a right hand and that the target gesture is formed with a first finger and thumb of the hand, it is determined to display a cursor at a target location within a sub-area of ​​the second region of the virtual keyboard corresponding to the first finger.

[0160] In some embodiments, the target location is a center location of the region.

[0161] In some embodiments, the display module 1901 includes: acquiring a displacement of the hand in response to the hand moving while holding the target gesture; determining a third position based on the first position and the displacement of the hand; The cursor is displayed moving from the first position to the third position.

[0162] In some embodiments, the execution module 1902 includes: in response to the key corresponding to the second position being a character key, inputting into an input box or a character display area of ​​the virtual keyboard the character indicated by the key corresponding to the second position; deleting a last character in an input box or a character display area of ​​the virtual keyboard in response to the key corresponding to the second position being a delete key; in response to the key corresponding to the second position being a line feed key, displaying an input cursor in an input box of the virtual keyboard to be fed to a next line; in response to the key corresponding to the second position being a confirmation key, canceling display of the virtual keyboard with the input content in the input box of the virtual keyboard as the target content.

[0163] In some embodiments, the display module 1901 further displays, in response to the presence of a candidate Chinese character corresponding to the input character combination in the character display area of ​​the virtual keyboard, a candidate Chinese character corresponding to the input character combination within the target range of the character display area.

[0164] In some embodiments, the virtual keyboard further comprises at least one virtual control; The display module 1901 further displays, in response to the hand being moved out of the target gesture, the cursor moving to track the movement of the hand; The execution module 1902 further executes a function corresponding to the target virtual control in response to detecting that the hand makes a target gesture and the cursor is located on a target virtual control of the at least one virtual control.

[0165] In some embodiments, the execution module 1902 includes: in response to the target virtual control being an uppercase input / lowercase input toggle control, toggle characters displayed within the virtual keyboard between uppercase input characters and lowercase input characters; in response to the target virtual control being a symbol input / alphabetic input switching control, switching characters displayed in the virtual keyboard between alphabetic and symbolic; In response to the target virtual control being a Chinese input / English input switching control, a character input mode of the virtual keyboard is switched between Chinese input and English input.

[0166] In some embodiments, the execution module 1902 further updates the display content of the target virtual control, so that the updated display content corresponds to a character switch or a character input mode switch in the virtual keyboard.

[0167] The device shown in the embodiments of the present application may be disposed in an electronic device or other device, but the embodiments of the present application are not limited thereto.

[0168] The embodiment of the present application provides a flexible and simple operation method for a scenario in which a key on a virtual keyboard is triggered by combining two operations, gesture and movement. In this method, a user can trigger the display of a cursor by simply making a target gesture with his / her hand, and then control the movement of the cursor by simply moving his / her hand while holding the target gesture. Such an operation is very simple and convenient, and the movement of the cursor can also be controlled to precisely select and trigger one key. This improves the granularity and accuracy of triggering a key, and makes it possible to realize character input in any form, so that the applicability is good. In addition, when a target gesture is detected, the display position of the cursor can also be determined based on the biometric characteristics of the target gesture, so that a user can flexibly perform gesture operations using different hands, reduce the cursor movement distance as much as possible, reduce the complexity of the operation, and improve the operation efficiency.

[0169] It should be noted that the key function executing device provided in the above embodiment is described only as an example of dividing each of the above function modules when executing a key function, but in actual application, the above functions are assigned to different function modules as necessary, that is, the internal configuration of the key function executing device is divided into different function modules to execute all or part of the above-described functions. In addition, the key function executing device provided in the above embodiment belongs to the same concept as the embodiment of the key function executing method, and the specific implementation process can be referred to the embodiment of the method. Here, the description is omitted.

[0170] FIG. 20 is a schematic diagram of the configuration of an electronic device provided in an embodiment of the present application. The electronic device 2000 includes one or more processors (CPU: Central Processing Units) 2001 and one or more memories 2002, although there may be large differences due to different specifications or performance. Here, at least one computer program is stored in the memory 2002, and when the at least one computer program is loaded and executed by the processor 2001, it realizes the key function execution method provided in each embodiment of the above method. The electronic device further includes other components for realizing the function of the device. For example, the electronic device further includes components such as a wired or wireless network interface and an input / output interface for input / output. The embodiment of the present application will not be described here.

[0171] The electronic device in the above embodiment of the method is realized as a terminal. For example, FIG. 21 is a block diagram of the configuration of a terminal provided in the embodiment of the present application. The terminal 2100 includes a processor 2101 and a memory 2102.

[0172] The processor 2101 may include one or more processing cores, for example a 4-core processor, an 8-core processor, etc. The processor 2101 processes digital signals. Processor (DSP: Digital Signal Processor or ), a field-programmable gate array (FPGA), or a programmable logic array (PLA). In some embodiments, the processor 2101 may include an artificial intelligence (AI) processor for processing computational operations related to machine learning.

[0173] The memory 2102 may include one or more computer-readable storage media. The computer-readable storage media may be non-transitory. In some embodiments, the non-transitory computer-readable storage media in the memory 2102 stores at least one instruction that, when executed by the processor 2101, implements a key function execution method provided in a method embodiment of the present application.

[0174] In some embodiments, the terminal 2100 optionally further comprises a peripheral interface 2103 and at least one peripheral. The processor 2101, the memory 2102, and the peripheral interface 2103 may be connected via a bus or signal lines. Each peripheral may be connected to the peripheral interface 2103 via a bus, signal line, or circuit board. Specifically, the peripheral includes at least one of a display 2104 or a camera component 2105.

[0175] The peripheral interface 2103 may be used to connect at least one peripheral device related to input / output (I / O) to the processor 2101 and the memory 2102. In some embodiments, the processor 2101, the memory 2102, and the peripheral interface 2103 are integrated on the same chip or circuit board. In some other embodiments, any one or two of the processor 2101, the memory 2102, and the peripheral interface 2103 may be implemented on separate chips or circuit boards. This embodiment is not limited thereto.

[0176] The display 2104 displays a user interface (UI). The UI may include graphics, text, icons, video, and any combination thereof. If the display 2104 is a touch display, the display 2104 is also capable of collecting touch signals on or above the surface of the display 2104. The touch signals may be input as control signals to the processor 2101 for processing. In this case, the display 2104 may be used to provide virtual buttons and / or a virtual keyboard, also referred to as soft buttons and / or a soft keyboard.

[0177] The camera component 2105 collects images or videos. Optionally, the camera component 2105 includes a front camera and a rear camera. Typically, the front camera is provided on the front panel of the terminal, and the rear camera is provided on the back of the terminal. In some embodiments, there are at least two rear cameras, each of which is any one of a main camera, a depth camera, a wide-angle camera, and a telephoto camera. This realizes a background blur function realized by combining the main camera and the depth camera, a panoramic shooting and a virtual reality (VR) shooting function realized by combining the main camera and the wide-angle camera, or other combined shooting functions.

[0178] In some embodiments, terminal 2100 further includes one or more sensors, including but not limited to an acceleration sensor, a gyro sensor, a pressure sensor, an optical sensor, and a proximity sensor.

[0179] As will be appreciated by those skilled in the art, the configuration shown in FIG. 21 is not intended to limit the terminal 2100, and the terminal 2100 may include more or fewer components than shown, may combine some components, or may employ a different arrangement of components.

[0180] The electronic device in the above method embodiments is realized as a server. For example, FIG. 22 is a schematic diagram of the configuration of a server provided in the embodiment of the present application. The server 2200 includes one or more processors (CPU: Central Processing Units) 2201 and one or more memories 2202, although there may be large differences due to different specifications or performance. Here, the memory 2202 stores at least one computer program, which, when loaded and executed by the processor 2201, realizes the key function execution method provided in each of the above method embodiments. Of course, the server has components such as a wired or wireless network interface and an input / output interface for input / output. The server includes other components for realizing the functions of the device, but the description is omitted here.

[0181] In an exemplary embodiment, a computer readable storage medium, such as a memory containing at least one computer program, is further provided, which, when executed by the processor, can cause the key function execution method in the above embodiment to be performed. For example, the computer readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0182] In an exemplary embodiment, a computer program product or computer program is further provided, the computer program product or computer program includes one or more program codes stored in a computer-readable storage medium. One or more processors of an electronic device read the one or more program codes from the computer-readable storage medium, and the one or more processors execute the one or more program codes to cause the electronic device to perform the key function execution method.

[0183] In some embodiments, the computer program according to the embodiments of the present application may be arranged to be executed on one computer device, on multiple computer devices at one location, or on multiple computer devices distributed at multiple locations and connected to each other via a communication network. The multiple computer devices distributed at multiple locations and connected to each other via a communication network may constitute a blockchain system.

[0184] As can be understood by those skilled in the art, all or part of the steps for implementing the above embodiments may be executed by hardware, or may be executed by instructing relevant hardware through a program. The program is stored in a computer-readable storage medium. The storage medium mentioned above may be a read-only memory, a magnetic disk, an optical disk, etc.

[0185] The above are only optional examples of the present application, and do not limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application should all be included within the protection scope of the present application.

Claims

1. 1. A method for performing a key function, comprising: a key function execution system displaying a virtual keyboard, the virtual keyboard including a first area and a second area, the first area and the second area each including at least two keys, the first area corresponding to a left hand and the second area corresponding to a right hand; in response to detecting a hand in a target gesture, the system displays a cursor at a first location based on a biometric feature of the target gesture, the first location being within an area of ​​the virtual keyboard corresponding to the biometric feature, and a location corresponding to the biometric feature; in response to the hand moving while holding the target gesture, the system displays the cursor moving to track the hand's movement; in response to the change in hand gesture, the system executes a function of a key corresponding to a second position of the virtual keyboard, the second position being a position where the cursor was located when the hand gesture changed; and The method includes:

2. the system displaying a cursor at a first location based on a biometric feature of the target gesture, the first location being within an area of ​​the virtual keyboard corresponding to the biometric feature, the step including: in response to the biometric characteristic indicating that the hand is a left hand, the system displaying a cursor at a target location within the first area of ​​the virtual keyboard; and in response to the biometric characteristic indicating that the hand is a right hand, the system displays a cursor at a target location within the second area of ​​the virtual keyboard. The method of claim 1.

3. The system displaying a cursor at a first location based on a biometric characteristic of the target gesture includes: in response to the biometric feature indicating that the hand is a left hand and that the target gesture is to be made with a first finger and thumb of the hand, the system displaying a cursor at a target location within a sub-area of ​​the first region corresponding to the first finger; in response to the biometric feature indicating that the hand is a right hand and that the target gesture is to be made with a first finger and thumb of the hand, the system displaying a cursor at a target location within a sub-area of ​​the second area corresponding to the first finger. The method of claim 1.

4. The method according to claim 2 or 3, wherein the target location is a center location of a region.

5. In response to the hand moving while holding the target gesture, the system displays that the cursor moves to track the hand's movement, in response to the hand moving while holding the target gesture, the system acquiring a displacement of the hand; the system determining a third position based on the first position and the displacement of the hand; and the system indicating that the cursor is moving from the first position to the third position. The method of claim 1.

6. The step of the system executing a function of a key corresponding to a second position of the virtual keyboard includes: in response to the key corresponding to the second position being a character key, the system inputs the character indicated by the key corresponding to the second position into an input box or a character display area of ​​the virtual keyboard; in response to the key corresponding to the second position being a delete key, the system deletes a last character in an input box or character display area of ​​the virtual keyboard; in response to the key corresponding to the second position being a line feed key, the system displays an input cursor in an input box of the virtual keyboard being fed to a next line; and in response to the key corresponding to the second position being a confirmation key, the system cancels display of the virtual keyboard with the input content in the input box of the virtual keyboard as the target content. The method of claim 1.

7. and in response to a presence of a candidate Chinese character corresponding to the input character combination in a character display area of ​​the virtual keyboard, the system displays the candidate Chinese character within a target range of the character display area. The method of claim 1.

8. The virtual keyboard further includes at least one virtual control, and the method further comprises: in response to detecting that a hand is not in the target gesture and that the hand is moving, the system indicating that the cursor moves to track the movement of the hand; and in response to detecting that the hand is making a target gesture and the cursor is located on a target virtual control of the at least one virtual control, the system executes a function corresponding to the target virtual control. The method of claim 1.

9. The step of the system executing a function corresponding to the target virtual control includes: in response to the target virtual control being an uppercase / lowercase input toggle control, the system toggles characters displayed in the virtual keyboard between uppercase input characters and lowercase input characters; in response to the target virtual control being a control for switching between symbol input and alphabet input, the system toggles characters displayed in the virtual keyboard between alphabetic and symbolic; in response to the target virtual control being a Chinese input / English input switching control, the system switches a character input mode of the virtual keyboard between Chinese input and English input. The method according to claim 8.

10. The step of the system executing a function corresponding to the target virtual control includes: The system further includes a step of updating the display content of the target virtual control, and the display content after the update is consistent with a change of a character or a change of a character input mode in the virtual keyboard.

10. The method of claim 9.

11. A key function execution system including an electronic device, a gesture tracking sensor, and a display device, the gesture tracking sensor and the display device being respectively connected to the electronic device; the display device displays a virtual keyboard, the virtual keyboard including a first area and a second area, the first area and the second area each including at least two keys, the first area corresponding to a left hand and the second area corresponding to a right hand; the gesture tracking sensor detects that a hand is in a target gesture; The display device further, in response to detecting the hand being in the target gesture, displays a cursor at a first location based on a biometric feature of the target gesture, the first location being within an area of ​​the virtual keyboard corresponding to the biometric feature and corresponding to the biometric feature; The gesture tracking sensor further detects that the hand is moving while holding the target gesture; the display device further displays, in response to the hand moving while holding the target gesture, that the cursor moves in accordance with the movement of the hand; the electronic device executes a function of a key corresponding to a second position of the virtual keyboard in response to the change in the hand gesture, the second position being a position where the cursor is located when the hand gesture is changed. system.

12. The system of claim 11 , wherein the display device is a virtual reality display device or a screen display.

13. A key function execution device, comprising: a display module; and an execution module; The display module displays a virtual keyboard, the virtual keyboard including a first area and a second area, the first area and the second area each including at least two keys, the first area corresponding to a left hand and the second area corresponding to a right hand; The display module further, in response to detecting the hand being in a target gesture, displays a cursor at a first location based on a biometric feature of the target gesture, the first location being within an area of ​​the virtual keyboard corresponding to the biometric feature and corresponding to the biometric feature; The display module further displays, in response to the hand moving while holding the target gesture, that the cursor moves in accordance with the movement of the hand; the execution module executes a function of a key corresponding to a second position of the virtual keyboard in response to the change in the hand gesture, the second position being a position where the cursor is located when the hand gesture is changed. Device.

14. An electronic device comprising one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and wherein the at least one computer program, when loaded and executed by the one or more processors, realizes the key function execution method described in any one of claims 1 to 10.

15. A computer program that causes an electronic device to execute a key function execution method described in any one of claims 1 to 10.

Images