Virtual keyboard based text input method and device

Abstract

A text input method and apparatus based on a virtual keyboard are provided, which relate to the field of human-computer interaction and solve the problem of how to complete text input based on a virtual keyboard. The method includes: an electronic device acquires an image including a first hand, detects keypoints of the first hand based on the image including the first hand (301), and projects a virtual keyboard onto the first hand based on the keypoints of the first hand (302). After acquiring an image including the first hand and fingers of a second hand (303), the electronic device identifies a contact or pressing state between the fingers of the second hand and the first hand based on the image including the fingers of the first hand and the second hand (304). When the fingers of the second hand and the first hand are in contact or pressing state, the electronic device determines the input text based on the position where the fingers of the second hand touch or press the first hand (305).

Description

[Technical field]

[0001] TECHNICAL FIELD Embodiments of the present application relate to the field of human-computer interaction, and in particular to a text input method and device based on a virtual keyboard. [Background technology]

[0002] Currently, virtual reality (VR) / augmented reality (AR) technology is changing to implement a three-dimensional (3D) immersive display experience, which can make the interaction method more suitable for reality. Therefore, virtual reality / augmented reality technology is expected to have a wide range of applications in various industries such as social networking, healthcare, education, games, and movies. For VR / AR devices, the biggest drawback is the text input mode. Generally, the user decides to place the light beam from the virtual space on the key of the virtual keyboard by rotating or translating the handle, and then press the handle to complete the input; or the user puts both hands in the air and presses the virtual keyboard to complete the input. These methods are inefficient and difficult to operate; long-term input makes the user feel pain. Therefore, how to complete text input based on the virtual keyboard is an urgent problem to be solved. Summary of the Invention

[0003] The present application provides a text input method and device based on a virtual keyboard to solve the problem of how to complete text input based on a virtual keyboard.

[0004] In order to achieve the aforementioned objectives, the following technical solutions are used in the embodiments of this application.

[0005] According to a first aspect, the present application provides a text input method based on a virtual keyboard. The method includes: after acquiring an image including a first hand, an electronic device detects key points of the first hand based on the image including the first hand, and projects a virtual keyboard on the first hand based on the key points of the first hand. For example, the virtual keys are projected on a palm or back of the first hand. Furthermore, after acquiring an image including the first hand and the fingers of a second hand, the electronic device identifies a contact or pressing state between the fingers of the second hand and the first hand based on the image including the fingers of the first hand and the fingers of the second hand. When the fingers of the second hand and the first hand are in a contact or pressing state, the electronic device determines the input text based on the position where the fingers of the second hand touch or press the first hand.

[0006] Thus, the virtual keyboard is projected based on the user's body part (e.g., the palm or the back of the hand), and the user can navigate the interface. excessive The present invention uses the brain to assist in locating the projected virtual keyboard without the need to gaze at the screen. This allows the user to efficiently and accurately press or touch the projected virtual keyboard, effectively improving the efficiency and accuracy of text input. The electronic device determines the input text by identifying the body position pressed or touched by the user. In addition, the text input method with the virtual keyboard is suitable for scenarios such as: Special limited to There is no need to worry about the user's posture, so the user can operate the device in any posture, which solves the problems of fatigue and pain caused by long periods of interrupted input, provides sufficient physical feedback, and effectively improves the user experience.

[0007] In a possible implementation, the electronic device identifying a contact or pressing state between the fingers of the second hand and the first hand based on an image including the fingers of the first hand and the second hand includes: the electronic device analyzing the image including the fingers of the first hand and the second hand by using a pre-configured neural network model; identifying a contact or pressing state between the fingers of the second hand and the first hand; and obtaining fingertip key points of the fingers of the second hand.

[0008] In a possible implementation, the electronic device identifying a contact or pressing state between the fingers of the second hand and the first hand based on an image including the fingers of the first hand and the fingers of the second hand includes: the electronic device uses a pre-configured neural network model to analyze the image including the fingers of the first hand and the fingers of the second hand and an optical flow of the fingers of the second hand; identifies a contact or pressing state between the fingers of the second hand and the first hand; and obtains fingertip key points of the fingers of the second hand. Thus, the electronic device can identify a contact state, pressing state, or pressing state between the fingers of the second hand and the first hand. Get away The accuracy of identification is improved.

[0009] In another possible implementation, the electronic device determining the input text based on the position where the finger of the second hand touches or presses the first hand includes: the electronic device determines the position where the finger of the second hand touches or presses the first hand based on the fingertip coordinate of the finger of the second hand and the coordinate of the virtual keyboard; and determines the input text based on the position where the finger of the second hand touches or presses the first hand. The key point may be a two-dimensional coordinate. Thus, the electronic device converts the position where the fingertip of the second hand points and is the virtual keyboard into a coordinate, and determines the input text based on the coordinate. This improves the accuracy of recognizing the input text.

[0010] In another possible implementation, the electronic device projecting a virtual keyboard onto the first hand based on key points of the first hand includes: the electronic device identifying an action of the first hand based on a plurality of frames of an image including the first hand; determining that the action of the first hand matches a predefined action, and projecting the virtual keyboard based on the key points of the first hand. For example, the predefined action may range from clenching a fist to extending a palm. this is The electronic device identifies a position where the virtual keyboard is to be projected, and accurately projects the virtual keyboard. of surely vinegar This makes it easier for users to enter text.

[0011] In another possible implementation, after the electronic device acquires the image including the fingers of the first hand and the second hand, the method further includes: the electronic device acquires a first hand image including the fingers of the second hand and the fingers of the first hand based on the image including the fingers of the first hand and the fingers of the second hand. but separation What we do Identify the fingers of the second hand and the first hand; Is When the user inputs a text message, an image including the first hand is acquired, or an image including the fingers of the first hand and the second hand is acquired. Thus, the contact, pressing, or separation of the fingers of the second hand and the first hand is identified by positioning and tracking the image including the fingers of the first hand and the fingers of the second hand in real time. This allows the input text to be displayed in a clear and easy to understand manner. of accurate To recognition Can be , improving the user experience.

[0012] In another possible implementation, the method further includes: the electronic device inputs a training set into the neural network model, and the neural network model outputs a classification result; calculates a loss function value of a loss function based on the classification result, the labels of the first type of training images, and the labels of the second type of training images; and obtains parameters of the neural network model when the loss function converges and the loss function value is equal to or less than a threshold. The training set includes a first type of training images, a second type of training images, labels of the first type of training images, and labels of the second type of training images. The first type of training images includes a plurality of images of the fingers of the second hand and the first hand in a contact or pressing state, and the second type of training images includes a plurality of images of the fingers of the second hand and the first hand in a separation state. Is The electronic device can then determine the contact state, pressing state, classification state, and classification result of the second hand and the first hand. Get away It can be identified in real time.

[0013] In another possible implementation, after the electronic device determines the input text based on the position where the fingers of the second hand touch or press the first hand, the method further includes: the electronic device displays the first hand, the fingers of the second hand, and the input text. Thus, the user can intuitively see the input text in real time, improving the user experience.

[0014] According to a second aspect, the present application further provides a text input device. For beneficial effects, please refer to the description of the first aspect. Details will not be described again here. The text input device has a function of implementing the operations in the method example of the first aspect. This function may be implemented by hardware, or may be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned functions. In a possible design, the text input device includes an image capture unit and a processing unit. The processing unit is configured to acquire key points of the first hand based on an image including the first hand. The processing unit is further configured to project a virtual keyboard on the first hand based on the key points of the first hand. The image capture unit is configured to acquire an image including the first hand and fingers of the second hand. The processing unit is further configured to identify a contact state or pressing state between the fingers of the second hand and the first hand based on the image including the first hand and fingers of the second hand. The processing unit is further configured to: determine the input text based on the position where the finger of the second hand touches or presses the first hand when the finger of the second hand and the first hand are in contact or pressing state. These units may perform corresponding functions in the method example of the first aspect. For details, please refer to the detailed description of the method example. The details will not be described again here.

[0015] According to a third aspect, an electronic device is provided, which may include a processor, a memory, a display, and a camera. The processor is coupled to the display, the camera, and the memory. The memory is configured to store computer program code. The computer program code includes computer software instructions. When the computer software instructions are executed by the electronic device, the electronic device performs the following operations: after acquiring an image including a first hand, the electronic device detects key points of the first hand based on the image including the first hand, and projects a virtual keyboard onto the first hand based on the key points of the first hand. Furthermore, after acquiring an image including the first hand and fingers of a second hand, the electronic device determines a contact or pressing state between the fingers of the second hand and the first hand based on the image including the first hand and fingers of the second hand. When the fingers of the second hand and the first hand are in a contact or pressing state, the electronic device determines the inputted text based on the position where the fingers of the second hand touch or press the first hand.

[0016] According to a fourth aspect, there is provided a computer readable storage medium comprising computer software instructions which, when executed on an electronic device, enable the electronic device to perform a text input method based on a virtual keyboard according to the first aspect or any of the possible implementations of the first aspect.

[0017] According to a fifth aspect, there is provided a computer program product, which, when executed on a computer, enables the computer to perform the virtual keyboard based text input method according to the first aspect or any of the possible implementations of the first aspect.

[0018] According to a sixth aspect, a chip system is provided. The chip system is for use in an electronic device. The chip system includes an interface circuit and a processor; the interface circuit and the processor are interconnected via a line. The interface circuit is configured to receive signals from a memory of the electronic device and to send signals to the processor, the signals including computer instructions stored in the memory. When the processor executes the computer instructions, the chip system performs a text input method based on a virtual keyboard according to the first aspect or any of the possible implementations of the first aspect.

[0019] It should be understood that the description of technical features, technical solutions, beneficial effects, or similar words in this application does not mean that all features and advantages can be implemented in any individual embodiment. Conversely, the description of a feature or beneficial effect may be understood to mean that at least one embodiment includes the particular technical feature, technical solution, or beneficial effect. Thus, the description of a technical feature, technical solution, or beneficial effect in this specification is not necessarily specific to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in the embodiments can be combined in any suitable manner. Those skilled in the art can understand that an embodiment can be implemented without one or more of the particular technical features or technical solutions, or beneficial effects in a particular embodiment. In other embodiments, additional technical features and beneficial effects may be further specified in a particular embodiment that does not reflect all embodiments. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is a schematic diagram of a structure of an electronic device according to an embodiment of the present application.

[0021] [Diagram 2] FIG. 2 is a schematic diagram of a television camera according to an embodiment of the present application;

[0022] [Diagram 3]1 is a flowchart of a text input method based on a virtual keyboard according to an embodiment of the present application.

[0023] [Figure 4] FIG. 2 is a schematic diagram of a hand portion according to an embodiment of the present application.

[0024] [Diagram 5] FIG. 1 is a schematic diagram of a first hand keypoints identification result according to an embodiment of the present application;

[0025] [Figure 6(a)] FIG. 2 is a schematic diagram of a virtual keyboard projected onto a first hand according to an embodiment of the present application.

[0026] [Figure 6(b)] FIG. 2 is a schematic diagram of a virtual keyboard projected onto a first hand according to an embodiment of the present application.

[0027] [Figure 7] 1 is a flowchart of a text input method based on a virtual keyboard according to an embodiment of the present application.

[0028] [Figure 8] FIG. 2 is a schematic diagram of pre-set actions according to an embodiment of the present application;

[0029] [Figure 9] FIG. 2 is a schematic diagram of a state between a finger of a second hand and a first hand according to an embodiment of the present application.

[0030] [Figure 10] FIG. 2 is a schematic diagram of a determined input text according to an embodiment of the present application;

[0031] [Figure 11(a)] FIG. 2 is a schematic diagram of a text input process based on a virtual keyboard according to an embodiment of the present application;

[0032] [Figure 11(b)] FIG. 2 is a schematic diagram of a text input process based on a virtual keyboard according to an embodiment of the present application;

[0033] [Figure 11(c)] FIG. 2 is a schematic diagram of a text input process based on a virtual keyboard according to an embodiment of the present application;

[0034] [Figure 12] 1 is a schematic diagram of a text input device according to an embodiment of the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] In this specification, claims, and accompanying drawings of the present application, the terms "first," "second," "third," etc. are intended to distinguish different objects but are not intended to limit the order to a particular order.

[0036] In the embodiments of the present application, words such as "example", "for example" and the like are used to denote giving an example, illustration or explanation. In the embodiments of the present application, any embodiment or design scheme described in "example" or "for example" should not be described as preferred or having more advantages over other embodiments or design schemes. Rather, the use of words such as "example" or "for example" is intended to present the relevant concept in a particular way.

[0037] The electronic device in the embodiment of the present application may be a device including a display and a camera, such as a television, a tablet computer, a projector, a mobile phone, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), or an augmented reality (AR) / virtual reality (VR) device. The specific form of the electronic device is not particularly limited in the embodiment of the present application.

[0038] 1 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. As shown in FIG. 1, the electronic device includes: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a power management module 140, an antenna, a wireless communication module 160, an audio module 170, a speaker 170A, a speaker box interface 170B, a microphone 170C, a sensor module 180, keys 190, an indicator 191, a display 192, a camera 193, etc. The sensor module 180 may include a distance sensor, an optical proximity sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, etc.

[0039] It may be understood that the structure shown in this embodiment does not constitute a specific limitation of the electronic device. In other embodiments, the electronic device may include more or fewer components than those shown, combine some components, separate some components, or have a different component arrangement. The components shown may be implemented by hardware, software, or a combination of software and hardware.

[0040] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be separate components or may be integrated into one or more processors.

[0041] In this embodiment, the processor 110 is configured to: receive an image from the camera 193 and including the first hand; obtain key points of the first hand based on the image including the first hand; and project a virtual keyboard onto the first hand based on the key points of the first hand. The processor 110 is further configured to: receive an image from the camera 193 and including the fingers of the first hand; identify a contact or pressing state between the fingers of the first hand and the fingers of the second hand based on the image of the first fingers and the fingers of the second hand; and identify the entered text based on the location where the fingers of the second hand touch or press the first hand when the fingers of the second hand and the first hand are in contact or pressing state.

[0042] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to the instruction operation code and the time series signal to complete the control of reading the instruction and executing the instruction.

[0043] The memory may be located in the processor 110 and is configured to store instructions and data. In some embodiments, the memory of the processor 110 is a cache. The memory may store instructions or data that have been used or have been used periodically by the processor 110. If the processor 110 needs to use the instructions or data again, the processor may retrieve the instructions or data directly from the memory. This avoids repeated accesses and reduces the wait time of the processor 110, thereby improving system efficiency.

[0044] In some embodiments, the processor 110 may include one or more interfaces, which may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a USB interface, etc.

[0045] The power management module 140 is configured to connect to a power supply. The power management module 140 may be further connected to the processor 110, the internal memory 121, the display 192, the camera 193, the wireless communication module 160, etc. The power management module 140 receives a power input and provides power to the processor 110, the internal memory 121, the display 192, the camera 193, the wireless communication module 160, etc. In other embodiments, the power management module 140 may alternatively be located in the processor 110.

[0046] The wireless communication function of the electronic device may be implemented using an antenna, a wireless communication module 160, etc. The wireless communication module 160 is applied to the electronic device and is used to communicate with a wireless local area network (WLAN). k、 WLAN) (e.g., wireless fidelity (Wi-Fi®) networks), Bluetooth® ( B The company may provide wireless communication solutions including Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC) technology, and infrared (IR) technology.

[0047] The wireless communication module 160 may be one or more components that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves by an antenna, performs frequency modulation and filtering processing on the electromagnetic wave signal, and transmits the processed signal to the processor 110. The wireless communication module 160 may further receive a signal to be transmitted from the processor 110, perform frequency modulation and amplification on the signal, and convert the signal into an electromagnetic wave for radiation via the antenna. In some embodiments, an antenna of the electronic device is coupled to the wireless communication module 160, so that the electronic device may communicate with networks and other devices according to wireless communication techniques.

[0048] The electronic phone implements display functions using a GPU, a display 192, an application processor, etc. The GPU is a microprocessor used for image processing and is connected to the display 192 and the application processor. The GPU is configured to perform mathematical and geometric calculations and is configured to perform graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0049] The display 192 is configured to display images, videos, etc. The display 192 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (flexLED), or a combination of these. ible The light-emitting diodes may be FLEDs, mini LEDs, micro LEDs, micro OLEDs, quantum dot light emitting diodes (QLEDs), etc.

[0050] The electronic device may implement the image capture function using an ISP, a camera 193, a video codec, a GPU, a display 192, an application processor, etc. The ISP is configured to process data fed back by the camera 193. In some embodiments, the ISP may be located in the camera 193.

[0051] The camera 193 is configured to capture still images or video. An optical image of an object is generated through a lens and projected onto a photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then sent to the ISP to convert the electrical signal into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format, such as RGB or YUV. In some embodiments, the electronic device may include one or N cameras 193, where N is a positive integer greater than 1. For example, in this embodiment, the position of the camera 193 on the electronic device is not limited. In this embodiment, the camera 193 is configured to capture an image including a first hand and an image including fingers of the first hand and a second hand. For example, the electronic device is a VR device. As shown in FIG. 2, the camera 193 may be disposed at a lower end of the VR device. Indeed, in the embodiment of the present application, the position of the camera 193 on the electronic device is not limited.

[0052] Alternatively, the electronic device may not include a camera, in other words the camera 193 is not located on the electronic device (e.g., a television). The electronic device may be externally connected to the camera 193 via an interface (e.g., USB interface 130). The external camera 193 may be secured to the electronic device using an external fastener (e.g., a camera bracket with a clip). For example, the external camera 193 may be secured to an edge, e.g., an upper edge, of the display 192 of the electronic device using an external fastener.

[0053] The digital signal processor is configured to process digital signals and may process other digital signals in addition to digital image signals. For example, if the electronic device selects a frequency, the digital signal processor is configured to perform a Fourier transform on the frequency energy. The video codec is configured to compress or decompress digital video. The electronic device may support one or more video codecs. In this way, the electronic device may play or record video in multiple coding formats, e.g., moving picture experts group (MPEG)-1, MPEG-2, MPEG-3, MPEG-4.

[0054] NPU is a neural-network (NN) computing processor. NPU uses the structure of biological neural networks, for example, the transfer mode between human brain neurons, to rapidly process input information; and can continuously self-learn. Applications such as intelligent perception of electronic devices, such as image recognition, face recognition, voice recognition, and text understanding, can be implemented using NPU.

[0055] The external memory interface 120 may be configured to connect to an external storage card, such as a microSD card, to expand the storage capabilities of the electronic device. The external storage card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos are stored on the external storage card.

[0056] The internal memory 121 may be configured to store computer executable program code, the executable program code including instructions. The processor 110 executes the instructions stored in the internal memory 121 to perform various functional applications and data processing of the electronic device. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store an operating system, applications required by at least one function (e.g., audio playback function, image playback function), etc. The data storage area may store data generated during use of the electronic device (e.g., audio data), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may further include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or a universal flash storage (UFS).

[0057] The electronic device may implement audio functions using the audio module 170, the speaker 170A, the microphone 170C, the speaker box interface 170B, an application processor, etc. For example, the audio functions include playing music, recording, etc. In the present application, the microphone 170C may be configured to receive voice commands transmitted by a user to the electronic device. The speaker 170A may be configured to feed back the decision command of the electronic device to the user.

[0058] Audio module 170 is configured to convert digital audio information into analog audio signals for output, and is also configured to convert analog audio input into digital audio signals. Audio module 170 may be further configured to encode and decode audio signals. In some embodiments, audio module 170 may be located in processor 110, or some functional modules within audio module 170 may be located in processor 110. Speaker 170A, also called a "horn," is configured to convert audio electrical signals into voice signals. Microphone 170C, also called a "mike" or "mic," is configured to convert voice signals into electrical signals.

[0059] The speaker box interface 170B is configured to connect to a wired speaker box. The speaker box interface 170B can be a USB interface 130, or a 3.5 mm open mobile terminal platform (OMTP) standard interface, or a cellular telephone industry association (CEA) standard interface. n It may be a CTIA (Computer Telecommunications Industry Association of the USA) standard interface.

[0060] The keys 190 include a power key, a volume key, etc. The keys 190 may be mechanical keys or touch keys. The electronic device may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device.

[0061] The indicator 191 may be an indicator light and may be configured to indicate that the electronic device is in a power-on mode, a standby mode, a power-off mode, etc. For example, when the indicator light is off, it may indicate that the electronic device is in a power-off mode; when the indicator light is green or blue, it may indicate that the electronic device is in a power-on mode; when the indicator light is red, it may indicate that the electronic device is in a standby mode.

[0062] It may be understood that the structure shown in this embodiment of the present application does not constitute a particular limitation of the electronic device. The electronic device may have more or fewer components than those shown in FIG. 1, may combine two or more components, or may have a different component configuration. For example, the electronic device may further include components such as a speaker box. The various components shown in FIG. 1 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing or application specific integrated circuits.

[0063] All the methods in the following embodiments can be implemented in an electronic device having the above-mentioned hardware structure. In the following embodiments, the method in the embodiment of the present application is described using an example in which the electronic device is a VR device or an AR device.

[0064] It is assumed that a user carries a VR device and is in a virtual environment. When the user needs to input text in the virtual environment, if the VR device takes an image including a first hand, the VR device may detect key points of the first hand based on the image including the first hand, and project a virtual keyboard onto the first hand based on the key points of the first hand. It may be understood that the virtual keyboard is a keyboard image presented in a virtual form in the virtual environment, and is not a physical keyboard. For example, the virtual keyboard is a virtual keyboard that the user sees in the virtual environment presented by the VR device. As another example, the virtual keyboard is a virtual keyboard that the user sees in the virtual environment presented by the AR device. In this case, the user can see that the virtual keyboard is projected onto the first hand in the virtual environment. Furthermore, after acquiring the image including the fingers of the first hand and the second hand, the electronic device identifies a contact state or pressing state between the fingers of the second hand and the first hand based on the image including the fingers of the first hand and the second hand; and determines the input text based on the position where the fingers of the second hand touch or press the first hand when the fingers of the second hand and the first hand are in a contact state or pressing state. Next, with reference to FIG. 3, the text input method based on the virtual keyboard provided in the embodiment will be described in detail.

[0065] S301: Obtain keypoints of a first hand based on an image including the first hand.

[0066] The electronic device captures a first image using a camera; j The electronic device detects a first image captured by a camera according to a hand detection algorithm; and determines that the first image contains a first hand. Object detection may mean finding an object of interest in an image. Specifically, the electronic device first selects some candidate regions on a given image, then extracts features from these regions, and classifies the object in the image using a trained classifier based on the extracted features.

[0067] Further, the electronic device captures an image including the first hand within the first image. The image including the first hand may be an image including the entire first hand or may be an image including a portion of the first hand, including but not limited to: the fingers of the first hand shown in FIG. 4(a), the palm of the first hand shown in FIG. 4(b), the back of the first hand shown in FIG. 4(c), some of the fingers of the first hand and some of the palm of the first hand shown in FIG. 4(d), and some of the fingers of the first hand and some of the back of the first hand shown in FIG. 4(e).

[0068] Based on the image including the first hand, key points of the first hand are identified according to a hand pose estimation algorithm. The hand pose estimation algorithm is an algorithm in which hand key points are detected by training a neural network model, and the pose of the hand is described based on the hand key points. The key points here are knuckle points or skeleton points. For example, FIG. 5 is a schematic diagram of a key point identification result of the first hand according to an embodiment of the present application. The electronic device may identify 21 key points of the hand according to the hand pose estimation algorithm. Alternatively, the electronic device may identify more or less than 21 key points of the hand according to the hand pose estimation algorithm.

[0069] S302: Project a virtual keyboard onto the first hand based on keypoints of the first hand.

[0070] The electronic device may use the key points of the first hand as two-dimensional coordinates and project a virtual keyboard onto the first hand based on the key points of the first hand. For example, the electronic device may project the virtual keyboard onto the first hand according to a computer vision affine algorithm. In a possible design, as shown in FIG. 6(a), the virtual keyboard is projected onto the palm of the first hand based on key points 0, 1, 5, and 17 of the palm of the first hand. In another possible design, as shown in FIG. 6(b), the virtual keyboard is projected onto the back of the first hand based on key points 0, 1, 5, and 17 of the back of the first hand. The virtual keyboard may be a numeric keyboard or a 9-key pinyin keyboard. This is not limited here.

[0071] In a possible implementation, as shown in Fig. 7, the electronic device identifies a first hand action based on multiple frames of an image including a first hand, and determines whether the first hand action matches a preset action (perform S701). If the first hand action matches the preset action, a virtual keyboard is projected onto the first hand based on key points of the first hand (perform S302); or, if the first hand action does not match the preset action, the electronic device uses a camera to take an image including the first hand (perform S702), and obtains key points of the first hand based on the image including the first hand (perform S301).

[0072] An action is a process of changing the position of a role's five organs (facial expression change), changing the position of the role's limbs (action change), and changing the relative position of the role and the environment in which the role is located (movement distance change). The electronic device may identify a pose of the first hand in multiple consecutive frames according to a hand pose estimation algorithm, and determine an action of the first hand based on the pose of the first hand in multiple consecutive frames.

[0073] The pre-set action is a hand action pre-set by the electronic device. An action that matches the pre-set action may be understood to be the same action as the pre-set action. If the electronic device determines that the first hand action is completely the same as the pre-set action, the electronic device considers that the first hand action matches the pre-set action, and the first hand is the target hand. For example, the pre-set action may be an action from clenching a fist to extending the palm. As another example, the pre-set action may be an action of inverting the palm to the back of the hand. As another example, the pre-set action may be an action of inverting the back of the hand to the palm. For example, FIG. 8 (a) is a schematic diagram of an action from clenching a fist to extending the palm. The electronic device distinguishes between the action of extending the palm to clench a fist and the action of clenching a fist to extending the palm. The electronic device distinguishes between the action of inverting the palm to the back of the hand, as shown in FIG. 8 (b).

[0074] It should be noted that after the virtual keyboard is projected on the first hand, the electronic device may continuously capture images including the first hand, recognize the continuous images including the first hand, determine the key points of the first hand, track the first hand based on the key points, and project the virtual keyboard based on the key points of the first hand. In this way, the localization of the first hand is implemented through real-time localization and tracking of the key points of the first hand, and the virtual keyboard can move with the first hand, and the effect of the virtual keyboard being in contact with the skin is implemented to ensure a good user experience. For example, the electronic device continuously identifies at least 20 frames of images including the first hand within each second, determines the key points of the first hand, tracks the first hand based on the key points of the first hand, and projects the virtual keyboard based on the key points of the first hand.

[0075] S303: Obtain an image including fingers of a first hand and a second hand.

[0076] The electronic device captures a second image using a camera; detects the second image captured by the camera according to an object detection algorithm; and determines that the second image includes the first hand and the fingers of the second hand. Further, the electronic device continuously captures images including the first hand and the fingers of the second hand; performs image recognition on the continuous images of the first hand and the fingers of the second hand to determine key points of the fingers of the second hand (executing S703); and tracks the fingers of the second hand based on the key points of the fingers of the second hand (executing S704). Thus, localizing the fingers of the second hand is implemented by locating and tracking the key points of the fingers of the second hand in real time; and determining the contact state, pressing state, or separation state between the fingers of the second hand and the first hand. Away Identified; Entered text of accurate To recognition Can be , improving user experience.

[0077] In this embodiment, the left-right relationship between the first hand and the second hand is not limited. For example, the first hand may be a left hand, and the second hand may be a right hand. As another example, the first hand may be a right hand, and the second hand may be a left hand. Optionally, after acquiring the second image and identifying that the second image includes the fingers of the first hand and the second hand, the electronic device may also identify the left-right relationship between the first hand and the second hand.

[0078] S304: Based on an image including the fingers of the first hand and the second hand, a contact or pressing state between the fingers of the second hand and the first hand is identified.

[0079] Using a neural network model, the contact state, pressing state, or classification between the fingers of the second hand and the first hand is calculated. AwayBefore being identified, a large number of images including the fingers of the first hand and the second hand are first captured using a camera, and the images including the fingers of the first hand and the second hand are analyzed. When the fingers of the second hand touch or press the first hand, the fingers of the second hand block the position where the fingers of the second hand touch or press the first hand. Therefore, the texture of the first hand changes. Texture means a pattern or line on the surface of an object, which is a linear pattern on an object. When a training set is constructed, the texture features (features) of the first hand in a large number of images including the fingers of the first hand and the second hand are labeled, and the images including the fingers of the first hand and the second hand and labeled with the texture features of the first hand form the training set.

[0080] It may be understood that the training set includes a first type of training images, a second type of training images, labels for the first type of training images, and labels for the second type of training images, where the first type of training images includes a plurality of images in which the fingers of the second hand and the first hand are in contact or pressed together, and the second type of training images includes a plurality of images in which the fingers of the second hand and the first hand are in separate contact or pressed together. I'm Contains multiple images.

[0081] When the neural network model is trained, the parameters of each layer of the neural network model are first initialized (in other words, an initial value is assigned to each parameter), and a training set is input to the neural network model. Then, the neural network model outputs a classification result. The classification result indicates the type of training images identified by the neural network model and present in the training set. Specifically, the neural network model generates a classification result for a certain amount of images in which the fingers of the second hand and the first hand are in contact with each other, a certain amount of images in which the fingers of the second hand and the first hand are in a pressing state, a certain amount of images in which the fingers of the second hand and the first hand are in a separating state, and a certain amount of images in which the fingers of the second hand and the first hand are in a separating state. Is In the classification process, the labels are generated by inputting the images for the touch state, the images for the pressing state, and the classification state to train the whole network. AwayThe image may be individually attached to the image.

[0082] A loss function value of the loss function is calculated based on the classification result, the label of the first type of training image, and the label of the second type of training image. The loss function value of the loss function may indicate an error between the classification result and the training set. If the loss function does not converge and the loss function value is equal to or greater than a certain threshold, the neural network model continues to be trained using the aforementioned method. If the loss function converges and the loss function value is less than a certain threshold, the parameters of the neural network model are obtained. Alternatively, the training of the neural network model is completed when all training images in the training set are used to complete the training. For images input into the neural network model with a size of 56×56 pixels or less, the weight value of the neural network model is 500K or less to improve the processing speed of image recognition by the processor.

[0083] Once the neural network model is inferred, an image including the fingers of the first hand and the second hand is input to the neural network model, and using the neural network model, the image including the fingers of the first hand and the fingers of the second hand is analyzed to identify whether the texture of the first hand changes, and thus, a contact state between the fingers of the second hand and the first hand, a pressing state between the fingers of the second hand and the first hand, or a separation state between the fingers of the second hand and the first hand can be identified. AwayThe neural network model may further recognize multiple consecutive frames of an image including the first hand and the fingers of the second hand; and may identify that the fingers of the second hand are separated from the first hand based on the change in texture of the first hand in the multiple consecutive frames. For example, FIG. 9(a) is a schematic diagram of the fingers of the second hand touching the first hand. When the fingers of the second hand touch the first hand, the texture change of the first hand is small. FIG. 9(b) is a schematic diagram of the fingers of the second hand pressing the first hand. When the fingers of the second hand press the first hand, the texture of the first hand changes greatly. FIG. 9(c) is a schematic diagram of the fingers of the second hand separating from the first hand. FIG. 9(d) is a schematic diagram of the fingers of the second hand separating from the first hand. In this case, the fingers of the second hand are above the palm of the first hand, but the fingers of the second hand are not touching the palm of the first hand.

[0084] As shown in FIG. 7, the electronic device detects a distance between the fingers of the second hand and the first hand based on an image including the fingers of the first hand and the fingers of the second hand. Get away If so (executing S306), the electronic device uses a camera to take an image including the first hand (executing S702), obtains an image including the first hand, and executes S301; or obtains an image including the fingers of the first hand and the second hand, and executes S303.

[0085] It should be noted that when the fingers of the second hand touch or press the first hand, the color of the fingers of the second hand changes with the force used by the fingers of the second hand. For example, when the fingers of the second hand press the first hand, the fingernails of the second hand become white and light red. As another example, when the fingers of the second hand touch the first hand, the fingernails of the second hand become light red. In the process from the time when the fingers of the second hand touch the first hand to the time when the fingers of the second hand leave the first hand, the fingernails of the second hand also change. Therefore, when identifying that the fingers of the second hand are touching, pressing, and releasing the first hand, the neural network model can determine the contact state, pressing state, or separation state between the fingers of the second hand and the first hand based on the images including the fingers of the first hand and the fingers of the second hand and the optical flow of the fingers of the second hand. Get awayTherefore, the contact state, pressing state, or separation state between the fingers of the second hand and the first hand can be identified. Get away The accuracy of identification is further improved.

[0086] Optionally, the image input to the neural network model may be a sub-image including the fingers of the second hand and captured from an image including the fingers of the first hand and the second hand.

[0087] In addition, the neural network model may further output fingertip keypoints of the fingers of the second hand to determine the input text based on where the fingertip keypoints of the fingers of the second hand touch or press the first hand.

[0088] In this application, the contact, pressing, or separation between the fingers of a second hand and a first hand is Away It can be understood that the state relationship between the fingers of the second hand and the first hand and output in the inference process may be different for neural networks obtained through training with different training data classification methods in the neural network training process.

[0089] S305: Determine entered text based on where the fingers of the second hand touch or press on the first hand.

[0090] Before determining the input text based on the position where the fingers of the second hand touch or press the first hand, it is first determined whether the fingers of the second hand are in the key area. In some embodiments, the key area is assumed to be a polygonal area formed by a plurality of points. A ray is drawn horizontally using the points of the fingertips of the second hand, and the amount of intersections between the ray and the sides of the polygon is determined. If the amount of intersections between the ray and the sides of the polygon is even, it is determined that the fingers of the second hand are not in the key area; if the amount of intersections between the ray and the sides of the polygon is odd, it is determined that the fingers of the second hand are in the key area.

[0091] Furthermore, the position where the fingers of the second hand touch or press the first hand is determined based on the fingertip coordinates of the fingers of the second hand and the coordinates of the virtual keyboard; the input text is determined based on the position where the fingers of the second hand touch or press the first hand. The virtual keyboard can be projected into a two-dimensional coordinate system, and the two-dimensional coordinates can be set to the boundary points (key point 0, key point 1, key point 5, key point 9, key point 13, key point 17 of the first hand) and keys of the virtual keyboard. The fingers of the second hand touch or press the first hand to determine the two-dimensional coordinates of the fingers of the second hand, and the coordinates in the two-dimensional coordinate system of the virtual keyboard can be determined based on the two-dimensional coordinates of the fingers of the second hand to determine the input text. For example, as shown in FIG. 10, it is determined that the input text is the number 3 based on the coordinates of the fingers of the second hand and the position where the fingers of the second hand touch or press the first hand.

[0092] Furthermore, the VR device can establish a virtual model of the first hand and a virtual model of the fingers of the second hand; based on the virtual model of the first hand and the virtual model of the fingers of the second hand, the text input process in the virtual world can be restored by a computer vision rendering method. Alternatively, the AR device can combine the virtual model of the first hand and the virtual model of the fingers of the second hand with the real world, so that the user intuitively understands the text input process.

[0093] It should be understood that in this embodiment, the location where the virtual keyboard is projected is not limited. After the electronic device captures an image including another body part, such as an arm, the electronic device identifies the skeletal points of the other body part, and projects the virtual keyboard onto the other body part based on the skeletal points of the other body part.

[0094] Referring to Figures 11(a), 11(b) and 11(c), the following uses an example in which the electronic device is a mobile phone to explain the text input process based on the virtual keyboard.

[0095] The mobile phone takes a first image, the first image including a first hand 1101 (shown in FIG. 11(a)). The mobile phone identifies key points of the first hand and projects a virtual keyboard 1102 (shown in FIG. 11(b)) onto the first hand based on the key points of the first hand. The mobile phone takes a second image, the first image including a first hand 1101 (shown in FIG. 11(a)). The mobile phone identifies key points of the first hand and projects a virtual keyboard 1102 (shown in FIG. 11(b)) onto the first hand based on the key points of the first hand. 2 The image includes a first hand 1101 and fingers 1103 of a second hand (shown in FIG. 11(c)). The mobile phone identifies a contact or pressing state between the fingers 1103 of the second hand and the first hand 1101 based on the image including the first hand 1101 and fingers 1103 of the second hand. The mobile phone first determines that the fingers 1103 of the second hand are within a range of an area where the number 8 is located based on the position where the fingers 1103 of the second hand touch or press the first hand 1101, and determines the input text. For example, if the position where the fingers 1103 of the second hand touch or press the first hand 1101 shown in FIG. 11(c) is the position of the number 8, the input text is determined to be the number 8.

[0096] It can be understood that to implement the functions of the above embodiments, the electronic device includes corresponding hardware structures and / or corresponding software modules for performing each function. Those skilled in the art should easily recognize that, in combination with the units and method steps described in the embodiments disclosed in the present application, the present application can be implemented by hardware or a combination of hardware and computer software. Whether the functions are performed by hardware or by hardware driven by computer software depends on the specific application scenario and design constraints of the technical solution.

[0097] 12 is a schematic diagram of the structure of a text input device according to an embodiment of the present application. These text input devices can be configured to implement the functions of the electronic device in the above method embodiments, and thus can also obtain the beneficial effects of the above method embodiments. In this embodiment of the present application, the text input device can be the electronic device shown in FIG. 1, or a module (e.g., a chip) used in the electronic device.

[0098] As shown in FIG. 12, the text input device 1200 includes an image capture unit 1210, a processing unit 1220, a display unit 1230, and a storage unit 1240. The text input device 1200 is configured to implement the functions of the electronic device in the embodiment of the method shown in FIG. 3 and FIG. 7. The image capture unit 1210 is configured to capture an image including a first hand and an image including fingers of the first hand and a second hand. The storage unit 1240 is configured to store instructions executed by the processing unit 1220, store input data required by the processing unit 1220 to execute the instructions, or store data generated after the processing unit 1220 executes the instructions. The processing unit 1220 may perform the functions of the processor 110 of the electronic device shown in FIG. 1. Specifically, the processing unit 1220 is configured to obtain key points of the first hand based on the image including the first hand. The processing unit 1220 is further configured to project a virtual keyboard onto the first hand based on the key points of the first hand. For example, the virtual keys are projected onto a palm or a back of the hand. The processing unit 1220 is further configured to identify a contact or pressing state between the fingers of the second hand and the first hand based on the image including the fingers of the first hand and the fingers of the second hand. The processing unit 1220 is further configured to: determine the entered text based on a position where the fingers of the second hand touch or press the first hand when the fingers of the second hand and the first hand are in contact or pressing state.

[0099] In a possible implementation, when the processing unit 1220 is configured to identify a contact or pressing state between the fingers of the second hand and the first hand based on an image including the fingers of the first hand and the second hand, the processing unit 1220 is specifically configured to: analyze the image including the fingers of the first hand and the second hand using a pre-configured neural network model; identify a contact or pressing state between the fingers of the second hand and the first hand; and obtain fingertip key points of the fingers of the second hand.

[0100] In another possible implementation, when the processing unit 1220 is configured to identify a contact or pressing state between the fingers of the second hand and the first hand based on an image including the fingers of the first hand and the second hand, the processing unit 1220 is specifically configured to: analyze the image including the fingers of the first hand and the second hand and the optical flow of the fingers of the second hand using a pre-configured neural network model; identify a contact or pressing state between the fingers of the second hand and the first hand; and obtain fingertip key points of the fingers of the second hand.

[0101] In another possible implementation, when the processing unit 1220 is configured to determine the input text based on the position where the fingers of the second hand touch or press the first hand, the processing unit 1220 is specifically configured to: determine the position where the fingers of the second hand touch or press the first hand based on the fingertip coordinates of the fingers of the second hand and the coordinates of the virtual keyboard, and determine the input text based on the position where the fingers of the second hand touch or press the first hand.

[0102] In another possible implementation, when the processing unit 1220 is configured to project a virtual keyboard onto the first hand based on key points of the first hand, the processing unit 1220 is specifically configured to: identify an action of the first hand based on multiple frames of an image including the first hand; determine that the action of the first hand matches a pre-defined action; and project the virtual keyboard based on the key points of the first hand. The pre-defined action ranges from clenching a fist to extending a palm.

[0103] In another possible implementation, the processing unit 1220 further: determines a distance between the fingers of the second hand and the first hand based on an image including the fingers of the first hand and the fingers of the second hand. but separation What you are doing Identify; separate the fingers of the second hand from the first hand Is when, using the image capture unit, capture an image including the first hand, or capture an image including the fingers of the first hand and the second hand.

[0104] In another possible implementation, the processing unit 1220 is further configured to input a training set to the neural network model. The neural network model outputs a classification result. The training set includes a first type of training image, a second type of training image, a label of the first type of training image, and a label of the second type of training image. The first type of training image includes a plurality of images of the fingers of the second hand and the first hand in a contact or pressing state, and the second type of training image includes a plurality of images of the fingers of the second hand and the first hand in a separated state. Is The processing unit 1220 is further configured to: calculate a loss function value of a loss function based on the classification result; and obtain parameters of the neural network model when the loss function converges and the loss function value is less than or equal to a threshold.

[0105] The display unit 1230 is configured to display the first hand, the second hand, and the entered text determined by the processing unit 1220.

[0106] For a more detailed description of the processing unit 1220, please directly refer to the relevant descriptions of the method embodiments shown in Figures 3 and 7. The details will not be described again here.

[0107] The processor in the embodiments of the present application may be a Central Processing Unit (CPU), another general-purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), or a combination thereof. -It may be understood that the general purpose processor may be an ASIC (Application Specific Integrated Circuit), a Field Programmable Gate Array (FPGA), another programmable logic device, a transistor logic device, a hardware component, or any combination thereof. A general purpose processor may be a microprocessor, any conventional processor, or the like.

[0108] The method steps in the embodiments of the present application may be implemented in a hardware manner or in a manner of executing software instructions by a processor. The software instructions may include corresponding software modules. The software modules may be stored in a Random Access Memory (RAM), a flash memory, a Read-Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Programmable Read-Only Memory (EPROM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a register, a hard disk, a removable hard disk, a CD-ROM, or other forms of storage medium well known in the art. For example, the storage medium is coupled to the processor such that the processor can read information from the storage medium and write information to the storage medium. Indeed, the storage medium may be a component of the processor. The processor and the storage medium may be located in an ASIC. In addition, the ASIC may be located in a network device or a terminal device. Indeed, the processor and the storage medium may exist as separate components in the network device or the terminal device.

[0109] All or part of the above-mentioned embodiments may be implemented by software, hardware, firmware, or any combination thereof. When software is used to implement the embodiments, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer programs and instructions. When the computer program or instructions are loaded and executed in a computer, all or part of the procedures or functions in the embodiments of the present application are executed. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer program or instructions may be stored in a computer-readable storage medium or transmitted from a computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instructions may be transmitted from a website, computer, server, or data center to another website, computer, server, or data center by wire or wirelessly. The computer-readable storage medium may be any available medium accessible by a computer, or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media, such as floppy disks, hard disks, magnetic tapes, optical media, such as digital video discs (DVDs), or semiconductor media, such as solid state drives. - It may also be a solid-state drive (SSD).

[0110] In the embodiments of the present application, unless otherwise specified or there is a logical contradiction, the terms and / or descriptions between different embodiments are consistent and can refer to each other, and the technical features of different embodiments can be combined based on their internal logical relationships to form a new embodiment.

[0111] In this application, "at least one" means one or more, and "plurality" means two or more. The term "and / or" represents an association relationship between associated objects, and represents that three relationships may exist. For example, "A and / or B" may represent three cases: only A is present, both A and B are present, and only B is present, and A and B may be singular or plural. In the text description of this application, the character " / " represents an "or" relationship between associated objects. In the formulas of this application, the character " / " represents a "split" relationship between associated objects.

[0112] It can be understood that various numbers in the embodiments of the present application are merely used for distinction to facilitate description, and are not used to limit the scope of the embodiments of the present application. The sequence numbers of the above processes do not mean the execution sequence. The execution sequence of the processes should be determined based on the functions and internal logic of the processes.

Claims

1. 1. A method for inputting text based on a virtual keyboard, comprising: obtaining key points of the first hand based on a first image including the first hand; projecting a virtual keyboard onto the first hand based on the keypoints of the first hand; successively acquiring second images including the fingers of the first hand and a second hand; identifying a contact or pressing state between the fingers of the second hand and the first hand based on whether a color of the fingernail changes in the plurality of consecutively acquired second images; determining input text based on a location where the fingers of the second hand touch or press the first hand when the fingers of the second hand and the first hand are in the contact or pressing state; The step of identifying the touch state or the pressing state comprises: analyzing the second image and the optical flow of the fingers of the second hand using a pre-configured neural network model; identifying the contact or pressing state between the fingers of the second hand and the first hand based on the second image and the analysis of the optical flow; acquiring fingertip keypoints of the fingers of the second hand; method.

2. The step of projecting a virtual keyboard onto the first hand based on the key points of the first hand includes: projecting the virtual keyboard onto the palm of the first hand based on key points of the palm of the first hand. The method of claim 1.

3. The step of projecting a virtual keyboard onto the first hand based on the key points of the first hand includes: projecting the virtual keyboard onto the back of the first hand based on key points on the back of the first hand. The method of claim 1.

4. The step of determining the entered text comprises: determining the locations where the fingers of the second hand touch or press the first hand based on fingertip coordinates of the fingers of the second hand and coordinates of the virtual keyboard; determining the entered text based on the location where the fingers of the second hand touch or press on the first hand.

4. The method according to claim 1 .

5. The step of projecting the virtual keyboard onto the first hand includes: identifying an action of the first hand based on a plurality of frames of images including the first hand; determining that the action of the first hand matches a predefined action; and projecting the virtual keyboard based on the key points of the first hand.

5. The method according to any one of claims 1 to 4.

6. The predefined actions range from clenching a fist to extending a palm. The method according to claim 5.

7. After the step of acquiring the second image, the method further comprises: identifying, based on the second image, that the fingers of the second hand and the first hand are separate; acquiring the first image or the second image when the fingers of the second hand and the first hand are separated.

7. The method according to any one of claims 1 to 6.

8. The method may further comprise: inputting a training set into the neural network model, the neural network model outputting a classification result; the training set including a first type of training images, a second type of training images, labels of the first type of training images, and labels of the second type of training images; the first type of training images including a plurality of images of the fingers of the second hand and the first hand in the contact state or the pressing state, and the second type of training images including a plurality of images of the fingers of the second hand and the first hand separated; Calculating a loss function value of a loss function based on the classification result; and obtaining parameters of the neural network model when the loss function converges and the loss function value is less than or equal to a threshold.

8. The method according to any one of claims 1 to 7.

9. After the step of determining the entered text, the method further comprises: displaying the first hand, the fingers of the second hand, and the entered text.

9. The method according to any one of claims 1 to 8.

10. a processing unit configured to acquire key points of a first hand based on a first image including the first hand, a processing unit further configured to project a virtual keyboard onto the first hand based on the key points of the first hand; and an image capture unit configured to capture a second image including the fingers of the first hand and a second hand in succession; the processing unit is further configured to identify a contact or pressing state between the fingers of the second hand and the first hand based on whether a color of the fingernail changes in the plurality of consecutively acquired second images; The processing unit is further configured to: determine input text based on a location where the fingers of the second hand touch or press the first hand when the fingers of the second hand and the first hand are in the contact state or the pressing state; When the processing unit is configured to identify the touch state or the press state, the processing unit: Analyzing the second image and optical flow of the fingers of the second hand using a pre-configured neural network model; identifying the contact or pressing state between the fingers of the second hand and the first hand based on the second image and the analysis of the optical flow; acquiring fingertip keypoints of the fingers of the second hand; A text input device.

11. When the processing unit is configured to project the virtual keyboard onto the first hand, the processing unit: configured to project the virtual keyboard onto the palm of the first hand based on key points of the palm of the first hand. The text input device according to claim 10.

12. When the processing unit is configured to project the virtual keyboard onto the first hand, the processing unit: configured to project the virtual keyboard onto the back of the first hand based on key points on the back of the first hand. The text input device according to claim 10.

13. When the processing unit is configured to determine the entered text, the processing unit: determining the locations where the fingers of the second hand touch or press the first hand based on fingertip coordinates of the fingers of the second hand and coordinates of the virtual keyboard; determining the entered text based on the location where the fingers of the second hand touch or press on the first hand. A text input device according to any one of claims 10 to 12.

14. When the processing unit is configured to project the virtual keyboard onto the first hand, the processing unit: Identifying an action of the first hand based on a plurality of frames of images including the first hand; determining that the action of the first hand matches a predefined action; and projecting the virtual keyboard based on the key points of the first hand. A text input device according to any one of claims 10 to 13.

15. The predefined actions range from clenching a fist to extending a palm.

15. A text input device according to claim 14.

16. The processing unit further comprises: Identifying, based on the second image, that the fingers of the second hand and the first hand are separate; acquiring the first image or acquiring the second image using the image capture unit when the fingers of the second hand and the first hand are separated. A text input device according to any one of claims 10 to 15.

17. The processing unit further comprises: inputting a training set into the neural network model, the neural network model outputting a classification result; the training set including a first type of training images, a second type of training images, labels of the first type of training images, and labels of the second type of training images; the first type of training images including a plurality of images of the fingers of the second hand and the first hand in the contact state or the pressing state, and the second type of training images including a plurality of images of the fingers of the second hand and the first hand separated; Calculate a loss function value of a loss function based on the classification result; and obtain parameters of the neural network model when the loss function converges and the loss function value is less than or equal to a threshold. A text input device according to any one of claims 10 to 16.

18. the text input device further comprising a display unit; the display unit configured to display the first hand, the fingers of the second hand, and the input text. A text input device according to any one of claims 10 to 17.

19. 10. An electronic device, the electronic device having a processor, a memory, a display, and a camera; the processor is coupled to the display, the camera, and the memory; the memory is configured to store computer program code; the computer program code comprises computer software instructions; and when the computer software instructions are executed by the electronic device, the electronic device is enabled to perform a text input method based on a virtual keyboard according to any one of claims 1 to 9.

20. A computer readable storage medium containing computer software instructions, comprising: A computer-readable storage medium, wherein when the computer software instructions are executed on an electronic device, the electronic device is enabled to perform the virtual keyboard based text input method according to any one of claims 1 to 9.

21. 10. A computer program product, which when executed on a computer, enables the computer to perform a method for inputting text based on a virtual keyboard according to any one of claims 1 to 9.

Images