A virtual keyboard-based input method and device

Abstract

This application provides an input method based on a virtual keyboard, including detecting a first input operation applied to an operation area; mapping the first input operation to the operation of a corresponding key in a key display area to obtain a second input operation for the virtual keyboard, wherein the operation area and the key display area are located in different positions; obtaining an input event based on the second input operation; and responding to the input event by displaying the input information corresponding to the input event in an input box. The input method provided in this application allows the operation area and key display area of ​​the virtual keyboard to be set independently. When applied to widescreen devices, the operation area can be located in an area of ​​the touch screen that is more easily accessible to the user's hands, enabling the user to perform input operations in an area more easily accessible to both hands. The operation input is mapped to the virtual keyboard for input, while the key display area is centrally displayed, allowing the user's gaze to focus on the key display area, reducing the distance the user's gaze travels and improving the user experience.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an input method and device based on a virtual keyboard. Background Technology

[0002] With the development and evolution of mobile terminal devices, widescreen devices are gradually becoming the mainstream trend as users seek a larger interactive range and field of view. On widescreen devices, inputting information through a graphical interface (such as typing) remains one of the most common input technologies.

[0003] Currently, the mainstream graphical user interface (GUI) input technology uses a virtual keyboard (or soft keyboard, which displays a keyboard on the screen), allowing users to input text, symbols, or other information supported by the keyboard by touching virtual keys on the screen. Current virtual keyboard designs primarily mimic the form of physical keyboards, but offer greater customizability and scalability compared to physical keyboards.

[0004] When virtual keyboards are applied to widescreen devices, the larger horizontal dimensions of these devices make it difficult for users to reach the keys in the middle of the keyboard when holding the device with both hands to input information. This results in unnatural, inflexible, and inefficient interaction. Summary of the Invention

[0005] This application provides an input method based on a virtual keyboard, which, when applied to widescreen devices, solves the interaction problem between the virtual keyboard and the user on widescreen devices, improves input efficiency, and enhances the user experience.

[0006] In a first aspect, this application provides an input method based on a virtual keyboard, comprising displaying an interactive interface of an electronic device, the interactive interface including an input box and a virtual keyboard, the virtual keyboard including a key display area and an operation area, wherein the key display area and the operation area are located in different positions; detecting a first input operation acting on the operation area; mapping the first input operation to the operation of a corresponding key in the key display area to obtain a second input operation for the virtual keyboard; obtaining an input event based on the second input operation; and responding to the input event by displaying the input information corresponding to the input event in the input box.

[0007] The virtual keyboard-based input method provided in this application sets the operation area and the key display area independently. When applied to widescreen devices, the operation area can be located in an area of ​​the touch screen that is more easily accessible to the user's hands, allowing the user to perform input operations in the operation area that is more easily accessible to both hands. The operation input is mapped to the virtual keyboard to realize input, and the key display area is centrally displayed, so that the user's eyes can be focused on the key display area, reducing the distance the user's eyes move and improving the user experience.

[0008] In one possible implementation, the first input operation includes a click operation; mapping the first input operation to the corresponding key operation in the key display area to obtain the second input operation for the virtual keyboard is specifically implemented as follows: obtaining the click position of the click operation in the operation area; mapping the click position to the key display area to obtain the mapped position; and obtaining the click operation for the target key in the virtual keyboard based on the mapped position.

[0009] In this possible implementation, the user's click position in the operation area is mapped to obtain the mapped position of the key display area. Then, the target key corresponding to the mapped position is found. In this way, the user can click on the target key in the virtual keyboard. That is to say, the user does not need to hit the key in the virtual keyboard. By inputting in the operation area that is more easily accessible to the user, the user can input the virtual keyboard, which removes the restriction of the virtual keyboard on both hands and greatly improves the input efficiency.

[0010] In another possible implementation, the click location is mapped to a key display area. A specific implementation of obtaining the mapped location is as follows: the click location is used as the input of a neural network model, and the output is the mapped location; or, the click location and the position information of each key in the virtual keyboard are used as the input of the neural network, and the output is the mapped location; or, the click location and the motion information of the touch object are used as the input of the neural network, and the output is the mapped location.

[0011] By establishing a mapping relationship between the coordinates of each position in the operation area and the coordinates of each position in the key display area of ​​the virtual keyboard through a neural network model, the mapping position on the virtual keyboard corresponding to the user's click position in the operation area can be quickly and accurately determined.

[0012] In another possible implementation, the first input operation includes a click operation, preceded by several historical click operations; the first input operation is mapped to the corresponding key operation in the key display area to obtain a second input operation for the virtual keyboard, preceded by: obtaining information on several historical click operations; and adjusting the click position of the current click operation based on the information on several historical click operations.

[0013] By using historical click operation information, the click position is corrected to prevent situations where the calculated mapping position is at the edge of the virtual key and the input information cannot be triggered.

[0014] In another possible implementation, the number of historical click operations includes the timing information of each historical click operation and the click position information of each historical click operation.

[0015] In another possible implementation, the virtual keyboard-based input method provided in this application further includes: responding to a second input operation by generating a feedback event corresponding to the second input operation in the key display area.

[0016] In one example, the second input action includes the target key being clicked, and the feedback event includes one or more of the following: the target key being highlighted, the target key vibrating, and the target key being pressed. In this way, the feedback event notifies the user that the target key has been successfully clicked, thereby improving the user's input experience.

[0017] In another possible implementation, the user performs input operations on the operation area using a touch object; the input method based on a virtual keyboard provided in this application further includes: obtaining the spatial position of the touch object; obtaining the target position of the touch object in the key display area based on the spatial position of the touch object and the mapping relationship, wherein the mapping relationship indicates the mapping relationship between each spatial position of the touch object and each position in the key display area; and displaying a preset style icon at the target position in the key display area.

[0018] For example, when a user hovers their finger over a certain position in the operation area, a preset style icon is displayed on the "A" key in the key display area. When the user's finger moves and hovers over another position in the operation area, a preset style icon is displayed on the "C" key in the key display area. The preset style icon indicates the target key of the virtual keyboard corresponding to the current position of the user's finger, making it easier for the user to interact with the virtual keyboard through the operation area.

[0019] It should be noted that the preset style icon is used to identify the mapping position on the key display area by the user's current operation in the operation area. The preset style icon can be any style icon. That is to say, the shape, fill effect, outline effect, color, transparency, texture and other visualization effect parameters of the preset style icon can be adjusted and combined arbitrarily. For example, different preset styles can be circular icons, square icons, cylindrical icons, etc. This application does not limit this.

[0020] In another possible implementation, the virtual keyboard-based input method provided in this application further includes: acquiring image information and / or capacitive signal distribution information and / or depth information distribution information of the touch body; and visually rendering and displaying the touch body at a target position in the key display area based on the image information and / or capacitive signal distribution information and / or depth information distribution information of the touch body.

[0021] In this possible implementation, the user's interaction with the virtual keyboard is visualized to further enhance the input experience.

[0022] Optionally, the touchpad can be the user's finger.

[0023] In another possible implementation, the operation area includes a first operation area and a second operation area; the key display area is located in the center of the touch display screen, and the first operation area and the second operation area are located on both sides of the key display area; or, the key display area is on one side of the touch display screen, the first operation area is on the other side of the touch display screen, and the second operation area overlaps with the key display area.

[0024] In this possible implementation, the first and second operation areas are distributed on both sides of the touch screen, making it easy for users to touch the widescreen device with their fingers when holding it with both hands. Input operations are performed in the operation areas, and the virtual keyboard keys are displayed in the center, so that the user's eyes can focus on the key display area, reducing the distance the eyes need to move and improving the user experience.

[0025] In another possible implementation, the operation area and the key display area are located on different touch screens; for example, in the field of multi-screen interaction or XR, in a multi-screen interaction scenario, a user can type information on one of the touch screens and then focus their gaze on the other screens to obtain feedback information on the typing input; in an XR interaction scenario, a user can type information on an additional touch screen device and then observe the interaction results on a head-mounted display to obtain typing-related visual feedback.

[0026] Alternatively, the operation area and the key display area may be located on different devices; for example, in the field of multi-device collaboration, users can link two or more devices, then perform typing operations on one device, and then observe the feedback effect of the typed information on the other device.

[0027] Of course, in addition to typing scenarios, the virtual keyboard-based input method provided in this application can also be applied to any situation where gesture information is obtained using a touch screen, such as any gesture operation scenario such as creation, editing, system operation, etc.

[0028] Secondly, this application also provides an input device based on a virtual keyboard, including a display module, a detection module, a mapping module, an acquisition module, and a response module. The display module displays the interactive interface of an electronic device, which includes an input box and a virtual keyboard. The virtual keyboard includes a key display area and an operation area. The detection module detects a first input operation applied to the operation area. The mapping module maps the first input operation to the operation of a corresponding key in the key display area, obtaining a second input operation on the virtual keyboard. The operation area and the key display area are located in different positions. The acquisition module obtains an input event based on the second input operation. The response module responds to the input event and displays the input information corresponding to the input event in the input box.

[0029] In one possible implementation, the first input operation includes a click operation; the mapping module is specifically used to obtain the click position of the click operation in the operation area; map the click position to the key display area to obtain the mapped position; and based on the mapped position, obtain the click operation for the target key in the virtual keyboard.

[0030] In another possible implementation, mapping the click location to the key display area to obtain the mapped location can be achieved by: using the click location as input to a neural network model and outputting the mapped location; or using the click location and the position information of each key in the virtual keyboard as input to a neural network and outputting the mapped location; or using the click location and the motion information of the touch object as input to a neural network and outputting the mapped location.

[0031] In another possible implementation, the first input operation includes a click operation, and the click operation is preceded by several historical click operations; the input device based on the virtual keyboard provided in this application further includes: an adjustment module; the adjustment module is used to acquire information of several historical click operations; and adjust the click position of the current click operation based on the information of several historical click operations.

[0032] In another possible implementation, the number of historical click operations includes the timing information of each historical click operation and the click position information of each historical click operation.

[0033] In another possible implementation, the response module is also used to: respond to the second input operation and generate a feedback event corresponding to the second input operation in the key display area.

[0034] In another possible implementation, the second input operation includes the target button being clicked, and the feedback event includes one or more of the following: the target button being highlighted, the target button vibrating, and the target button displaying that it has been pressed.

[0035] In another possible implementation, the user performs input operations on the operation area via a touch object; the input device based on a virtual keyboard provided in this application further includes: an acquisition module for acquiring the spatial position of the touch object; an acquisition module for obtaining the target position of the touch object in the key display area based on the spatial position of the touch object and the mapping relationship, wherein the mapping relationship indicates the mapping relationship between each spatial position of the touch object and each position in the key display area; and a display module for displaying a preset style icon at the target position in the key display area.

[0036] In another possible implementation, the acquisition module is further configured to acquire image information and / or capacitance signal distribution information and / or depth information distribution information of the touch body; the display module is further configured to, based on the image information and / or capacitance signal distribution information and / or depth information distribution information of the touch body, visualize and render the touch body at a target position in the key display area of ​​the virtual keyboard.

[0037] Optionally, the touchpad can be the user's finger.

[0038] In another possible implementation, the operating area includes a first operating area and a second operating area; the key display area is located in the center of the touch screen, and the first and second operating areas are located on either side of the key display area; or, the key display area is on one side of the touch screen, the first operating area is on the other side of the touch screen, and the second operating area overlaps with the key display area.

[0039] In another possible implementation, the operation area and the key display area are located on different touch screens; or, the operation area and the key display area are located on different devices.

[0040] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores instructions that, when executed by the processor, cause the method described in the first aspect to be implemented.

[0041] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the method described in the first aspect to be implemented.

[0042] Fifthly, embodiments of this application also provide a computer program or computer program product, the computer program or computer program product including instructions that, when executed, cause a computer to perform the method described in the first aspect.

[0043] In a sixth aspect, embodiments of this application also provide a chip including at least one processor and a communication interface, wherein the processor is used to execute the method described in the first aspect. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of a separate virtual keyboard on a display screen;

[0045] Figure 2 This is a schematic diagram of an adaptive magnifying virtual keyboard on a display screen;

[0046] Figure 3 This is a schematic diagram of a floating virtual keyboard on a display screen.

[0047] Figure 4 This is a schematic diagram showing that the operating area and the key display area are consistent.

[0048] Figure 5 A schematic diagram illustrating an application scenario of an input method based on a virtual keyboard provided in an embodiment of this application is shown.

[0049] Figure 6 A flowchart illustrating the input method based on a virtual keyboard provided in an embodiment of this application;

[0050] Figure 7 A schematic diagram showing the distribution of the operation area and the virtual keyboard display area on the screen is shown;

[0051] Figure 8 A schematic diagram showing the distribution of the operation area and key display area on the display screen is shown.

[0052] Figure 9 This diagram illustrates another distribution of the operation area and key display area on the screen.

[0053] Figure 10 and Figure 11 The diagrams show how a widescreen device is placed on a support surface, and how a user interacts with the display screen using both hands.

[0054] Figure 12 This illustrates a mapping relationship between position coordinates in the operating area and position coordinates in the key display area;

[0055] Figure 13 This diagram illustrates multiple key points captured by the screen sensor during typing.

[0056] Figure 14 for Figure 13 A magnified view of a portion of the image;

[0057] Figures 15 to 17 Several style examples of preset style icons are shown for identifying the mapped positions on the key display area of ​​the virtual keyboard;

[0058] Figure 18 A schematic diagram of the structure of an input device based on a virtual keyboard provided in an embodiment of this application;

[0059] Figure 19 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0060] The term "and / or" used in this article describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0061] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first input operation" and "second input operation," etc., are used to distinguish different input operations, not to describe a specific order of input operations.

[0062] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0063] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0064] When users interact with the virtual keyboard on current widescreen devices, there are currently three main ways to arrange the virtual keyboard in current technology:

[0065] Split Design: This design distributes the main interface of the virtual keyboard on both sides of the screen. The left and right ends of the virtual keyboard have high-frequency interactive / operable virtual buttons, making it convenient for users to hold and input information. Figure 1 As shown.

[0066] Adaptive scaling: This design aligns the width of the virtual keyboard with the main screen of the widescreen device, while adjusting the display height of the virtual keyboard proportionally. The virtual key layout fills the entire virtual keyboard interface, such as... Figure 2 As shown.

[0067] Floating: This design typically displays the virtual keyboard proportionally centered in the device during initialization, but users can move the virtual keyboard to adjust its position for easier input. Figure 3 As shown, this type of keyboard displays information in a relatively focused manner, so the user's gaze does not have to span the entire screen.

[0068] With current technology, when users interact with these virtual keyboards, they all need to directly interact with the virtual keys, i.e., press the corresponding virtual key; otherwise, the virtual keyboard cannot provide feedback on the corresponding input information. Figure 4 As shown.

[0069] When users input using the virtual keyboard designed above on widescreen devices (such as foldable computers, foldable phones, tablets, etc.), there will be a conflict between the control area of ​​their hands and the central area of ​​their line of sight, which will lead to:

[0070] Users using Figure 1 and Figure 2 When using the virtual keyboard shown, although there is enough space for both hands to operate, the eyes need to keep an eye on the entire virtual keyboard interface.

[0071] Users using Figure 3 When using the virtual keyboard shown, although the virtual keyboard occupies a small area of ​​the interface and improves the space utilization of the interface, the space for hand interaction is limited, resulting in unnatural and inflexible interaction and reduced input efficiency.

[0072] One proposed solution provides a virtual keyboard that can predict keystroke positions based on typing behavior. This technology allows the keyboard to adapt to the user's input habits, thereby improving input efficiency.

[0073] This solution collects user typing behavior information, including but not limited to typing pattern information, previous keystroke behavior information, details of the virtual keyboard used for typing, and contextual parameter information related to the device and user interaction with the displayed virtual keyboard. It then extracts and fuses multiple features to obtain multiple feature vectors, trains a neural network to identify known features, and learns positional features that affect typing behavior, thereby predicting the corresponding user's virtual keyboard keystroke position.

[0074] This approach requires a large amount of sample data, and data collection and model training take a long time to accumulate. Moreover, the trained model lacks universality, resulting in poor versatility of the virtual keyboard formed by the fusion training model.

[0075] To address the aforementioned issues, this application provides an input method based on a virtual keyboard. The operation area (hereinafter referred to as the operation area for ease of description) and the key display area of ​​the virtual keyboard are set independently. When applied to widescreen devices, the operation area can be located in an area of ​​the touch screen that is more easily accessible to the user's hands, allowing the user to perform input operations in the operation area that is more easily accessible to both hands. The input operation is mapped to the virtual keyboard for input, and the key display area is centrally displayed, allowing the user's gaze to be focused on the key display area, reducing the distance the user's gaze moves and improving the user experience.

[0076] Figure 5 A schematic diagram illustrating an application scenario of an input method based on a virtual keyboard provided in an embodiment of this application is shown. For example... Figure 5 As shown, users can hold the widescreen device with both hands, and use their thumbs as touchpads to interact with the touchscreen to perform input operations.

[0077] The widescreen device mentioned in the embodiments of this application can be understood as a device with a large screen width, such as a foldable screen computer, a dual-screen computer, a foldable screen phone, a widescreen phone, a widescreen tablet, etc.

[0078] certainly, Figure 5 The illustration shown is merely one application scenario example of the virtual keyboard-based input method provided in this application embodiment, and does not constitute a limitation on the embodiments of this application. The input method provided in this application embodiment can also be applied to other application scenarios. For example, a widescreen device is placed on a supporting object, and the user's hands act as touch objects to interact with the touchscreen to achieve input operations. Another example is its application in multi-screen interaction scenarios, where a user can type information on one touchscreen and then focus their gaze on other screens to obtain typing input feedback. Yet another example is its application in XR interaction scenarios, where a user can type information on an additional touchscreen device and then observe the interaction results on a head-mounted display to obtain typing-related visual feedback. Yet another example is its application in multi-device collaboration scenarios, where a user can associate two or more devices, perform typing operations on one device, and then observe the typing feedback effect on another device. Furthermore, the touch object can also be other objects that can interact with the touchscreen, such as a stylus.

[0079] Of course, in addition to typing scenarios, the input method based on a virtual keyboard provided in this application embodiment can also be applied to any situation where gesture information is obtained using a touch screen, such as any gesture operation scenario such as creation, editing, system operation, etc.

[0080] The following uses the typing scenario on a widescreen device as an example to introduce the detailed implementation of the input method based on a virtual keyboard provided in this application embodiment. The implementation of other scenarios is similar. For specific implementation, please refer to the detailed implementation introduction of the typing scenario on a widescreen device. For the sake of brevity, this application embodiment will not repeat the details.

[0081] Figure 6 This is a flowchart illustrating a virtual keyboard-based input method provided in an embodiment of this application. This method can be applied to any widescreen device, achieving a user-friendly typing interaction, reducing the distance the user's eye moves, and improving the user's typing input experience. Figure 6 As shown, the input method based on a virtual keyboard provided in this application embodiment includes at least steps S601 to S604.

[0082] In step S601, the interactive interface of the electronic device is displayed.

[0083] When a user needs to input information, they perform an action on the touchscreen display (e.g., tapping an input box) to bring up a virtual keyboard for input. For example, the electronic device detects the user's tap on the input box and displays an input interface, which includes an input box and a virtual keyboard. The virtual keyboard includes a key display area and an operation area, wherein the key display area and the operation area are located in different positions on the touchscreen display.

[0084] Figure 7 A schematic diagram of an input interface is shown. For example... Figure 7 As shown, the virtual keyboard's key display area is centered on the screen, while the operation area is located on both sides of the key display area, close to the edge of the screen, making it easy for fingers to reach and input.

[0085] In step S602, a first input operation acting on the operation area is detected.

[0086] Users input data through the operation area on the touchscreen of a widescreen device, and the touchscreen detects the user's input through sensors within the operation area.

[0087] The input operation can be an operation in which the touch object (such as the user's finger) directly contacts the touch screen, or it can be an input operation performed near the top of the touch screen without direct contact with the touch screen.

[0088] For example, when a user taps a location in the operation area of ​​the touchscreen with their finger, the sensor on the touchscreen detects that the location has been tapped and generates a corresponding signal to inform the processor that a tap has been detected on the screen.

[0089] The sensor on the touchscreen can be any sensor capable of performing the functions required in the embodiments of this application. For example, the sensor on the touchscreen can be a pressure sensor, which senses pressure signals and converts them into electrical signals. There are many types of pressure sensors, such as resistive pressure sensors and inductive pressure sensors. When a touch operation is applied to the touchscreen display, the widescreen device detects the intensity of the touch operation based on the pressure sensor. The widescreen device can also calculate the touch position based on the detection signal from the pressure sensor. In some examples, touch operations applied to the same touch position but with different intensities can correspond to different operation commands. For example, when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS message is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS message is executed.

[0090] For example, the sensors on a touchscreen can also be touch sensors (such as capacitive touch sensors). Touch sensors detect touch operations applied to or near them. The touch sensor can then transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen.

[0091] In step S603, the first input operation is mapped to the operation of the corresponding key in the key display area to obtain the second input operation for the virtual keyboard.

[0092] After detecting the first input operation applied to the operation area, the input operation is mapped and converted into a second input operation for the virtual keyboard, so that information can be input without hitting the virtual keyboard.

[0093] The operation area can be set in a position that is easily accessible to the user's fingers, so that the user can perform input operations in the operation area; the key display area can be set in a certain area of ​​the display screen to reduce the user's eye movement.

[0094] Figure 8 A schematic diagram showing the distribution of the operation area and the virtual key display area on the screen is shown. For example... Figure 8 As shown, the operation area includes a first operation area and a second operation area. The key display area is centered on the screen, with the first and second operation areas located on either side of the key display area. The first and second operation areas are positioned close to the sides of the screen, facilitating input via thumbs when the user holds the device with both hands. The centered virtual keyboard eliminates the need for the user's gaze to traverse the entire screen, reducing eye movement and making information input more efficient.

[0095] The key display area refers to the area where each key on the virtual keyboard is displayed.

[0096] Figure 9 This diagram illustrates another distribution of the operating area and key display area on the screen. (Example) Figure 9 As shown, the key display area is located on the right side of the screen, one operation area is located on the left side of the screen, and another operation area is located on the right side of the screen, partially overlapping with the key display area. Thus, only the mapping between the operation area on the left and the virtual keyboard needs to be established; the operation area on the right can input information by directly pressing the target key on the virtual keyboard.

[0097] In one example, the key display area and the operation area can use the same background color in the input interface. This background color should be different from the background color of other areas in the input interface to distinguish them and make it easier for users to find the key display area and the operation area. Optionally, the operation area of ​​the virtual keyboard can also include prompts to guide the user to perform input operations in this area. For example, the operation area of ​​the virtual keyboard may display the prompt text "Please perform input operations here," or display a prompt gesture "A rendered virtual finger" to guide the user to perform input operations in this area.

[0098] In some other examples, the widescreen device can be placed on a support surface, allowing users to input data using the five fingers of both hands on the input area, further increasing input efficiency. Figure 10 and Figure 11 As shown.

[0099] There are several ways to map the operations of the operation area to the virtual keyboard.

[0100] For example, a mapping table can be created by statistically analyzing the mapping relationship between the coordinates of each position in the operation area and the coordinates of each position in the key display area. This mapping table records the mapping relationship between the coordinates of each position in the operation area and the coordinates of each position in the key display area.

[0101] When a user taps a location in the operating area, the sensor in the touchscreen detects the coordinates of the tapped location and retrieves the corresponding key display area coordinates by looking up a mapping table.

[0102] For example, by determining the mapping formula between the coordinates of each position in the operation area and the coordinates of each position in the key display area, the coordinates of the position corresponding to the user's operation in the operation area can be substituted into the mapping formula to calculate the mapped position coordinates in the key display area.

[0103] In one example, the mapping formula could be y = ax + b, mapping the user's actual trigger position to the target position on the virtual keyboard. Figure 8 For example, the mapping formula corresponding to the left operating region is y = 0.85x + 0.45, and the mapping formula corresponding to the right operating region is y = 0.85x - 0.2.

[0104] For example, the ranges of parameters a and b in the mapping formula y = ax + b in the left operation region are: a ∈ [0.5, 1.5], b ∈ (0, 1); the ranges of parameters a and b in the mapping formula y = ax + b in the right operation region are: a ∈ [0.5, 1.5], b ∈ (-1, 0).

[0105] For example, a mapping relationship between the coordinates of each position in the operation area and the coordinates of each position in the key display area can be established using neural networks. Figure 12 This illustrates a mapping relationship between the position coordinates in the operation area and the position coordinates in the key display area.

[0106] In one example, the neural network can take only the coordinates of the actual position the user triggers in the operating area as input and output the mapped position coordinates in the virtual keyboard.

[0107] In another example, for a specific virtual keyboard, the neural network's input can also include the actual coordinates of the user's touch points in the operating area and the position information of each virtual key on the virtual keyboard, outputting the mapped position coordinates on the virtual keyboard. By training corresponding neural networks for different virtual keyboard layouts, the prediction of mapped position coordinates on the virtual keyboard can be made more accurate.

[0108] In another example, the neural network's input can also include the actual coordinates of the user's touch location within the operating area and the motion information of the touch object (such as velocity and acceleration), outputting the mapped position coordinates in the virtual keyboard. By combining the touch object's motion information, the mapped position coordinates in the virtual keyboard can be predicted more accurately.

[0109] In another example, the input to the neural network may also include the actual position coordinates of the user's touch in the operation area, the position information of each virtual key in the virtual keyboard, and the motion information of the touch object (such as speed and acceleration), and the output is the mapped position coordinates in the virtual keyboard.

[0110] Understandably, the training samples for a neural network will differ depending on its input. For example, in the first example, the neural network's input only includes the coordinates of the user's actual trigger position. The training samples for this network include the coordinates of the user's trigger position within the operating area and the label of the sample, which is the mapped coordinates of the key display area. In the second example, the neural network's input includes the coordinates of the user's actual trigger position within the operating area and the position information of each virtual key on the virtual keyboard. The training samples for this network also include the coordinates of the user's trigger position within the operating area, the position information of each virtual key on the virtual keyboard, and the label of the sample, which is the mapped coordinates of the key display area of ​​the virtual keyboard.

[0111] Optionally, the neural network can be a multilayer perceptron (MLP) neural network.

[0112] After obtaining the mapping position of the actual trigger position on the virtual keyboard, the click operation for the target key on the virtual keyboard is obtained based on the mapping position.

[0113] For example, if the actual trigger location is mapped to the position of the "S" key on the virtual keyboard, then the operation of clicking the "S" key will be obtained.

[0114] In some other examples, to avoid the mapping position corresponding to the input operation falling on the edge of the target key or in the middle of two keys, thus preventing the input information from being triggered, the input operation could be as follows: the user clicks point a1 in the operation area with their finger. Point a1 maps to the middle position between the key "a" and key "s" in the virtual keyboard's key display area. In this case, since it cannot be determined whether the "a" key or the "s" key is triggered, the input information cannot be triggered. This application embodiment also proposes a landing point correction mechanism, that is, adjusting the user's input operation in the operation area before executing step S603.

[0115] Taking typing as an example, users input information by clicking multiple times in the operation area. When the landing point (also known as the click position) of the current click is determined, the click position is adjusted based on information from previous click operations. Optionally, the previous click operations include the sequence information of each previous click operation and the click position information of each previous click operation.

[0116] Regarding the user's typing behavior, the information received by the display's sensor module is actually a series of time-correlated keystroke information. The trajectory of the information typed by the user during actual typing may deviate due to different preceding and following keys associated with the current virtual key. For example, if a key above key "A" (such as key "Q") is pressed before key "A", the keystroke will land slightly below key "A" (e.g., key "Q"). Figure 13 and Figure 14 The "plus sign" indicates the landing point. If you press a key below key "A" (e.g., key "Z") before pressing key "A", the landing point will fall slightly above key "A" (e.g., key "Z"). Figure 13 and Figure 14 (The asterisk in the diagram indicates the landing point).

[0117] In such cases, the actual landing point information collected by the sensor is combined with the timing information f(pos_t,…,pos_0) for correction, thereby preventing the input information from failing to be triggered because the calculated mapped point is at the edge of the virtual button. For example, if the landing point of the previous click operation is above the landing point of the current click operation, the landing point of the current click operation is adjusted upwards to correct the deviation; if the landing point of the previous click operation is below the landing point of the current click operation, the landing point of the current click operation is adjusted downwards to correct the deviation.

[0118] In step S604, an input event is obtained based on the second input operation.

[0119] After receiving the user's second action on the virtual keyboard, an input event is generated for the virtual keyboard. For example, if the second action on the virtual keyboard is a click on the keys "H", "U", "A", "W", "E", and "I" in sequence, then the input event will be the input of "H", "U", "A", "W", "E", and "I" in sequence.

[0120] In step S605, in response to the input event, the input information corresponding to the input event is displayed in the input box.

[0121] After the input event is determined, respond to the input event and display the input information in the input box, such as "HUAWEI".

[0122] In some other examples, the method provided in this application embodiment may further include: responding to a second input operation mapped to the virtual keyboard during the user's information input through the operation area. That is, in response to the second input operation, generating a feedback event corresponding to the second input operation in the key display area of ​​the virtual keyboard.

[0123] Optionally, the second input operation includes the target key being clicked, and the feedback event includes one or more of the following: the target key being highlighted, the target key vibrating, the target key being pressed, and the actual touch point being displayed on the target key. In this way, the user is notified that the target key has been successfully clicked through the feedback event, and the user input experience is improved.

[0124] In some other examples, during the process of a user inputting information through the operation area, the position of the user's finger is also mapped in real time to the key display area of ​​the virtual keyboard, so that the user knows that the current position of the finger in the operation area corresponds to the position on the virtual keyboard.

[0125] For example, a user performs input operations by using their finger in the operation area; the spatial position of the finger is obtained by a sensor (e.g., the spatial position of the finger can be detected by a capacitive sensor); based on the spatial position of the finger and the mapping relationship, the target position of the finger in the key display area of ​​the virtual keyboard is obtained, and the mapping relationship indicates the mapping relationship between each spatial position of the finger and each position in the key display area of ​​the virtual keyboard; a preset style icon is displayed at the target position in the key display area of ​​the virtual keyboard.

[0126] For example, when a user hovers their finger over a certain position in the operation area, the virtual keyboard's key display area shows a preset style icon on the "A" key. When the user's finger moves and hovers over another position in the operation area, the virtual keyboard's key display area shows a preset style icon on the "C" key. The preset style icon indicates the target key on the virtual keyboard corresponding to the current position of the user's finger, making it easier for the user to interact with the virtual keyboard through the operation area.

[0127] It should be noted that the preset style icon is used to identify the mapped position of the user's current operation in the operation area on the virtual keyboard's key display area. The preset style icon can be any style icon, meaning that various visual effect parameters such as the shape, fill effect, outline effect, color, transparency, and texture of the preset style icon can be adjusted and combined arbitrarily. For example, different preset styles can be used for circular icons (see...). Figure 15 ), square icon, cylindrical icon (see Figure 16 Preset styles can also be represented by pictographic graphics. For example, if the touchpad is a user's finger, the preset style image can be a fingerprint (see...). Figure 17 This application does not limit the specific implementation method of the preset style icons.

[0128] In another example, during user input, the process also includes visually mapping the user's finger onto the virtual keyboard's key display area. For instance, image information and / or capacitance signal distribution information and / or depth information of the user's finger are acquired; a two-dimensional image of the user's finger is obtained based on the image information and / or capacitance signal distribution information and / or depth information of the touch surface; and this two-dimensional image of the user's finger is then visualized and rendered at a target position in the virtual keyboard's key display area. Specifically, RGB image information of the user's finger is acquired using a camera on a widescreen device; a two-dimensional image of the user is obtained based on this RGB image information; and this two-dimensional image of the user's finger is rendered and displayed at a target position in the virtual keyboard's key display area (i.e., the position of the finger in the operating area is mapped to the position in the virtual keyboard's key display area). This allows the user to see in real-time the mapping position of their finger in the operating area onto the virtual keyboard's key display area as their finger moves in the operating area.

[0129] Alternatively, the rendering and display of the user's finger image can be achieved in other ways, such as by detecting the distribution of capacitive signals and / or depth information of the finger through sensors on the touch screen, and then using this distribution information to render and display the finger image at the target position of the virtual keyboard's key display area (i.e., the position of the finger in the operation area is mapped to the position of the virtual keyboard's key display area).

[0130] It should be explained that, in the context of the embodiments of this application, the meaning of the operating area refers to a certain area on the surface of the touch screen and the area above that area, for example... Figure 8 The operation area shown includes the portion defined by the operation area on the touchscreen, and the space above that portion. When a user's finger enters the operation area, the sensors on the device begin to collect the finger's movements, such as movement, clicking, and gestures within the operation area. The finger is then visually mapped onto the virtual keyboard's key display area, allowing the user to understand which key position on the virtual keyboard corresponds to the finger's movement within the operation area. For example, if the finger moves between points a1 and c1 in the operation area, the virtual keyboard's key display area will show the finger moving between keys A and C. When the finger moves to point c1 and clicks on the touchscreen surface at point c1, the virtual keyboard's key display area will show that key C has been clicked. Key C will generate a feedback event (e.g., highlighting, vibration) to notify the user that key C has been clicked. In response to this key click event, the user inputs the character "C" and displays the character "C" in the input box of the virtual keyboard's display area.

[0131] Based on the same concept as the aforementioned embodiment of an input method based on a virtual keyboard, this application also provides an input device 1800 based on a virtual keyboard. This input device 1800 can be deployed on electronic devices to achieve a more user-friendly input interaction on widescreen devices. The input device 1800 includes components for implementing... Figure 5-17 The units or modules of each step in the input method shown.

[0132] Figure 18 This is a schematic diagram of the structure of an input device based on a virtual keyboard, provided as an embodiment of this application. Figure 18 As shown, the virtual keyboard-based input device 1800 includes at least a display module 1801, a detection module 1802, a mapping module 1803, an acquisition module 1804, and a response module 1805. The display module 1801 displays the interactive interface of the electronic device, which includes an input box and a virtual keyboard. The virtual keyboard includes a key display area and an operation area, wherein the key display area and the operation area are located in different positions. The detection module 1802 detects a first input operation applied to the operation area. The mapping module 1803 maps the first input operation to the operation of a corresponding key in the key display area of ​​the virtual keyboard, thus obtaining a second input operation for the virtual keyboard. The acquisition module 1804 obtains an input event based on the second input operation. The response module 1805 responds to the input event by displaying the input information corresponding to the input event in the input box.

[0133] In one possible implementation, the first input operation includes a click operation; the mapping module 1803 is specifically used to obtain the click position of the click operation in the operation area; map the click position to the key display area of ​​the virtual keyboard to obtain the mapped position; and based on the mapped position, obtain the click operation for the target key in the virtual keyboard.

[0134] In another possible implementation, the click location is mapped to the key display area of ​​the virtual keyboard. A specific implementation of obtaining the mapped location is as follows: the click location is used as the input of a neural network model, and the output is the mapped location; or, the click location and the position information of each key in the virtual keyboard are used as the input of the neural network, and the output is the mapped location; or, the click location and the motion information of the touch object are used as the input of the neural network, and the output is the mapped location.

[0135] In another possible implementation, the first input operation includes a click operation, and several historical click operations are included before the click operation; the virtual keyboard-based input device 1800 provided in this application also includes: an adjustment module 1806; the adjustment module is used to acquire information of several historical click operations; and adjust the click position of the current click operation based on the information of several historical click operations.

[0136] In another possible implementation, the number of historical click operations includes the timing information of each historical click operation and the click position information of each historical click operation.

[0137] In another possible implementation, the response module 1805 is also used to: respond to the second input operation and generate a feedback event corresponding to the second input operation in the key display area of ​​the virtual keyboard.

[0138] In another possible implementation, the second input operation includes the target button being clicked, and the feedback event includes one or more of the following: the target button being highlighted, the target button vibrating, and the target button displaying that it has been pressed.

[0139] In another possible implementation, the user performs input operations on the operation area via a touch object; the virtual keyboard-based input device 1800 provided in this application also includes an acquisition module 1807, which is used to acquire the spatial position of the touch object; the obtaining module 1804 is further used to obtain the target position of the touch object in the key display area of ​​the virtual keyboard based on the spatial position of the touch object and the mapping relationship, wherein the mapping relationship indicates the mapping relationship between each spatial position of the touch object and each position of the key display area of ​​the virtual keyboard; the display module 1801 is used to display a preset style icon at the target position of the key display area of ​​the virtual keyboard.

[0140] In another possible implementation, the acquisition module 1807 is further configured to acquire image information and / or capacitance signal distribution information and / or depth information distribution information of the touch body; the display module 1801 is further configured to visualize and render the touch body at a target position in the key display area of ​​the virtual keyboard based on the image information and / or capacitance signal distribution information and / or depth information distribution information of the touch body.

[0141] Optionally, the touchpad can be the user's finger.

[0142] In another possible implementation, the operation area includes a first operation area and a second operation area; the key display area of ​​the virtual keyboard is located in the center of the touch screen, and the first operation area and the second operation area are located on both sides of the key display area of ​​the virtual keyboard; or, the key display area of ​​the virtual keyboard is on one side of the touch screen, the first operation area is on the other side of the touch screen, and the second operation area overlaps with the key display area of ​​the virtual keyboard.

[0143] In another possible implementation, the operation area and the key display area are located on different touch screens; or, the operation area and the key display area are located on different devices.

[0144] The virtual keyboard-based input device 1800 according to the embodiments of this application can correspondingly execute the methods described in the embodiments of this application, and the above and other operations and / or functions of each module in the virtual keyboard-based input device 1800 are respectively for implementing Figure 5-17 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0145] This application also provides an electronic device, including at least one processor, a memory, and a communication interface, wherein the processor is used to execute... Figure 5-17 The method described.

[0146] Figure 19 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0147] like Figure 19As shown, the electronic device 1900 includes at least one processor 1901, a memory 1902, a communication interface 1903, and a display screen 1904. The processor 1901, memory 1902, communication interface 1903, and display screen 1904 are communicatively connected, either via a wired (e.g., bus) or wireless connection. The communication interface 1903 is used to send and / or receive data from other devices. The memory 1902 stores computer instructions, which the processor 1901 executes to perform the methods described in the aforementioned method embodiments. This addresses the interaction issue between the virtual keyboard and the user on widescreen devices, improving input efficiency and enhancing the user experience. The display screen 1904 is a touchscreen display that shows the input interface, which includes a key display area and an operation area for the virtual keyboard. The operation area and the key display area of ​​the virtual keyboard are located in different positions.

[0148] It should be understood that, in the embodiments of this application, the processor 1901 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0149] The memory 1902 may include read-only memory and random access memory, and provides instructions and data to the processor 1901. The memory 1902 may also include non-volatile random access memory.

[0150] The memory 1902 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0151] It should be understood that the electronic device 1900 according to the embodiments of this application can perform the implementation of the embodiments of this application. Figure 5-17 The method shown is described in detail above, and will not be repeated here for the sake of brevity.

[0152] Embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein when the computer instructions are executed by a processor, the aforementioned method is implemented.

[0153] An embodiment of this application provides a chip including at least one processor and an interface, wherein the at least one processor determines program instructions or data through the interface; the at least one processor is used to execute the program instructions to implement the method mentioned above.

[0154] Embodiments of this application provide a computer program or computer program product that includes instructions that, when executed, cause a computer to perform the methods mentioned above.

[0155] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0156] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, processor-executed software modules, or a combination of both. The software modules can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0157] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An input method based on a virtual keyboard, characterized in that, include: The interactive interface of the display electronic device includes an input box and a virtual keyboard; the virtual keyboard includes a key display area and an operation area, wherein the key display area and the operation area are located in different positions. A first input operation acting on the operation area was detected; The first input operation is mapped to the operation of the corresponding key in the key display area to obtain the second input operation for the virtual keyboard; Based on the second input operation, an input event is obtained; In response to the input event, the input information corresponding to the input event is displayed in the input box.

2. The method according to claim 1, characterized in that, The first input operation includes a click operation; The step of mapping the first input operation to the corresponding key in the key display area to obtain a second input operation for the virtual keyboard includes: Obtain the click position of the click operation within the operation area; Map the clicked position to the key display area to obtain the mapped position; Based on the mapped position, a click operation is obtained for the target key in the virtual keyboard.

3. The method according to claim 2, characterized in that, The step of mapping the clicked position to the key display area to obtain the mapped position includes: The click location is used as the input to the neural network model, and the mapped location is output. Alternatively, the click location and the position information of each key in the virtual keyboard can be used as input to a neural network to output the mapped position; Alternatively, the click location and the motion information of the touch object can be used as input to a neural network to output the mapped location.

4. The method according to any one of claims 1-3, characterized in that, The first input operation includes a click operation, and the click operation is preceded by several historical click operations; The step of mapping the first input operation to a corresponding key in the key display area to obtain a second input operation for the virtual keyboard, before the following steps are included: Obtain information about the aforementioned historical click operations; Based on the information from the aforementioned historical click operations, the click position of the current click operation is adjusted.

5. The method according to claim 4, characterized in that, The plurality of historical click operations include the timing information of each historical click operation and the click position information of each historical click operation.

6. The method according to any one of claims 1-5, characterized in that, Also includes: In response to the second input operation, a feedback event corresponding to the second input operation is generated in the key display area.

7. The method according to claim 6, characterized in that, The second input operation includes the target button being clicked, and the feedback event includes one or more of the following: the target button being highlighted, the target button vibrating, and the target button being pressed.

8. The method according to any one of claims 1-7, characterized in that, Users can input operations in the operation area using a touchpad; The method further includes: Obtain the spatial position of the touch object; Based on the spatial position and mapping relationship of the touch body, the target position of the touch body in the key display area is obtained, and the mapping relationship indicates the mapping relationship between each spatial position of the touch body and each position of the key display area; A preset style icon is displayed at the target position in the key display area.

9. The method according to claim 8, characterized in that, Also includes: Acquire image information and / or capacitance signal distribution information and / or depth information distribution information of the touch object; Based on the image information and / or capacitance signal distribution information and / or depth information distribution information of the touch object, the touch object is visualized and rendered at the target position in the key display area.

10. The method according to claim 8 or 9, characterized in that, The touchpad is the user's finger.

11. The method according to any one of claims 1-10, characterized in that, The operating area includes a first operating area and a second operating area; The key display area is located in the center of the touch screen, and the first operation area and the second operation area are located on both sides of the key display area; Alternatively, the key display area is on one side of the touch screen, the first operation area is on the other side of the touch screen, and the second operation area overlaps with the key display area.

12. The method according to any one of claims 1-10, characterized in that, The operation area and the key display area are located on different touch screens; Alternatively, the operation area and the key display area may be located on different devices.

13. An electronic device comprising a memory and a processor, characterized in that, The memory stores instructions that, when executed by a processor, cause the method described in any one of claims 1-12 to be implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it causes the method as described in any one of claims 1-12 to be implemented.

Images