Input method and device based on virtual keyboard

Abstract

The invention provides an input method based on a virtual keyboard. The input method comprises the steps that first input operation acting on an operation area is detected; the first input operation is mapped into operation of corresponding keys in a key position display area, second input operation for the virtual keyboard is obtained, and the positions of the operation area and the key position display area are different; based on the second input operation, obtaining an input event; responding to the input event, and displaying input information corresponding to the input event in the input box. According to the input method, the operation area and the key position display area of the virtual keyboard are independently arranged, when the input method is applied to the wide-screen equipment, the operation area can be located in the area, easier to touch by hands, in the touch display screen, so that a user performs input operation in the operation area easier to touch by the hands, and the user experience is improved. Operation input is mapped to the virtual keyboard to achieve input, the key position display area is displayed in a centralized mode, the sight of a user can be concentrated in the key position display area, the sight moving distance of the user is reduced, and user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an input method and device based on a virtual keyboard. Background Art

[0002] With the development and evolution of mobile terminal devices, users are seeking a wider interactive range and field of view, and wide-screen devices are gradually becoming the mainstream trend. On wide-screen devices, users input information (such as typing) through a graphical interface, which is still one of the most common input technologies.

[0003] The current mainstream graphical interface input technology is to use a virtual keyboard (or soft keyboard, that is, displaying a keyboard on the screen), so that users can input text, symbols or information supported by the keyboard by touching the virtual keys displayed on the screen. The current virtual keyboard design is mainly designed in the form of a physical keyboard, but it has higher customizability and scalability than a physical keyboard.

[0004] When a virtual keyboard is applied to a widescreen device, due to the large horizontal size of the widescreen device, when the user holds the device with both hands to input information, the keys in the middle of the virtual keyboard are difficult to reach, resulting in unnatural and inflexible interaction and low input efficiency. Summary of the invention

[0005] The present application provides an input method based on a virtual keyboard, which, when applied to a wide-screen device, solves the interaction problem between the virtual keyboard of the wide-screen device and the user, improves input efficiency, and improves user experience.

[0006] In a first aspect, the present application provides an input method based on a virtual keyboard, including an interactive interface of an electronic device, the interactive interface including an input box and a virtual keyboard, the virtual keyboard including a key display area and an operation area, wherein the key display area and the operation area are located at different positions; detecting a first input operation acting on the operation area; mapping the first input operation to an operation of a corresponding key in the key display area to obtain a second input operation for the virtual keyboard; based on the second input operation, obtaining an input event; in response to the input event, displaying input information corresponding to the input event in the input box.

[0007] In the input method based on the virtual keyboard provided by the present application, the operation area and the key display area are set independently of each other. When applied to a wide-screen device, the operation area can be located in an area of ​​the touch display screen that is easier for human hands to reach, so that the user can perform input operations in the operation area that is easier for both hands to reach, and the operation input is mapped to the virtual keyboard to realize input. The key display area is displayed in a centralized manner, and the user's line of sight can be concentrated on the key display area, thereby reducing the distance the user's line of sight moves and improving the user experience.

[0008] In one possible implementation, the first input operation includes a click operation; the first input operation is mapped to the operation of the corresponding key in the key display area, and a specific implementation of the second input operation for the virtual keyboard is: obtaining the click position of the click operation in the operation area; mapping the click position to the key display area to obtain the mapping position; based on the mapping position, obtaining the click operation for the target key in the virtual keyboard.

[0009] In this possible implementation, the user's click position in the operation area is mapped and converted to obtain the mapping position of the key display area, and then the target key corresponding to the mapping position is found, so that the user's click operation on the target key in the virtual keyboard is realized. In other words, the user does not need to hit the keys in the virtual keyboard, but can realize the input operation on the virtual keyboard by performing input operations in the operation area that is easier for the user to reach, which removes the restriction of the virtual keyboard on both hands and greatly improves the input efficiency.

[0010] In another possible implementation, the click position is mapped to a key display area, and a specific implementation of obtaining the mapping position is: using the click position as the input of the neural network model, and outputting the mapping position; or, using the click position and the position information of each key in the virtual keyboard as the input of the neural network, and outputting the mapping position; or, using the click position and the movement information of the touch body as the input of the neural network, and outputting the mapping position.

[0011] A neural network model is used to establish a mapping relationship between the coordinates of each position in the operation area and the coordinates of each position in the key display area of ​​the virtual keyboard, so as to quickly and accurately determine the mapping position on the virtual keyboard corresponding to the user's click position in the operation area.

[0012] In another possible implementation, the first input operation includes a click operation, which includes several historical click operations before the click operation; the first input operation is mapped to the operation of the corresponding key in the key display area to obtain a second input operation for the virtual keyboard, which also includes: obtaining information of several historical click operations; based on the information of several historical click operations, adjusting the click position of the current click operation.

[0013] The click position is corrected through historical click operation information to prevent the situation where the typing information cannot be triggered because the calculated mapping position is at the edge of the virtual key.

[0014] In another possible implementation, the plurality of historical click operations include timing information of each of the plurality of historical click operations and click position information of each of the historical click operations.

[0015] In another possible implementation, the virtual keyboard-based input method provided by the present application further includes: in response to the second input operation, generating a feedback event corresponding to the second input operation in the key position display area.

[0016] In one example, the second input operation includes a target button being clicked, and the feedback event includes one or more of the target button being highlighted, the target button vibrating, and the target button appearing to be pressed. In this way, the feedback event reminds the user that the target button has been successfully clicked, thereby improving the user input experience.

[0017] In another possible implementation, the user performs input operations in the operation area through the touch body; the input method based on the virtual keyboard provided in the present application also includes: obtaining the spatial position of the touch body; based on the spatial position of the touch body and the mapping relationship, obtaining the target position of the touch body in the key display area, the mapping relationship indicates the mapping relationship between each spatial position of the touch body and each position of the key display area; displaying a preset style icon at the target position of the key display area.

[0018] Exemplarily, when the user hovers the finger over a certain position in the operation area, the key display area displays a preset style icon on the "A" key; when the user moves the finger to hover over another position in the operation area, the key display area displays a preset style icon on the "C" key. The preset style icon identifies the target key of the virtual keyboard mapped to the current position of the user's finger, making it easy for the user to interact with the virtual keyboard through the operation area.

[0019] It should be noted that the preset style icon is used to identify the mapping position of the user's current operation in the operation area on the key display area. The preset style icon can be an icon of any style. That is to say, the shape, fill effect, outline effect, color, transparency, texture and other visualization effect parameters of the preset style icon can be adjusted and arbitrarily combined. For example, different preset styles can be circular icons, square icons, cylindrical icons, etc., and this application does not limit this.

[0020] In another possible implementation, the input method based on the virtual keyboard provided by the present application also includes: obtaining image information and / or capacitance signal distribution information and / or depth information distribution information of the touch body; based on the image information and / or capacitance signal distribution information and / or depth information distribution information of the touch body, visually rendering the touch body at the target position of the key display area.

[0021] In this possible implementation, the interaction between the user and the virtual keyboard is visually simulated to further enhance the input experience.

[0022] Optionally, the touch body is a finger of a user.

[0023] In another possible implementation, the operation area includes a first operation area and a second operation area; the key position display area is at the center of the touch display screen, and the first operation area and the second operation area are located on both sides of the key position display area; or, the key position display area is on one side of the touch display screen, the first operation area is on the other side of the touch display screen, and the second operation area overlaps with the key position display area.

[0024] In this possible implementation, the first operation area and the second operation area are distributed on both sides of the touch display screen, which is convenient for the user to reach with his fingers when holding the wide-screen device with both hands, and perform input operations in the operation area. The keys of the virtual keyboard are displayed in the center, and the user's line of sight can be concentrated on the key display area, which reduces the line of sight movement distance and improves the user experience.

[0025] In another possible implementation, the operation area and the key display area are located on different touch display screens; for example, in the multi-screen interaction field or the XR field, in a multi-screen interaction scenario, the user can type typing information on one of the touch screens, and then focus their eyes on other screens to obtain feedback information on the typing input; in an XR interaction scenario, the user can type typing information on an additional touch screen device, and then observe the interaction results on the head-mounted display to obtain visual feedback related to typing.

[0026] Alternatively, the operation area and the key display area are located on different devices; for example, in the field of multi-device collaboration, the user can associate two or more devices, then perform typing operations on one of the devices, and then observe the typing information feedback effect on another device.

[0027] Of course, in addition to typing scenarios, the virtual keyboard-based input method provided in this application can also be applied to any occasion where gesture information is obtained using a touch screen, such as creation, editing, system operation, and any other gesture operation scenarios.

[0028] In the second aspect, the present application also provides an input device based on a virtual keyboard, including a display module, a detection module, a mapping module, a obtaining module and a response module, wherein the display module is used to display an interactive interface of the electronic device, the interactive interface including an input box and a virtual keyboard, the virtual keyboard including a key display area and an operation area; the detection module is used to detect a first input operation acting on the operation area; the mapping module is used to map the first input operation to an operation of a corresponding key in the key display area, and obtain a second input operation for the virtual keyboard, wherein the operation area and the key display area are located at different positions; the obtaining module is used to obtain an input event based on the second input operation; the response module is used to respond to the input event and display the input information corresponding to the input event in the input box.

[0029] In one possible implementation, the first input operation includes a click operation; the mapping module is specifically used to obtain the click position of the click operation in the operation area; the click position is mapped to the key display area to obtain the mapping position; based on the mapping position, a click operation for the target key in the virtual keyboard is obtained.

[0030] In another possible implementation, the click position is mapped to the key display area, and a specific implementation of obtaining the mapping position is: using the click position as the input of the neural network model, and outputting the mapping position; or, using the click position and the position information of each key in the virtual keyboard as the input of the neural network, and outputting the mapping position; or, using the click position and the movement information of the touch body as the input of the neural network, and outputting the mapping position.

[0031] In another possible implementation, the first input operation includes a click operation, which includes several historical click operations before the click operation; the virtual keyboard-based input device provided in the present application also includes: an adjustment module; the adjustment module is used to obtain information of several historical click operations; based on the information of several historical click operations, adjust the click position of the current click operation.

[0032] In another possible implementation, the plurality of historical click operations include timing information of each of the plurality of historical click operations and click position information of each of the historical click operations.

[0033] In another possible implementation, the response module is further used to: respond to the second input operation and generate a feedback event corresponding to the second input operation in the key display area.

[0034] In another possible implementation, the second input operation includes a target key being clicked, and the feedback event includes one or more of the target key being highlighted, the target key being vibrated, and the target key being displayed as being pressed.

[0035] In another possible implementation, the user performs input operations in the operation area through the touch body; the input device based on the virtual keyboard provided in the present application also includes: an acquisition module, which is used to acquire the spatial position of the touch body; the acquisition module is also used to acquire the target position of the touch body in the key display area based on the spatial position of the touch body and the mapping relationship, and the mapping relationship indicates the mapping relationship between each spatial position of the touch body and each position of the key display area; the display module is used to display a preset style icon at the target position of the key display area.

[0036] In another possible implementation, the acquisition module is also used to acquire image information and / or capacitance signal distribution information and / or depth information distribution information of the touch body; the display module is also used to visually render and display the touch body at the target position of the key display area of ​​the virtual keyboard based on the image information and / or capacitance signal distribution information and / or depth information distribution information of the touch body.

[0037] Optionally, the touch body is a finger of a user.

[0038] In another possible implementation, the operation area includes a first operation area and a second operation area; the key display area is located at the center of the touch display screen, and the first operation area and the second operation area are located on both sides of the key display area; or, the key display area is on one side of the touch display screen, the first operation area is on the other side of the touch display screen, and the second operation area overlaps with the key display area.

[0039] In another possible implementation, the operation area and the key position display area are located on different touch display screens; or, the operation area and the key position display area are located on different devices.

[0040] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores instructions, and when the instructions are executed by the processor, the method described in the first aspect is implemented.

[0041] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0042] In a fifth aspect, an embodiment of the present application further provides a computer program or a computer program product, wherein the computer program or the computer program product comprises instructions, which, when executed, cause a computer to execute the method described in the first aspect.

[0043] In a sixth aspect, an embodiment of the present application further provides a chip, comprising at least one processor and a communication interface, wherein the processor is used to execute the method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of a split virtual keyboard on a display screen;

[0045] Figure 2 A schematic diagram of an adaptive magnified virtual keyboard on a display screen;

[0046] Figure 3 A schematic diagram of a floating virtual keyboard on a display screen;

[0047] Figure 4 This is a schematic diagram showing that the operation area is consistent with the key display area;

[0048] Figure 5 A schematic diagram of an application scenario of an input method based on a virtual keyboard provided in an embodiment of the present application is shown;

[0049] Figure 6 A flowchart of an input method based on a virtual keyboard provided in an embodiment of the present application;

[0050] Figure 7 A schematic diagram showing the distribution of an operation area and a display area of ​​a virtual keyboard on a display screen is shown;

[0051] Figure 8 A schematic diagram showing the distribution of an operation area and a key position display area on a display screen is shown;

[0052] Fig. 9 Another schematic diagram showing the distribution of the operation area and the key position display area on the display screen is shown;

[0053] Fig.10 and Fig.11 Schematic diagrams showing a widescreen device placed on a supporting surface and a user using both hands to interact with the display screen for input;

[0054] Fig.12 A mapping relationship between position coordinates in an operation area and position coordinates in a key display area is shown;

[0055] Fig.13 A schematic diagram showing multiple landing points collected by the screen sensor during typing;

[0056] Fig.14 for Fig.13 A partial enlarged view of

[0057] Figures 15 to 17 Several examples of styles of preset style icons for identifying mapping positions on the key position display area of ​​the virtual keyboard are shown respectively;

[0058] Fig.18 A schematic diagram of the structure of an input device based on a virtual keyboard provided in an embodiment of the present application;

[0059] Fig.19 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The term "and / or" mentioned in this article is a kind of relationship that describes the association of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0061] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first input operation and a second input operation are used to distinguish different input operations rather than to describe a specific order of the input operations.

[0062] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0063] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0064] When a user interacts with a virtual keyboard on a current widescreen device, there are three main ways to arrange the virtual keyboard in current technology, namely:

[0065] Separate: This design distributes the main operation interface of the virtual keyboard on both sides of the screen edge. There are interactive / operable high-frequency virtual keys on the left and right ends of the virtual keyboard, which is convenient for users to input information. Figure 1 shown.

[0066] Adaptive zoom: This design keeps the width of the virtual keyboard consistent with the main display screen of the widescreen device, and adjusts the display height of the virtual keyboard in a certain proportion. The virtual key layout covers the entire virtual keyboard interface, such as Figure 2 shown.

[0067] Floating: This design generally displays the virtual keyboard in the center of the device in proportion during initialization, but the user can move the virtual keyboard to facilitate input, such as Figure 3 This form of keyboard displays information relatively concentratedly, and the user's line of sight does not have to span the entire screen.

[0068] Under the current technical implementation, when users interact with the above virtual keyboards, they need to directly interact with the virtual keys, that is, hit the corresponding virtual keys, otherwise the virtual keyboard cannot feedback the corresponding input information, such as Figure 4 shown.

[0069] When users use the virtual keyboard designed above to input on wide-screen devices (such as foldable screen computers, foldable screen mobile phones, tablets, etc.), there will be a conflict between the control area of ​​both hands and the central area of ​​the line of sight, which will lead to:

[0070] Users are using Figure 1 and Figure 2 When using the virtual keyboard shown, although both hands have enough space to operate, both eyes need to take care of the entire virtual keyboard interface;

[0071] Users are using Figure 3 When using the virtual keyboard shown in the figure, although the operation interface of the virtual keyboard occupies a small space, which improves the space utilization of the operation interface, the interaction space of both hands is limited, resulting in unnatural and inflexible interaction and reduced input efficiency.

[0072] A related solution provides a method that supports using typing behavior to predict keystroke positions. A virtual keyboard using this technology can adapt itself according to the user's input habits to improve input efficiency.

[0073] The solution collects the user's typing behavior information, including but not limited to typing pattern information, previous keystroke behavior information, virtual keyboard detail information used for typing, and context parameter information related to the device and user interaction of the displayed virtual keyboard, and then extracts and fuses multiple features to obtain multiple feature vectors. The solution trains a neural network to identify known features and learns position features that affect typing behavior, thereby predicting the virtual keyboard keystroke position of the corresponding user.

[0074] This solution needs to be based on a large amount of sample data. Data collection and model training require a long time to accumulate, and the trained model is not universal, resulting in poor universality of the virtual keyboard that integrates the training model.

[0075] In response to the above problems, the present application provides an input method based on a virtual keyboard, in which the operation area of ​​the virtual keyboard (hereinafter referred to as the operation area for the sake of ease of description) and the key display area are set independently of each other. When applied to a wide-screen device, the operation area can be located in an area of ​​the touch display screen that is easier for human hands to reach, so that the user can perform input operations in the operation area that is easier for both hands to reach, and the operation input is mapped to the virtual keyboard to realize input. The key display area is displayed centrally, and the user's line of sight can be concentrated on the key display area, thereby reducing the distance the user's line of sight moves and improving the user experience.

[0076] Figure 5 A schematic diagram of an application scenario of an input method based on a virtual keyboard provided in an embodiment of the present application is shown. Figure 5 As shown, the user can hold the wide screen device with both hands, and the thumbs of both hands of the user act as touch bodies to interact with the touch screen to implement input operations.

[0077] The widescreen device mentioned in the embodiments of the present application can be understood as a device with a larger screen width. For example, the widescreen device can be a folding screen computer, a dual-screen computer, a folding screen mobile phone, a widescreen mobile phone, a widescreen tablet, etc.

[0078] certainly, Figure 5 What is shown is only an example of an application scenario of the input method based on the virtual keyboard provided in the embodiment of the present application, and does not constitute a limitation on the embodiment of the present application. The input method provided in the embodiment of the present application can also be applied to other application scenarios, for example, a widescreen device is placed on a supporting object, and the user's hands interact with the touch screen as touch bodies to implement input operations. For another example, applied to a multi-screen interaction scenario, the user can type typing information on one of the touch screens, and then focus their sight on other screens to obtain feedback information of the typing input. For another example, applied to an XR interaction scenario, the user can type typing information on an additional touch screen device, and then observe the interaction results on the head-mounted display to obtain visual feedback related to typing. For another example, applied to a multi-device collaborative scenario, the user can associate two or more devices, then type on one of the devices, and then observe the typing information feedback effect on another device. And the touch body can also be other objects that can interact with the touch screen, such as a stylus.

[0079] Of course, in addition to typing scenarios, the virtual keyboard-based input method provided in the embodiment of the present application can also be applied to any occasion where gesture information is obtained using a touch screen, such as creation, editing, system operation, and any other gesture operation scenarios.

[0080] Taking the typing scenario on a widescreen device as an example, the following introduces the detailed implementation of the virtual keyboard-based input method provided in the embodiment of the present application. The implementation of other scenarios is similar. For specific implementations, please refer to the detailed implementation introduction of the typing scenario on a widescreen device. For the sake of brevity, the embodiment of the present application will not be repeated.

[0081] Figure 6 The flowchart of the input method based on the virtual keyboard provided in the embodiment of the present application is a schematic diagram. The method can be applied to any wide-screen device to achieve friendly typing interaction, reduce the user's line of sight movement distance, and improve the user's typing input experience. Figure 6 As shown, the virtual keyboard-based input method provided in the embodiment of the present application at least includes steps S601 to S604.

[0082] In step S601, an interactive interface of the electronic device is displayed.

[0083] When the user needs to input information, the user operates on the touch display screen (for example, the user clicks on the input box) to call out the virtual keyboard for input operation. For example, the electronic device detects the user's operation of clicking on the input box and displays an input interaction interface, the input interaction interface includes the input box and the virtual keyboard, the virtual keyboard includes a key display area and an operation area, wherein the key display area and the operation area are located at different positions on the touch display screen.

[0084] Figure 7 A schematic diagram of an input interaction interface is shown. Figure 7 As shown, the key position display area of ​​the virtual keyboard is displayed in the center of the display screen, and the operation areas are located on both sides of the key position display area, close to the edge of the display screen, so as to be easily accessible by fingers for input operations.

[0085] In step S602 , a first input operation applied to the operation area is detected.

[0086] The user performs input operations in the operation area of ​​the touch screen of the wide screen device, and the touch screen detects the input operations performed by the user in the operation area through the sensors therein.

[0087] The input operation may be an operation in which a touch object (such as a user's finger) directly contacts the touch screen, or may be an input operation performed near the top of the touch screen without directly contacting the touch screen.

[0088] For example, a user clicks a certain position in the operation area of ​​the touch screen with a finger, and the sensor on the touch screen detects that the position is clicked, and generates a corresponding signal to inform the processor that a certain position on the screen is detected to be clicked.

[0089] The sensor on the touch screen can be any sensor that can realize the functions required by the embodiments of the present application. For example, the sensor provided on the touch screen can be a pressure sensor, which is used to sense pressure signals and can convert pressure signals into electrical signals. There are many types of pressure sensors, such as resistive pressure sensors and inductive pressure sensors. When a touch operation acts on the touch display screen, the widescreen device detects the touch operation intensity according to the pressure sensor. The widescreen device can also calculate the touch position according to the detection signal of the pressure sensor. In some examples, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with a touch operation intensity less than a first pressure threshold acts on a short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on a short message application icon, an instruction to create a new short message is executed.

[0090] For another example, the sensor provided on the touch screen may also be a touch sensor (e.g., a capacitive touch sensor). The touch sensor is used to detect a touch operation applied thereto or thereabout. The touch sensor may transmit the detected touch operation to an application processor to determine the type of touch event. A visual output related to the touch operation may be provided through a display screen.

[0091] In step S603, the first input operation is mapped to an operation of a corresponding key in the key position display area to obtain a second input operation for the virtual keyboard.

[0092] After detecting a first input operation acting on the operation area, the input operation is mapped and converted into a second input operation for the virtual keyboard, so that information input can be achieved without hitting the virtual keyboard.

[0093] The operation area can be set at a position that is easily accessible to the user's fingers, so that the user can perform input operations in the operation area; the key display area can be concentrated in a certain area of ​​the display screen to reduce the user's line of sight movement.

[0094] Figure 8 FIG. 1 shows a schematic diagram of the distribution of an operation area and a virtual key position display area on a display screen. Figure 8 As shown, the operation area includes a first operation area and a second operation area, the key position display area is displayed in the center of the display screen, and the first operation area and the second operation area are respectively located on both sides of the key position display area. The first operation area and the second operation area are close to both sides of the display screen, which is convenient for users to input information with their thumbs when holding the device with both hands. The virtual keyboard is displayed in the center, and the user's line of sight does not have to span the entire screen, which reduces the distance the user's line of sight moves and makes information input more efficient.

[0095] The key position display area means each key position display area of ​​the virtual keyboard, which is used to display each key of the virtual keyboard.

[0096] Fig. 9 Another schematic diagram of the distribution of the operation area and the key position display area on the display screen is shown. Fig. 9 As shown, the key display area is located on the right side of the display screen, one operation area is located on the left side of the display screen, and another operation area is located on the right side of the display screen and partially overlaps with the key display area. In this way, only the mapping relationship between the operation area on the left and the virtual keyboard is required, and the operation area on the right can realize information input operation by directly hitting the target key in the virtual keyboard.

[0097] In one example, the key display area and the operation area can use the same background color in the input interface, which is different from the background color of other areas in the input interface to distinguish them from other areas, making it easier for users to find the key display area and the operation area. Optionally, the operation area of ​​the virtual keyboard also includes prompt information to prompt the user to perform input operations in this area. For example, the prompt text "Please perform input operations here" is displayed on the operation area of ​​the virtual keyboard, or the prompt gesture "a rendered virtual finger" is displayed on the operation area of ​​the virtual keyboard to prompt the user to perform input operations in this area.

[0098] In some other examples, the widescreen device may be placed on a support surface, and the five fingers of both hands may be used to perform input operations in the operation area to further increase the input efficiency, such as Fig.10 and Fig.11 shown.

[0099] There are many ways to map the operations of the operating area to the virtual keyboard.

[0100] For example, the mapping relationship between each position coordinate in the operation area and each position coordinate in the key display area can be statistically analyzed to create a mapping table, which records the mapping relationship between each position coordinate in the operation area and each position coordinate in the key display area.

[0101] When the user clicks a certain position in the operation area, the sensor in the touch screen detects the coordinates of the clicked position, and obtains the mapping position coordinates of the key display area corresponding to the coordinates of the clicked position by looking up the mapping table.

[0102] For example, by determining a mapping formula between each position coordinate in the operation area and each position coordinate in the key display area, the position coordinates corresponding to the user's operation in the operation area can be brought into the mapping formula to calculate the mapping position coordinates in the key display area.

[0103] In one example, the mapping formula may be y=ax+b, mapping the actual trigger position of the user to the target position in the virtual keyboard. Figure 8 For example, the mapping formula corresponding to the left operation area is y=0.85x+0.45, and the mapping formula corresponding to the right operation area is y=0.85x-0.2.

[0104] Exemplarily, the value ranges of parameters a and b in the mapping formula y=ax+b of the left operating area are: a∈[0.5,1.5], b∈(0,1); the value ranges of parameters a and b in the mapping formula y=ax+b of the right operating area are: a∈[0.5,1.5], b∈(-1,0).

[0105] For another example, a mapping relationship between the coordinates of each position in the operation area and the coordinates of each position in the key display area can be established by means of a neural network. Fig.12 A mapping relationship between position coordinates in an operation area and position coordinates in a key display area is shown.

[0106] In one example, the input of the neural network may be only the position coordinates actually triggered by the user in the operation area, and the output may be the mapped position coordinates in the virtual keyboard.

[0107] In another example, for a specific virtual keyboard, the input of the neural network may also include the position coordinates actually triggered by the user in the operation area and the position information of each virtual key in the virtual keyboard, and output the mapping position coordinates in the virtual keyboard. For different virtual keyboard layouts, the corresponding neural network is trained to achieve more accurate prediction of the mapping position coordinates in the virtual keyboard.

[0108] In another example, the input of the neural network may also include the position coordinates actually triggered by the user in the operation area and the motion information of the touch body (such as speed and acceleration, etc.), and output the mapping position coordinates in the virtual keyboard. By combining the motion information of the touch body, the mapping position coordinates in the virtual keyboard can be predicted more accurately.

[0109] In another example, the input of the neural network may also include the position coordinates actually triggered by the user in the operation area, the position information of each virtual key in the virtual keyboard, and the motion information of the touch body (such as speed and acceleration, etc.), and output the mapped position coordinates in the virtual keyboard.

[0110] It is understandable that the input of the neural network is different, and the training samples of the neural network will also be different. For example, the input of the neural network in the first example only includes the position coordinates actually triggered by the user, and the training samples of the neural network include the position coordinates in the operation area and the label of the sample, and the label is the mapped position coordinates of the key display area. The input of the neural network in the second example includes the position coordinates actually triggered by the user in the operation area and the position information of each virtual key in the virtual keyboard. The training samples of the neural network include the position coordinates in the operation area and the position information of each virtual key in the virtual keyboard, and the label of the sample, and the label is the mapped position coordinates of the key display area of ​​the virtual keyboard.

[0111] Optionally, the neural network may be a multilayer perceptron (MLP) neural network.

[0112] After obtaining the mapping position of the actual trigger position on the virtual keyboard, a click operation on a target key in the virtual keyboard is obtained according to the mapping position.

[0113] For example, the mapping position of the actual trigger position on the virtual keyboard is the position of the key "S", and the key "S" is clicked.

[0114] In some other examples, in order to avoid the mapping position corresponding to the input operation falling on the edge of the target key or in the middle of two keys, resulting in the inability to trigger the typing of information. The input operation may be, for example, that the user clicks point a1 in the operation area with a finger, and the point a1 is mapped to the key position display area of ​​the virtual keyboard as the middle position between key "a" and key "s". At this time, since it is impossible to determine whether the "a" key or the "s" key is triggered, the typing of information cannot be triggered. . The embodiment of the present application also proposes a landing point correction mechanism, that is, before executing step S603, the user's input operation in the operation area is adjusted.

[0115] Taking typing operation as an example, the user inputs information by performing multiple click operations in the operation area. When determining the landing point of the current click operation (also referred to as the click position), the click position of the current click operation is adjusted according to the information of the historical click operation. Optionally, the historical click operation includes the timing information of each historical click operation and the click position information of each historical click operation.

[0116] In response to the user's typing behavior, the information received by the sensor module of the display screen is actually a series of landing point information that is related in time sequence. The trajectory of the user typing information during the actual typing process may be deviated due to the different previous and next keys associated with the current virtual key. For example, when a key above key "A" (such as key "Q") is pressed before key "A", the landing point will fall on the lower part of key "A" (such as key "Q"). Fig.13 and Fig.14 If you press a key below "A" (such as "Z") before pressing "A", the point of impact will fall slightly above "A" (such as Fig.13 and Fig.14 The points indicated by the “asterisk” in the figure).

[0117] For such cases, the actual landing point information collected by the sensor is combined with the timing information f(pos_t,…,pos_0) to correct the failure to trigger the typing information due to the calculated mapping point being at the edge of the virtual key. For example, if the landing point of the previous click operation of the current click operation is above the landing point of the current click operation, the landing point of the current click operation is adjusted upward for deviation correction; if the landing point of the previous click operation of the current click operation is below the landing point of the current click operation, the landing point of the current click operation is adjusted downward for deviation correction.

[0118] In step S604, an input event is obtained based on the second input operation.

[0119] After obtaining the second operation of the user on the virtual keyboard, an input event for the virtual keyboard is generated. For example, if the second operation on the virtual keyboard is clicking the keys "H", "U", "A", "W", "E", and "I" in sequence, the input event is inputting "H", "U", "A", "W", "E", and "I" in sequence.

[0120] In step S605, in response to the input event, input information corresponding to the input event is displayed in the input box.

[0121] After the input event is determined, in response to the input event, input information, such as “HUAWEI”, is displayed in the input box.

[0122] In some other examples, the method provided by the embodiment of the present application may further include: in the process of the user inputting information through the operation area, responding to the second input operation mapped to the virtual keyboard. That is, in response to the second input operation, a feedback event corresponding to the second input operation is generated in the key display area of ​​the virtual keyboard.

[0123] Optionally, the second input operation includes clicking the target button, and the feedback event includes one or more of highlighting the target button, vibrating the target button, displaying that the target button is pressed, and displaying the target button at the actual touch point. In this way, the user is reminded through the feedback event that the target button is successfully clicked, thereby improving the user input experience.

[0124] In some other examples, when the user inputs information through the operation area, the position of the user's finger is also mapped in real time to the key display area of ​​the virtual keyboard, so that the user can understand that the current position of the finger in the operation area corresponds to the position on the virtual keyboard.

[0125] Exemplarily, a user performs input operations in an operation area using a finger; the spatial position of the finger is obtained through a sensor (for example, the spatial position of the finger can be detected by a capacitive sensor); based on the spatial position of the finger and a mapping relationship, a target position of the finger in a key display area of ​​a virtual keyboard is obtained, and the mapping relationship indicates a mapping relationship between various spatial positions of the finger and various positions of the key display area of ​​the virtual keyboard; a preset style icon is displayed at the target position of the key display area of ​​the virtual keyboard.

[0126] For example, when the user hovers the finger over a certain position in the operation area, the key position display area of ​​the virtual keyboard displays a preset style icon on the "A" key. When the user moves the finger to hover over another position in the operation area, the key position display area of ​​the virtual keyboard displays a preset style icon on the "C" key. The preset style icon indicates that the current position of the user's finger is mapped to the target key of the virtual keyboard, making it easy for the user to interact with the virtual keyboard through the operation area.

[0127] It should be noted that the preset style icon is used to identify the mapping position of the user's current operation in the operation area on the key display area of ​​the virtual keyboard. The preset style icon can be an icon of any style. That is to say, the shape, fill effect, outline effect, color, transparency, texture and other visualization effect parameters of the preset style icon can be adjusted and combined arbitrarily. For example, different preset styles can be circular icons (see Fig.15 ), square icon, cylindrical icon (see Fig.16 ), etc. The preset style can also be represented by pictographic graphics. For example, if the touch body is a user's finger, the preset style image can be a finger fingerprint (see Fig.17 ), this application does not limit the specific implementation method of the preset style icon.

[0128] In another example, during the user input process, the user's finger is also visually mapped to the key display area of ​​the virtual keyboard for display. For example, the image information and / or capacitance signal distribution information and / or depth information distribution information of the user's finger is obtained; based on the image information and / or capacitance signal distribution information and / or depth information distribution information of the touch body, a two-dimensional image of the user's finger is obtained, and the two-dimensional image of the user's finger is visually rendered and displayed at the target position of the key display area of ​​the virtual keyboard. Specifically, the RGB image information of the user's finger is acquired through the camera of the widescreen device, and the two-dimensional image of the user is obtained based on the RGB image information, and the two-dimensional image of the user's finger is rendered and displayed to the target position of the key display area of ​​the virtual keyboard (that is, the position of the finger in the operation area is mapped to the position of the key display area of ​​the virtual keyboard) for display. The user's finger moves in the operation area, and the user's finger rendered on the key display area of ​​the virtual keyboard moves with it, so that the user can understand in real time the mapping position of the finger in the operation area to the key display area of ​​the virtual keyboard.

[0129] Alternatively, rendering and displaying the user's finger image may also be achieved by other means, such as obtaining the finger's capacitance signal distribution and / or depth information distribution through sensor detection on the touch screen, and using these distribution information to render and display the finger image to the target position of the key display area of ​​the virtual keyboard (that is, the position of the finger in the operation area is mapped to the position of the key display area of ​​the virtual keyboard) for display.

[0130] It should be explained that the operating area mentioned in the context of the embodiments of the present application means a certain area on the touch screen surface and the area above the area, for example Figure 8 The operation area shown includes the part framed by the operation area on the touch screen, and the space area above the part. When the user's finger enters the operation area, the sensor on the device starts to work to collect the finger's operation, such as movement, clicking, gestures, etc. in the operation area, and visually maps the finger to the key position display area of ​​the virtual keyboard, so that the user can understand the position of the key position display area of ​​the virtual keyboard corresponding to the finger's operation in the operation area. For example, if the finger moves between point a1 and point c1 in the operation area, the key position display area of ​​the virtual keyboard will show that the finger moves between key A and key C. When the finger moves to point c1 in the operation area and clicks on the touch screen surface at point c1, the key position display area of ​​the virtual keyboard will show that key C is clicked by the finger, and key C will generate a feedback event (such as highlighting, vibration, etc.) to inform the user that key C is clicked. In response to the event that key C is clicked, the character "C" is input, and the character "C" is displayed in the input box of the virtual keyboard display area.

[0131] Based on the same concept as the aforementioned embodiment of the input method based on a virtual keyboard, the embodiment of the present application further provides an input device 1800 based on a virtual keyboard. The input device 1800 based on a virtual keyboard can be deployed on an electronic device to achieve a more user-friendly input interaction on a wide-screen device. The input device 1800 includes a Figure 5-17 The units or modules of each step in the input method shown.

[0132] Fig.18 The structure diagram of an input device based on a virtual keyboard provided in an embodiment of the present application is shown in FIG. Fig.18 As shown, the input device 1800 based on the virtual keyboard at least includes a display module 1801, a detection module 1802, a mapping module 1803, an acquisition module 1804 and a response module 1805, wherein the display module 1801 is used to display the interactive interface of the electronic device, the interactive interface includes an input box and a virtual keyboard, the virtual keyboard includes a key display area and an operation area, wherein the key display area and the operation area are located at different positions; the detection module 1802 is used to detect a first input operation acting on the operation area; the mapping module 1803 is used to map the first input operation to the operation of the corresponding key in the key display area of ​​the virtual keyboard, and obtain a second input operation for the virtual keyboard; the acquisition module 1804 is used to obtain an input event based on the second input operation; the response module 1805 is used to respond to the input event and display the input information corresponding to the input event in the input box.

[0133] In one possible implementation, the first input operation includes a click operation; the mapping module 1803 is specifically used to obtain the click position of the click operation in the operation area; map the click position to the key display area of ​​the virtual keyboard to obtain the mapping position; based on the mapping position, obtain the click operation for the target key in the virtual keyboard.

[0134] In another possible implementation, the click position is mapped to the key display area of ​​the virtual keyboard, and a specific implementation of obtaining the mapping position is: using the click position as the input of the neural network model, and outputting the mapping position; or, using the click position and the position information of each key in the virtual keyboard as the input of the neural network, and outputting the mapping position; or, using the click position and the movement information of the touch body as the input of the neural network, and outputting the mapping position.

[0135] In another possible implementation, the first input operation includes a click operation, which includes several historical click operations before the click operation; the virtual keyboard-based input device 1800 provided in the present application also includes: an adjustment module 1806; the adjustment module is used to obtain information of several historical click operations; based on the information of several historical click operations, adjust the click position of the current click operation.

[0136] In another possible implementation, the plurality of historical click operations include timing information of each of the plurality of historical click operations and click position information of each of the historical click operations.

[0137] In another possible implementation, the response module 1805 is further used to: respond to the second input operation and generate a feedback event corresponding to the second input operation in the key position display area of ​​the virtual keyboard.

[0138] In another possible implementation, the second input operation includes a target key being clicked, and the feedback event includes one or more of the target key being highlighted, the target key being vibrated, and the target key being displayed as being pressed.

[0139] In another possible implementation, the user performs input operations in the operation area through the touch body; the input device 1800 based on the virtual keyboard provided in the present application also includes an acquisition module 1807, which is used to acquire the spatial position of the touch body; the acquisition module 1804 is also used to obtain the target position of the touch body in the key display area of ​​the virtual keyboard based on the spatial position of the touch body and the mapping relationship, and the mapping relationship indicates the mapping relationship between each spatial position of the touch body and each position of the key display area of ​​the virtual keyboard; the display module 1801 is used to display a preset style icon at the target position of the key display area of ​​the virtual keyboard.

[0140] In another possible implementation, the acquisition module 1807 is also used to acquire the image information and / or capacitance signal distribution information and / or depth information distribution information of the touch body; the display module 1801 is also used to visually render and display the touch body at the target position of the key display area of ​​the virtual keyboard based on the image information and / or capacitance signal distribution information and / or depth information distribution information of the touch body.

[0141] Optionally, the touch body is a finger of a user.

[0142] In another possible implementation, the operation area includes a first operation area and a second operation area; the key position display area of ​​the virtual keyboard is in the center of the touch display screen, and the first operation area and the second operation area are located on both sides of the key position display area of ​​the virtual keyboard; or, the key position display area of ​​the virtual keyboard is on one side of the touch display screen, the first operation area is on the other side of the touch display screen, and the second operation area overlaps with the key position display area of ​​the virtual keyboard.

[0143] In another possible implementation, the operation area and the key position display area are located on different touch display screens; or, the operation area and the key position display area are located on different devices.

[0144] The virtual keyboard-based input device 1800 according to the embodiment of the present application may correspond to executing the method described in the embodiment of the present application, and the above and other operations and / or functions of each module in the virtual keyboard-based input device 1800 are respectively to implement Figure 5-17 For the sake of brevity, the corresponding processes of each method in are not repeated here.

[0145] The present application also provides an electronic device, including at least one processor, a memory and a communication interface, wherein the processor is used to execute Figure 5-17 The method described.

[0146] Fig.19 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0147] like Fig.19As shown, the electronic device 1900 includes at least one processor 1901, a memory 1902, a communication interface 1903 and a display screen 1904. Among them, the processor 1901, the memory 1902, the communication interface 1903 and the display screen 1904 are connected in communication, and the communication connection can be achieved by wired means (such as a bus) or by wireless means. The communication interface 1903 is used to send and / or receive data sent by other devices; the memory 1902 stores computer instructions, and the processor 1901 executes the computer instructions and executes the method in the aforementioned method embodiment to solve the interaction problem between the virtual keyboard of the widescreen device and the user, improve input efficiency, and improve user experience. The display screen 1904 is a touch display screen, and the display screen displays an input interface. The input interface includes a key display area and an operation area of ​​the virtual keyboard, and the operation area and the key display area of ​​the virtual keyboard are located in different positions.

[0148] It should be understood that in the embodiment of the present application, the processor 1901 may be a central processing unit CPU, and the processor 1901 may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0149] The memory 1902 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1901. The memory 1902 may also include a nonvolatile random access memory.

[0150] The memory 1902 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0151] It should be understood that the electronic device 1900 according to the embodiment of the present application can execute the implementation of the embodiment of the present application. Figure 5-17 The method shown, the detailed description of the implementation of this method can be found above, for the sake of brevity, it will not be repeated here.

[0152] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer instructions are executed by a processor, the above-mentioned method is implemented.

[0153] An embodiment of the present application provides a chip, which includes at least one processor and an interface, wherein the at least one processor determines program instructions or data through the interface; the at least one processor is used to execute the program instructions to implement the method mentioned above.

[0154] An embodiment of the present application provides a computer program or a computer program product, wherein the computer program or the computer program product comprises instructions, and when the instructions are executed, the computer is caused to execute the above-mentioned method.

[0155] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0156] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented by hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0157] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. An input method based on a virtual keyboard, characterized in that: include: Displaying an interactive interface of an electronic device; the interactive interface includes an input box and a virtual keyboard; the virtual keyboard includes a key display area and an operation area, wherein the key display area and the operation area are located at different positions; detecting a first input operation acting on the operation area; Mapping the first input operation to an operation of a corresponding key in the key position display area to obtain a second input operation for the virtual keyboard; Based on the second input operation, obtaining an input event; In response to the input event, input information corresponding to the input event is displayed in the input box.

2. The method according to claim 1, characterized in that The first input operation includes a click operation; The mapping the first input operation to the operation of the corresponding key in the key position display area to obtain the second input operation for the virtual keyboard includes: Obtaining the click position of the click operation in the operation area; Mapping the click position to the key position display area to obtain a mapping position; Based on the mapping position, a click operation on a target key in the virtual keyboard is obtained.

3. The method according to claim 2, characterized in that The step of mapping the click position to the key position display area to obtain the mapping position includes: Using the click position as input of a neural network model and outputting the mapped position; Alternatively, the click position and the position information of each key in the virtual keyboard are used as inputs of a neural network, and the mapping position is output; Alternatively, the click position and the movement information of the touch object are used as inputs of a neural network, and the mapping position is output.

4. The method according to any one of claims 1 to 3, characterized in that: The first input operation includes a click operation, and the click operation includes a number of historical click operations; Mapping the first input operation to an operation of a corresponding key in the key position display area to obtain a second input operation for the virtual keyboard also includes: Obtaining information of the several historical click operations; Based on the information of the several historical click operations, the click position of the current click operation is adjusted.

5. The method according to claim 4, characterized in that The several historical click operations include the timing information of each historical click operation in the several historical click operations and the click position information of each historical click operation.

6. The method according to any one of claims 1 to 5, characterized in that: Also includes: In response to the second input operation, a feedback event corresponding to the second input operation is generated in the key display area.

7. The method according to claim 6, characterized in that The second input operation includes a target button being clicked, and the feedback event includes one or more of the target button being highlighted, the target button vibrating, and the target button being pressed.

8. The method according to any one of claims 1 to 7, characterized in that: The user performs input operations in the operation area through the touch body; The method further comprises: Acquiring the spatial position of the touch object; Based on the spatial position of the touch body and a mapping relationship, obtaining a target position of the touch body in the key display area, wherein the mapping relationship indicates a mapping relationship between each spatial position of the touch body and each position of the key display area; A preset style icon is displayed at a target position in the key position display area.

9. The method according to claim 8, characterized in that Also includes: Acquiring image information and / or capacitance signal distribution information and / or depth information distribution information of the touch-sensitive object; Based on the image information and / or the capacitance signal distribution information and / or the depth information distribution information of the touch object, the touch object is visually rendered and displayed at a target position in the key position display area.

10. The method according to claim 8 or 9, characterized in that: The touch object is the finger of the user.

11. The method according to any one of claims 1 to 10, characterized in that: The operation area includes a first operation area and a second operation area; The key position display area is located in the center of the touch display screen, and the first operation area and the second operation area are located on both sides of the key position display area; Alternatively, the key position display area is on one side of the touch display screen, the first operation area is on the other side of the touch display screen, and the second operation area overlaps with the key position display area.

12. The method according to any one of claims 1 to 10, characterized in that: The operation area and the key display area are located on different touch screens; Alternatively, the operation area and the key position display area are located on different devices.

13. An electronic device comprising a memory and a processor, characterized in that: Instructions are stored in the memory, and when the instructions are executed by the processor, the method according to any one of claims 1 to 12 is implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images