Methods and devices for dynamically adapting a virtual keyboard

Abstract

Method for providing a dynamically adapted virtual keyboard, the method comprising the following: on a display of a mobile computing device, displaying at least part of an electronic form that includes input objects for accepting user input; Automatic parsing of the electronic form to recognize attributes of at least one of the input objects; Automatically determining, based on parsing results, whether the electronic form includes a non-text input object; In response to the detection that the electronic form includes a non-text input object, a corresponding non-text component for virtual keyboards is automatically added to a virtual keyboard; and Displaying the virtual keyboard with the added non-text component for virtual keyboards on the mobile computing device's display; where the added non-text component is configured for virtual keyboards to allow a user to invoke a function associated with the corresponding non-text input object in the electronic form from the virtual keyboard.

Description

TECHNICAL AREA

[0001] The present disclosure relates generally to data processing systems. More specifically, the present disclosure relates to virtual keyboards for data processing systems, such as a portable (handheld) mobile computing device. GENERAL STATE OF THE ART

[0002] When a person (the "user") interacts with a software application on a data processing system, the software application can cause the data processing system to display various input objects for capturing information from the user. These input objects can display text fields and other user interface (UI) objects. For example, an email application might display a pop-up window prompting the user to enter their login credentials, an accounting application might display a user interface for recording business expenses, or a web browser application might display a webpage for registering on a social media website or a window for participating in an online chat.

[0003] For the purposes of this disclosure, the nouns "display" and "monitor" refer to the hardware component that the data processing system uses to present visual information to the user. The noun "screen" refers to all content displayed on a monitor at any given time. The noun "window" refers to a portion of the display used by an application to jointly display a set of related content elements. The noun "page" refers to a set of related content elements intended to be displayed together in a window, although it may not be possible for all elements to fit into the window at once; and scrolling, page turning, or zooming in may be necessary to bring some parts of the page into the window.The noun "electronic form" refers to a page containing one or more input fields for capturing information from the user. For example, a data processing system might display a screen containing multiple windows. One of these windows could be a web page of a web-based email application. This page might include input fields for registering with the email application. Accordingly, this page could be called an electronic form. Part of the electronic form might be visible in the window. The user can scroll down or peg through the window to see more of the electronic form.

[0004] Filling out electronic forms on a desktop or laptop computer with a full-size monitor, a separate full-size keyboard, and a mouse or other pointing device can be relatively straightforward. The full-size monitor allows the user to view most of the text fields and other UI objects of the electronic form. The full-size keyboard may include a tab key, enabling easy navigation between adjacent text fields of the electronic form. Furthermore, the mouse allows the user to interact with certain input objects on the electronic form (e.g., buttons for functions such as "Confirm," "Save," or "Submit"), provided the electronic form is small enough or the monitor screen is large enough to display these input objects along with the rest of the form.

[0005] However, in recent years, several data processing systems have been developed to operate without separate input devices such as a keyboard and mouse. For example, a portable (handheld) mobile computing device (a "handheld device") can use touchscreen technology to provide both visual output and user input on the screen. Specifically, the handheld device can display a collection of buttons or other objects ("virtual keyboard components") on its screen that together resemble a keyboard. And when the user touches the virtual keyboard components, which look like keys, the handheld device can respond as if the user had pressed corresponding keys on an actual keyboard.For the purposes of this disclosure, the term "virtual keyboard" refers to a collection of selectable virtual keyboard components (e.g., alphanumeric buttons) displayed by a data processing system in a predetermined portion of the screen. This portion of the screen may be referred to as a virtual keyboard image or window. In contrast, the term "conventional keyboard" refers to a physical keyboard distinct from the display. Similarly, the term "key" refers to a part of a conventional keyboard that is pressed to provide user input, whereas the term "virtual key" refers to a selectable virtual keyboard component represented as part of a virtual keyboard. As stated above, virtual keys may also be referred to as buttons.

[0006] Typically, a handheld device's operating system (OS) provides a standard virtual keyboard, and applications running on the device can use this standard virtual keyboard by default. Because such a virtual keyboard can be used by multiple applications, it can be referred to as a general-purpose virtual keyboard. In addition, third parties can create virtual keyboards with other features for installation and use on handheld devices. For example, Swype™ virtual keyboard from Swype, Inc., allows a user to select a set of virtual keys without having to lift their finger between them, and a handheld device can allow its user to install and use the Swype™ keyboard instead of a standard virtual keyboard.Since such a virtual keyboard can be used by multiple applications, it can also be called a general-purpose virtual keyboard. Furthermore, a software application developer has the option of creating a new virtual keyboard specifically for that application. The developer might even create other virtual keyboards with different virtual keys for each different screen or electronic form within that application. These types of virtual keyboards can be referred to as application-specific virtual keyboards or form-specific virtual keyboards. However, there are many reasons why developers avoid application-specific and form-specific virtual keyboards; these include reasons related to the time and cost involved in creating and maintaining such virtual keyboards.Consequently, applications for handheld devices typically use virtual general-purpose keyboards provided by the OS.

[0007] Virtual keyboards allow handheld device developers to produce smaller devices with fewer mechanical parts compared to devices with a monitor and a separate conventional keyboard. However, filling out electronic forms on a handheld device with a virtual keyboard can be challenging. For example, some users prefer the tactile feel of a physical keyboard with keys that can be pressed. Furthermore, screen size can pose significant challenges, as the handheld device may use a small screen that offers very little space for displaying information to the user.

[0008] For example, if an electronic form for a handheld device contains a lot of content, the user may need to zoom in on part of the form due to the small screen size. This is necessary to make the text or other UI objects in that part large enough to read or use. However, zooming in typically reduces the portion of the electronic form visible on the screen, making parts of the form that extend beyond the screen's edge unreadable. These challenges are further compounded if a large portion of the screen is occupied by the virtual keyboard. Additionally, the virtual keyboard may lack virtual keys that function like the Tab key on a conventional keyboard to aid navigation within the electronic form.

[0009] For this reason, the user may need to perform additional operations to enlarge the view, as well as operations to move the view and to select, in order to view and select the input object in those parts of the electronic form that have been enlarged beyond the screen.

[0010] US 2005 / 0200907A1 describes a web server for generating display control information for splitting and displaying a form in a style compatible with a client device used by the user, comprising: an HTTP request receiving unit for receiving a form request from the client device, an application code database for storing a screen definition of the form that is the subject of the form request and code from a validator for performing a validation of an input value entered into an input field of the form, a form splitting unit for splitting the form according to the determined terminal capacity using the read screen definition of the form and the read code from the validator, and a screen generation unit for generating screen information to be displayed on the client device using a splitting result.

[0011] US 2008 / 0158160A1 describes a method for completing electronic forms, comprising identifying data fields in an electronic form, receiving user input for at least one data field of the electronic form, retrieving or selecting candidates for data field entries for the electronic form from a database based on the received user input and a user identification, and proposing a data field entry based on the selection.

[0012] DE 20 2008 000 258 U1 describes a portable electronic device comprising: a touchscreen display, one or more processors, memory, and a program, wherein the program is stored in the memory and is configured to be executed by the one or more processors.The program includes the following: instructions to display, in a first area of ​​the touchscreen display, a current string entered by a user using a keyboard; instructions to display, in a second area of ​​the touchscreen display, the keyboard, which includes a spacebar key; instructions to display a suggested replacement string in the spacebar key of the keyboard; instructions to replace the current string in the first area with the suggested replacement string when the user performs a predetermined gesture related to the spacebar key on the keyboard; and instructions to retain the current string in the first area when the user performs a second predetermined gesture related to the touchscreen display. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The functions and advantages of the present invention will become apparent from the accompanying claims, the following detailed description of one or more exemplary embodiments, and the corresponding figures. The following applies: Fig. Figure 1 is a block diagram of a data processing system with a virtual keyboard module according to at least one example implementation; Fig. 2A, Fig. 2B and Fig. Figure 2C shows schematic diagrams illustrating different output configurations according to at least one example implementation; Fig. Figure 3 shows a flowchart of a process for dynamically adapting a virtual keyboard according to at least one example implementation; and Fig. Figures 4A to 4D show block diagrams illustrating components for providing virtual keyboards according to various alternative embodiments. DETAILED DESCRIPTION OF ONE OR MORE VERSIONS

[0014] A virtual general-purpose keyboard can be set up like a standard QWERTY keyboard by default. In addition, the user can change the available virtual keys or buttons by pressing an option button on the virtual keyboard. For example, pressing a "SYM" button can change the virtual keyboard to display number symbols and general symbols instead of alphabetic characters. A virtual general-purpose keyboard with virtual buttons that do not change unless requested by the user (e.g., by pressing a "SYM" or a "Shift" button) can be referred to as a static virtual keyboard.

[0015] When a person uses a handheld device with a static virtual keyboard to interact with an electronic form, they may have to perform many different operations, and at least some of these operations may be quite difficult. To begin, the user may need to zoom in to enlarge the text and other UI objects in the electronic form to a readable size. The user may then need to select an initial input field (for example, a username field). In response to the user selecting an alphanumeric input field, the operating system may automatically cause a virtual keyboard to appear (or "pop up") in a predetermined part of the screen, along with a blinking cursor at the beginning of the selected input field.When an input field is selected, the field can be described as active or in focus. Similarly, the virtual keyboard, when displayed, can be considered active, and when not displayed, it can be considered inactive. When the virtual keyboard is active, its window can cover the lower half of the display area, significantly reducing the available space for viewing the electronic form. Consequently, the virtual keyboard will likely obscure text fields and other UI objects within the electronic form (e.g., command buttons for functions such as "Confirm," "Save," or "Submit").

[0016] In any case, after the virtual keyboard is displayed, the user can use it to enter alphanumeric data for the active input field. After entering the desired data for the first input field, the user may need to zoom in or pan the view of the electronic form on the screen to display the next input field in the portion of the screen not occupied by the virtual keyboard. The user must then zoom in to make the text in the electronic form large enough to be legible and to verify that the next desired input field has been located. The user can then select the input field and use the virtual keyboard to enter further data.The process of panning, zooming, selecting, and entering data can continue until the user has entered all the data required for the electronic form. The user can then perform further panning and zooming operations to locate and select a UI object for the desired action (e.g., "Confirm," "Save," or "Submit"). Furthermore, if input objects are small and / or too close together, selecting the desired input object can be difficult during this process.

[0017] According to the present disclosure, in at least one embodiment, the virtual keyboard module parses the electronic form being used and dynamically modifies a virtual keyboard based on the determined attributes of the form. Thus, the virtual keyboard module provides a dynamically adapted, general-purpose virtual keyboard.

[0018] For example, if the virtual keyboard module detects that an electronic form has more than one input object, the virtual keyboard module can automatically add one or more navigation buttons to the virtual keyboard (e.g., a "Next" and "Previous" button), allowing the user to move the focus from the current input object to the next or previous input object.

[0019] Furthermore, the virtual keyboard module can automatically detect command buttons in the form and, in response, automatically add corresponding command buttons to the virtual keyboard. For example, if the electronic form is a login popup window with text fields for a username and password, as well as "Confirm" and "Cancel" buttons, the virtual keyboard module can automatically add a "Confirm" button to the virtual keyboard. When the user taps the "Confirm" button on the virtual keyboard, the virtual keyboard module can convert this input into an entry indicating that the user has selected the "Confirm" button in the form, and the underlying application can then process the input accordingly.Furthermore, the virtual keyboard module can automatically detect other input objects in the form and, in response, automatically add corresponding virtual keyboard components to the virtual keyboard. For example, if the electronic form includes an input object for selecting a color, the virtual keyboard module can automatically add a color picker to the virtual keyboard, and if the electronic form contains a text field for receiving date data, the virtual keyboard module can add a date picker to the virtual keyboard.

[0020] Fig. Figure 1 is a block diagram of a data processing system 20 with a virtual keyboard module 60 according to an exemplary embodiment. In the illustrated embodiment, the data processing system 20 is a portable (handheld) mobile computing device 20 with at least one processor 30, data storage 34, and a display 40 that responds to the processor 30. In the illustrated embodiment, the virtual keyboard module 60 is implemented as software stored in a data storage 34 and executed by the processor 30 to provide the functionality described below. An operating system 50 and one or more applications 52 are also stored in the data storage 34, and when these components are executed on the processor 30, they can call the virtual keyboard module 60.

[0021] In the embodiment of Fig. 1 The virtual keyboard module 60 comprises a parsing engine 62, a layout engine 64, and an input manager 66. As described in more detail below, the parsing engine 62, layout engine 64, and input manager 66 can work together to ensure the dynamic adaptation of the virtual keyboard based on attributes of the electronic form with which the user is interacting. In Fig. The components of a specific design architecture are described below. In alternative embodiments, other architectures can be used to provide dynamically adapted general-purpose virtual keyboards. For example, instead of using a central virtual keyboard module containing the parsing engine, layout engine, and input manager as subcomponents, the functionality can be implemented using a virtual keyboard system that employs one or more equal-level modules working together to provide a dynamically adapted general-purpose virtual keyboard. Furthermore, some of the functionality can be provided by other modules. For example, some or all of the input manager's functions can be implemented as part of a text subsystem, as part of the operating system, or as part of another module.Similarly, some or all of the functions of the parsing engine and other components can be implemented as part of another module. Many additional variations can be used in different implementations to provide the functionality described in this text.

[0022] Fig. Figures 4A through 4D show block diagrams illustrating components for providing virtual keyboards according to various alternative implementation architectures 410-425. For example, according to architecture 410, the virtual keyboard manager, parsing engine, layout engine, and input manager are implemented as relatively independent, equal-ranking components at the same hierarchical level as the OS. Architecture 411 is similar, except that the layout engine is implemented as part of the virtual keyboard module. Architecture 412 is similar to architecture 411, except that the parsing engine is implemented as part of the input manager. Architecture 413 is similar to architecture 411, except that the virtual keyboard module is implemented as part of the layout engine. In this hierarchy, the layout engine can also select from several different virtual keyboard modules.Architecture 414 is similar to architecture 413, except that the parsing engine is implemented as part of the input manager. In this hierarchy, the layout engine can also select from several different virtual keyboard modules. Architecture 415 is similar to architecture 411, but the input manager is also implemented as part of the virtual keyboard module. For architectures 410-415, the components are not directly part of the operating system.

[0023] In Architecture 416, the components are implemented as relatively independent, equal-ranking components in the OS. Architecture 417 is like Architecture 416, except the layout engine is implemented as part of the virtual keyboard module. Architecture 418 is like Architecture 417, except the parsing engine is implemented as part of the input manager. Architecture 419 is like Architecture 416, except the virtual keyboard module is implemented as part of the layout engine. Architecture 420 is like Architecture 419, except the parsing engine is implemented as part of the input manager. For Architectures 419 and 420, the layout engine can select from multiple virtual keyboards.

[0024] Architecture 421 is like the Fig. Architecture 1 is shown, however, the virtual keyboard module is implemented as part of the OS. Architecture 422 is similar to architecture 421, however, the parsing engine is implemented relatively independently of the virtual keyboard module.

[0025] In architectures 423-425, one or more components for providing virtual keyboards are implemented as part of an application. For example, according to architecture 423, the parsing engine is implemented as part of the application, with the other components embedded according to X, where X corresponds to any architecture from the set comprising architectures 410-422 (the parsing engine from those architectures being replaced by the parsing engine in the application). Similarly, for architecture 424, Y denotes any architecture from the set comprising architectures 410-422, the virtual keyboard module, layout engine, and parsing engine from those architectures being replaced by the virtual keyboard module, layout engine, and parsing engine from architecture 424.Likewise, for architecture 425, Z denotes any architecture from the set comprising architectures 410-422, wherein the virtual keyboard module, layout engine and parsing engine from these architectures is replaced by the virtual keyboard module, layout engine and parsing engine from architecture 425.

[0026] Several different programming languages ​​or combinations thereof can be used to implement the required components in various environments. For example, a combination of C and C++ can be used, along with specific UI hooks for the respective UI technology (e.g., JavaScript for Hypertext Markup Language (HTML), Qt Meta Language or Qt Modeling Language (QML), and C++ for QML, etc.). Implementation details may differ for implementations designed for various platforms or devices, such as devices using the Android operating system, devices using the iPhone operating system, devices using the Windows CE operating system, and so on.

[0027] Fig. 2A, Fig. 2B and Fig. Figure 2C shows schematic diagrams illustrating different output configurations on the display 40 of the mobile device 20. For example, it shows Fig. Figure 2A represents a sequence of graphical user interface (GUI) screens displayed on the display 40 by the application 52 in conjunction with the operating system (OS) 50 and the virtual keyboard module 60. The application 52 can be a web browser application or another type of application that allows user input. To illustrate one or more functions of one or more embodiments of the present invention, the illustrated screens show various options and operations that enable the user to log in to an email account. Of course, the same functions, similar functions, and / or related functions from different types of applications can be used for many other purposes.

[0028] The screen 100 in Fig. 2A represents a window that uses the entire space on the display 40 to show various elements to the user. These elements include some explanations and instructions in text form, followed by two text fields 112 and 114, and a command button 118. For the purposes of this disclosure, the term "text field" means an input field that accepts alphabetic input, numeric input, or both alphabetic and numeric (alphanumeric) input. Fig. In sections 2A to 2C, text field 112 is used to obtain a user ID from the user, and text field 114 is used to obtain a password from the user. Command button 118 is used to trigger a registration function or event after the user has entered a user ID and password. For the purposes of this disclosure, objects that accept alphabetic or numeric input (e.g., text fields 112 and 114) are referred to as text input fields or text input objects. In contrast, input objects that do not accept alphabetic or numeric input may be referred to as non-text input objects. Similarly, virtual keyboard components that are not text buttons (such as alphabetic and numeric buttons) may be referred to as non-text virtual keyboard components.

[0029] As shown in the dashed circle 122 on screen 120, the user can tap on text field 112 to enter a user ID.

[0030] In response, the virtual keyboard module 60, as shown in screen 140, can open a keyboard window 160 over the lower half of the display 40, leaving a truncated electronic form window 150 at the top of the display 40. (In contrast, the electronic form window 150 is not displayed in screen 120, since in this case the form window 150 is area-equal to display 40 (i.e., it occupies the entire area of ​​it)). For the purposes of this disclosure, the keyboard window 160 can also be referred to as the virtual keyboard 160. As shown in screen 140, when the virtual keyboard 160 is open or active, some important parts of the electronic form used by the application 52 for user input are not visible in the electronic form window 150.

[0031] However, before the virtual keyboard module 60 displays the virtual keyboard 160, the virtual keyboard module 60 performs a variety of operations to dynamically adapt the virtual keyboard 160 based on the attributes of the electronic form being used.

[0032] Fig. Figure 3 shows a flowchart of a process for dynamically adapting a virtual keyboard according to at least one example implementation. The illustrated process can begin when the user taps a text field of an electronic form, such as text field 112. As shown in block 300, in response to the detection of a tap on a text field, the mobile device 20 transfers control to the virtual keyboard module 60. As shown in block 310, the virtual keyboard module 60 can then determine whether the content of the electronic form has changed. For example, if the virtual keyboard module 60 is being called for the first time for the current electronic form, the virtual keyboard module 60 concludes that the content has changed.In addition, the virtual keyboard module 160 can conclude that the content has been changed if (a) a component providing information to the form provides updated data; (b) a loss of network connectivity is detected for a form that is retrieved from one or more online sources, or for a form that requires a connection to accept or proceed with user input; or (c) other changes are detected.

[0033] If the content has been changed, the parsing engine 62 can parse the electronic form – as shown in Block 312. Specifically, the parsing engine 62 identifies or recognizes various input objects or widgets in the form and captures various attributes associated with these objects. For example, the parsing engine 62 can, for the input shown in screen 100, Fig. The parsing engine 62 can capture information about the electronic form shown in Figure 2A. It recognizes that the form contains text field 112, that text field 112 is labeled "User ID," and that text field 112 accepts alphanumeric data up to a certain length. Parsing engine 62 can capture similar information for text field 114. Furthermore, parsing engine 62 can recognize that the electronic form includes command button 118, that command button 118 is labeled "Register!", and that command button 118 is configured to trigger a specific event or function when selected. In other cases, electronic forms may contain many other types of input objects, and parsing engine 62 can capture the relevant attribute for these objects.The input objects for which the parsing engine extracts 62 attributes include, without restriction, the following object types: dropdown lists, combo boxes, date fields, time fields, numeric fields, addresses, colors, maps, contacts, Uniform Resource Locators (URLs) or other file paths, audio / video objects, signatures or handwritten input (e.g., input of kanji characters).

[0034] The Parsing Engine 62 can also capture attributes for input objects based on tags associated with those objects. For example, an electronic form might include class tags to identify input objects intended for a specific type of data. These class tags adhere to conventions such as those of a particular microformat. For instance, geographic coordinates can be represented in an HTML document using the following tag set and class: Meeting Point at 52.48 , -1.89

[0035] An electronic form written in HTML can use the same classes for encoding a set of fields that prompt the user to enter an exact location: Meeting point at <form class="geo"> <input type="textbox" class="latitude" name="lat"> 52.48, <input type="textbox" class="longitude" name="long"> -1.89 <input type="'submit"> < / form>

[0036] When the form is parsed, the parsing engine 62 can recognize that the input field "lat" represents a latitude coordinate and the input field "long" represents a longitude coordinate. Similar techniques can be used to parse electronic forms with input fields for specifying a location as an address, and to parse forms with other types of input fields.

[0037] During or after parsing the electronic form, the parsing engine 62 can arrange some or all of the extracted information into one or more lists to be used for generating dynamic buttons for the virtual keyboard. For example, when parsing the electronic form from screen 120, the parsing engine 62 can generate the following structured list of UI element descriptions: <1, user ID, textbox, focus> <2, password, textbox, nofocus> <3, Register!, submit, nofocus>

[0038] Parsing Engine 62 can also populate the structured list of UI element descriptions with state information, such as whether a checkbox has been checked, which item has been selected in a dropdown list, and so on. Furthermore, the parsing engine can populate the list with contextual information based on previous usage. For example, Parsing Engine 62 can create a list or tree of state data that includes information about previous entries in a text field.

[0039] If the electronic form is a structured UI, the Parsing Engine 62 can examine the object memory of the UI framework that describes the final on-screen layout to extract the relevant input objects and attributes from the form. Structured UIs include, without limitation, interfaces based on HTML, QML, the GNU Image Manipulation Program (GIMP) Toolkit (GTK), the Enlightenment Foundation Libraries (EFL), the Clutter software library, and the Extensible Application Markup Language (XAML). If the electronic form is not a structured UI, the Parsing Engine 62 can examine other aspects of the form to extract the relevant input objects and attributes.

[0040] After parsing the electronic form, or after determining that the content has not been modified, the process passes to block 316, where virtual keyboard module 60 evaluates the current application and system context associated with the form. For example, for application context, virtual keyboard module 60 can determine which input object is in focus. For system context, virtual keyboard module 60 can determine which application type is using the electronic form, or what the general intention of the person using this application is. For example, virtual keyboard module 60 can determine that the user intends to take a photo, create a contact entry, or achieve some other end result.

[0041] Therefore, at least in one implementation, the virtual keyboard module automatically recognizes the control semantics or meanings of the input objects in an electronic form and dynamically adjusts the input layout for a virtual keyboard based on this recognized control semantics. A control has a specific meaning when used in a particular context. The virtual keyboard module can recognize this meaning and translate it into further functions within the virtual keyboard. For example, an email client might use a plain text field to receive an "To:" address from the user. The developer can go beyond this basic functionality and declare the text field as an "email input" field.The added declaration provides further semantics or meaning, which then allows the text field to, for example, perform input validation to ensure that the user enters a correct email address. A virtual keyboard module according to the present disclosure can use this and other types of control semantics to provide improved virtual keyboards. Furthermore, the virtual keyboard module can derive new control semantics or meanings based on the layout of the electronic form. For example, an email client might display a form with fields for "To:", "Subject," and "Text," as well as a "Submit" button. The keyboard copy of the "Submit" button, while designed as a default action for the form, might disable itself if the "To:" field does not contain a valid email address.

[0042] Furthermore, any user interaction with the system or the virtual keyboard, as well as general context changes within the system, can cause the virtual keyboard module 60 to rescan and parse the electronic form and dynamically change the virtual keyboard. As an example of a system context change, if a background installation completes that provides a new input mode (e.g., a map location picker), and the user interacts with an electronic form that has an input object corresponding to an address entry, the virtual keyboard module 60 can dynamically add the map location picker to the virtual keyboard.

[0043] As shown in Block 318, after the virtual keyboard module 60 parses the electronic form and evaluates the context, the Layout Engine 64 uses the parsing results and the extracted context data to determine which dynamic virtual keyboard components should be included in the virtual keyboard. In other words, the Layout Engine 64 determines the layout and appearance of the virtual keyboard based on the structured list of UI elements and the extracted context data.

[0044] If the user, for example, refers again to screen 120 in Fig. 2A - when the user taps on text field 112, the virtual keyboard module 60 can respond using the parsing engine 62 to parse the electronic form and analyze the context as described above. Then, using the layout engine 64, it dynamically determines which virtual keyboard components should be included in the virtual keyboard based on the parsing and analysis results. For example, with reference to screen 120, the layout engine 64 can determine that the virtual keyboard should include the following virtual keyboard components: a) Navigation buttons (e.g. “Next” and “Previous”) to move the focus from the current input object to the previous or next input object; b) the label for the next input object, which is displayed in the navigation button “Next” (either with or without other text, such as “Next:”); c) more or less conventional alphanumeric virtual keys; d) a command button to trigger the event or action linked to the "Register!" command button on the electronic form; and e) the label for the “Register!” button, which is displayed in the corresponding command button in the virtual keyboard.

[0045] Layout Engine 64 can decide that the navigation buttons should be taken into account in response to the finding that the electronic form generally contains more than one input object and specifically more than one text field.

[0046] As shown in Block 320, Input Manager 66 can then result in a dynamically adapted virtual keyboard that includes these functions to be displayed on Display 40. For example, Input Manager 66—as shown in Screen 140 of the Fig. Figure 2A shows a virtual keyboard 160, which includes a navigation button 162 for "Previous," a navigation button 164 for "Next," and various more or less conventional alphanumeric buttons 166. As shown, the input manager 66 can also include a command button 168 for "Register!" in the virtual keyboard 160. Furthermore, the input manager 66 can configure the command button 168 to cause the function or event associated with the command button 118 to be invoked in response to the user's selection of the command button 168.

[0047] In other embodiments or when used with other electronic forms, the input manager can add other types of dynamic virtual keyboard components to the virtual keyboard. Such virtual keyboard components can include, without limitation, the following object types: drop-down lists, combo boxes, date fields, time fields, spin boxes (numeric fields linked to up and down arrows), address pickers, color pickers, address pickers for maps, contact pickers, file browsers, audio / video pickers with preview, signature pads, and handwriting input (e.g., for entering kanji characters).

[0048] Referring again to screen 140, the input manager 66 can also set the focus to the text field 112. For the purposes of this disclosure, navigation control involves controlling which input object is in focus. In contrast, cursor control involves controlling the position of the cursor or caret in an input field.

[0049] With renewed reference to Fig. 3. The input manager 66 can then receive user input and process this user input as described in blocks 322 and 324. The process then returns to block 310, where the virtual keyboard module 60 determines whether the content of the electronic form has been changed and performs further operations as described above. As shown in blocks 318 and 320, the virtual keyboard module 60 can then proceed with dynamically adjusting or changing the virtual keyboard 160.

[0050] For example, when processing user input, the input manager 66 can populate text field 112 with an entry (e.g., "JSmith") based on the alphanumeric buttons tapped by the user. Then, when the user taps the "Next" button (as shown in screen 170 of the Fig. (2B, represented by the dashed circle 172), the input manager 66 can automatically scroll or paginate the electronic form upwards until the text box 114 above the virtual keyboard 160 becomes visible and position the focus in the text box. Furthermore, the layout engine 64 can adjust the "Next" and "Previous" buttons accordingly. For example, the layout engine 64 can update the text in the navigation button 164 "Register!" to indicate that the user can move the focus to the command button 118 by tapping the navigation button 164. In another embodiment, the layout engine 64 includes labels of corresponding input objects in the "Previous" and "Next" buttons.

[0051] As shown in screen 180, while the focus is on text field 114, the input manager 66 can populate text field 114 with input based on the alphanumeric buttons tapped by the user.

[0052] Furthermore, the user can – as shown in the dashed circle 192 in screen 190 – Fig. Figure 2C illustrates this – the dynamically generated command button 168 “Register!” within the virtual keyboard 160 is tapped to trigger the registration function, even though the original command button 118 “Register!” of the electronic form cannot be displayed on screen 40. In response to the detection that the user has tapped command button 168, the input manager 66 can automatically invoke the function or event originally associated with command button 118.

[0053] Furthermore, the virtual keyboard module 60 can provide additional functions to enhance the user experience when interacting with electronic forms on the mobile device 20. For example, the virtual keyboard module 60 can configure one or more virtual keyboard components with the ability to detect two or more input modes (e.g., short tap and press and hold, or short tap and long tap), and the virtual keyboard module 60 can provide two or more functions depending on which input mode is used. For example, the virtual keyboard module 60 can equip the navigation buttons in the virtual keyboard 160 with additional functionality that can be accessed when the user presses and holds a navigation button instead of just tapping it.The dashed circle 202 in screen 200 represents the user pressing and holding the navigation button 162. As shown in screen 200, this additional or secondary functionality in one embodiment includes providing a pop-up menu 210 that lists all input objects in the form, so that the user can easily move the focus directly to a desired object by simply tapping the label for the field in the pop-up menu. In other embodiments, the virtual keyboard module may provide other secondary functions.

[0054] As previously described, a parsing method can parse an electronic form to determine the presence of control semantics associated with the input objects in the form. The method can also include dynamically adapting a virtual keyboard in response to the types and attributes of these input objects.

[0055] In one example implementation, a virtual keyboard module parses an electronic form that is generated and displayed by a software application. The software application can be a web browser, an email application, a word processor, or virtually any type of application.

[0056] After parsing the electronic form and analyzing the context, the virtual keyboard module displays a virtual keyboard that is dynamically adapted to the form. For example, the virtual keyboard module can add a specialized virtual keyboard component to the virtual keyboard, which is displayed along with more or less conventional alphanumeric buttons. Alternatively, the virtual keyboard module can display a virtual keyboard that contains the specialized virtual keyboard components but no alphanumeric buttons. For example, if the focus is moved to an input object on the electronic form that is associated with the color picker, the virtual keyboard module can populate the virtual keyboard with a color picker. Alternatively, the virtual keyboard module can also display some other virtual keyboard components (e.g.,Consider the color selection for navigation buttons and command buttons in the virtual keyboard.

[0057] The dynamic integration of virtual keyboard components can enable users to navigate between text fields more efficiently, as they no longer need to move the view to activate a different text field. The dynamic virtual keyboard can also provide users with simpler data entry mechanisms. Furthermore, the dynamic consideration of command buttons such as "Confirm" or "Save" (or other virtual keyboard components) can allow users to perform actions that would otherwise require moving the electronic form.

[0058] In the description above, some functions were referred to as input objects. However, it should be clear that the term "input object" is not limited to functions implemented through object-oriented programming, but also covers functions for receiving user input implemented using other types of programming methods, including procedural programming methods, structural markup languages ​​(e.g., HTML, XAML, etc.), and other methods.

[0059] With regard to the principles and exemplary embodiments described and illustrated in this text, it is evident that the illustrated embodiments can be modified in terms of their arrangement and details without deviating from such principles. Furthermore, while the preceding discussion focused on specific embodiments, other configurations are also possible. Therefore, phrases such as "in one embodiment," "in another embodiment," or similar expressions used in this text generally refer to possible embodiments and are not intended to limit the invention to specific configurations of embodiments. As used here, the terms can refer to the same or different embodiments, which can be combined into other embodiments.

[0060] Furthermore, components described as communicating with or responding to one another need not be in continuous communication unless expressly stated otherwise. It should also be clear that the hardware and software components described herein represent functional elements that are reasonably self-contained, such that each element can be designed, constructed, or updated independently of the others. In alternative embodiments, many of the components can be implemented as hardware, software, or combinations of hardware and software to provide the functionality described and illustrated herein. For example, alternative embodiments include machine-retrievable media encoding instructions or control logic for executing the operations of the invention. Such embodiments may also be referred to as program products.Such machine-retrievable media can include, without limitation, tangible storage media such as magnetic disk storage, optical disks, RAM, read-only memory (ROM), flash memory, etc. In at least one embodiment, the instructions for all components can be stored in a single non-volatile, machine-retrievable medium. In at least one other embodiment, two or more non-volatile, machine-retrievable media can be used to store the instructions for the components. For example, instructions for one component can be stored in one medium and instructions for another component in a different medium.

[0061] Alternatively, some of the instructions for a component can be stored in one medium, and the remainder of the instructions for that component (as well as instructions for other components) can be stored in one or more other media. Instructions can also be used in a distributed environment and stored locally and / or externally for access by a single- or multi-processor machine. In some embodiments, parts or all of the control logic for implementing the described operations can be implemented in hardware logic (e.g., as part of an integrated circuit (IC) chip, a programmable gate array (PGA), an application-specific integrated circuit (ASIC), etc.).

[0062] Similarly, even if one or more example processes have been described with respect to certain operations performed in a specific sequence, numerous modifications can be applied to these processes to derive numerous alternative embodiments of the present invention. For example, alternative embodiments may include processes that use fewer than all of the operations disclosed herein, processes that use additional operations, and processes in which the individual operations disclosed herein have been combined, split, rearranged, or otherwise modified.

[0063] Given the wide variety of useful implementations that can be easily derived from the example implementations described here, this detailed description is intended merely as an illustration and should not be understood as limiting the scope of the invention. The invention therefore claims all implementations that fall within the scope of the following claims and all equivalents to such implementations.

Claims

[1] Method for providing a dynamically adapted virtual keyboard, the method comprising: on a display of a mobile computing device, displaying at least part of an electronic form that includes input objects for accepting user input; Automatic parsing of the electronic form to recognize attributes of at least one of the input objects; Automatically determining, based on parsing results, whether the electronic form includes a non-text input object; In response to the detection that the electronic form includes a non-text input object, a corresponding non-text component for virtual keyboards is automatically added to a virtual keyboard; and Displaying the virtual keyboard with the added non-text component for virtual keyboards on the mobile computing device's display; where the added non-text component is configured for virtual keyboards to allow a user to invoke a function associated with the corresponding non-text input object in the electronic form from the virtual keyboard. [2] Method according to claim 1, wherein: the non-text input object in the electronic form includes a command button; and the added non-text component for virtual keyboards in the virtual keyboard includes a corresponding command button configured to allow the user to invoke the function associated with the command button in the electronic form from the virtual keyboard. [3] Method according to claim 1, wherein the non-text input object in the electronic form comprises an object from the group consisting of a command button and a list box. [4] Method according to claim 1, wherein: The process of displaying at least one part of the electronic form includes displaying a first part of the electronic form that does not include the non-text input object on a first part of the display; and The process of displaying the virtual keyboard with the added non-text component for virtual keyboards involves displaying the virtual keyboard on a second part of the display, while the first part of the electronic form, which does not include the non-text input object, is displayed on the first part of the display. [5] The method of claim 4, further comprising: Enabling the user to interact indirectly with the non-text input object in the electronic form via the virtual keyboard, even though the non-text input object is located in a second, unrepresented part of the electronic form. [6] Method according to claim 1, wherein the electronic form comprises an input screen generated by a software application. [7] Method according to claim 6, wherein the software application comprises a web browser. [8] Method according to claim 1, wherein: The process of parsing the electronic form includes the recognition of two or more input objects in the electronic form; and The procedure also includes the automatic addition of one or more navigation control buttons to the virtual keyboard in response to the detection of two or more input objects in the electronic form. [9] The method of claim 8, further comprising: Detect that the user has selected one of the added navigation control buttons in the virtual keyboard; In response to this, the focus is automatically moved to a different input object; the automatic determination of whether the other input object is displayed; and In response to the determination that the other input object is not displayed, the electronic form is automatically scrolled or flipped through the display to cause the other input object to be displayed. [10] Method according to claim 9, wherein: the two or more input objects comprise a first text input field and a second text input field; and The process of automatically moving the focus to a different input object includes automatically moving the focus from the first text input field to the second text input field. [11] The method of claim 1, wherein the method further comprises adding a virtual keyboard component to the virtual keyboard, wherein the virtual keyboard component is configured such that, in response to user interaction with the virtual keyboard component, a list of input objects from the electronic form is displayed in the virtual keyboard. [12] The method of claim 11, further comprising: Detect that the user has selected one of the input objects from the list displayed in the virtual keyboard; and In response, the focus is automatically moved to the selected input object. [13] The method of claim 11, further comprising: Detect that the user has selected one of the input objects from the list displayed in the virtual keyboard; In response to this, the focus is automatically moved to the selected input object; the automatic determination of whether the selected input object is displayed; and In response to the determination that the selected input object is not displayed, the electronic form is automatically scrolled or paginated on the display to cause the selected input object to be displayed. [14] Comprising at least one non-volatile, machine-retrievable medium: Instructions which, when executed by a mobile device, enable the mobile device to perform the method described in any one of claims 1 to 13. [15] Mobile computing device comprising: a processor; at least one local storage device that responds to the processor; and instructions in the local storage device which, when executed by the mobile device, enable the mobile computing device to perform the method described in any one of claims 1 to 13.

Images