Techniques for identifying user interface elements and systems and devices using the same

Abstract

To provide touch interfaces that have sufficient responsiveness and accuracy to be used with many applications.SOLUTION: Embodiments of the present disclosure relate generally to techniques for identifying elements in a user interface (UI), and more particularly, techniques for determining UI elements selected on a contact-sensitive user interface, and using those techniques to provide one or more haptic responses.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] (Priority Claim) This application claims the benefit under 35 U.S.C. § 119(e) of U.S. Provisional Application No. 62 / 507,902, filed May 18, 2017, the disclosure of which is incorporated herein by this reference in its entirety.

[0002] FIELD OF THE INVENTION Embodiments of the present disclosure relate generally to techniques for identifying elements in a user interface (UI), and more particularly to techniques for determining a selected UI element on a touch-sensitive user interface, and using such techniques to provide one or more haptic responses. [Background technology]

[0003] Touch interfaces incorporating touch sensing are used in a variety of applications, including, for example, tablet computers, personal computers, smartphones, and other consumer products. Touch interfaces are also used as control panels in automobiles, appliances (e.g., refrigerators, ovens, washer / dryers, etc.), heating and air conditioning control systems, security systems, and automatic teller machines (ATMs). Touch interfaces in these applications can be, for example, touchpads or can incorporate a screen and a graphical user interface (GUI).

[0004] In general, there is a need for a touch interface that is responsive and accurate enough for use in many applications. [Brief explanation of the drawings]

[0005] Objects and advantages of embodiments of the present disclosure will become apparent to those skilled in the art from the following detailed description taken in conjunction with the accompanying drawings. The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Patent and Trademark Office upon request and payment of the necessary fee. [Figure 1] FIG. 10 is a swim diagram illustrating the process of generating and using a search task list to identify touched UI elements within a GUI. [Figure 2] 1 is a flowchart of a process for generating a search task list according to one embodiment of the present disclosure. [Figure 3] 1 is a flowchart of a process for extracting UI elements from a UI configuration definition according to one embodiment of the present disclosure. [Figure 4] 1 is a flowchart of a process for generating an intermediate search tree according to one embodiment of the present disclosure. [Figure 5] 1 is a flowchart of a process for generating a search task tree according to one embodiment of the present disclosure. [Figure 6] 1 is a flowchart of a process for generating a search task list according to one embodiment of the present disclosure. [Figure 7] 1 is a flowchart of a process for determining whether a touch occurred within a UI element according to one embodiment of the disclosure. [Figure 8] 1 illustrates an embodiment of a wireless GUI with UI elements. [Figure 9] 9 illustrates UI elements of the wireless GUI of FIG. 8 grouped according to an embodiment of the present disclosure. [Figure 10] 1 illustrates a tree-like structure of UI elements formed in accordance with an embodiment of the present disclosure. [Figure 11] 1 illustrates one embodiment of a system incorporating a search task list. [Figure 12] 1 illustrates one embodiment of a wireless GUI including features and parameters associated with at least some of the UI elements of the wireless GUI. [Figure 13]12 illustrates an embodiment of the system of FIG. 11 integrated as a subsystem into an automobile head unit.

[0006] Disclosure Some embodiments of the present disclosure generally relate to a method for creating instructions for searching for elements of a graphical user interface (GUI) displayed on a touch-sensitive screen, including the steps of parsing a GUI definition and identifying elements of the GUI responsive to the parsing, creating records containing entries for the identified elements, associating the identified elements with groups of similarly positioned elements, arranging the records for the identified elements in a tree-like structure, collapsing identified elements in the same group into a single leaf within the tree-like structure, optimizing the tree-like structure, and creating a search instruction list responsive to the tree-like structure.

[0007] Some embodiments of the present disclosure generally relate to a computer program product that enables a computer to generate executable instructions for making elements of a graphical user interface (GUI) searchable. The program product may include a computer-readable medium and software instructions on the computer-readable medium. The software instructions on the computer-readable medium adapt a computer to perform the following operations: parse a GUI definition and identify elements of the GUI responsive to the parsed GUI definition, create records including entries for the identified elements, associate the identified elements with groups of similarly positioned elements, arrange the records of the identified elements in a tree-like structure, collapse identified elements in the same group into a single leaf within the tree-like structure, optimize the tree-like structure, and create a search instruction list responsive to the tree-like structure.

[0008] Some embodiments of the present disclosure generally relate to a microcontroller operably coupled to a touchscreen configured to display a graphical user interface (GUI). The microcontroller includes at least one processor and one or more executable instructions stored on a non-transitory storage medium that, when executed by the processor, are adapted to cause the processor to determine a location of a sensed touch on the touchscreen and identify a GUI element associated with the touch location corresponding to the sensed touch.

[0009] Some embodiments of the present disclosure generally relate to a method for identifying elements of a graphical user interface (GUI) displayed on a touchscreen, including determining a location of a sensed touch on the touchscreen, executing one or more search instructions corresponding to the location, each search instruction of the one or more search instructions corresponding to a GUI element and adapted to return search results when executed, and identifying the GUI elements corresponding to the search results.

[0010] Some embodiments of the present disclosure generally relate to a system. The system includes a display subsystem and a touch subsystem. The display subsystem is configured to control a display. The touch subsystem includes a touch sensor and a touch controller. The touch controller is configured to determine a location of a touch sensed by the touch sensor, execute one or more search instructions corresponding to the location and a search tree, where each search instruction of the one or more search instructions corresponds to a GUI element and is adapted to return search results when executed, identify the GUI element corresponding to the search result, and generate a haptic control message corresponding to the identified GUI element. DETAILED DESCRIPTION OF THE INVENTION

[0011] In the following detailed description, reference is made to the accompanying drawings, which form a part of this specification and which show, by way of illustration, specific embodiments in which the present disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present disclosure. However, other embodiments may be used, and changes in structure, material, and process may be made without departing from the scope of the present disclosure. The figures presented herein are not intended to be actual illustrations of any particular method, system, device, or structure, but merely idealized representations used to explain embodiments of the present disclosure. The figures presented herein are not necessarily drawn to scale. Similar structures or components in various figures may retain the same or similar numbering for the convenience of the reader. However, similarity in numbering does not imply that the structures or components are necessarily of the same size, composition, configuration, or any other characteristic.

[0012] It will be readily understood that the components of the embodiments, as generally described and illustrated in the figures herein, could be arranged and designed in a wide variety of different configurations. Thus, the following description of various embodiments is not intended to limit the scope of the disclosure, but is merely representative of various embodiments. While various aspects of the embodiments may be presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0013] The following description may include examples to assist those skilled in the art in enabling the disclosed embodiments. The use of the terms "exemplary," "example," and "for example" means that the associated description is explanatory, and the scope of the present disclosure is intended to encompass examples and legal equivalents. The use of such terms is not intended to limit the embodiments or the scope of the present disclosure to specific components, steps, features, functions, etc.

[0014] Furthermore, the specific implementations shown and described are merely examples and should not be construed as the only way to implement the present disclosure, unless otherwise specified herein. Elements, circuits, and functions may be shown in block diagram form so as not to obscure the present disclosure in unnecessary detail. Conversely, the specific implementations shown and described are merely exemplary and should not be construed as the only way to implement the present disclosure, unless otherwise specified herein. Furthermore, the block definitions and partitioning of logic among various blocks are exemplary specific implementations. It will be readily apparent to one skilled in the art that the present disclosure can be implemented with numerous other partitioning solutions. For the most part, details regarding timing considerations and the like have been omitted; such details are not necessary to obtain a complete understanding of the present disclosure and are within the capabilities of those skilled in the art.

[0015] Those skilled in the art will understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout this specification may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof. Some figures may illustrate signals as a single signal for clarity of presentation and explanation. Those skilled in the art will understand that a signal may represent a bus of signals, which may have various bit widths, and that the present disclosure may be implemented with any number of data signals, including a single data signal.

[0016] The various illustrative logic blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed using a general-purpose processor, a special-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor (which may also be referred to herein as a host processor or simply a host) may be a microprocessor, although the processor may alternatively be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. A general-purpose computer including a processor is considered a special-purpose computer, and the general-purpose computer is configured to execute computing instructions (e.g., software code) related to the embodiments of the present disclosure.

[0017] The embodiments may be described in terms of a process that is depicted as a flowchart, a flow diagram, a structure diagram, or a block diagram. While a flowchart may describe operational acts as a sequential process, many of these acts can be performed in a different order, in parallel, or substantially simultaneously. Additionally, the order of acts may be rearranged. A process may correspond to a method, a thread, a function, a procedure, a subroutine, a subprogram, etc. Furthermore, the methods disclosed herein may be implemented in hardware, software, or both. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media, such as any medium that facilitates transfer of a computer program from one place to another.

[0018] Any reference to elements herein using designations such as "first," "second," etc. does not limit the quantity or order of those elements unless such limitation is expressly stated. Rather, these designations may be used herein as a convenient method of distinguishing between two or more elements or instances of an element. Thus, reference to a first element and a second element does not imply that only two elements may be used or that the first element must precede the second element in any way. Additionally, unless otherwise specified, a set of elements may include one or more elements.

[0019] As used herein, the term "substantially" when referring to a given parameter, characteristic, or condition means and includes the extent to which one of ordinary skill in the art would understand that the given parameter, characteristic, or condition is met with small variations, such as within acceptable manufacturing tolerances. As an example, depending on the particular parameter, characteristic, or condition that is substantially met, the parameter, characteristic, or condition may be at least 90% met, at least 95% met, or even at least 99% met.

[0020] Various embodiments described in this disclosure generally relate to techniques for determining a selected UI element on a touch-sensitive user interface and using such techniques to provide one or more haptic responses. For purposes of understanding the embodiments described herein, a contact sensor can respond to an object (such as a finger or stylus) contacting a touch area of the touch interface or the proximity of the object to the touch area. In this disclosure, "touch" generally refers to physical contact between an object and a touch-sensitive area, but can also encompass the proximity of an object that produces a correspondence measurable by a contact sensor. Furthermore, a touch-sensitive area refers to a physical area on the touch interface where a contact sensor can respond to the contact of an object.

[0021] As used herein, a touch-sensitive GUI refers to a touch interface integrated with a GUI. For example, a GUI typically includes one or more display areas and active / activatable areas. In this disclosure, a display area is an area of a user interface that displays information to a user. An activatable area is an area of a GUI, such as a button, slider, or menu, where a user can take some action on the user interface. Some display areas are also activatable areas that display information and allow some action to be taken. In a touch-sensitive GUI, an active area can be activated by touching the touch-sensitive area where the active area appears (e.g., tapping a GUI button on a touchscreen). Active areas can be displayed as GUI elements / objects of all different shapes and sizes, such as buttons, sliders, selectable panes, menus, etc.

[0022] Generally, when a touch is sensed in a touch-sensitive area, a process is used to determine the active area of the GUI that corresponds to the touch, if applicable. For example, if an "Enter" button is tapped, the contact is measured and, corresponding to the measured contact, a process determines that the contact was an Enter button. Because the Enter button is an active area, an event is generated in the touch-sensitive GUI and / or the underlying application program that launches the GUI.

[0023] Additionally, when a particular GUI element is associated with an active area, actuators integrated with the touch interface can provide one or more physical responses, commonly referred to as haptic responses. These can be in the form of force, vibration, or movement, and can mimic surface texture, ridges, edges, button press / click interactions, and other simulated sensations and responses. In the case of a GUI, the haptic response can be localized to the GUI element with which the user interacts. For example, when a user touches a GUI button, the haptic response can provide the feel of the button with raised edges, as if the button were pressed, or as if it had a rough texture.

[0024] Various embodiments described herein may refer to creating and updating electronic records. Electronic records may be in the form of data files, and updating an electronic record may include inserting or deleting data entries into one or more fields of the record. Alternatively, it may refer to class objects and instantiated objects that, at run time, have state information and variables that match the described record. Both situations are contemplated in various embodiments described herein.

[0025] Various embodiments of the present disclosure relate to techniques for identifying touched GUI elements on a touch-sensitive interface. These techniques and related structures are particularly efficient in terms of memory usage and responsiveness. Furthermore, compared to other techniques, they have low interface data storage requirements and perform a small number of tasks at run time to identify UI elements.

[0026] Some embodiments of the present disclosure relate to a method for creating a list of optimized search tasks that can be performed to identify touched GUI elements on a touch-sensitive interface. The search tasks can be processor-executable instructions that, when executed, return a success or failure message, if applicable, to a subsystem searching for the touched GUI element. In one embodiment, the search tasks are created based on a definition file that maps various elements and their locations within the GUI. The search tasks are optimized for various performance parameters.

[0027] In one embodiment, the search task list can be performed by an embedded device, such as a touch controller, compared to traditional touch-sensitive GUIs known to the inventors of the present disclosure that can perform the search within a display subsystem (e.g., an automobile head unit). Performing the search for GUI elements within the touch controller saves time communicating with the display subsystem and the time it takes for the subsystem to react and communicate with, for example, a haptic feedback subsystem. The time saved improves the responsiveness of the touch-sensitive GUI compared to traditional touch-sensitive GUIs, and also reduces, from the user's perspective, the time between when the user touches the screen and when the user receives feedback responsive to the touch.

[0028] Additionally, the creation of the task search list is configurable, and depending on the GUI, a set of common features can be selected for optimized implementation for a particular application. For example, in some embodiments, the creation process can be optimized for GUIs that include paging, drop-down menus, and pop-up windows that hide other GUI elements, elements of a particular shape, or elements that move or change shape when touched.

[0029] 1 illustrates the overall operation of a system according to various embodiments of the present disclosure. At operation 112, the software application tool 102 is configured to process the UI definition file to create a search task list of conditionally executable instructions (operation 108) that can be performed to identify UI elements that have touched the touch-sensitive screen, if applicable.

[0030] The search task list may be stored in a non-transitory storage memory accessible by one or more processors that are part of the touch system (ACT 110). When a touch event occurs on the touch interface, the touch sensor 106 may detect the touch (ACT 118) and provide one or more signals representing the touch to the touch processor. The touch processor 104 determines the location on the touch interface where the contact occurred (ACT 112) and, if applicable, searches for and identifies the contacted UI element responsive to that determination (ACT 114). In one embodiment, the touch processor 104 may provide the search results to a graphical user interface subsystem (ACT 116).

[0031] One embodiment of a process for creating a search task list is described with reference to Figures 2, 3, 4, 5, 6, and 7. An embodiment of the present disclosure utilizes a search tree-like structure that organizes UI elements according to tree and grid techniques. Various UI elements are divided into related groups that are treated like a grid and organized into a search tree, and then various search tasks are generated. The search tasks are conditioned using instructions to optimize the execution of the search. Those skilled in the art will understand that other algorithms, such as divide-and-conquer techniques, can be used to divide the screen(s) into searchable areas.

[0032] 2 illustrates one embodiment of a process for generating a search task list. At operation 202, a UI structural definition is loaded and parsed to identify screens, sub-screens, and UI elements on the screens and sub-screens. The UI structural definition can be an electronic file, a database, raw data, etc. At operation 204, elements are grouped to divide the searchable space into one or more searchable regions with element groups. At operation 206, the groups are linked into a tree-like structure based on the searchable space. At operation 208, search tasks are associated with branches and nodes of the search tree to form a search tree and optimize the search tree. At operation 210, the conditional tasks of the search tree are stored in a list, which is a task list that can be executed by a processor.

[0033] In one embodiment, the software application tool 102 that generates the search task list can be configured to write the results of one or more of operations 202, 204, 206, 208, and 210 to an output file. This output file can be used by a debugging tool to review the results of the process. The same debugging tool can be configured to use a text version of the search task list and run it in a virtual test environment (e.g., a DOS executable) to verify that the search task list is ready for operation.

[0034] FIG. 3 illustrates one embodiment of a method 300 for extracting UI elements from a UI configuration definition. In one embodiment, the UI configuration definition is an XML definition of a portion of a display that is transformed by the configuration generation features of an application tool. The application tool parses the structure definition and grabs the elements defined in the definition structure. At operation 302, each UI element is loaded, and at operation 304, it is determined whether the UI element is a known UI element. If it is not a known UI element (i.e., this is the first time the element has been identified in the structure definition), at operation 306, the process creates a new element definition for that type of UI element (e.g., button, knob, slider, etc.). After creating the new definition, or if the element is a known UI element, at operation 308, it is determined whether to assign the element to an existing group. In one embodiment, the determination of whether to assign to an existing group is based on common characteristics of the elements, such as various predetermined parameters, for example, the type of element, the screen on which the UI element is displayed, the layer location, the type of response associated with the element (e.g., visual, tactile, audio, etc.), etc. If it is determined that the element should be assigned to a new group, then at operation 310 a new group record is created with parameters associated with the element. After the new group record is created, or if it is determined that the element should be assigned to an existing group, then at operation 312 inputs for the new element are inserted into the new element's group record. In one embodiment, the inputs include fields for the element ID and the element's location (i.e., the element's coordinates on the screen). At operation 314 it is determined whether there are more UI elements, and if so, the process runs for each remaining UI element identified in the UI configuration definition. At operation 316 the process returns the element(s), element definition(s), and group(s).

[0035] In one embodiment not illustrated in FIG. 3 , if a UI configuration definition includes more than one screen definition, each such screen is assigned a screen ID, which is a parameter of the UI elements and can also be incorporated as a parameter of each group. Each screen can also include subscreens, which are defined regions of the displayed GUI within which some UI elements change dynamically, while UI elements outside of those regions remain unchanged. By way of non-limiting example, regions with dynamic elements include swappable panes, scrollable menus, activatable information panes, navigation buttons, etc.

[0036] FIG. 4 illustrates a process 400 for creating a search tree according to one embodiment of the present disclosure. This process involves determining how to divide each screen identified in the UI definition into searchable regions, with each searchable region containing one or more groups of elements. In the embodiment of the process shown in FIG. 4, a dividing line (x coordinate, y coordinate) is selected that divides the groups of elements, so that at least one group is on one side of the dividing line and at least one other group is on the other side of the dividing line. Effectively, the screen is now divided into two searchable regions by a shared boundary along the dividing line. Groups are divided in a recursive manner until no further division of the groups is possible.

[0037] In another embodiment, the screen or searchable area is divided simultaneously in the x and y coordinate directions, which may result in subdivisions of up to four groups. This technique can also result in subdivisions of less than four, for example, divisions of three groups and one empty searchable area.

[0038] In still other embodiments, a circle, square, and / or polygon may be used to define a portion of the screen that is not subdivided into a searchable area, excluding that portion from the searchable area.

[0039] Proceeding through the process shown in FIG. 4, operation 402 loads a first searchable area having two or more groups. For the first iteration, this should be the entire screen, including all groups. In this embodiment, an initial searchable area record exists with the area defined to encompass the entire screen, including all elements and groups. Operation 404 selects a grid line that divides the initial searchable area into two searchable areas, each with some of the groups. Operation 406 creates new records, sorts the groups between the initial record and the new record, and updates these records with their respective searchable regions. The dividing line is recorded as a split / division of the two searchable areas. The first searchable area and the groups and elements therein are linked to the division, which is then linked to the new searchable area and the groups and elements therein. At run time, there are class objects for the elements, the groups, and the split / division.

[0040] For each searchable region that contains more than one group of elements, the process is performed recursively (act 408) to divide the searchable region.

[0041] In particular, in one embodiment, elements are defined by a reference to the element's definition (e.g., element ID) and, in this embodiment, a translation of the element to its origin, thereby reducing interface memory requirements as there is no need to define each element individually.

[0042] If the screen is completely split, there is now an intermediate search tree containing divisions / splits, UI elements and groups of UI elements, and the links between them.

[0043] At operation 410, a group-level search task is created for each group. A group-level task is a process step or series of process steps that may include (i) a task to determine if a touch or contact event occurred within (or outside) a UI element, (ii) a task to modify the search area in some way, and (iii) a task to prepare for the next task.

[0044] Each group-level task can include an indication of the next task to be performed in the event of success or failure. For example, each task can include a bit with an "offset" to the next task address. Additionally, each group-level task can receive arguments when it is executed. In some embodiments, the previous task can provide arguments or set an environment bit / flag to indicate which arguments are available to the next task.

[0045] In one embodiment, an index can be generated using an offset in group coordinates (position angle). Every element in a searchable area can be assigned a different ID, when configured, offset by the index from the base ID of the searchable area. The result is an element ID and an offset value. Separate provisions exist for modifying either the response (e.g., haptic) or the element ID - thus, one group element may return a single element ID but multiple response IDs, while another may return one response ID for several element differences.

[0046] The group-level search task can be inserted into a group record, inserted into a search task list, or inserted into an intermediate record. Once the group-level search task is completed, operation 412 returns the intermediate search tree.

[0047] Although not illustrated in Figure 4, in one embodiment, an environment variable can be set for each task, indicating what to return when the task is executed, when the task is successful, and if it is the final task, if applicable. As a non-limiting example, the environment variable could be a haptic ID, a value that controls how to modify the element ID and haptic ID for elements within a group shape, etc. An environment flag can also be set that indicates the data that should be sent to accompany the description of the next task. Using certain constraints and the correct environment variables, for example, the definition of a circle can be reduced from 7 bytes to 2 bytes.

[0048] FIG. 5 illustrates one embodiment of an optimization process 500 performed on the intermediate search tree. At operation 504, all elements are grouped by a common characteristic. Examples of common characteristics include element type, position within a layer, position relative to another element in another layer (e.g., all behind the same element), display group, shape, etc. In one embodiment, a common characteristic can be selected to optimize the search process. For example, if elements are grouped by layer position with elements in the top layer at the top of the search tree, they will be searched first. As another example, in applications where "paging" exists (i.e., layers of a user interface can be swiped to expose underlying layers or one layer can be overlaid on another), grouping by display group allows for control of all display elements using a single control—e.g., retrieving all elements in a display group, applying changes to elements responsive to a control setting, turning all elements in a display group on or off, etc. In various embodiments, identifiers such as layer IDs, position IDs, shape IDs, etc., can be used to identify groups organized by common characteristics.

[0049] At operation 506, the search tasks are inserted into each element's search tree and split to form intermediate search task trees. At operation 508, for each task, the intermediate search task tree is reordered to ensure a single pass. At operation 510, redundant or inefficient search tasks are eliminated. At operation 512, the optimized search task tree is returned.

[0050] FIG. 6 illustrates one embodiment of a process 600 for creating a search task list. At operation 602, a class object for a search task tree is loaded, and at operation 604, an instruction word (i.e., a search task) is created from the class object and inserted into the search task list. In the embodiment shown in FIG. 6, the instruction word includes a task code field and a jump field. In one embodiment, the instruction includes a data field. Any failure (i.e., elements are different) and any split requires a jump to another instruction unless the next task immediately follows the current task in memory. At operation 606, a task code is inserted into the task code field, and at operation 608, a jump value is inserted into the jump field.

[0051] In some embodiments, some or all of the jump values are not inserted until all tasks have been inserted into the search task list. In other embodiments, the jump values can be inferred from the search task tree.

[0052] At operation 610, the various tasks in the search task list are concatenated in memory to form a conditional search task list that, if all objects are in the list (operation 612), is returned by a process at operation 614. The search task list and the search tree may be stored in memory.

[0053] Task instructions may vary depending on container size limitations (i.e., byte limitations) available in the particular environment in which the search task list is implemented. In one embodiment, the data associated with each task instruction may vary depending on system conditions including instruction interface requirements (8-bit, 12-bit, 16-bit, etc.), available memory, etc. As a non-limiting example, an instruction to search within an octagonal UI element may be made simply by x and y coordinate data and the number of sides. However, additional data may be included if the instruction interface and other memory requirements permit.

[0054] FIG. 7 illustrates a search process for determining whether a touch occurred within a UI element, according to one embodiment of the present disclosure. A search of the search tree is performed using the provided data and the search task list. At operation 702, executable instructions for each task, along with payload data for each task, are sequentially provided to the processor interface and executed at operation 704. When searching the search tree, at operation 706, it is determined whether a touch occurred within the UI element, and the result of each task is true / false, success / fail, indicating whether a touch occurred within the UI element. At operation 708, the processor loads and receives the next task instructions and associated data in response to the current task being successful. That is, if the result of operation 706 is success, the next task in the task list is executed.

[0055] If the result is a failure, alternate task instructions and associated data responsive to the result are loaded and received by the processor. If an alternate task exists (operation 714), the alternate task location is provided at operation 716 and the process loops back to operation 702 to load the task from the alternate location on the processor. When the search is over, the UI element is either found or not found. If the UI element is found, operation 710 returns the found result and operation 712 returns the element's ID, as well as any preference / responsiveness parameters. If the element is not found, operation 720 returns the not found result.

[0056] In one embodiment, the search process shown in Figure 7 can be a firmware application running on a processor (microcontroller) on the touch. The touch processor can have one or more search tasks executed by the search process stored in flash memory. In one embodiment, the search tasks can be stored in RAM associated with the display controller, and the search tasks can be provided to the touch processor during a setup or provisioning process and kept accessible to the search process.

[0057] At this point, the inventors realize that the described embodiments offer several advantages over alternative approaches. Memory requirements are significantly reduced—up to 50% from linear search, pure grid, or pure search tree approaches—and still provide an improvement over combined grid / tree approaches. This is in part because the number of search operations performed is reduced. Because the number of search operations is reduced, response cycles are significantly shorter than alternative approaches (including conventional methods). For example, for a 1200 x 1200 touch-sensitive GUI, cycle times of less than 36 μs were achieved, compared to alternative approaches that ranged from 72 μs (pure grid) to 1200 μs (linear). This difference translates to an extremely responsive touch interface for the user. This touch interface can be made more sophisticated by the designer, with numerous elements having different response characteristics.

[0058] 8, 9, and 10 illustrate the processes illustrated and described with reference to FIGS. 2-7 in connection with a radio application GUI as one non-limiting example of a GUI that can be used with embodiments of the present disclosure. The radio GUI 810 illustrated in FIG. 8 includes eight types of UI elements, totaling 144 UI elements, summarized in table 820.

[0059] Figure 9 illustrates UI elements grouped according to the method described with reference to Figure 3. In this embodiment, grouped elements 832, 834, 836, 838, 840, 842, 844, 846, and 848 have similar touch characteristics (e.g., haptic feedback in response to touch), physical location on the screen, and shape.

[0060] FIG. 10 shows one embodiment of a tree-like structure 850 formed using the tree and grid method described with reference to FIG.

[0061] 11 illustrates a system 1000 and associated tools 1040 capable of implementing the search methods described herein, according to one embodiment of the present disclosure. The system 1000 includes microcontroller firmware 1010 having thereon a GUI element search function 1012 and a response determination function 1014 for GUI elements. A processor executing the microcontroller firmware 1010 is coupled to a response driver 1018 that can receive control signals from the microcontroller firmware 1010 and, in turn, drive the response of a touch-sensitive interface 1020. In one embodiment, the touch-sensitive interface 1020 is a touchscreen with one or more actuators, and the response driver 1018 is a haptic driver configured to generate control signals that excite the actuators. The sensing circuit 1022 can generate one or more measurement signals in response to a contact at the touch-sensitive interface 1020. Touch measurement and processing 1016 can determine touch information (e.g., location, type, etc.) in response to measurement signals from sensing circuitry 1022 and provide this touch information to response determination 1014 and GUI element retrieval function 1012. Control signals received by response driver 1018 can be based, for example, at least in part, on the touch information such that haptic feedback is provided in a right location on touch-sensitive interface 1020.

[0062] 11 also illustrates a tool 1040 that can implement a search list generation process to create an element search task list and element response information in accordance with one embodiment of the present disclosure. A search list creation application program 1044 is configured to implement the process described with reference to FIGS. 2-6, a GUI definition XAML file 1042, to generate an element search task list. The application 1044 can provide an element search task list 1046 and element response information 1048 as files to the microcontroller firmware 1010. In one embodiment, this application can also provide a search tree, which can be incorporated into a search task.

[0063] In one or more embodiments of the firmware, the firmware can include force measurement and processing functionality to incorporate force level information about touch events. In such embodiments, the force level information and GUI element ID and haptic response details returned by the element search function can be used by a haptic sequencer to generate haptic control signals responsive to the force level, GUI element ID, and haptic response details.

[0064] The system of FIG. 11 can be incorporated into a variety of consumer products, appliances, and machines that utilize touch interfaces and touch control panels, including automobiles.

[0065] FIG. 12 illustrates a simplified version of a radio GUI 1210 for an automobile touch control panel. Three regions are specifically designated Region 1, Region 2, and Region 3. Region 1 is a center button 1212 of a rotary dial 1214 for temperature control. Haptic feedback is provided with haptic profile ID#4 (vibration) in response to touch events with a strong force level. Region 2 is the rotary dial 1214, also for temperature control. Haptic feedback is provided with haptic profile ID#3 (friction) in response to touch events with a weak force level. Finally, Region 3 is a button 1216 for presenting a menu for automobile settings. Haptic feedback is provided with haptic profile ID#2 (click) in response to touch events with a strong force level, and with haptic profile ID#3 (friction) in response to touch events with a weak force level.

[0066] Figure 13 illustrates the system of Figure 11 and the GUI of Figure 12 incorporated into an automobile control system commanded by a head unit 1310, whose haptic effects are controlled by a microcontroller. In this embodiment, the touch controller 1320 and shape search function 1324 are part of the automobile subsystem, and the automobile head unit 1310 responds to touches with haptic feedback without direct intervention of the head unit's processing circuitry. The touch controller 1320 is configured to operate a touch state machine that, from touch location and force level information, identifies the touched screen button and includes the button location for triggering the haptic effect.

[0067] In this embodiment, force processing 1326 and touch processing 1322 are integrated into one controller component, and the head unit screen 1332 contains definitions of several geometric object descriptions (screen display design 1336, and search tree definition 1338) that are each required to derive a range of haptic effects activated directly by the touch controller 1320 and performed by the haptic device 1350.

[0068] For example, after a touch on the display 1330, the touch controller 1320 receives force and touch information from force processing 1326 and touch processing 1322. This information can include a force measurement and the touch location on the display. Shape search 1324 provides UI element information, if applicable, corresponding to the UI element displayed on the display 1330 where the touch occurred. If no UI element exists that corresponds to the location on the display, shape search 1324 provides a null search result. While searching for shape information for the UI element, shape search 1324 can use definitions stored in the head unit 1310. In one embodiment, shape search 1324 can receive the definitions during a provisioning process, for example, when the touch controller 1320 is integrated with the head unit 1310 or when the head unit 1310 is powered on. If shape search 1324 identifies a UI element, the haptic information is used by haptic control 1328 to send a haptic activation message to the haptic device 1350 that includes a haptic effect and the location of the haptic effect. The haptic activation message may include a parameter representing the level of the haptic effect (e.g., weak, medium, strong). The haptic device 1350 searches for the haptic effect definition in a haptic library 1352 stored in the haptic device. The haptic device 1350 then controls actuators on the display 1330 so that a particular area of the display exhibits the requested haptic effect. In particular, different haptic devices may have different haptic libraries, so the effects may vary between devices.

[0069] In this embodiment, the GUI definition is an XAML file, which is an XML implementation for a graphical user interface. An XAML file contains a hierarchical list of drawing instructions for the screen elements of the GUI's UI. In an XAML file, there are tags associated with the GUI elements. For example, "Width," "Height," and "Horizontal Alignment" are all valid tags for a particular element.

[0070] Many of the functional units described herein may be shown, described, or labeled as modules, threads, or other segregations of programming code to more specifically emphasize their implementation independence. The modules may be implemented at least partially in hardware in one form or another. For example, a module may be implemented as a hardware circuit comprising custom VLSI circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A module may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, or the like.

[0071] Modules may also be implemented using software or firmware stored in physical storage devices (e.g., computer-readable storage media), memory, or combinations thereof, for execution by various types of processors.

[0072] An identified module of executable code may, for example, comprise one or more physical or logical blocks of computer instructions that may be organized as, for example, a thread, an object, a procedure, or a function. Nevertheless, an identified executable module may comprise different instructions stored in different locations that need not be physically located together but that, when logically coupled together, comprise a module and achieve the specified purpose of the module.

[0073] In practice, a module of executable code may be a single instruction, or many instructions, and may be distributed across several different code segments, among different programs, and across several storage devices or memory devices. Similarly, operational data may be identified in modules and illustrated herein, and may be embodied in any suitable form and organized within any suitable type of data structure. Operational data may be collected as a single data set, or may be distributed in different locations across different storage devices, or may exist, at least in part, simply as electronic signals over a system or network. If a module or portion of a module is implemented in software, the software portion is stored in one or more physical devices, referred to herein as computer-readable media.

[0074] In some embodiments, the software portion is stored in a non-transitory state such that the software portion, or a representation thereof, remains in the same physical location for a period of time. Furthermore, in some embodiments, the software portion is stored in one or more non-transitory storage devices comprising hardware elements capable of storing non-transitory states and / or signals representing the software portion, although other portions of the non-transitory storage device may perform signal modification and / or transmission. Examples of non-transitory storage devices include flash memory and random-access memory (RAM). Another example of a non-transitory storage device includes read-only memory (ROM), which may store signals and / or states representing the software portion for a period of time. However, the ability to store signals and / or states is not diminished by the additional function of transmitting signals identical to or representing the stored signals and / or states. For example, a processor may access a ROM to obtain signals representing the stored signals and / or states in order to execute the corresponding software instructions.

[0075] On a practical level, software that enables a computer system to perform the operations described herein can be supplied on any one of a variety of media. Moreover, the actual implementation of the techniques and operations of the present invention is actually statements written in a computer language. Such computer language statements, when executed by a computer, cause the computer to act according to the particular content of the statements. Furthermore, software that enables a computer system to act according to the present invention can be provided in any number of forms, including, but not limited to, original source code, assembly code, object code, machine code, compressed or encrypted versions thereof, and any and all equivalents.

[0076] Those skilled in the art will recognize that "medium" or "computer-readable medium," as used herein, can comprise a diskette, tape, compact disc, integrated circuit, ROM, CD, DVD, BLU-RAY, cartridge, flash memory, memory stick, or card, or any other non-destructive storage medium usable by a computer, including those now known or later developed.

[0077] It will be recognized that actual software may be "written" to disk, "embodied" in an integrated circuit, "carried" over a communications circuit, "stored" in a memory chip, or "loaded" into cache memory, but for purposes of this application, the software will simply be referred to as being "in" or "on" a computer-readable medium. Accordingly, the terms "in" or "on" are intended to encompass the above-mentioned and all equivalent and possible ways in which software may be associated with a computer-readable medium.

[0078] For the sake of brevity, the term "computer program product" will therefore be used to refer to a computer-readable medium, as defined above, having any form of software that enables a computer system to operate in accordance with any embodiment of the present invention.

[0079] While the present disclosure has been described herein with reference to certain illustrated embodiments, those skilled in the art will recognize and appreciate that the present invention is not so limited. Rather, numerous additions, deletions, and modifications can be made to the illustrated and described embodiments without departing from the scope of the invention as claimed below, along with their legal equivalents. In addition, features of one embodiment can be combined, as contemplated by the inventors, with features of other disclosed embodiments and still fall within the scope of the present disclosure.

Claims

1. 1. A method of generating instructions for retrieving elements of a graphical user interface (GUI) displayed on a touch-sensitive screen, comprising: parsing a GUI definition and identifying elements of the GUI responsive to said parsing; creating a record containing an input of the identified elements; associating the identified elements with groups of similarly positioned elements; arranging the identified element records in a tree-like structure; collapsing the identified elements in the same group into a single leaf in said tree-like structure; optimizing the tree-like structure; creating a search command list responsive to said tree-like structure.

2. The method of claim 1 , further comprising creating one or more group search instructions responsive to each one of the groups.

3. grouping all identified elements by common characteristics; reordering the tree-like structure to ensure that all identified element records can be traversed in a single path; and eliminating redundant or inefficient configurations responsive to said reordering.

4. 4. The method of claim 3, further comprising the step of traversing all nodes of the tree-like structure and, as each node is visited, inserting one or more search instructions into a search instruction list responsive to the visited node.

5. The method of claim 4 , wherein the search instruction list is an extensible markup language (XML) file.

6. 1. A computer program product that enables a computer to create executable instructions for retrieving elements of a graphical user interface (GUI), comprising: a computer-readable medium; The software instructions on the computer readable medium cause the computer to perform the following operations: parsing a GUI definition and identifying elements of the GUI that are responsive to the parsed GUI definition; creating a record containing an entry of said identified elements; associating the identified elements with groups of similarly positioned elements; arranging the records of the identified elements in a tree-like structure; collapsing the identified elements in the same group into a single leaf in said tree-like structure; optimizing the tree-like structure; creating a search command list responsive to said tree-like structure; and software instructions adapted to enable said computer program product to perform the above.

7. 7. The computer program product of claim 6, wherein the software instructions are further adapted to enable the computer to perform the operation of creating one or more group search instructions responsive to each one of the groups.

8. The software instructions may cause the computer to: grouping all identified elements by common characteristics; reordering the tree-like structure to ensure that all identified element records can be traversed in a single path; The computer program product of claim 7 , further adapted to perform an operation to eliminate redundant or inefficient settings from the records responsive to the reordering.

9. 9. The computer program product of claim 8, wherein the software instructions further adapt the computer to traverse all of the nodes of the tree-like structure and, as each node is visited, insert one or more search instructions into a search instruction list responsive to the visited node.

10. 10. The computer program product of claim 9, wherein the search instruction list is an Extensible Markup Language (XML) file.

11. a microcontroller operably coupled to a touch screen configured to display a graphical user interface (GUI), at least one processor; One or more executable instructions stored on a non-transitory storage medium, the instructions, when executed by the processor, cause the processor to: determining a location of the sensed touch on the touchscreen; and identifying a GUI element associated with a location of the touch that corresponds to the sensed touch.

12. The microcontroller of claim 11 , wherein the instructions further adapt the processor to identify a user feedback response responsive to the identified GUI element.

13. The microcontroller of claim 12 , wherein the instructions further adapt the processor to generate one or more control signals responsive to the user feedback response associated with the identified GUI element.

14. The microcontroller of claim 13 , wherein the user feedback response is a tactile response.

15. The microcontroller of claim 14 , wherein the control signal is configured to control a haptic driver to excite one or more actuators within the touchscreen.

16. The microcontroller of claim 15 , further comprising determining a force level associated with the touch, wherein the processor generates the one or more control signals responsive to the force level.

17. The one or more executable instructions include one or more conditional search instructions that, when executed by the processor, cause the processor to: comparing the location of the touch with a location of one or more elements of the GUI; selecting a GUI element responsive to said comparison; and providing a haptic command and an element identifier corresponding to the selected GUI element.

18. The microcontroller of claim 17 , wherein the haptic command is configured to instruct a haptic driver to excite one or more actuators in a touchscreen that are responsive to the haptic command.

19. 1. A method for identifying elements of a graphical user interface (GUI) displayed on a touch screen, comprising: determining a location of the sensed touch on the touchscreen; executing one or more search instructions responsive to the location, each search instruction of the one or more search instructions corresponding to a GUI element and adapted to return search results when executed; and identifying a GUI element responsive to the search result.

20. 20. The method of claim 19, further comprising identifying a user feedback response responsive to the identified GUI element.

21. 21. The method of claim 20, further comprising generating one or more control signals responsive to a user feedback response associated with the identified GUI element.

22. The method of claim 21 , wherein the user feedback response is a haptic response.

23. 23. The method of claim 22, wherein the control signals are configured to control a haptic driver to excite one or more actuators in the touchscreen.

24. 24. The method of claim 23, further comprising determining a force level associated with the touch, wherein the processor generates the one or more control signals responsive to the force level.

25. 20. The method of claim 19, wherein each search instruction includes a payload that is inserted in response to the search tree.

26. 1. A system comprising: a display subsystem configured to control a display; A touch subsystem comprising a touch sensor and a touch controller, the touch controller comprising: determining a location of the sensed touch on the touch sensor; executing one or more search instructions responsive to the location and the search tree, each search instruction of the one or more search instructions corresponding to a GUI element and adapted and executed to return search results when executed; identifying a GUI element responsive to the search results; and generating a haptic control message responsive to the identified GUI element.

27. 27. The system of claim 26, further comprising a haptic feedback subsystem configured to generate haptic feedback at the display.

28. 27. The system of claim 26, wherein some of the one or more search instructions comprise a jump instruction associated with another of the one or more search instructions and are configured, when executed, to load the other search instruction.

29. 30. The system of claim 28, wherein the jump instruction enables the one or more search instructions to be executed in a tree order.

30. 27. The system of claim 26, further comprising an automobile head unit.

Images