Technique for identifying user interface elements and systems and devices using the technique

Abstract

To provide a microcontroller determining a UI element selected on a contact-responsive user interface, a program, and a method for providing one or more tactile responses.SOLUTION: A process by a microcontroller determines a touched place detected on a touch screen, and executes one or more search instructions stored as a list of conditionally executable instructions responding to a determined place in order to identify a GUI (graphical user interface) element associated with the touched place. Each search instruction corresponds to a GUI element, is executed in the same consecutive sequence as sequenced one or more instructions, and returns the executed search results. The process also creates a control signal responding to a user feedback response responding to the identified GUI element.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] (Priority Claim) This application claims the benefit under 35 U.S.C. § 119(e) of U.S. Provisional Application No. 62 / 507,902, filed May 18, 2017, the disclosure of which is hereby incorporated by reference in its entirety.

[0002] FIELD OF THEINVENTION Embodiments of the present disclosure relate generally to techniques for identifying elements in a user interface (UI), and more specifically, to techniques for determining a selected UI element on a touch-sensitive user interface and using such techniques to provide one or more haptic responses. [Background technology]

[0003] Touch interfaces incorporating touch sensing are used in a variety of applications including, for example, tablet computers, personal computers, smartphones, and other consumer products. Touch interfaces are also used as control panels in automobiles, appliances (e.g., refrigerators, ovens, washer / dryers, etc.), heating and air conditioning control systems, security systems, and automatic teller machines (ATMs). Touch interfaces in these applications can be, for example, touchpads or can incorporate a screen and a graphical user interface (GUI).

[0004] In general, there is a need for a touch interface that is responsive and accurate enough for use in many applications. [Brief description of the drawings]

[0005] Objects and advantages of the embodiments of the present disclosure will become apparent to those skilled in the art from the following detailed description taken in conjunction with the accompanying drawings: The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee. [Figure 1] FIG. 1 is a swim diagram illustrating a process for generating and using a search task list to identify contacted UI elements within a GUI. [Diagram 2] 1 is a flowchart of a process for generating a search task list according to one embodiment of the present disclosure. [Diagram 3] 1 is a flowchart of a process for extracting UI elements from a UI configuration definition according to one embodiment of the present disclosure. [Figure 4] 1 is a flowchart of a process for generating an intermediate search tree according to one embodiment of the present disclosure. [Diagram 5] 1 is a flowchart of a process for generating a search task tree according to one embodiment of the present disclosure. [Figure 6] 1 is a flowchart of a process for generating a search task list according to one embodiment of the present disclosure. [Figure 7] 1 is a flowchart of a process for determining whether a touch occurred in a UI element according to one embodiment of the disclosure. [Figure 8] 1 illustrates an embodiment of a wireless GUI with UI elements. [Figure 9] 9 illustrates UI elements of the wireless GUI of FIG. 8 grouped according to an embodiment of the present disclosure. [Figure 10] 1 illustrates a tree-like structure of UI elements formed in accordance with an embodiment of the present disclosure. [Figure 11] 1 illustrates one embodiment of a system incorporating a search task list. [Figure 12] 1 illustrates one embodiment of a wireless GUI including features and parameters associated with at least some of the UI elements of the wireless GUI. [Figure 13]12 illustrates one embodiment of the system of FIG. 11 integrated as a subsystem into an automobile head unit.

[0006] Disclosure Some embodiments of the present disclosure generally relate to a method of creating instructions for searching elements of a graphical user interface (GUI) displayed on a touch-sensitive screen, the method including the steps of parsing a GUI definition and identifying elements of the GUI responsive to the parsing, creating records containing entries for the identified elements, associating the identified elements with groups of similarly positioned elements, arranging the records of the identified elements in a tree-like structure, collapsing identified elements in the same group into a single leaf in the tree-like structure, optimizing the tree-like structure, and creating a search instruction list responsive to the tree-like structure.

[0007] Some embodiments of the present disclosure generally relate to a computer program product that enables a computer to create executable instructions for making elements of a graphical user interface (GUI) searchable. The program product may include a computer-readable medium and software instructions on the computer-readable medium. The software instructions on the computer-readable medium adapt a computer to perform the following operations: parse a GUI definition and identify elements of the GUI responsive to the parsed GUI definition, create a record including an entry for the identified elements, associate the identified elements with groups of similarly positioned elements, arrange the records of the identified elements in a tree-like structure, collapse the identified elements in the same group into a single leaf in the tree-like structure, optimize the tree-like structure, and create a search instruction list responsive to the tree-like structure.

[0008] Some embodiments of the present disclosure generally relate to a microcontroller operatively coupled to a touchscreen configured to display a graphical user interface (GUI). The microcontroller includes at least one processor and one or more executable instructions stored on a non-transitory storage medium that, when executed by the processor, are adapted to cause the processor to determine a location of a sensed touch on the touchscreen and identify a GUI element associated with the touch location corresponding to the sensed touch.

[0009] Some embodiments of the present disclosure generally relate to a method for identifying elements of a graphical user interface (GUI) displayed on a touch screen, the method including determining a location of a sensed touch on the touch screen, executing one or more search instructions corresponding to the location, each search instruction of the one or more search instructions corresponding to a GUI element and adapted to return search results when executed, and identifying the GUI elements corresponding to the search results.

[0010] Some embodiments of the disclosure generally relate to a system. The system includes a display subsystem and a touch subsystem. The display subsystem is configured to control a display. The touch subsystem includes a touch sensor and a touch controller. The touch controller is configured to determine a location of a touch sensed at the touch sensor, execute one or more search instructions corresponding to the location and a search tree, where each search instruction of the one or more search instructions corresponds to a GUI element and is adapted to return search results when executed, identify a GUI element corresponding to the search results, and generate a haptic control message corresponding to the identified GUI element. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] In the following detailed description, reference is made to the accompanying drawings, which form a part of this specification and which show, by way of illustration, specific examples of embodiments in which the present disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present disclosure. However, other embodiments may be used, and changes in structure, materials, and processes may be made without departing from the scope of the present disclosure. The figures presented herein are not intended to be actual diagrams of any particular method, system, device, or structure, but are merely idealized representations used to explain embodiments of the present disclosure. The figures presented herein are not necessarily drawn to scale. Similar structures or components in various figures may retain the same or similar numbering for the convenience of the reader. However, similarity in numbering does not imply that the structures or components are necessarily of the same size, composition, configuration, or any other characteristic.

[0012] It will be readily understood that the components of the embodiments as generally described and illustrated in the figures herein could be arranged and designed in a wide variety of different configurations. Thus, the following description of various embodiments is not intended to limit the scope of the disclosure, but is merely representative of various embodiments. Although various aspects of the embodiments may be presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0013] The following description may include examples to assist those skilled in the art in enabling the disclosed embodiments. The use of the terms "exemplary," "example," and "for example" means that the associated description is explanatory, and the scope of the disclosure is intended to encompass examples and legal equivalents, and the use of such terms is not intended to limit the embodiments or the scope of the disclosure to specific components, steps, features, functions, etc.

[0014] Furthermore, the specific implementations shown and described are merely examples and should not be construed as the only way to implement the present disclosure unless otherwise specified herein. Elements, circuits, and functions may be shown in block diagram form so as not to obscure the present disclosure in unnecessary detail. Conversely, the specific implementations shown and described are merely exemplary and should not be construed as the only way to implement the present disclosure unless otherwise specified herein. Furthermore, the block definitions and partitioning of logic between various blocks are exemplary specific implementations. It will be readily apparent to one skilled in the art that the present disclosure can be implemented with numerous other partitioning solutions. For the most part, details regarding timing considerations and the like have been omitted, and such details are not necessary to obtain a complete understanding of the present disclosure and are within the capabilities of one skilled in the art.

[0015] Those skilled in the art will understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout this specification may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof. Some figures may illustrate signals as a single signal for clarity of display and explanation. Those skilled in the art will understand that a signal may represent a bus of signals, which may have various bit widths, and that the present disclosure may be implemented with any number of data signals, including a single data signal.

[0016] The various example logic blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed using a general purpose processor, a special purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general purpose processor (which may also be referred to herein as a host processor or simply a host) may be a microprocessor, but alternatively the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. A general purpose computer with a processor is considered a special purpose computer, and the general purpose computer is configured to execute computing instructions (e.g., software code) related to the embodiments of the present disclosure.

[0017] The embodiments may be described in terms of a process that is depicted as a flowchart, a flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe operational acts as a sequential process, many of these acts can be performed in a different order, in parallel, or substantially simultaneously. Additionally, the order of the acts may be rearranged. A process may correspond to a method, a thread, a function, a procedure, a subroutine, a subprogram, etc. Furthermore, the methods disclosed herein may be implemented in hardware, software, or both. If implemented in software, the functions may be stored on or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media, such as any medium that facilitates transfer of a computer program from one place to another.

[0018] Any reference to elements herein using designations such as "first," "second," etc. does not limit the quantity or order of those elements unless such limitation is expressly stated. Rather, these designations may be used herein as a convenient way of distinguishing between two or more elements or instances of an element. Thus, reference to a first element and a second element does not imply that only two elements may be used or that the first element must precede the second element in any way. In addition, unless otherwise stated, a set of elements may include one or more elements.

[0019] As used herein, the term "substantially" when referring to a given parameter, characteristic, or condition means and includes the extent to which one of ordinary skill in the art would understand that the given parameter, characteristic, or condition is met with small variations, such as, for example, within acceptable manufacturing tolerances. As an example, depending on the particular parameter, characteristic, or condition that is substantially met, the parameter, characteristic, or condition may be at least 90% met, at least 95% met, or even at least 99% met.

[0020] Various embodiments described in this disclosure generally relate to techniques for determining a selected UI element on a touch-sensitive user interface and using such techniques to provide one or more haptic responses. For purposes of understanding the embodiments described herein, a contact sensor can respond to an object (such as a finger or stylus) contacting a touch region of a touch interface or to the proximity of the object to the touch region. In this disclosure, "contact" generally refers to physical contact between an object and a touch-sensitive region, but can also encompass the proximity of an object that produces a correspondence measurable by a contact sensor. Additionally, a touch-sensitive region refers to a physical region on a touch interface where a contact sensor can respond to contact of an object.

[0021] A touch-sensitive GUI, as used herein, refers to a touch interface integrated with a GUI. For example, a GUI typically includes one or more display areas and active / activatable areas. In this disclosure, a display area is an area of ​​a user interface that displays information to a user. An activatable area is an area of ​​a GUI, such as a button, slider, or menu, where a user can take some action on the user interface. Some display areas are also activatable areas that display information and can take some action. In a touch-sensitive GUI, an active area can be activated by touching a touch-sensitive area where the area is displayed (e.g., tapping a GUI button on a touch screen). An active area can be displayed as GUI elements / objects, such as buttons, sliders, selectable panes, menus, etc., of all different shapes and sizes.

[0022] In general, when a touch is sensed in a touch-sensitive area, a process is used to determine an active area of ​​the GUI that corresponds to the touch, if applicable. For example, when an "Enter" button is tapped, the contact is measured and, corresponding to the measured contact, a process determines that the contact was an Enter button. Because the Enter button is an active area, an event is generated in the touch-sensitive GUI and / or the underlying application program that launches the GUI.

[0023] Additionally, when a particular GUI element is associated with an active area, actuators integrated with the touch interface can provide one or more physical responses, commonly referred to as haptic responses. These can be in the form of force, vibration, or movement, and can mimic surface textures, ridges, edges, button press / click interactions, and other simulated sensations and responses. In the case of a GUI, the haptic response can be localized to the GUI element with which the user interacts. For example, when a user touches a GUI button, the haptic response can provide the feel of the button with raised edges, as if the button was pressed, or as if it had a rough texture.

[0024] Various embodiments described herein may refer to creating and updating electronic records. Electronic records may be in the form of data files, and updating an electronic record may include inserting or deleting data entries into one or more fields of the record. Alternatively, it may refer to class objects and instantiated objects that, at run time, have state information and variables that match the described record. Both situations are contemplated in various embodiments described herein.

[0025] Various embodiments of the present disclosure relate to techniques for identifying touched GUI elements on a touch-sensitive interface. These techniques and related structures are particularly efficient in terms of memory usage and responsiveness. Furthermore, compared to other techniques, they have low interface data storage requirements and perform a small number of tasks at run time to identify the UI elements.

[0026] Some embodiments of the present disclosure relate to a method for creating a list of optimized search tasks that can be performed to identify touched GUI elements on a touch-sensitive interface. A search task can be a processor-executable instruction that, when executed, returns a success or failure message, if applicable, to a subsystem searching for the touched GUI element. In one embodiment, the search task is created based on a definition file that maps various elements and their locations in the GUI. The search task is optimized for various efficiency parameters.

[0027] In one embodiment, the search task list can be performed by an embedded device, such as a touch controller, as compared to a conventional touch-sensitive GUI known to the inventors of this disclosure that can perform the search in a display subsystem (e.g., an automobile head unit). Performing the search for GUI elements in the touch controller saves time to communicate with the display subsystem, as well as the time it takes for the subsystem to react and communicate with, for example, a haptic feedback subsystem. The time saved increases the responsiveness of the touch-sensitive GUI as compared to a conventional touch-sensitive GUI, and also reduces the time from the user's perspective between when the user touches the screen and when the user receives feedback responsive to the touch.

[0028] Additionally, the creation of the task search list is configurable, and depending on the GUI, a set of common features can be selected for optimized implementation for a particular application. For example, in some embodiments, the creation process can be optimized for GUIs that include paging, drop-down menus, and pop-up windows that hide other GUI elements, elements of a particular shape, or elements that move or change shape when touched.

[0029] 1 illustrates the overall operation of a system according to various embodiments of the present disclosure. At act 112, the software application tool 102 is configured to process the UI definition file to create a search task list of conditionally executable instructions (act 108) that can be performed to identify UI elements that have contacted the touch-sensitive screen, if applicable.

[0030] The search task list may be stored in a non-transitory storage memory that is accessible by one or more processors that are part of the touch system (ACT 110). When a touch event occurs on the touch interface, the touch sensor 106 may detect the touch (ACT 118) and provide one or more signals representing the touch to the touch processor. The touch processor 104 determines where on the touch interface the touch occurred (ACT 112) and searches for and identifies the touched UI element (ACT 114), if applicable, that is responsive to the determination. In one embodiment, the touch processor 104 may provide the search results to a graphical user interface subsystem (ACT 116).

[0031] An embodiment of a process for creating a search task list is described with reference to Figures 2, 3, 4, 5, 6, and 7. An embodiment of the present disclosure utilizes a search tree-like structure that organizes UI elements according to tree and grid techniques. Various UI elements are divided into related groups that are treated like a grid and organized into a search tree, and then various search tasks are generated. The search tasks are conditioned to optimize the execution of the search using instructions. Those skilled in the art will understand that other algorithms, such as divide and conquer, can be used to divide the screen(s) into searchable areas.

[0032] 2 illustrates one embodiment of a process for generating a search task list. At operation 202, a structural definition of the UI is loaded and parsed to identify screens, sub-screens, and UI elements on the screens and sub-screens. The UI configuration definition can be an electronic file, a database, raw data, etc. At operation 204, elements are grouped and the searchable area is divided into one or more searchable areas with element groups. At operation 206, the groups are linked into a tree-like structure based on the searchable areas. At operation 208, search tasks are associated with the branches and nodes of the search tree to form a search tree and optimize the search tree. At operation 210, the conditional tasks of the search tree are stored in a list, which is a task list that can be executed by a processor.

[0033] In one embodiment, the software application tool 102 that generates the search task list can be configured to write the results of one or more of operations 202, 204, 206, 208, and 210 to an output file. This output file can be used by a debugging tool to review the results of the process. The same debugging tool can be configured to use a text version of the search task list and run it in a virtual test environment (e.g., a DOS executable) to verify that the search task list is ready to run.

[0034] FIG. 3 illustrates one embodiment of a method 300 for extracting UI elements from a UI configuration definition. In one embodiment, the UI configuration definition is an xml definition of a portion of a display that is transformed by the configuration generation features of an application tool. The application tool parses the structure definition and grabs the elements defined in the definition structure. At operation 302, each UI element is loaded, and at operation 304, it is determined whether the UI element is a known UI element. If it is not a known UI element (i.e., this is the first time the element has been identified in the structure definition), at operation 306, the process creates a new element definition for that type of UI element (e.g., button, knob, slider, etc.). After creating the new definition, or if the element is a known UI element, at operation 308, it is determined whether to assign the element to an existing group. In one embodiment, the determination of whether to assign to an existing group is based on common characteristics of the elements, such as various predefined parameters, for example, the type of element, the screen on which the UI element is displayed, the location of the layer, the type of response associated with the element (e.g., visual, tactile, audio, etc.). If it is determined that the element is to be assigned to a new group, then at operation 310 a new group record is created with parameters associated with the element. After the new group record is created, or if it is determined that the element is to be assigned to an existing group, then at operation 312 inputs for the new element are inserted into the new element's group record. In one embodiment, the inputs include fields for the element ID and the element's location (i.e., the coordinates of the element on the screen). At operation 314 it is determined whether there are more UI elements, and if so, the process runs for each remaining UI element identified in the UI configuration definition. At operation 316 the process returns the element(s), element definition(s), and group(s).

[0035] In one embodiment not illustrated in FIG. 3, if a UI configuration definition contains more than one screen definition, each such screen is assigned a screen ID that is a parameter of the UI elements. It may also be incorporated as a parameter of each group. Each screen may also contain sub-screens, which are defined regions of the displayed GUI within which some UI elements change dynamically, while UI elements outside such regions remain unchanged. As non-limiting examples, regions with dynamic elements include swappable panes, scrollable menus, activatable information panes, navigation buttons, etc.

[0036] FIG. 4 illustrates a process 400 for creating a search tree, according to one embodiment of the disclosure. In this process, a decision is made as to how to divide each screen identified in the UI definition into searchable regions, each searchable region containing one or more groups of elements. In the embodiment of the process shown in FIG. 4, a dividing line (x coordinate, y coordinate) is selected that divides the groups of elements, so that at least one group is on one side of the dividing line and at least one other group is on the other side of the dividing line. In effect, the screen is now divided into two searchable regions by a shared boundary along the dividing line. Groups are divided in a recursive manner until the groups cannot be further divided.

[0037] In another embodiment, the screen or searchable area is divided in the x and y coordinate directions simultaneously, which may result in a subdivision of up to four groups. This technique can also result in a subdivision of less than four, for example, a division of three groups and one empty searchable area.

[0038] In yet other embodiments, such that a portion of the screen is not subdivided into searchable regions, circles, squares, and / or polygons may be used to define that portion to exclude it from the searchable region.

[0039] Proceeding through the process shown in FIG. 4, operation 402 loads a first searchable area with two or more groups. For the first iteration, this should be the entire screen, including all groups. In this embodiment, an initial searchable area record exists with the area defined to encompass the entire screen, including all elements and groups. Operation 404 selects a grid line that divides the initial searchable area into two searchable areas, each with some of the groups. Operation 406 creates new records, sorts the groups between the initial record and the new record, and updates these records with their respective searchable regions. The dividing line is recorded as a split / division of the two searchable areas. The first searchable area and the groups and elements therein are linked to the division, which is then linked to the new searchable area and the groups and elements therein. At run time, there is a class object for the element, a class object for the group, and a class object for the split / division.

[0040] For each searchable region that contains more than one group of elements, the process is performed recursively (act 408) to split the searchable region.

[0041] In particular, in one embodiment, elements are defined by a reference to the element's definition (e.g., element ID) and, in this embodiment, a translation to the element's origin, thereby reducing interface memory requirements since there is no need to define each element individually.

[0042] If the screen is completely split, then at this point there is an intermediate search tree that contains divisions / splits, UI elements and groups of UI elements, and the links between them.

[0043] At operation 410, a group-level search task is created for each group. A group-level task is a process step or a series of process steps that may include (i) a task to determine if a touch or contact event occurred within (or outside) a UI element, (ii) a task to modify the search area in some way, and (iii) a task to prepare the next task.

[0044] Each group-level task can include an indication of the next task to be performed in case of success or failure. For example, each task can include a bit with an "offset" to the next task address. Additionally, each group-level task can receive arguments when it is executed. In some embodiments, the previous task can provide arguments or set an environment bit / flag to indicate what arguments are available to the next task.

[0045] In one embodiment, an offset of the group coordinates (position angle) can be used to generate an index. Every element in a searchable area can be assigned a different ID, if configured, offset by the index from the searchable area's base ID. The result is an element ID and an offset value. There are separate provisions to modify either the response (e.g., haptics) or the element ID - so one group element may return a single element ID but multiple response IDs, and another may return one response ID for several element differences.

[0046] The group-level search task may be inserted into the group record, inserted into the search task list, or inserted into an intermediate record. Once the group-level search task is completed, operation 412 returns the intermediate search tree.

[0047] Although not illustrated in FIG. 4, in one embodiment, an environment variable can be set for each task, which indicates what to return when the task is executed, when the task is successful, and when it is the final task, if applicable. As a non-limiting example, the environment variable can be a haptic ID, a value that controls how to modify the element ID and haptic ID for elements in a group shape, etc. An environment flag can also be set that indicates the data that should be sent to accompany the description of the next task. With certain constraints and the correct environment variables, the definition of a circle, for example, can be reduced from 7 bytes to 2 bytes.

[0048] FIG. 5 illustrates one embodiment of an optimization process 500 performed on the intermediate search tree. At operation 504, all elements are grouped by a common characteristic. Examples of common characteristics include element type, position within a layer, position relative to another element in another layer (e.g., all behind the same element), display group, shape, etc. In one embodiment, the common characteristic can be selected to optimize the search process. For example, if elements are grouped by layer position with elements in the top layer at the top of the search tree, they will be searched first. As another example, in an application where there is "paging" (i.e., layers of a user interface can be swiped to expose layers below, or one layer can be overlaid on top of another), grouping by display group allows for a single control to be used to control all display elements - e.g., to search all elements in a display group, apply changes to elements that respond to a control setting, turn all elements in a display group on or off, etc. In various embodiments, identifiers such as layer IDs, position IDs, shape IDs, etc. can be used to identify groups organized by common characteristics.

[0049] At operation 506, the search tasks are inserted into each element's search tree and split to form intermediate search task trees. At operation 508, for each task, the intermediate search task tree is reordered to ensure a single pass. At operation 510, redundant or inefficient search tasks are eliminated. At operation 512, the optimized search task tree is returned.

[0050] FIG. 6 illustrates one embodiment of a process 600 for creating a search task list. At operation 602, a class object of a search task tree is loaded, and at operation 604, an instruction word (i.e., a search task) is created from the class object and the instruction word (i.e., a search task) is inserted into the search task list. In the embodiment shown in FIG. 6, the instruction word includes a task code field and a jump field. In one embodiment, the instruction includes a data field. Any failure (i.e., elements are different) and any split requires a jump to another instruction unless the next task is immediately after the current task in memory. At operation 606, a task code is inserted into the task code field, and at operation 608, a jump value is inserted into the jump field.

[0051] In some embodiments, some or all of the jump values ​​are not inserted until all tasks have been inserted into the search task list, in other embodiments the jump values ​​can be inferred from the search task tree.

[0052] At operation 610, the various tasks of the search task list are concatenated in memory to form a conditional search task list that, if all objects are in the list (operation 612), is returned by a process at operation 614. The search task list and the search tree may be stored in memory.

[0053] Task instructions may vary depending on container size limitations (i.e., byte limitations) available in the particular environment in which the search task list is implemented. In one embodiment, the data associated with each task instruction may vary depending on system conditions including instruction interface requirements (8-bit, 12-bit, 16-bit, etc.), available memory, etc. As a non-limiting example, an instruction to search within an octagonal UI element may be made simply by x and y coordinate data and the number of sides. However, additional data may be included if the instruction interface and other memory requirements permit.

[0054] FIG. 7 illustrates a search process for determining whether a touch has occurred within a UI element, according to one embodiment of the disclosure. A search of the search tree is performed using the provided data and the search task list. At operation 702, executable instructions for each task are sequentially provided to the processor's interface along with payload data for each task, which are executed at operation 704. When searching the search tree, at operation 706, it is determined whether a touch has occurred within a UI element, and the result of each task is true / false, success / failure, indicating whether a touch has occurred within the UI element. At operation 708, the processor loads and receives the next task instructions and associated data in response to the result of the current task being successful. That is, if the result of operation 706 is success, the next task in the task list is executed.

[0055] If the result is a failure, alternate task instructions and associated data responsive to the result are loaded and received by the processor. If an alternate task exists (operation 714), the alternate task location is provided at operation 716 and the process loops back to operation 702 to load the task from the alternate location on the processor. When the search is over, the UI element is either found or not found. If the UI element is found, operation 710 returns the found result and operation 712 returns the element's ID, as well as any preference / responsiveness parameters. If the element is not found, operation 720 returns the not found result.

[0056] In one embodiment, the search process shown in Figure 7 can be a firmware application running on a processor (microcontroller) on the touch. The touch processor can have one or more search tasks executed by the search process stored in flash memory. In one embodiment, the search tasks can be stored in RAM associated with the display controller and provided to the touch processor during a configuration or provisioning process and kept accessible to the search process.

[0057] At this point, the inventors realize that the described embodiments provide several advantages over alternative approaches. Memory requirements are significantly reduced - up to 50% from linear search, pure grid, or pure search tree approaches - and still provide an improvement over the combined grid / tree approach. This is in part because the number of search operations performed is reduced. Because the number of search operations is reduced, response cycles are significantly shorter than alternative approaches (including conventional approaches). For example, for a 1200x1200 touch-sensitive GUI, cycle times of less than 36μs were achieved compared to alternative approaches that ranged from 72μs (pure grid) to 1200μs (linear). The difference is an extremely responsive touch interface for the user. This touch interface can be made more sophisticated with numerous elements with different response characteristics for the designer.

[0058] Figures 8, 9, and 10 illustrate the processes illustrated and described with reference to Figures 2-7 in relation to a GUI for a radio application, as one non-limiting example of a GUI that may be used with embodiments of the present disclosure. The radio GUI 810 illustrated in Figure 8 includes eight types of UI elements, totaling 144 UI elements, summarized in table 820.

[0059] Figure 9 illustrates UI elements grouped according to the method described with reference to Figure 3. In this embodiment, grouped elements 832, 834, 836, 838, 840, 842, 844, 846, and 848 have similar touch characteristics (e.g., haptic feedback in response to touch), physical location on the screen, and shape.

[0060] FIG. 10 illustrates one embodiment of a tree-like structure 850 formed using the tree and grid method described with reference to FIG.

[0061] FIG. 11 illustrates a system 1000 and associated tools 1040 capable of implementing the search methods described herein, according to one embodiment of the disclosure. The system 1000 includes a microcontroller firmware 1010 having thereon a GUI element search function 1012 for GUI elements and a response determination 1014. A processor executing the microcontroller firmware 1010 is coupled to a response driver 1018 that can receive control signals from the microcontroller firmware 1010 and subsequently drive responses of the touch-sensitive interface 1020. In one embodiment, the touch-sensitive interface 1020 is a touch screen with one or more actuators, and the response driver 1018 is a haptic driver configured to generate control signals that excite the actuators. The sensing circuit 1022 can generate one or more measurement signals in response to a contact at the touch-sensitive interface 1020. Touch measurement and processing 1016 can determine touch information (e.g., location, type, etc.) in response to measurement signals from sensing circuitry 1022 and provide this touch information to response determination 1014 and GUI element search function 1012. Control signals received by response driver 1018 can be based, for example, at least in part, on the touch information such that haptic feedback is provided at a right location on touch-sensitive interface 1020.

[0062] Also shown in Figure 11 is a tool 1040 that can implement a search list generation process to create an element search task list and element response information in accordance with one embodiment of the present disclosure. A search list creation application program 1044 is configured to implement the process described with reference to Figures 2-6, a GUI definition XAML file 1042 to generate an element search task list. The application 1044 can provide an element search task list 1046 and element response information 1048 as files to the microcontroller firmware 1010. In one embodiment, this application can also provide a search tree, which can be incorporated into the search task.

[0063] In one or more embodiments of the firmware, the firmware can include force measurement and processing functionality to incorporate force level information for touch events. In such embodiments, the force level information and GUI element ID and haptic response details returned by the element search function can be used by a haptic sequencer to generate haptic control signals responsive to the force level, GUI element ID, and haptic response details.

[0064] The system of FIG. 11 can be incorporated into a variety of consumer products, appliances, and machines that utilize touch interfaces and touch control panels, including automobiles.

[0065] FIG. 12 illustrates a simplified version of a radio GUI 1210 for an automobile touch control panel. Three regions are specifically referred to as region 1, region 2, and region 3. Region 1 is a central button 1212 with a rotary dial 1214 for temperature control. Haptic feedback is provided with haptic profile ID#4 (vibration) in response to touch events with a strong force level. Region 2 is a rotary dial 1214, also for temperature control. Haptic feedback is provided with haptic profile ID#3 (friction) in response to touch events with a weak force level. Finally, region 3 is a button 1216 for presenting a menu for automobile settings. Haptic feedback is provided with haptic profile ID#2 (click) in response to touch events with a strong force level, and with haptic profile ID#3 (friction) in response to touch events with a weak force level.

[0066] Figure 13 illustrates the system of Figure 11 and the GUI of Figure 12 integrated into an automobile control commanded by a head unit 1310, whose haptic effects are controlled by a microcontroller. In this embodiment, the touch controller 1320 and shape search function 1324 are part of the automobile subsystem, and the automobile head unit 1310 responds to touches with haptic feedback without direct intervention of the head unit's processing circuitry. The touch controller 1320 is configured to operate a touch state machine that identifies the touched screen button from the touch location and force level information, including the button location for triggering the haptic effect.

[0067] In this embodiment, force processing 1326 and touch processing 1322 are integrated into one controller component, and the head unit screen 1332 contains the definition of several geometric object descriptions (screen display design 1336, and search tree definition 1338) that are each required to derive the range of haptic effects activated directly by the touch controller 1320 and performed by the haptic device 1350.

[0068] For example, after a touch on the display 1330, the touch controller 1320 receives force and touch information from the force processing 1326 and the touch processing 1322. This information can include a force measurement and a touch location on the display. The shape search 1324 provides UI element information, if applicable, corresponding to a UI element displayed on the display 1330 where the touch occurred. If there is no UI element corresponding to the location on the display, the shape search 1324 provides a null search result. While searching for the shape information of the UI element, the shape search 1324 can use a definition stored in the head unit 1310. In one embodiment, the shape search 1324 can receive the definition during a provisioning process, for example, when the touch controller 1320 is integrated with the head unit 1310 or when the head unit 1310 is powered on. If the shape search 1324 identifies a UI element, the haptic information is used by the haptic control 1328 to send a haptic activation message to the haptic device 1350 including a haptic effect and a location of the haptic effect. The haptic activation message may include a parameter representing the level of the haptic effect (e.g., weak, medium, strong). The haptic device 1350 searches for the haptic effect definition in a haptic library 1352 stored in the haptic device. The haptic device 1350 then controls actuators of the display 1330 such that a particular area of ​​the display exhibits the requested haptic effect. In particular, different haptic devices may have different haptic libraries, so the effects may vary between devices.

[0069] In this embodiment, the GUI definition is an XAML file, which is an XML implementation for a graphical user interface. An XAML file contains a hierarchical list of drawing instructions for the screen elements of the GUI's UI. In an XAML file, there are tags associated with the GUI elements. For example, "Width", "Height", and "Horizontal Alignment" are all valid tags for a particular element.

[0070] Many of the functional units described herein may be shown, described, or labeled as modules, threads, or other segregations of programming code to more specifically emphasize their implementation independence. The modules may be implemented at least partially in hardware in one form or another. For example, the modules may be implemented as hardware circuits comprising custom VLSI circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. The modules may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, or the like.

[0071] Modules may also be implemented using software or firmware stored in physical storage devices (e.g., computer-readable storage media), memory, or combinations thereof, for execution by various types of processors.

[0072] An identified module of executable code may, for example, comprise one or more physical or logical blocks of computer instructions which may, for example, be organized as a thread, an object, a procedure, or a function. Nevertheless, an identified executable module may comprise different instructions stored in different locations which need not be physically located together but which, when logically coupled together, comprise a module and achieve a specified purpose of the module.

[0073] In practice, a module of executable code may be a single instruction, or many instructions, and may be distributed across several different code segments among different programs, and across several storage devices or memory devices. Similarly, operational data may be identified in modules and illustrated herein, and may be embodied in any suitable form and organized in any suitable type of data structure. The operational data may be collected as a single data set, or may be distributed in different locations across different storage devices, or may exist, at least in part, simply as electronic signals on a system or network. If a module or a portion of a module is implemented in software, the software portion is stored in one or more physical devices, referred to herein as computer-readable media.

[0074] In some embodiments, the software portion is stored in a non-transitory state such that the software portion or a representation thereof remains in the same physical location for a period of time. Furthermore, in some embodiments, the software portion is stored in one or more non-transitory storage devices that comprise hardware elements capable of storing non-transitory states and / or signals representative of the software portion, although other parts of the non-transitory storage devices may perform signal modification and / or transmission. Examples of non-transitory storage devices include flash memory and random-access-memory (RAM). Another example of a non-transitory storage device includes read-only memory (ROM), which may store signals and / or states representative of the software portion for a period of time. However, the ability to store signals and / or states is not diminished by the further ability to transmit signals that are identical to or representative of the stored signals and / or states. For example, a processor may access a ROM to obtain signals representative of the stored signals and / or states in order to execute the corresponding software instructions.

[0075] On a practical level, software that enables a computer system to perform the operations described herein can be supplied on any one of a variety of media. Moreover, the actual implementation of the techniques and operations of the present invention are actually statements written in a computer language. Such computer language statements, when executed by a computer, cause the computer to act according to the particular content of the statements. Moreover, the software that enables a computer system to act according to the present invention can be provided in any number of forms, including, but not limited to, original source code, assembly code, object code, machine code, compressed or encrypted versions thereof, and any and all equivalents.

[0076] Those skilled in the art will recognize that "medium" or "computer readable medium" as used herein can comprise a diskette, tape, compact disc, integrated circuit, ROM, CD, DVD, BLU-RAY, cartridge, flash memory, memory stick, or card, or any other non-destructive storage medium usable by a computer, including those now known or later developed.

[0077] It will be appreciated that actual software may be "written" on disk, "embodied" in an integrated circuit, "carried" over a communications circuit, "stored" in a memory chip, or "loaded" into a cache memory, but for purposes of this application, the software will simply be referred to as being "in" or "on" a computer-readable medium. Thus, the terms "in" or "on" are intended to encompass the above-mentioned and all equivalent and possible ways in which software may be associated with a computer-readable medium.

[0078] For simplicity, therefore, the term "computer program product" will be used to refer to a computer readable medium as defined above, the medium having any form of software that enables a computer system to operate in accordance with any embodiment of the present invention.

[0079] While the present disclosure has been described herein with respect to certain illustrated embodiments, those skilled in the art will recognize and understand that the present invention is not so limited. Rather, numerous additions, deletions, and modifications can be made to the illustrated and described embodiments without departing from the scope of the present invention as claimed below together with their legal equivalents. In addition, features of one embodiment can be combined with features of another disclosed embodiment as contemplated by the inventors and still fall within the scope of the present disclosure.

Claims

1. a microcontroller operably coupled to a touch screen configured to display a graphical user interface (GUI), At least one processor; One or more executable instructions stored on a non-transitory storage medium that, when executed by the processor, cause the processor to: generating a list of search tasks linked in memory at consecutive physical memory addresses to form a conditional search task list; determining a location of a sensed touch on the touch screen; and executing one or more instructions of the conditional search task list to identify a GUI element associated with the touch location corresponding to the sensed touch; The microcontroller, wherein the search tasks in the conditional search task list are executed in the same sequential order as the search tasks are ordered in a non-transitory memory.

2. The microcontroller of claim 1 , wherein the one or more instructions enable the processor to identify a user feedback response responsive to the identified element of the GUI.

3. The microcontroller of claim 2 , wherein the instructions further adapt the processor to generate one or more control signals responsive to the user feedback response associated with the identified GUI element.

4. The microcontroller of claim 3 , wherein the user feedback response is a haptic response.

5. The microcontroller of claim 4 , wherein the control signals are configured to control a haptic driver to excite one or more actuators in the touchscreen.

6. The microcontroller of claim 5 , further comprising determining a force level associated with the touch, wherein the processor creates the one or more control signals responsive to the force level.

7. The one or more executable instructions include one or more conditional search instructions that, when executed by the processor, cause the processor to: comparing the location of the touch with a location of one or more elements of the GUI; selecting a GUI element responsive to said comparison; and providing a haptic command and an element identifier corresponding to the selected GUI element.

8. The microcontroller of claim 7 , wherein the haptic command is configured to instruct a haptic driver to excite one or more actuators in a touchscreen responsive to the haptic command.

9. generating a list of search tasks linked in memory at consecutive physical memory addresses to form a conditional search task list; determining a location of the sensed touch on the touch screen; sequentially executing one or more search tasks of the conditional search task list responsive to the determined locations, each search task of the conditional search task list corresponding to an element of a graphical user interface (GUI) and returning search results when executed; identifying a GUI element responsive to the search results; The method of claim 1, wherein the one or more search tasks are executed in the same sequential order as the one or more search tasks are ordered in the non-transitory memory.

10. The method of claim 9 , further comprising identifying a user feedback response responsive to the identified GUI element.

11. The method of claim 10 , further comprising generating one or more control signals responsive to a user feedback response associated with the identified GUI element.

12. The method of claim 11 , wherein the user feedback response is a haptic response.

13. The method of claim 12 , wherein the control signals are configured to control a haptic driver to excite one or more actuators in the touchscreen.

14. The method of claim 13 , further comprising determining a force level associated with the touch, wherein a processor creates the one or more control signals responsive to the force level.

15. The method of claim 9 , wherein each search task includes a payload that is inserted in response to a search tree.

16. 1. A system comprising: a display subsystem configured to control a display; 1. A touch subsystem comprising a touch sensor and a touch controller, the touch controller comprising: generating a list of one or more search tasks concatenated in memory at contiguous physical memory addresses to form a conditional search task list; determining a location of a touch sensed at the touch sensor; Sequentially executing one or more search tasks of the conditional search task list responsive to the location and search tree, each search task of the conditional search task list corresponding to a GUI element and adapted and executed to return search results when executed; identifying a GUI element responsive to the search results; and generating a haptic control message responsive to the identified GUI element. The one or more search tasks are executed in the same sequential order as the one or more search tasks are ordered in the non-transitory memory.

17. The system of claim 16 , further comprising a haptic feedback subsystem configured to generate haptic feedback at the display.

18. 17. The system of claim 16, wherein some of the one or more search tasks comprise a jump instruction associated with another search task in the conditional search task list and are configured to load the other search task when executed.

19. 20. The system of claim 18, wherein the jump instruction enables one or more search tasks of the conditional search task list to be executed in a tree-like order.

20. 20. The system of claim 16, further comprising an automobile head unit.

Images