Touch classifier system using contextual data

Abstract

A method of providing classified touch data to a computer program executing on a device comprises: (a) assembling a map of touch signals from a touch sensor arranged on an electronic display and comprising a plurality of intersections of row electrodes and column electrodes, the graph comprises a corresponding touch signal value of each cross point in the plurality of cross points and defines at least one touched area of the touch sensor; (b) providing contextual data relating to user interface content currently presented on the electronic display; (c) calculating classified touch data corresponding to the graph of the touch signal based at least in part on the graph and the context data; and (d) providing the classified touch data to an operating system of the device.

Description

Background Art

[0001] Capacitive touch sensors are input components found in a variety of electronic devices. In some devices, capacitive touch sensors consist of optically transparent material patterned over or integrated into the display panel. This touchscreen display configuration, common in laptops, tablets, smartphones, and gaming systems, provides combined touch input and display functionality. Summary of the Invention

[0002] One aspect of the present disclosure relates to a device comprising an electronic display, a touch sensor disposed on the electronic display and comprising a plurality of intersections of row electrodes and column electrodes, and touch sensor logic, a context engine, and a touch classifier system operatively coupled thereto. The touch sensor logic is configured to assemble a map of touch signals from the touch sensor, the map comprising a corresponding touch signal value for each of the plurality of intersections and defining at least one touched area of ​​the touch sensor. The context engine is configured to collect and provide context data related to user interface content currently presented on the electronic display. The touch classifier system is configured to calculate classified touch data corresponding to the map of touch signals based at least in part on the map and the context data, and to provide the classified touch data to an operating system of the device.

[0003] Another aspect of the present disclosure relates to a method for providing classified touch data to a computer program executed on a device having an integrated electronic display. The method includes: (a) assembling a map of touch signals from a touch sensor arranged on the electronic display and including a plurality of intersections of row electrodes and column electrodes, the map including a corresponding touch signal value for each of the plurality of intersections and defining at least one touched area of ​​the touch sensor; (b) providing contextual data related to user interface content currently presented on the electronic display; (c) calculating classified touch data corresponding to the map of touch signals based at least in part on the map and the contextual data; and (d) providing the classified touch data to an operating system of the device.

[0004] This Summary is intended to introduce some concepts in a simplified form that are further described in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. The claimed subject matter is not limited to implementations that address any disadvantages noted in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1A Aspects of an exemplary handheld device having a touch sensor disposed on an electronic display are shown.

[0006] Figure 1B Aspects of two example handheld devices are shown, each having a touch sensor disposed on an electronic display.

[0007] Figure 2 Aspects of an example capacitive touch sensor are shown.

[0008] Figure 3 Aspects of an example capacitive touch sensor and associated touch sensor logic are shown.

[0009] Figure 4 Aspects of a graph of an example touch signal are shown.

[0010] Figure 5 Aspects of an example computer system are shown.

[0011] Figure 6A Aspects of an example layout of user interface elements on a display are shown.

[0012] Figure 6B Shown with Figure 6A The example layout corresponds to various aspects of the context data.

[0013] Figure 7 Aspects of an example method of providing classified touch data to a computer program executing on a device are shown.

[0014] Figure 8 Aspects of an example pen configured to provide input to a capacitive touch sensor are shown.

[0015] Figure 9 Shown Figure 5 Additional aspects of the example computer system. DETAILED DESCRIPTION

[0016] In many cases, touchscreen display devices may operate in a manner that encourages false or unintentional touch events, potentially degrading the user experience if these events are processed as input. This problem is particularly troublesome for relatively small handheld devices with little or no non-functional bezel—that is, touch sensing extends all the way to the edges of the device, where users may grip the edges during normal operation. Many touchscreen device users have experienced situations where a gripping motion of the thumb or palm is mistakenly interpreted as valid touch input, resulting in the closing of running applications, unintended scrolling, and so on. Some useful remedies for this problem attempt to distinguish valid from invalid touches by identifying geometric features of the fingertips (rather than other parts of the hand) in the touch signal map from the touch sensor. However, this approach is difficult to implement, especially in relatively noisy signal environments. For example, in "floating ground" operation, where the device is neither connected to AC power nor held by the user's bare hand, excessive noise may be observed.

[0017] The present disclosure provides a powerful solution to solve the above problems and provides additional advantages. Broadly speaking, the method of the present disclosure collects certain contextual data from the device's operating system and provides this data, along with an (X, Y) graph of the touch signal, to a touch classifier system executed at one or more processing levels on the device. The touch classifier system determines which touch areas are valid and which are invalid, and reports this information to the OS. Contextual data can include various types of data, such as device orientation, user hand habits, etc. One of the most important features of contextual data is the layout of user interface elements presented on the display at any given time. Such a layout includes the position and size of various icons and other markers, and the touch classifier system can compare any touched area identified by the graph with these icons and markers. Compared to existing methods, the solutions proposed in this article can reduce the false alarm rate in valid touch detection. By rejecting invalid touch events in the early stages of the processing pipeline, these solutions can also reduce the latency and power consumption of valid touch detection.

[0018] Turning now to the accompanying drawings, Figure 1AAspects of an example handheld device 10 having an electronic display 12 are shown. Devices consistent with the present disclosure include laptop computers, tablet computers, smartphones, and handheld gaming systems. Other examples include touch screen displays installed in cars or kiosks, as well as virtually any other type of touch screen display device. While the size, functionality, and form factor of the device are not particularly limited, certain aspects of the present solution are particularly suitable for handheld devices that can be held by a user with their hand or fingers touching the front surface 14 of the device. In some examples, the electronic display 12 is a liquid crystal display (LCD). In some examples, the electronic display is a light emitting diode (LED) display, such as an active matrix organic LED (AMOLED) or a quantum LED (QLED) display. Other types of electronic displays are also contemplated.

[0019] Device 10 includes a capacitive touch sensor 16 disposed on top of electronic display 12, i.e., on front surface 14. The touch sensor is configured to acquire touch signals in response to a touch from a user's finger or stylus on the front surface, thereby locating at least one touch point caused by the user. Touch point 18A is the contact point between the user's fingertip 20 and front surface 14. Figure 1B The touch point 18B in FIG. 1 is the contact touch point between the user pen 22 and the front surface.

[0020] Figure 2Various aspects of an example capacitive touch sensor 16 are shown in detail. The capacitive touch sensor includes a series of row electrodes 24 that intersect with a series of column electrodes 26. The touch sensors considered here can include any number N of row electrodes and any number M of column electrodes, providing MN intersections or points of intersection. Although it is typical for the row electrodes to be aligned horizontally and the column electrodes to be aligned vertically, this is not necessary as the terms "row" and "column" are used interchangeably in this description. In the illustrated example, a series of row electrodes 24 are arranged on one side of a dielectric layer 28 and a series of column electrodes 26 are arranged on the opposite side of the dielectric layer. In some examples, the dielectric layer can be 50μm to 100μm, although other thickness ranges are also contemplated. The dielectric layer can include a polymer film, such as polyethylene terephthalate (PET). In a curved, flexible and / or bendable touch screen system, the dielectric layer can be curved, flexible and / or bendable. The row electrodes 24 and column electrodes 26 each include a conductive material 30 distributed in the form of narrow (e.g., 1 mm wide) elongated strips on opposing surfaces of the dielectric layer 26. In some examples, adjacent electrodes may be spaced 1 mm to 5 mm apart. The composition of the conductive material 30 is not particularly limited. The conductive material may include a metal microwire mesh, metal particles or metal island films, or a degenerately doped semiconductor film such as indium tin oxide (ITO). Regardless of the embodiment, it is desirable that the capacitive touch sensor include row and column electrodes with relatively low resistance and high optical transmittance.

[0021] Now turn Figure 3 , the row electrodes 24 and column electrodes 26 of the capacitive touch sensor 16 are addressed by touch sensor logic 32, which is operatively coupled to the touch sensor 16. The touch sensor logic is configured to sense contact on or near the front surface of the device, including the coordinates (X, Y) directly below the touch point 18 on the front surface. To this end, the touch sensor logic includes row drive circuitry 34 and column sense circuitry 36. The terms "row drive" and "column sense" are intuitive for configurations in which the drive signal is driven by the row electrodes and sensed via the column electrodes. Based on this intuitive understanding, the terms "column signal" and "touch signal" may be used interchangeably in this disclosure. Of course, the opposite configuration is also foreseeable.

[0022] Column sensing circuitry 36 is configured to sense column signals from the series of column electrodes 26. In the illustrated example, the column sensing circuitry includes M column amplifiers, each coupled to a corresponding column electrode. Row driver circuitry 34 includes a local row counter 38 in the form of an N-bit shift register, having an output that drives each of the N row electrodes 24. The local row counter is clocked by a row driver clock 40. The local row counter includes a blanking input for temporarily forcing all output values ​​to zero, regardless of the value stored in the shift register. Stimulation for one or more rows can be provided by filling the local row counter with 1s at each output to be stimulated and 0s elsewhere, then switching the blanking signal using the desired modulation from modulation clock 42. In the illustrated example, the output voltage can only assume two values, corresponding to the 1 or 0 stored in each bit of the local row counter. In other examples, the output voltage can assume a wider range of values, for example, to reduce the harmonic content of the output waveform or to lower radiated emissions. In some examples, row driver circuitry 34 may include one or more additional registers offset from the local row counter 38 and blanked by a modulation clock of a different frequency. Such variations are capable of stimulating two or more row electrodes (eg, electrodes separated by a fixed number of intermediate rows) simultaneously.

[0023] like Figure 3 The row driver circuit 34 shown applies an excitation pulse to each row electrode 24 in turn. In this way, the row driver circuit is configured to simultaneously drive one or more row electrodes in a series of row electrodes while leaving one or more other row electrodes in the series undriven. During periods when the front surface is untouched, none of the column amplifiers registers an output exceeding the threshold. However, the electrical impedance near the intersection of a row electrode 24 and a column electrode 26 is responsive to the proximity of a finger or stylus to that intersection: when a user places a fingertip on the front surface, the fingertip capacitively couples one or more row electrodes 24 near the touch point 18 (e.g., the intersection) with one or more column electrodes 26 also near that touch point. Capacitive coupling causes the largest signal changes on the column electrodes directly behind the touch point, and smaller changes on the column electrodes on either side of the touch point, with these changes gradually decreasing with increasing distance from the touch point.

[0024] The column sense circuit 36 ​​returns the value of the column that provides the maximum received signal as the X coordinate of the touch point. The touch sensor logic 32 determines which rows are excited when the maximum signal is received and returns the value of the row as the Y coordinate of the touch point. In some examples, the column sense circuit 36 ​​may also return a Z coordinate that varies based on the signal strength received at the coordinates (X, Y). Accordingly, the touch sensor logic 32 can distinguish between a force touch associated with a strong signal, a light touch associated with a weaker signal, and a hover associated with an even weaker but detectable signal. Alternatively or additionally, the touch sensor logic can provide a varying output based on the contact area, e.g., the number of electrodes touched, which can also be related to the touch force. For pen detection (see below), the shape of the touch track on the electrode can be used to distinguish ink and / or hover from noise.

[0025] A brief review Figure 1B , pen 22 can replace the user's fingertip to perform touch input on electronic device 10. Accordingly, the touch sensor can be configured to sense the position of pen tip 42 relative to front surface 14. Like the user's fingertip, the tip of the passive pen includes a high dielectric constant material that capacitively couples row and column electrodes at the touch point. The passive pen provides better touch accuracy than a fingertip and can reduce smudges on the front surface. Although touch screen input devices are typically slender cylindrical, this shape is not required. Figure 1B Also shown is another touch input device 22' that can be used with a large-format electronic device 10'. It should be noted that touch input devices of various shapes and sizes are contemplated herein, and all references to "pen" are equally applicable to other touch input devices. In addition to being able to track touch points faster and more accurately than passive pens, active pens also provide greater touch accuracy. Figure 8 Various aspects of active pen tracking are described in the context of

[15] .

[0026] As mentioned above, Figure 3The touch sensor logic 32 in is configured to provide appropriate drive signals to the drive electrodes of the touch sensor and to digitize the signals obtained on the sense electrodes. To this end, the touch sensor logic can take the form of one or more application-specific integrated circuits (ICs), typically including at least one low-power firmware processor 44. The touch sensor logic is also configured to assemble a map of touch signals obtained by the touch sensor. The map can be implemented using any suitable data structure, including a matrix of touch signal values ​​collected at each intersection of the row electrodes and the column electrodes. Each row i of the matrix is ​​a digitized form of the output of the column sensing circuit 36, obtained from each column electrode when the i-th row electrode is energized. Accordingly, the map includes a corresponding touch signal value for each of the multiple intersections and can define at least one touched area of ​​the touch sensor. Typically, the touch signal values ​​need to be denoised. Figure 4 Various aspects of such a graph are shown in one non-limiting example. FIG. 46 shows a first touched area 48A and a second touched area 48B superimposed on the graph. First touched area 48A is the area touched by two fingers of the user. Second touched area 48B is the area touched by the user's thumb on one side of the device when the user is holding the device. The term "blob" may be used to refer to a continuous matrix element in a graph corresponding to a given touched area, and the term "heat map" may be used to refer to the graph.

[0027] Back again Figure 1A , device 10 includes an inertial measurement unit (IMU) 50. The IMU may include an electronic accelerometer and an electronic gyroscope, which are configured to provide six degrees of freedom (DOF) measurements of linear acceleration and angular acceleration of the electronic device. In some examples, the IMU may also include an electronic compass based on a magnetometer. Thus, the IMU can be configured to provide estimates of the relative orientation and absolute orientation of the device.

[0028] The apparatus 10 also includes a computer 52 . Figure 5 Certain aspects of a computer are shown schematically. The computer 52 has a logic system 54 and a computer memory system 56 operatively coupled to the logic system. The logic system may include at least one processor, such as a central processing unit (CPU) 58 and / or a graphics processing unit (GPU). The computer memory system stores, among other data, program instructions that, when executed by the logic system, cause the logic system to perform various operations. In the illustrated example, the computer memory system stores instructions for an operating system (OS) 60 of the device 10, including a kernel 62 and one or more application programs (not shown). The instructions stored in the computer memory system may cause the logic system to instantiate one or more machines or engines described herein. In the example shown, the computer memory system stores instructions for an operating system (OS) 60 of the device 10, including a kernel 62 and one or more application programs (not shown). The instructions stored in the computer memory system may cause the logic system to instantiate one or more machines or engines described herein. Figure 5In the example shown, the instructions cause the logic system to instantiate a context engine 64 and at least a portion of an engine system referred to herein as a "touch classifier system" 66. However, it should be understood that in some examples, a portion of the touch classifier system is executed in the touch sensor logic 32 rather than in the computer 52 (see below).

[0029] The context engine 64 is configured to collect and provide context data 68 related to at least the user interface (UI) content currently presented on the electronic display 12. Figure 5 In the example shown, the context engine is part of the OS kernel 62 of the device 10, which provides the technical advantage of direct observation of various UI update events requested by applications. Context data can be collected from any application running on the device, including the OS shell, or from the graphics layer itself. Generally speaking, the context data not only indicates the currently running application; it also includes the screen position and size of each UI element, and an indication of the touch events that each UI element can "handle" in the sense of event handling. The context data provides a basis for distinguishing between valid and invalid touched areas, independent of the touch signal, which provides the technical effect of reducing the possibility of accidental touches triggering UI responses. Figure 6A shows an example layout of user interface elements on a display, Figure 6B Shows the Figure 6A Context data 68A corresponding to the example layout in.

[0030] In a modern application framework, each UI element displayed on the display is drawn by a method linked to a programmatic object when that object is created or updated. Because each drawing method includes a call to the OS, the OS "knows" the layout of the UI elements on the display at any given time, as well as the dimensions of each UI element. The OS also knows which touch events a given UI element must respond to and which touch events it can ignore. With this configuration, the context engine 64 can be configured to "detect" various object creation and update events and assemble, maintain and / or update a file of context data 68. The file can be in any suitable format, including JSON format. In some examples, the file can be a memory mapped file that can be provided to any engine of the touch classifier system 66 with low latency.

[0031] The touch classifier system 66 is configured to calculate classified touch data corresponding to the graph of touch signals based in part on the graph itself and in part on the contextual data 68. The classified touch data includes at least one pair of (X, Y) coordinates for each valid touched area in the graph. The classified touch data may also include an indication of the class of the touch (e.g., one finger or two fingers). The touch classifier system is further configured to provide the classified touch data to the OS 60 of the device 10. A technical effect of the touch classifier system is to incorporate contextual data into downstream determinations of touch validity, thereby improving the accuracy of determinations and reducing false positives. A technical effect of the touch classifier system is to incorporate contextual data into downstream determinations of touch validity, thereby improving the accuracy of determinations and reducing false positives. In Figure 5 In the example shown, the touch classifier system 66 includes a firmware engine 70 , a middleware engine 72 , and a software engine 74 .

[0032] Instructions and / or other logic, including those in the firmware engine 70, are executed on the firmware processor 44 of the touch sensor logic 32. The logic coordinates the digitized outputs of the column sense amplifiers with the row counters to generate a (possibly noisy) graph of capacitance or admittance as a function of column position X and row position Y. For example, the firmware engine performs a denoising routine to subtract common-mode noise that may be injected by an electronic display, power supply, or mains, etc. One non-limiting strategy for electromagnetic noise suppression on a capacitive touch sensor is to place at least one untouched sensing electrode slightly away from the touch point and use that electrode as a reference—i.e., an indicator of common-mode noise, which is then subtracted from the signal recorded at the touch point. The firmware engine then performs a rough segmentation of the denoised graph. To do this, the segmentation algorithm can compare each denoised signal value with a threshold to determine whether the corresponding electrode intersection (X, Y) is part of the touched area. The segmentation algorithm then identifies matrix elements with the same numerical index corresponding to continuous touched areas, and identifies matrix elements with different numerical indices corresponding to discontinuous touched areas, such as Figure 4 shown.

[0033] The instructions comprising the middleware engine 72 are executed at the OS level, on the CPU or GPU of the computer system 52, or on an auxiliary processor. The middleware engine may be configured to roughly categorize the matrix elements corresponding to each area touched by shape.

[0034] Instructions comprising the software engine 72 are executed at the OS level, on the CPU or GPU of the computer system 52. The software engine is configured to classify each matrix element corresponding to a touched area into one of a number of predefined categories. These categories may include "single finger," "two fingers," "more than two fingers," or "palm," among others. Categories such as "single finger" or "two fingers" are classified as valid touches, while other categories are classified as invalid touches. In some examples, the software engine may include a neural network trained to perform the classification; in some examples, deterministic logic may be used. Whether through a neural network or a deterministic approach, the software engine may be configured to fuse the output of the middleware engine 72 with other inputs or indications (e.g., input from a pen tracking system). In one example, indications of valid pen signals may be used according to a suitable heuristic algorithm to reduce the probability that a given touched area is associated with a valid category. In other words, when valid pen signals are also detected, the matrix element corresponding to a given touched area is more likely to be classified as invalid. In another example, the software engine may fuse input from the IMU 50 with the output of the middleware engine to influence the classification. The software engine 74 is also configured to generate and provide touch reports that may be disclosed to any application executing on the device 10 having appropriate permissions.

[0035] exist Figure 5 In the example shown, the firmware engine 70, the middleware engine 72, and the software engine 74 each receive context data 68 related to the user interface content currently presented on the electronic display. In other examples, only one or two engines may receive context data.

[0036] Figure 7 Aspects of an example method 76 of providing classified touch data to a computer program executing on a device are shown. The device has an integrated electronic display and a touch sensor disposed on the electronic display, as described above.

[0037] At 78A of method 76, a touch sensor acquires touch signals on a plurality of sense electrodes. The term "touch signals" refers collectively to a set of touch signals from each of the sense electrodes, typically coordinated with an output of a row counter. At 78B, touch sensor logic operatively coupled to the touch sensor assembles a map of the touch signals from the touch sensor. In a scenario related to method 76, the map includes a plurality of intersections of row and column electrodes, the map includes a corresponding touch signal value for each of the plurality of intersections, and defines at least one touched area of ​​the touch sensor. At 78C, a context engine of the device collects and provides context data related to user interface content currently displayed on the electronic display. A touch classifier system receives the context data.

[0038] At 78D, the touch classifier system of the device calculates classified touch data corresponding to the map of touch signals. At 78E, the touch classifier system provides the classified touch data calculated at 78D to an operating system of the electronic device.

[0039] The classified touch data is based in part on the map of touch signals assembled at 78B and in part on the contextual data provided at 78C. As described above, the classified touch data includes at least one pair of (X, Y) coordinates for each touched area of ​​the map and may also include an indication of the touch category (e.g., one finger or two fingers). Generally speaking, the touch classifier system can employ variable strictness or tolerance to determine whether a valid touch has been received in a given touched area. One technical effect of this feature is that it provides a baseline metric that can be used in conjunction with a variable threshold, i.e., a threshold that is adjusted upward or downward based on the contextual data 68, to ensure the aforementioned advantages. The strictness or tolerance can be parameterized in a variety of ways—for example, through a threshold or acceptance interval that is compared to some metric of the touched area. Some useful metrics can quantify the size of the touched area—for example, its diameter or circumference. Smaller touched areas are more likely to be caused by an intentional touch with a finger, while larger touched areas are more likely to be caused by an accidental touch with the palm or side of the hand. Accordingly, the validity judgment can be made more stringent by lowering the size threshold or more lenient by raising the size threshold, both of which are controlled by the contextual data. In this way, the classified touch data can be calculated based partly on the size of the touched area shown in the figure and partly on the contextual data. In addition to size, the shape of each touched area can also be evaluated for similarity to the shape of a finger touch, rather than a touch on another part of the hand. As with size, the similarity metric can also be compared to a threshold that depends on the contextual data to determine whether the touched area received a valid touch. Evaluating the validity of touch input based on the size and / or shape of the touched area has the technical advantage of providing a baseline indication of whether the touch corresponds to an intentional touch by the user's finger.

[0040] Another useful metric is the duration of a touch in a given touched area. A purposeful touch may last only a fraction of a second, while a touch caused by a user holding the device or resting their hand on the device while writing may last longer. In some examples, calculating the classified touch data includes reporting a touch on a touched area if the touched area disappears within a time interval responsive to the contextual data.

[0041] Another useful metric is the distance between the touch area and the edge of the display. In typical operating scenarios, user activity is likely concentrated toward the center of the display, rather than the edges. Therefore, in some examples, classified touch data is calculated at least in part based on the position of the touched area relative to the display. As described below, the thresholds or acceptance intervals used with any of these metrics can be made stricter or looser by adjusting them in response to contextual data.

[0042] In some examples, the most useful feature of the contextual data for evaluating whether a touch area receives a valid touch is the layout of UI elements on the display. For example, in examples where the size of a touch area is compared to a threshold to determine validity, the threshold can vary based on one or more aspects of the UI elements co-located with the touched area on the display. This feature enables detailed UI layout to influence which touch areas are recognized as valid and which are recognized as invalid, thereby providing an important technical effect of reducing the frequency of false touch detections when the touch area does not fit the UI elements (if any) in the touched area. Larger UI elements (such as icons) may increase the threshold, while smaller UI elements (such as scroll bars) may lower the threshold. In these examples, calculating classified touch data may include reporting a touch on the touched area if the touched area is smaller than the co-located UI element or smaller than the spacing between adjacent co-located UI elements. In other examples where size thresholds are used to parameterize strictness or tolerance, the threshold can be set higher for areas of the display where accidental touches may occur (e.g., edges); and lower for areas of the display where touch input is expected (such as a virtual keyboard or within a text box). In some examples, calculating the classified touch data may include reporting a touch on the touched area only when the touched area is within a predetermined input-receiving UI element (e.g., a virtual keyboard or text box), and not reporting a touch if the touched area is outside the input-receiving UI element. In examples where the touch input is expected to be within the confines of a virtual keyboard, the valid touched area may be limited to the size of the key.

[0043] More generally, in a heuristic sense, the thresholds can be made stricter for areas of the display where user input is less likely. Such areas can include areas without user interface elements, and / or areas where clicking would transfer input focus to a different application. For example, calculating classified touch data can include reporting a touch on a touched area only when a user interface element (e.g., an icon) is within a threshold distance of the touched area. In another example, if a web browser is open and covers 80% of the display, a strict threshold for valid touch input outside the browser client area can be enforced so that inadvertent hand contact while hovering does not steal the web browser's focus. Similarly, a strict threshold for valid touch input can also be used near scroll bars to prevent erroneous scrolling.

[0044] In some examples, calculating the classified touch data includes reporting a touch on the touched area if the touched area spans only one user interface element. Consider the following scenario: a tablet is being used in desktop mode, and a set of icons that can perform functions are presented on the display, perhaps on the side where the user is holding the tablet, and the user's grip may extend across two or three icons. Because contextual data establishes the size of each icon, and simultaneous selection of multiple icons is not a valid action, touch input from the user's grip can be rejected.

[0045] In some cases, the context data provided by the context engine can reflect (e.g., indicate) the input / output components currently being used by the device. Such components may include, for example, a camera, microphone, speaker, touchpad, or external monitor. If a physical keyboard (wireless or wired) is connected to the device, or a microphone is used to receive voice input, the likelihood of receiving purposeful touch input is reduced, and the touch classifier system can be configured to reflect this in the classified touch data. Similarly, the classified touch data can reflect whether the device is receiving simultaneous pen signals, and a touch on the touched area is only reported in the absence of simultaneous pen signals. In some examples, the context data shows which wireless network the device is connected to, as this indicates the environment in which the device is being used, and this is reflected in the classified touch data. For example, being connected to a cellular network may indicate that the device is being used in a vehicle, where accidental touches are more likely. In summary, the above-described feature of recording certain aspects of the hardware state in the context data provides a technical advantage in that it allows this information to be used in a heuristic manner to influence the determination of whether a touch on a given touched area is intentional.

[0046] Additional examples and scenarios are also contemplated. For example, contextual data can reflect the physical orientation of the device. This orientation can be displayed via accelerometer output, for example, and can reflect whether the device is being used in portrait or landscape mode. This information, along with the user's handedness, can be used to inform whether a touched area near a certain border is receiving a valid touch. Thus, in some examples, contextual data reflects the user's handedness.

[0047] For ease of explanation, the examples above assume that the various decisions are made using deterministic and / or fuzzy logic. However, the reader should keep in mind that these decisions can also be made using a trained neural network that processes both graph and contextual data simultaneously, i.e., it receives and fuses both forms of data at the input layer of neurons. A neural network can be trained to make these decisions using training data consisting of a set of appropriately labeled graph and contextual data files.

[0048] Compared to existing approaches, the solutions in this paper reduce false positives for valid touch detection. They also reduce latency and power consumption for valid touch detection by rejecting invalid touch events earlier in the processing pipeline and significantly limiting the area of ​​the touch sensor where significant touch events must be processed in the CPU or GPU. In some examples, processing overhead can also be reduced by updating context information only when the UI layout on the display changes.

[0049] Figure 8 Aspects of an example active pen 22' are shown. Figure 7 , a probe electrode 80 is disposed at the tip 42' of the active pen 22'. The probe electrode is operatively coupled to associated sensing logic 82 and injection logic 56. The sensing logic and injection logic are coupled to, and may be partially embodied in, a microprocessor 86 that is configured to perform digital signal processing (DSP). The microprocessor is operatively coupled to a computer memory 88. The sensing logic 82 includes linear analog components that are configured to maintain the probe electrode 80 at a constant voltage and convert any current entering or leaving the probe electrode into a proportional current sensing voltage. The sensing logic may include an analog-to-digital (A / D) converter 90 that converts the current sensing voltage into digital data for subsequent processing.

[0050] Instead of capacitively coupling the row and column electrodes of a capacitive touch sensor through a dielectric, the sensing logic 82 of the active pen 22' senses the stimulus pulse from the row electrode 24 (located behind the touch point 18B) and responsively injects charge into the column electrode 26 also located behind the touch point. To this end, the injection logic 56 is configured to control the injection of charge from the probe electrode 80 into the column electrode directly beneath it. The injected charge appears to the column sensing circuit 36 ​​as an electrostatic pulse delivered by capacitive coupling between the column electrode and the charged row electrode that intersects at the touch point.

[0051] In some examples, the sensing logic 82 and injection logic 56 are active during non-overlapping time windows of each touch sensing frame, so that charge injection and charge sensing can be performed at the same probe electrode 80. In this embodiment, the touch sensor logic 32 stimulates a series of row electrodes 24 during the time window when the sensor logic is active, but suspends row stimulation during the time window when the pen 22' can inject charge. This strategy provides an additional advantage in that it enables the touch sensor logic 32 to distinguish between touch points caused by the pen 22' and touch points caused by a fingertip or palm. If the column sensing circuit 36 ​​detects charge from the column electrode 26 during the pen 22's charge injection time window (when no row electrodes 24 are stimulated), the detected touch point 18B must be a touch point of the pen. However, if the column sensing logic detects charge during the pen's charge sensing window (when the row electrodes 24 are stimulated), the detected touch point may be a touch point of a finger, hand, passive pen, etc.

[0052] In some examples, row electrodes 24 stop injecting charge during the pen resolution window and are reused as sensing electrodes within that window. In a configuration that supports this functionality, each row driver in row counter 38 has a tri-state output that includes a high-impedance state, and each row electrode 24 is matched to a row sense amplifier (not shown, but similar to the column sense amplifiers in column sense circuitry 36). When the row driver is in the high-impedance state, the row sense amplifier provides a touch signal for calculating the Y coordinate of the pen position, similar to the touch signal from the column sense circuitry used to calculate the X coordinate. In other examples and scenarios, this functionality can be used even during the window in which a finger touch is resolved—that is, undriven row electrodes can be reused as sensing electrodes. Continuing with the description of an active pen, when receiving a signal from a row electrode 24, pen 22' can inject a charge pulse with an amplitude proportional to the strength of the received signal. The capacitive touch sensor 16 receives the electrostatic signal from pen 22' and calculates the Y coordinate, which can be the row providing the maximum signal from the pen, or a function of the signal received at that row and an adjacent row.

[0053] Active sensing and subsequent charge injection enable very small touch points 18B to be accurately located without the need for long integration times, thereby avoiding an increase in touch sensing latency. However, this approach introduces certain challenges associated with noise suppression. Various solutions (e.g., code division multiple access or frequency division multiple access) can be applied to eliminate strong interference from the transmitting direction in the receiving direction. A capacitive touch sensor may need to receive two signals simultaneously (one from the row electrode 24 and the other from the probe electrode 80). Another solution is to require the pen 22' to take a more active role in determining the coordinates of the touch point. In the illustrated example, the sensing logic 82 of the active pen 22' includes a remote row counter 92 that is synchronized with the local row counter 38 of the touch sensor logic 32. This feature enables the pen and touch screen to have shared timing, but without the need for a wired connection.

[0054] When probe electrode 80 touches the front surface, sensing logic 82 receives a waveform that persists as long as the touch is maintained. The waveform reaches its maximum amplitude at the moment row electrode 24 (i.e., the electrode directly below probe electrode 80) is activated. Sensing logic 82 is configured to sample the waveform at each increment of remote row counter 92 and determine when the maximum amplitude is sensed. For example, this determination can be performed once per frame.

[0055] Because the active pen 22' and the touch sensor logic 32 have shared timing due to synchronized row counters, the state of the local row counter 92 at the maximum sensed amplitude directly reports the row coordinate of the touch point 18B, i.e., the Y coordinate. To utilize this information, the Y coordinate must be transmitted back to the touch sensor logic 32. To this end, the pen includes a transceiver 94 configured to wirelessly transmit the calculated row coordinates to the touch screen's row sensing logic. This disclosure encompasses various modes of transmitting data (including the Y coordinate) from the pen to the touch screen.

[0056] In various variations described above, the touch sensor logic 32 is configured to provide a pen signal in response to motion of the pen on the associated touch sensor 16. In these examples, the pen signal comprises the analog output of the column sense circuit 36 ​​referenced to a time defined by the sense logic 82 of the pen 22'.

[0057] As described above, the methods described herein can be associated with a computer system of one or more computing devices. Such methods and processes can be implemented as applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0058] Figure 8A schematic diagram of a computer system 52 is provided that is configured to provide some or all of the computer system functionality disclosed herein. The computer system 52 may take the form of a personal computer, an application server computer, or any other computing device.

[0059] Computer system 52 includes a logic system 54 and a computer memory system 56. Computer system 52 may optionally include a display system 96, an input system 98, a network system 100, and / or other systems not shown.

[0060] The logic system 54 includes one or more physical devices configured to execute instructions. For example, the logic system can be configured to execute instructions that are part of at least one operating system (OS), application, service, and / or other program construct. The logic system can include at least one hardware processor (e.g., a microprocessor, a central processing unit, a central processing unit (CPU), and / or a graphics processing unit (GPU)) configured to execute software instructions. Additionally or alternatively, the logic system can include at least one hardware or firmware device configured to execute hardware or firmware instructions. The processor of the logic system can be single-core or multi-core, and the instructions executed on it can be configured for sequential, parallel, and / or distributed processing. The various components of the logic system can be selectively distributed across two or more independent devices that can be remotely located and / or configured for coordinated processing. Various aspects of the logic system can be virtualized and executed by a remotely accessible, networked computing device configured for cloud computing.

[0061] Computer memory system 56 includes at least one physical device configured to temporarily and / or permanently store computer system information, such as data and instructions executable by logic system 54. When the computer memory system includes two or more devices, the devices may be co-located or remotely located. Computer memory system 56 may include at least one volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location read addressable, file read addressable, and / or content read addressable computer storage device. Computer memory system 56 may include at least one removable and / or internal computer memory device. As the logic system executes instructions, the state of computer memory system 56 may be transformed—for example, to store different data.

[0062] Various aspects of the logic system 54 and the computer memory system 56 may be integrated into one or more hardware logic components. Any such hardware logic component may include at least one program or application specific integrated circuit (PASIC / ASIC), program or application specific standard product (PSSP / ASSP), system on chip (SOC), or complex programmable logic device (CPLD), among others.

[0063] The logic system 54 and the computer memory system 56 can work together to instantiate one or more logical machines or engines. In this specification, the terms "machine" and "engine" collectively refer to a combination of hardware, firmware, software, instructions, and / or any other components that work together to provide computer system functionality. In other words, machines and engines are by no means abstract concepts, but always have concrete forms. A machine or engine can be instantiated by a single computing device, or a machine or engine can include two or more subcomponents instantiated by two or more different computing devices. In some embodiments, a machine or engine includes a local component (e.g., a software application executed by a computer system processor) that cooperates with a remote component (e.g., a cloud computing service provided by a network of one or more server computer systems). The software and / or other instructions that give a particular machine or engine its functionality can optionally be stored on one or more computer memory devices as one or more unexecuted modules.

[0064] The machines and engines (as described above) can be implemented using any suitable combination of machine learning (ML) and artificial intelligence (AI) techniques. Non-limiting examples of techniques that can be incorporated into embodiments of one or more machines include support vector machines, multi-layer neural networks, convolutional neural networks (e.g., spatial convolutional networks for processing images and / or videos, and / or any other suitable convolutional neural networks that are configured to convolve and pool features in one or more temporal and / or spatial dimensions), recurrent neural networks (e.g., long short-term memory networks), associative memories (e.g., lookup tables, hash tables, Bloom filters, neural Turing machines, and / or neural random access memories), unsupervised spatial and / or clustering methods (e.g., nearest neighbor algorithms, topological data analysis, and / or k-means clustering), and / or graphical models (e.g., (hidden) Markov models, Markov random fields, (hidden) conditional random fields, and / or AI knowledge bases)).

[0065] When a display system 96 is included, the display system can be used to present a visual representation of the data stored in the computer memory system 56. In some examples, the visual representation can take the form of a graphical user interface (GUI). The display system can include one or more display devices utilizing virtually any type of technology. In some embodiments, the display system can include one or more virtual, augmented, or mixed reality displays.

[0066] When an input system 98 is included, the input system 98 may include one or more input devices or be connected to one or more input devices. The input device may include a sensor device or a user input device. Examples of user input devices include a keyboard, a mouse, or a touch screen.

[0067] When network system 100 is included, the network system can be configured to communicatively couple computer system 52 with one or more other computer systems. The network system can include wired and / or wireless communication devices compatible with one or more different communication protocols. The network system can be configured to communicate over a personal network, a local area network, and / or a wide area network.

[0068] In summary, one aspect of the present disclosure relates to a device comprising an electronic display, a touch sensor disposed on the electronic display (comprising a plurality of intersections of row and column electrodes), and operatively coupled touch sensor logic, a context engine, and a touch classifier system. The touch sensor logic is configured to assemble a map of touch signals from the touch sensor, the map comprising a corresponding touch signal value for each of a plurality of intersections and defining at least one touched area of ​​the touch sensor. The context engine is configured to collect and provide context data related to user interface content currently presented on the electronic display. The touch classifier system is configured to calculate classified touch data corresponding to the map of touch signals based at least in part on the map and the context data, and to provide the classified touch data to an operating system of the device.

[0069] In some embodiments, the operating system includes a context engine. In some embodiments, the context data reflects the input and output components being used on the device. In some embodiments, the device also includes an accelerometer, and the context data reflects the orientation of the device. In some embodiments, the context data reflects which wireless network the device is connected to. In some embodiments, the classified touch data is calculated at least in part based on the size of at least one touched area shown in the figure. In some embodiments, the classified touch data is calculated at least in part based on the position of at least one touched area relative to the display. In some embodiments, the context data reflects one or both of the position of the pen and the orientation of the pen. In some embodiments, the context data reflects the user's handedness.

[0070] Another aspect of the present disclosure is directed to a method for providing classified touch data to a computer program executed on a device having an integrated electronic display. The method includes: (a) assembling a map of touch signals from a touch sensor disposed on the electronic display, the touch sensor comprising a plurality of intersections of row and column electrodes, the map comprising a corresponding touch signal value corresponding to each of the plurality of intersections and defining at least one touched area of ​​the touch sensor; (b) providing contextual data related to user interface content currently presented on the electronic display; (c) calculating classified touch data corresponding to the map of touch signals based at least in part on the map and the contextual data; and (d) providing the classified touch data to an operating system of the device.

[0071] In some embodiments, calculating the classified touch data includes reporting a touch on the at least one touched area if the user interface element is within a threshold distance of the at least one touched area. In some embodiments, calculating the classified touch data includes reporting a touch on the at least one touched area if the at least one touched area spans only one user interface element. In some embodiments, calculating the classified touch data includes reporting a touch on the at least one touched area if the at least one touched area disappears within a predetermined time interval. In some embodiments, calculating the classified touch data includes reporting a touch on the at least one touched area if the at least one touched area is within a predetermined user interface element that receives input, and not reporting a touch if the at least one touched area is outside of the predetermined user interface element that receives input. In some embodiments, calculating the classified touch data includes reporting a touch on the at least one touched area if no pen signal is received simultaneously on the device. In some embodiments, calculating the classified touch data includes reporting a touch on the at least one touched area if the at least one touched area is within a predetermined size range of a co-located user interface element. In some embodiments, calculating the classified touch data includes reporting a touch on the at least one touched area if the at least one touched area is smaller than a co-located user interface element or a spacing between adjacent, co-located user interface elements.

[0072] One aspect of the present disclosure relates to a handheld device comprising an electronic display, a touch sensor disposed on the electronic display (comprising a plurality of intersections of row and column electrodes), and operatively coupled touch sensor logic, a context engine, and a touch classifier system. The touch sensor logic is configured to assemble a map of touch signals from the touch sensor, the map comprising a corresponding touch signal value for each of a plurality of intersections and defining at least one touched area of ​​the touch sensor. The context engine is configured to collect and provide context data related to user interface content currently presented on the electronic display. The touch classifier system is configured to calculate classified touch data corresponding to the map of touch signals based at least in part on the map and the context data, and to provide the classified touch data to an operating system of the handheld device.

[0073] In some embodiments, the touch classifier system includes a firmware engine that executes on a dedicated touch processor chip of the touch sensor logic. In some embodiments, the touch classifier system includes a software engine that executes on a processor of a handheld device.

[0074] The present disclosure is illustrated using the accompanying drawings as examples. Components, process steps, and other elements that are substantially identical in one or more of the accompanying drawings are identified in a coordinated manner, and repetitive descriptions are minimized. However, it should be noted that elements identified in a coordinated manner may also differ to some extent. Furthermore, it should be noted that the accompanying drawings are schematic diagrams and are generally not drawn to scale. Rather, the various drawing scales, aspect ratios, and numbers of components shown in the drawings may be intentionally distorted to more clearly illustrate certain features or relationships.

[0075] It should be understood that the configurations and / or methods described herein are for illustrative purposes only, and these specific embodiments or examples should not be considered as limitations on the present invention, as there are many possible variations. The specific routines or methods described herein may represent one or more processing strategies. Thus, the various operations shown and / or described may be performed in the order shown and / or described, in other orders, in parallel, or in an omitted manner. Similarly, the order of the above-described processes may also be changed. In this spirit, the phrase "at least in part based on" is intended to remind the reader that the functions and / or conditional logic shown herein neither require nor exclude appropriate additional logic executed in conjunction with the shown logic to provide additional benefits. Values ​​based at least in part on A and B are based on A, also based on B, and may or may not be based on factors other than A and B.

[0076] The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents thereof.

Claims

1. A device (10), comprising: electronic display (12); a touch sensor (16) disposed on the electronic display and comprising a plurality of intersections of row electrodes (24) and column electrodes (26); touch sensor logic (32) operatively coupled to the touch sensor and configured to assemble (78B) a map (46) of touch signals from the touch sensor, the map including a corresponding touch signal value for each of the plurality of intersections and defining at least one touched area (48) of the touch sensor; a context engine (64) configured to collect and provide (78C) context data related to user interface content currently presented on the electronic display; as well as a touch classifier system (66) configured to compute (78D) classified touch data corresponding to the map of touch signals based at least in part on the map and the contextual data, and The classified touch data is provided (78E) to an operating system (60) of the device.

2. The device of claim 1, wherein the operating system comprises the context engine. 3 . The device of claim 1 , wherein the context data reflects which input-output component is being used on the device. The device of claim 1 , further comprising an accelerometer, wherein the contextual data reflects an orientation of the device. The device of claim 1 , wherein the context data reflects which wireless network the device is connected to. 6 . The device of claim 1 , wherein the classified touch data is calculated based at least in part on a size of the at least one touched area displayed in the map.

7. The device of claim 1, wherein the classified touch data is calculated based at least in part on a position of the at least one touched area relative to the display.

8. The apparatus of claim 1, wherein the contextual data reflects one or both of a pen position and a pen orientation.

9. The device of claim 1, wherein the contextual data reflects a user's handedness.

10. A method (76) of providing classified touch data to a computer program executed on a device (10) having an integrated electronic display (12), the method comprising: assembling (78B) a map (46) of touch signals from a touch sensor (16) disposed on the electronic display and including a plurality of intersections of row electrodes (24) and column electrodes (26), the map including a corresponding touch signal value for each of the plurality of intersections and defining at least one touched area (48) of the touch sensor; providing (78C) contextual data related to user interface content currently presented on the electronic display; calculating (78D) classified touch data corresponding to the map of touch signals based at least in part on the map and the contextual data; as well as The classified touch data is provided (78E) to an operating system (60) of the device.

11. The method of claim 10, wherein calculating the classified touch data comprises: A touch on the at least one touched area is reported if the user interface element is within a threshold distance of the at least one touched area.

12. The method of claim 10, wherein calculating the classified touch data comprises: If the at least one touched area spans only one user interface element, a touch on the at least one touched area is reported.

13. The method of claim 10, wherein calculating the classified touch data comprises: If the at least one touched area disappears within a predetermined time interval, a touch on the at least one touched area is reported.

14. The method of claim 10, wherein calculating the classified touch data comprises: reporting a touch on the at least one touched area if the at least one touched area is within a predetermined input-receiving user interface element; And if the at least one touched area is outside the predetermined input receiving user interface element, the touch is not reported.

15. The method of claim 10, wherein calculating the classified touch data comprises: If no pen signal is simultaneously received on the device, a touch on the at least one touched area is reported.

16. The method of claim 10, wherein calculating the classified touch data comprises: A touch on the at least one touched area is reported if the at least one touched area is within a predetermined size for co-located user interface elements.

17. The method of claim 10, wherein calculating the classified touch data comprises: If the at least one touched area is smaller than the co-located user interface element, or smaller than the spacing between adjacent co-located user interface elements, a touch on the at least one touched area is reported.

18. A handheld device (10) comprising: electronic display (12); a touch sensor (16) disposed on the electronic display and comprising a plurality of intersections of row electrodes (24) and column electrodes (26); touch sensor logic (32) operatively coupled to the touch sensor and configured to assemble (78B) a map (46) of touch signals from the touch sensor, the map including a corresponding touch signal value for each of the plurality of intersections and defining at least one touched area (48) of the touch sensor; a context engine (64) configured to collect and provide (78C) context data related to user interface content currently presented on the electronic display; as well as A touch classifier system (66) is configured to calculate (78D) classified touch data corresponding to the map of the touch signals based at least in part on the map and the contextual data, and to provide (78E) the classified touch data to an operating system (60) of the handheld device.

19. The handheld device of claim 18, wherein the touch classifier system comprises a firmware engine executing on a dedicated touch processor chip of the touch sensor logic.

20. The handheld device of claim 18, wherein the touch classifier system comprises a software engine executing on a processor of the handheld device.

Images