SYSTEMS AND / OR METHODS FOR CREATING AND DETECTING CHANGES ASSOCIATED WITH ELECTRIC FIELDS - Patent application

Abstract

The detection system has an interface including a substrate supporting a conductive coating. The substrate is provided with electrodes. A multiplexer supplies current to the electrodes. A demultiplexer receives voltages from the electrodes and provides corresponding signals to a controller. The controller receives these signals and determines from them an action to be performed in relation to the interface by applying an algorithmic approach. Static interactions can be recognized and machine learning can be used for gesture recognition and / or identification of other interaction types. The technology can be used in a wide range of applications where it is desirable to sense interactions with a defined area, such as in the case of touch, gestures, hovering, and / or the like.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. patent application Ser. No. 63 / 156,033, filed March 3, 2021, Ser. No. 63 / 272,101, filed October 26, 2021, and Ser. No. 63 / 291,076, filed December 17, 2021, the entire contents of each of which are incorporated by reference herein.

[0002] FIELD OF THEINVENTION Certain exemplary embodiments of the present invention relate to systems and / or methods involving electric field sensing technology, and more particularly, to systems and / or methods utilizing electrical impedance tomography (EIT), electrical capacitive (ECT) tomography, electrical field tomography (EFT), and / or other electric field sensing technologies for use in connection with a variety of applications including, for example, electronic devices, automated guided vehicles (AGVs), robots, furniture, guarded spaces, architectural / sensing walls and windows, vending machines, and / or other machines / apparatus / articles. 2. Background and Summary of the Invention

[0003] For thousands of years, much of human experience has been mediated through the five senses: seeing, hearing, touch or feeling, tasting, and smelling. Indeed, while early humans had to rely on their ability to see and hear to help keep themselves safe from predators, modern humans no doubt rely on their ability to smell to help them take action to improve when food is burning in the oven.

[0004] From a technology perspective, there is an ongoing effort to build immersive environments that engage multiple senses. Some of the more obvious desires include building virtual reality (VR) and augmented reality (AR) environments. However, technologies that are considered more general also seek to engage the senses in interesting and integrated ways. Consider, for example, a smartphone, which includes a touchscreen display that itself engages both vision and touch. Touch interactions may result in haptic (additional touch-related) feedback, as well as sound effects (accompanying auditory cues).

[0005] Touch-sensitive displays are used in a variety of different devices and for a variety of different uses. Touch-sensitive displays are used as the primary (or sometimes only) means of interacting with, for example, mobile phones and tablet computing devices. Automated teller machines (ATMs), airport and railroad check-in kiosks, laptops, video game systems, vehicle and instrument control panels, and other devices also often include touch-sensitive displays. Some smart televisions are equipped with touch-sensing capabilities.

[0006] Much research has been done into touch sensing functionality, which is perhaps not surprising given the breadth of devices in which such functionality has been deployed. Nevertheless, those skilled in the art continue to search for new ways to create larger, more accurate, and more affordable touch sensing systems that operate faster. Size and speed are both issues / challenges. Often, there is a tradeoff between the speed at which touch sensing related functions can be detected and determined and the accuracy associated with the detection. Scaling up to larger touch sensing surfaces can exacerbate these issues, as more hardware may need to be included to obtain a desired level of accuracy, which may require longer times to perform touch detection related operations. For example, there is a simultaneous desire to enable advanced touch-related features such as multi-touch input, gesture-related operations, and action-based interactivity including hovering, which may require signal processing systems including more complex hardware and / or software.

[0007] While engineers continue to build devices and systems that create immersive experiences for interacting with humans by engaging sensory receptors in interesting ways, there is a simultaneous need to build human-like senses into the devices and systems. Computer vision provides an analogue of human vision and is used in cameras, optical character recognition (OCR), medical screening, and other technologies. Speech recognition, text-to-speech, and other technologies provide an analogue of human hearing. Mechanical, piezoelectric, and / or other pressure-sensing technologies in machines provide an analogue of the human sense of touch.

[0008] The technical field of sensor fusion takes outputs from these and / or other technologies and uses them to accomplish tasks. For example, a cleaning robot may "see" dirt being sucked up, "feel" it encounter a table leg, etc. As another example, a video game system may have some vestibular sense of its orientation in real three-dimensional space, "hear" voice commands and provide relevant or irrelevant auditory feedback, "see" where the player is looking and provide relevant or irrelevant visual cues, provide haptic feedback in the form of haptics based on external inputs and / or in-game events, etc.

[0009] Certain exemplary embodiments relate to these and / or other technical fields. More specifically, the inventors have recognized that electrical tomography-related technologies can be leveraged to provide multiple different types of sensing technologies, potentially in the same application or device. For example, certain exemplary embodiments relate to touch sensing systems and / or methods that leverage electrical tomography-related technologies, including electrical impedance tomography (EIT), electrical capacitance tomography (ECT), and electric field tomography (EFT) technologies, to provide large interfaces that provide improved accuracy and shorter processing times, e.g., in connection with touch sensing related functions. In touch sensing systems, for example, electrical tomography-related technologies can be leveraged to detect touch and hover-related actions that can augment or replace computer-mediated senses of vision, touch, and / or the like. More specifically, certain exemplary embodiments can "see" when a gesture is made, "predict" when a touch will occur, "feel" where and / or when a touch will occur, etc.

[0010] More broadly, certain illustrative embodiments have the ability to "see," "predict," and "sense" potential human or other interactions and understand them in a manner relevant to the particular application to which these illustrative embodiments are applied. Such applications may include, for example, industrial safety system related applications, office system related applications, public or other information display systems, and / or the like.

[0011] Certain exemplary embodiments are advantageously operable to sense the presence of humans and / or other objects that affect the electric field. These other objects may include, for example, contaminants such as metallic elements or liquids that carry an electric charge, pests, vehicles or other machinery (including potential runaway equipment), boxes, metal particles, solid waste, etc. Certain exemplary embodiments are advantageously operable to sense the identity of stored energy, such as stored electricity, pneumatic pressure, or hydraulic pressure.

[0012] Certain illustrative embodiments advantageously incorporate hardware architectures and / or software algorithms that provide high speed data processing and decision-making capabilities, high accuracy, and the like.

[0013] The features, aspects, advantages, and example embodiments described herein may be combined to realize further embodiments.

[0014] These and other features and advantages may be better and more completely understood by reference to the following detailed description of the illustrative embodiments taken in conjunction with the drawings. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a schematic diagram illustrating how data can be obtained for different applications, including, for example, touch sensing related applications.

[0016] [Figure 2A] FIG. 13 illustrates how a four-electrode adjacent measurement approach can work in relation to a simplified eight-electrode system. [Figure 2B] FIG. 13 illustrates how a four-electrode adjacent measurement approach can work in relation to a simplified eight-electrode system.

[0017] [Diagram 3] FIG. 1 is a schematic diagram of control electronics that can be used to process collected data for different applications, including, for example, touch sensing related applications.

[0018] [Figure 4] FIG. 1 is a schematic diagram of a substrate that is conceptually segmented into different regions, which may be useful for performing accurate interactive processing in relation to large substrates.

[0019] [Diagram 5] FIG. 1 is a schematic diagram illustrating how a cluster of data points can be machine-learned to correspond to an interactive touch or other interaction location on an interface.

[0020] [Figure 6] FIG. 13 is a schematic diagram of how a series of collected points can be collected into a frame and subsequently mapped to a right swipe gesture.

[0021] [Figure 7] FIG. 1 is a schematic diagram of control electronics that may be used for electrical tomography mediated operations, in accordance with certain exemplary embodiments.

[0022] [Figure 8] 1 is a flowchart illustrating a first algorithmic approach (a "four-loop" algorithm) for identifying interactions (such as touch-related interactions) in association with an interface, in accordance with certain exemplary embodiments.

[0023] [Figure 9] FIG. 9 illustrates how the flowchart of FIG. 8 may operate in a first phase, in accordance with certain exemplary embodiments.

[0024] [Figure 10] 9 illustrates how the flowchart of FIG. 8 may operate in a second phase, in accordance with certain exemplary embodiments.

[0025] [Figure 11] 11 is a flowchart illustrating a second algorithmic approach (a "divider loop" algorithm) for identifying interactions (such as touch-related interactions) in relation to an interface, in accordance with certain exemplary embodiments.

[0026] [Figure 12] 12 is a flowchart illustrating how interactions identified using the technique of FIG. 11 can be more precisely identified and / or characterized, in accordance with certain illustrative embodiments.

[0027] [Figure 13] FIG. 12 illustrates how the flowchart of FIG. 11 may operate in a first phase, in accordance with certain exemplary embodiments.

[0028] [Figure 14A] 12 is a flowchart illustrating further details regarding a one-way processing loop applied using the example approach of FIG. 11, in accordance with a particular example embodiment.

[0029] [Figure 14B] 12 is a flowchart illustrating further details regarding a bidirectional processing loop applied using the example approach of FIG. 11, in accordance with certain exemplary embodiments.

[0030] [Figure 15] FIG. 14 is similar to FIG. 13, except that additional electrodes are applied to improve accuracy, in accordance with certain exemplary embodiments.

[0031] [Figure 16] 11 is a flowchart illustrating a third algorithmic approach (a "discrete loop" algorithm) for identifying interactions (such as touch-related interactions) in association with an interface, in accordance with certain exemplary embodiments.

[0032] [Figure 17] FIG. 17 illustrates how the flowchart of FIG. 16 may operate in a first phase, in accordance with certain exemplary embodiments.

[0033] [Figure 18] FIG. 2 illustrates generally how points may be collected and used for gesture recognition in accordance with certain exemplary embodiments.

[0034] [Figure 19A] 11A-11C are diagrams that help explain how hover-related interactions may be managed, in accordance with certain illustrative embodiments. [Figure 19B] 11A-11C are diagrams that help explain how hover-related interactions may be managed, in accordance with certain illustrative embodiments. [Figure 19C] 11A-11C are diagrams that help explain how hover-related interactions may be managed, in accordance with certain illustrative embodiments.

[0035] [Figure 20A] FIG. 13 provides information on how voltage-related measurements can be used in connection with the discrete loop algorithmic approach described herein, in accordance with certain exemplary embodiments. [Figure 20B] FIG. 13 provides information on how voltage-related measurements can be used in connection with the discrete loop algorithmic approach described herein, in accordance with certain exemplary embodiments.

[0036] [Figure 21]FIG. 1 illustrates voltage-related signals collected in association with a simple touch interaction using hardware of a specific exemplary embodiment.

[0037] [Figure 22] FIG. 13 illustrates voltage-related signals collected in association with a swipe interaction using hardware of a particular exemplary embodiment.

[0038] [Figure 23A] FIG. 1 illustrates a machine having a modular detection system installed thereon based on the technology disclosed herein, in accordance with certain exemplary embodiments.

[0039] [Figure 23B] FIG. 1 illustrates the disclosed techniques being used to create a protected area for a machine in a custom manner suited to the particular machine, in accordance with certain illustrative embodiments.

[0040] [Figure 23C] FIG. 1 illustrates the techniques disclosed herein being used for active enabling of defined areas, in accordance with certain example embodiments.

[0041] [Figure 24A] FIG. 1 is an illustration of a "skin" applied to a robot, in accordance with certain illustrative embodiments.

[0042] [Figure 24B] 1 illustrates the techniques disclosed herein being used on a mobile cart in a manner that helps close vision gaps, in accordance with certain exemplary embodiments.

[0043] [Figure 24C] FIG. 1 is a diagram of an exemplary skinned cobot that may implement the techniques disclosed herein for the above and / or other purposes, in accordance with certain exemplary embodiments.

[0044] [Figure 25A] FIG. 1 illustrates the techniques disclosed herein implemented on a mobile loader, in accordance with certain exemplary embodiments.

[0045] [Figure 25B] FIG. 11 is an illustration of defined shipping areas provided in the floor and sensors defined on the machine in accordance with certain illustrative embodiments;

[0046] [Figure 25C] FIG. 13 illustrates a conductive area provided outside a shipping area that can be used to guide a suitably equipped automated loading machine, in accordance with certain illustrative embodiments.

[0047] [Figure 26A] 1A-1C illustrate smart fencing solutions that can be used in conjunction with or in place of existing fences in different illustrative embodiments;

[0048] [Figure 26B] FIG. 1 illustrates how the techniques disclosed herein can be used to provide real-time compliance-related updates, according to certain example embodiments.

[0049] [Figure 26C] FIG. 1 illustrates a defined walking path that may benefit from the techniques disclosed herein, according to certain example configurations.

[0050] [Figure 27] 1A-1C are diagrams illustrating how the techniques disclosed herein can be used to provide a holistic view of an entire operating area, in accordance with certain exemplary embodiments.

[0051] [Figure 28] FIG. 10 is a diagram that illustrates a schematic demonstration of a fourth approach for determining interaction details, in accordance with certain illustrative embodiments; [Figure 29] FIG. 10 is a diagram that illustrates a schematic demonstration of a fourth approach for determining interaction details, in accordance with certain illustrative embodiments; [Diagram 30] FIG. 10 is a diagram that illustrates a schematic demonstration of a fourth approach for determining interaction details, in accordance with certain illustrative embodiments; [Diagram 31] FIG. 10 is a diagram that illustrates a schematic demonstration of a fourth approach for determining interaction details, in accordance with certain illustrative embodiments; [Diagram 32] FIG. 10 is a diagram that illustrates a schematic demonstration of a fourth approach for determining interaction details, in accordance with certain illustrative embodiments; [Diagram 33] FIG. 10 is a diagram that illustrates a schematic demonstration of a fourth approach for determining interaction details, in accordance with certain illustrative embodiments;

[0052] [Diagram 34] 10A-10C are schematic illustrations of different exemplary horizontal balance paths that may result in different exemplary embodiments;

[0053] [Diagram 35] 34 is a flowchart summarizing the operation of the exemplary approach described in relation to FIGS. 28-33.

[0054] [Diagram 36] FIG. 13 is an example diagram illustrating how slave lines can be used to improve accuracy, in accordance with certain illustrative embodiments.

[0055] [Figure 37] FIG. 1 illustrates variations of an example matrix approach that may be used to improve accuracy, in accordance with certain example embodiments.

[0056] [Figure 38] FIG. 13 illustrates how the center of gravity is biased towards higher electrode density in certain exemplary embodiments.

[0057] [Figure 39] FIG. 1 illustrates an example three-electrode system with an example center of gravity and an example balance path, which may be similar to certain example embodiments.

[0058] [Diagram 40] FIG. 13 illustrates an exemplary arrangement in which electrodes are provided only on opposing edges of a conductive region, in accordance with certain exemplary embodiments.

[0059] [Figure 41A] FIG. 1 is a system block diagram for a sensing system in accordance with certain illustrative embodiments.

[0060] [Figure 41B] FIG. 1 illustrates an exemplary platform architecture stack, in accordance with certain illustrative embodiments.

[0061] [Diagram 42] 11 is a diagram illustrating the contents of an interaction template in accordance with certain illustrative embodiments;

[0062] [Figure 43A] 1A-1C are diagrams illustrating different object tracking related systems incorporating a sensing system in accordance with certain illustrative embodiments. [Figure 43B] 1A-1C are diagrams illustrating different object tracking related systems incorporating a sensing system in accordance with certain illustrative embodiments.

[0063] [Figure 44A] FIG. 1 is an example diagram illustrating how the techniques disclosed herein can be used in connection with consumer electronics devices, in accordance with certain illustrative embodiments. [Figure 44B] FIG. 1 is an example diagram illustrating how the techniques disclosed herein can be used in connection with consumer electronics devices, in accordance with certain illustrative embodiments. [Figure 44C] FIG. 1 is an example diagram illustrating how the techniques disclosed herein can be used in connection with consumer electronics devices, in accordance with certain illustrative embodiments. [Fig.44D] FIG. 1 is an example diagram illustrating how the techniques disclosed herein can be used in connection with consumer electronics devices, in accordance with certain illustrative embodiments.

[0064] [Figure 45A] FIG. 1 is an example diagram illustrating how the techniques disclosed herein may be used in connection with a workplace-related system, in accordance with certain illustrative embodiments. [Figure 45B] FIG. 1 is an example diagram illustrating how the techniques disclosed herein may be used in connection with a workplace-related system, in accordance with certain illustrative embodiments. [Figure 45C] FIG. 1 is an example diagram illustrating how the techniques disclosed herein may be used in connection with a workplace-related system, in accordance with certain illustrative embodiments. [Figure 45D] FIG. 1 is an example diagram illustrating how the techniques disclosed herein may be used in connection with a workplace-related system, in accordance with certain illustrative embodiments. [Figure 45E] FIG. 1 is an example diagram illustrating how the techniques disclosed herein may be used in connection with a workplace-related system, in accordance with certain illustrative embodiments. [Fig.45F] FIG. 1 is an example diagram illustrating how the techniques disclosed herein may be used in connection with a workplace-related system, in accordance with certain illustrative embodiments.

[0065] [Figure 46A] FIG. 1 is a diagram of an example of how the sensing techniques disclosed herein can be used in different vehicle-related systems, in accordance with certain illustrative embodiments. [Figure 46B] FIG. 1 is a diagram of an example of how the sensing techniques disclosed herein can be used in different vehicle-related systems, in accordance with certain illustrative embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0066] Certain exemplary embodiments of the present invention relate to systems and / or methods that utilize electrical tomography-related technologies, including electrical impedance tomography (EIT), electric field tomography (EFT), and electrical capacitance tomography (ECT)-related technologies. The electrical tomography-related techniques of certain exemplary embodiments enable advanced functionality, such as human sensing, gesture detection, multi-touch interactivity, hovering-related functionality, and the like. This advanced technology can be advantageously used to help computer-based systems "see," "predict," and "feel" aspects of their environment similar to how humans would use their sensory receptors.

[0067] In general, EIT is a technique that allows visualization of the spatial distribution of electrical impedance (or conductivity) inside an object, such as the human body. EIT-enabled devices typically use voltage measurements on the surface of the object as current passes through the volume as initial data for "image reconstruction" (e.g., of the electric field). Both fast and effective reconstruction algorithms and highly accurate electrical measurements are useful for EIT-enabled systems. Since measurements in EIT-enabled systems can be made fairly quickly, the high processing speed of the measured information allows many processes to be visualized in practically real time.

[0068] The EFT approach examines the interaction of high-frequency electric fields with inhomogeneous conductive media without contacting electrodes. Unlike EIT-enabled approaches, EFT-enabled approaches do not require the injection of currents into the medium from the outside. The interaction involves a high-frequency redistribution of free charges in the medium, which results in small, regular phase shifts of the electric field in the area surrounding the object. This type of phenomenon is called Maxwell-Wagner relaxation. Measuring the perturbation of the electric field using a set of electrodes (which may be placed around the object, for example) allows the reconstruction of the internal structure of the medium. The electric field can be used to image dielectric (non-conductive) objects, similar to ECT-enabled approaches.

[0069] In the electrical tomography approach described herein, generally, data is acquired, processed, and classified. Then, a decision can be made regarding the interaction (e.g., touch-sensing related interaction in the case of a touch detection system). In the acquisition phase, data is collected using electrodes. Signals from the electrodes are digitized using an analog-digital converter (ADC). The acquired data is filtered using a multiplexer to enable detection (e.g., individual and multiple touch point detection in a touch detection system). The filtered data is stored, clustered, and calibrated. This may be done first to generate machine learning (ML) models related to the touch-sensing related interactions and later to enable classification based on those models. For example, ML models may be generated to determine what gestures were made, when and where a hovering motion was made, when and where a touch motion was made, etc., which may be useful in a variety of different applications including touch sensing applications, industrial safety applications, etc. A decision regarding the type, time, and location of the interaction is made based on the generated ML models. As will become clearer from the following description, certain illustrative embodiments may use the same or similar techniques to implement Z direction sensing, e.g., for features related to hovering and / or human detection.

[0070] FIG. 1 is a schematic diagram illustrating how data can be acquired for different applications, including, for example, touch-sensing and hovering-related applications. As shown in FIG. 1, a substrate 102 supports a conductive coating 104. The substrate 102 can be formed of glass, plastic, or some other material. Depending on the application, the substrate 102 may be formed of a transparent material. The conductive coating 104 may be a single layer or a multi-layer coating having a sheet resistance suitable for touch-related and hovering-related applications. The conductive coating 104 may include or consist of, for example, a single layer including ITO, a layer stack including ITO with a protective layer (e.g., a silicon-containing layer such as silicon oxide, silicon nitride, silicon oxynitride, etc.), and / or the like. A silver-based layer stack may also be used in connection with the conductive coating 104. The conductive coating 104 is blanket coated over the surface of the substrate 102. In certain exemplary embodiments, patterning of the conductive coating 104 is not required, for example, due to the processing approaches described herein. However, even if patterning into discrete electrodes or the like is not performed, edge removal or the like may be performed.

[0071] Composite materials may be used for the conductive coating 104 formed on the substrate 102. For example, co-sputtering may be used to deposit composite coatings with different sheet resistances and capacitances appropriate for the electrical tomography technique used and the desired response performance. In this regard, co-sputtering of titanium, silicon, aluminum, and / or other materials may be performed to form nitride and / or oxide layers. Composite layers including titanium nitride and silicon nitride may be used in certain exemplary embodiments. Each of these materials contributes to the layer resistance and reactance, respectively. For example, different compositions of TiN:SiyN (e.g., x,y percentages) and layer thicknesses affect the sheet resistance and reactance. Alloy targets with N2 reactive sputtering may be used in certain exemplary embodiments to achieve sheet resistances of, for example, 500 to 30,000 ohms / square, which may be desirable in certain EIT-related embodiments. Testing results by the assignee indicate that a sheet resistance of 4,000 to 9,000 ohms / square is an optimized range that helps achieve a high signal-to-noise ratio in association with certain exemplary embodiments of the technology disclosed herein. In different exemplary embodiments, conductive paints, conductive films, and / or other materials may be used.

[0072] In addition to being compatible with regular shapes such as squares, rectangles, etc., the techniques disclosed herein are compatible with a wide variety of substrate geometries. That is, the generation of an electric field on and / or within a substrate is independent of the substrate geometry and can be characterized and measured as it is formed based on electrodes (e.g., electrodes provided on and / or within the substrate) applied to the substrate. As a result, the substrate itself can be any shape required or desired for the end application, including, for example, regular or irregular shapes that can be linear or non-linear. The substrate can be flat or curved. In different exemplary embodiments, one or more portions of the substrate can be flat and one or more other portions of the substrate can be curved. The electrodes generate an electric field that can be characterized and measured in whole or in part, for example, using techniques described in more detail below.

[0073] Similarly, the substrate may include recesses, through-holes, protrusions, or other features. This may be desirable, for example, in tabletop applications where it is desirable to have holes to accommodate table legs, card tables where it is desirable to have a player tracking module accessible through the glass or other substrate, but with openings to accommodate chips and card shufflers, museum map displays where there may be an interactive geography including mountains and valleys that can be explored, as well as numerous other applications. Handles and / or the like may also be provided for some applications and may be considered forms of protrusions from the surface. Again, the ability to accommodate such features is possible, since based on the electrode placement, regardless of the "topography" of the substrate (which may include the presence / absence of some or all of the through-holes, recesses, protrusions, and / or other top or bottom surface features), the electrodes generate an electric field that is characterized and measured in whole or in part, for example, using techniques described in more detail below.

[0074] As alluded to above, depending on the substrate and / or features provided thereon or therein, the substrate may be two-dimensional or three-dimensional in nature, and the techniques disclosed herein may work with either arrangement.

[0075] The ability to use any geometry and any topography is advantageous because it creates flexibility for end applications. Other technologies, including other touch technologies, are not compatible in the same blanket manner. For example, in projected capacitive (p-cap) touch technology, the complexity of the conductive wire / mesh for non-linear geometries is typically very high. Assume, for example, that there is a grid or conductive mesh of wires required to connect each line to a controller. For complex geometries, it would be very difficult, if not impossible, to do, especially when space is at a premium. The same problem arises with holes in the structure. There will be portions of the wire or mesh that will be cut off because of the hole, thereby disconnecting the wire or mesh from the controller. Attempting to bypass the hole can be very complicated. The complexity increases for more complex geometries and topographies. In surface capacitive (s-cap) technology, the calculations to determine the touch location are based on resistivity measurements taken from each corner of a linear shape. For complex shapes, the equations become very complicated and are specific to each shape. This makes the s-cap technique difficult to use, especially in relation to a variety of different geometries and topographies. The same problem arises for holes, making the calculations especially complicated in that the electric field within the material is complex and discontinuous.

[0076] In the example of FIG. 1, the electrodes 106 are disposed around the edge of the substrate 102. These electrodes 106 are equally spaced in the example of FIG. 1, and a total of 64 electrodes are shown for illustrative purposes. It will be understood that the number of electrodes 106 may be varied based on the substrate size, substrate geometry, and / or topography, among other factors. It will be understood that in different exemplary embodiments, much fewer electrodes may be provided. For example, in some sensing applications, a single pair of electrodes may be used. For data acquisition, in one form, two adjacent electrodes are excited with a signal, as described in more detail below. The voltage is measured between every other adjacent electrode pair. With this approach, a total of 3,904 measurements are made. Further details of this approach are provided below. Meanwhile, it will be understood that the number of measurements made is equal to n×(n−3), where n is equal to the number of electrodes 106 around the substrate 102. In the example of FIG. 1, the lines between the electrodes indicate the connection of only one electrode pair.

[0077] The approach to current injection and voltage measurement summarized in the immediately preceding paragraph is known as the 4-pole adjacent measurement approach. Figures 2A and 2B show how the 4-pole adjacent measurement approach can work in the context of a simplified 8-electrode system. As shown in Figure 2A, a small AC current is injected into a pair of electrodes (including electrodes 1 and 2 in this case). This current injection creates an electric field. The voltage difference induced on the surface is measured for each pair of electrodes adjacent to one another. This includes the voltage differences between electrodes 3-4, 4-5, 5-6, 6-7, 7-8, etc. The voltage measurements are stored in an array, matrix, or other structure. In the example of Figure 2A, array V has memory locations v1, v2, etc. for each different voltage measurement.

[0078] After the loop of FIG. 2A is completed, the loop of FIG. 2B is executed. That is, in FIG. 2B, a small AC current is injected into another electrode pair (in this case, including electrodes 7 and 8). A different electric field is generated. Now, the voltage difference induced on the surface is measured for each electrode pair adjacent to each other. This includes the voltage difference for electrodes 1-2, 2-3, 3-4, 4-5, 5-6, etc. Voltage measurements are not performed on the electrode pair where the current is injected to help reduce the noise associated with the contact impedance. The same procedure is continued for all electrodes, which provides enough information to reconstruct an "image" of the electric field across the substrate. In the example of FIG. 2A-2B, a total of 40 measurements (i.e., 8×(8-3)=8×5=40) are obtained based on the above formula. These measurements help to reconstruct all or part of the electric field.

[0079] This exemplary approach relies on surface conductivity. It will be understood that intrabody conductivity can be measured and then used in different exemplary embodiments. When intrabody conductivity is used, the conductive material may be embedded in a substrate or other surface. For example, a conductive article may be co-extruded or otherwise included in a complex shaped body as an integral part thereof. In other cases, a mating component may be mounted on an external component, and the mating component may have a conductive coating formed thereon. Individually, the mating component essentially functions according to the surface conductivity related techniques as described above. However, because it is mated "below" or "under" the external component, the overall electric field for the joined components essentially functions as an intrabody measurement. The mating component may be sized, shaped, and arranged (e.g., contoured) to complement or otherwise match the contour of the outer component. The mating component and the outer component may be spaced apart from each other or may be joined together such that they share a contact surface.

[0080] In certain exemplary embodiments, a diffuse or "rough" substrate surface and / or a surface having a diffuser may be provided. For example, acid etched glass or other surfaces having a conductive coating thereon may be used to enable touch sensing related functions using the techniques disclosed herein. As another example, a chrome plated matte finish automobile trim surface may enable touch sensing related functions using the techniques disclosed herein. In different exemplary embodiments, other diffuse or rough surfaces may be used for other applications.

[0081] 3 is a schematic diagram of control electronics that can be used to process data collected using an EIT type approach for different applications, including, for example, touch sensing related applications. In FIG. 3, a direct digital system (DDS) module 302 generates a voltage for a voltage controlled current source (VCCS) 304. The VCCS 304 converts the voltage to a current that is then injected into electrodes 106 arranged around the periphery of the substrate 102 using a multiplexer 306.

[0082] After current is injected into a pair of electrodes 106, the voltage developed across the remaining electrodes 106 can be measured using the approach described above in connection with FIGS. 2A-2B. A multiplexer 306 is used to assist in performing this measurement simultaneously. A differential amplifier (DA) 308 is used in making the measurements. A low-pass filter (LPF) 310 removes noise from the received voltage. Typically, the LPF 310 is implemented in hardware. A controller 312 controls the switching between current injection and voltage measurement.

[0083] The controller 312 also transfers the measured and filtered data to an off-board mobile, PC, or other electronic device 314. The connection between the controller 312 and the electronic device 314 may be a wired or wireless connection. In the latter case, Wi-Fi, Bluetooth, or other wireless communication protocols may be used. The device 314 allows the user to "see" or "feel" a reconstructed "image" of the electric field created by the current injection by processing the measured voltage. The device 314 also helps to perform interaction detection (such as touch location detection) using applications executed thereon. In certain exemplary embodiments, the controller 312 can perform functions otherwise performed by a PC, making the PC an optional component in different exemplary embodiments.

[0084] FIG. 4 is a schematic diagram of a substrate conceptually segmented into different regions, which may be useful in the context of a large substrate (e.g., a large glass substrate) to perform precise interaction processing. In the example of FIG. 4, four overlapping segments (S1, S2, S3, and S4) are shown. More or fewer segments may be provided based on the size of the substrate, the desired accuracy, etc. As previously mentioned, the electrodes are disposed around the periphery of the substrate (e.g., glass) surface. Each of the electrodes is connected to a respective channel of a multiplexer. This approach allows for parallel processing of different regions. Also, because each electrode is separately connected to a different channel of the multiplexer, the signal generated by the interaction (e.g., touch, gesture, hovering, etc.) can be easily filtered to locate the region where the interaction occurred, since, at least in an EIT-based technology environment, the potential difference in that particular region will be greatest in the region where the interaction occurred. Additionally, each electrode may be connected to a multiplexer for task management (e.g., to apply a current or measure a voltage). In some demonstrative embodiments, each region may be processed on its own separate processing core, eg, operating separately from other operations performed in conjunction with other processing cores.

[0085] In the classification phase, the measured and filtered data may be processed using machine learning (ML) algorithms for high accuracy, non-touch, single touch, multi-touch, gesture, hovering, air gesture, and / or other interaction identification. A simple calibration process on the sensing surface (e.g., glass) may be used for each of these and / or other potential interaction types. In certain exemplary embodiments, different machine learning models may be developed for each different interaction type. This approach has been found to be very accurate in terms of recognizing the type and location of the interaction. It has also been found to potentially increase sensitivity and reduce variations caused by the manufacturing process (e.g., local non-uniformity of the conductive layer in terms of composition, sheet resistance, etc.), while also helping to account for issues of different substrate sizes and / or shapes (including linear or non-linear shapes, curved substrates, etc.), surface features (holes, protrusions, etc.), and / or the like. X and Y positions may be identified based on multiple regression models, etc., and these models may be used to support real-time tracking. That is, multiple regression models can be used for individual touches or other interaction locations, and multiple regression models can be used to track individual touches or other interaction locations over time, such as for purposes of gesture identification, as described below. Figure 5 is a schematic diagram showing how clusters of data points can be machine-learned to correspond to interactive touches or other interaction locations on an interface.

[0086] ML algorithms can also be used for gesture identification. Gestures that can be modeled and subsequently recognized include, for example, right swipes, left swipes, up swipes, down swipes, circling, and / or other gestures. To gather data used in developing a model and then recognizing gestures based on the trained model, interactions (touch interactions, hover interactions, and / or the like) are collected as a series of frames. The set of frames is processed continuously for real-time reconstruction and recognition of the gesture. FIG. 6 is a schematic diagram of how a series of collected points can be collected in frames and subsequently mapped to a right swipe gesture, which may be performed as an "air gesture" on or in close proximity to the substrate. Different exemplary embodiments can collect, for example, x, y, z, and t information without capturing the full frame itself.

[0087] FIG. 7 is a schematic diagram of control electronics that may be used for electrical tomography-mediated operations, according to certain exemplary embodiments. Essentially, FIG. 7 provides hardware for an electrical tomography-mediated interaction detection system for use with an interface, for example, as shown in FIG. 1, comprising a conductive surface (e.g., a substrate having a conductive surface thereon, such as a coating) and a plurality of electrodes provided around the edge of the substrate. In certain exemplary embodiments, the electrodes may be arranged in a regular pattern around each side of the substrate, and optionally, the electrodes are spaced equidistant from each other. In certain exemplary embodiments, the electrodes may be arranged in pairs on opposing sides of the substrate, the pairs forming rows and columns across the substrate. In certain exemplary embodiments, one electrode may be provided for a given surface area. For example, in certain exemplary embodiments, the electrodes may be arranged in a regular pattern around each side of the substrate, and the pairs may form rows and columns across the substrate. In certain exemplary embodiments, one electrode may be provided for a given surface area. For example, in certain exemplary embodiments, the electrodes may be arranged in a regular pattern around each side of the substrate, and the electrodes may be spaced equidistant from each other. 2 One electrode (or multiple electrodes) may be provided for every 300 cm 2 of the substrate surface area (e.g., 2(In the example, one electrode per electrode). This dimensionality has been found to provide good accuracy for large interfaces, including interfaces having diagonals of 65 inches or 67 inches or more. Of course, the electrode-surface area "density" can be adjusted upwards or downwards within or outside this range, for example, to affect accuracy.

[0088] The system of Figure 7 includes a controller 702, which may be a microcontroller board or embedded microcontroller running software, firmware, and / or the like. The controller 702 may be configured to perform individual high speed algorithms, filtering operations, and application level operations. Point tracking may be performed and simple gestures may be resolved at the controller level via algorithms. Exemplary algorithms are described in more detail below.

[0089] With respect to the controller 702, an integrated ADC 704 and / or software bandpass filter (SBPF) 706 may be provided to reduce the need for additional hardware units, thereby both speeding up and simplifying the overall hardware architecture. The controller 702 may be configured to perform at least some tasks in parallel (e.g., through multi-threading with a multi-core processor). These parallelizable tasks may include, for example, some or all of the following: current injection, analog-to-digital conversion via the ADC 704, filtering via the SBPF 706, point cloud and / or gesture recognition, etc.

[0090] The controller 702 is connected to the controlled high precision multi-channel multiplexer 708. An adjustable current regulator circuit (ACRC) 710 is also connected to the controller 702 and the multiplexer 708. The ACRC 710 can operate in one of several modes and can be considered, for example, as a main driver for applying an adaptable current to the conductive surface. In other words, the ACRC 710 is configured to control the amperage of the current that the multiplexer 708 is configured to provide to the electrodes. In this regard, in a first mode, for example, the ACRC 710 dynamically adjusts the amperage of the current such that the amperage of the current is increased until the current leakage between the multiplexer channels exceeds a threshold level and then is decreased such that the current leakage between the multiplexer channels is below the threshold level. This stepped or staged approach is advantageous because it allows high currents to pass through the multiplexer 708, resulting in better field creation, which in turn may allow for higher detection accuracy by avoiding excessive noise generation, cross-channel leakage, etc. that may adversely affect field creation.

[0091] In this first mode of operation, the ACRC 710 is configured to begin increasing the amperage from a baseline level. This baseline level may be preprogrammed (e.g., 20 mA) in certain exemplary embodiments. In certain exemplary embodiments, the baseline level may be based on the amperage actually provided to the multiplexer 708 before restarting the touch sensitive detection system. The increase in amperage in the first mode may be continuous or in stepwise increments. The increments may be related to (e.g., equal to, a multiple of, etc.), for example, a threshold level, a level of current leakage detectability, and / or the like. In certain exemplary embodiments, threshold levels of 5 mA or less may be possible. For a substrate having a 65 inch or 67 inch diagonal, 30 mA may be appropriate for current injection. For example, for a larger surface area, a larger value may be used.

[0092] In certain exemplary embodiments, the ACRC 710 may be dynamic in the sense that it may monitor current leakage during detection and / or other operations, may check accuracy, may monitor noise on the return voltage, etc. In this regard, in certain exemplary embodiments, if the current leakage creeps upward or there is too much noise returning to the measured voltage, the current provided via the multiplexer 708 may be reduced. Similarly, if accuracy decreases, the current may be increased. These decisions may be made via the controller 702 based on continuous feedback through the entire architecture loop described.

[0093] In certain exemplary embodiments, the ACRC 710 may be directly controlled by the controller 702 based on the injected effective current and the measured voltage. Machine learning may be used for the ACRC 710 adjustment. That is, ML algorithms may help determine how much current to apply to the surface of the interface based on characteristics such as, for example, interface dimensions / size, surface material, sheet resistance, etc. Furthermore, the controller 702 may adjust the ACRC 710 based on dynamically obtained SNR levels in certain exemplary embodiments. Initial parameters may be set during a training period and adapted in a real-time process based on existing characteristics as the actual type ("IS") may show differences from the trained ("SHOULD") type. Thus, in certain exemplary embodiments, the ACRC 710 provides the ability to automatically adjust the current on the surface when a leak is detected, either via an ML approach or a more programmatic or rule-based step-up / step-down procedure.

[0094] The ACRC 710 may additionally or alternatively be operable in a second mode in which the ACRC 710 causes the multiplexer (e.g., multiplexer channel) to provide a constant (e.g., static, or held, unregulated) amperage. The constant amperage in certain exemplary embodiments may be pre-programmed based on known characteristics of the touch detection system. These known characteristics may include, for example, substrate size and sheet resistance characteristics (such as the known sheet resistance of a conductive coating blanket coated on a substrate). In certain exemplary embodiments, the constant amperage may be the last level provided when operating in the first mode.

[0095] The ACRC 710 has parallel outputs to the multiplexer 708, for example, in certain exemplary embodiments, each channel is a low dropout channel providing up to 500 mA of output. In some exemplary embodiments, the outputs can be parallelized such that current is provided in parallel to the multiplexer 708 and potentially to the electrodes. As will be appreciated by those skilled in the art, the algorithmic approaches described below can be modified to account for this parallelized operation. The use of multi-core processors and the like can also be used in certain exemplary embodiments to facilitate processing. By providing parallelized current outputs, faster overall processing speeds can be achieved. In certain exemplary embodiments, different amperages can be provided across different channels of the multiplexer 708.

[0096] The exemplary system of FIG. 7 illustrates a demultiplexer 712, separate from the multiplexer 708, configured to receive voltages from the electrodes and provide corresponding signals to the controller 702 (e.g., via a direct connection to the controller 702 or its components, such as the ADC 704, SBPF 706, etc.). In certain exemplary embodiments, a direct connection between the controller 702 and the demultiplexer 712 is not required. Thus, the controller 702 is further configured to receive signals from the demultiplexer 712 and determine therefrom an operation performed in association with the interface (e.g., a touch, gesture, hovering, or other interaction performed in association with the interface). Different voltages may be provided across different channels of the demultiplexer 712, for example, based on a disturbance to the electric field. Although the example of FIG. 7 has a multiplexer 708 separate from the demultiplexer 712, it will be appreciated that some exemplary embodiments may use a single multiplexer for operations related to both multiplexing and demultiplexing.

[0097] In certain exemplary embodiments, an analog power supply (APS) 714 may be interposed between the controller 702 and the ACRC 710. The APS 714 is configured to provide power to the ACRC 710, for example, via a direct digital synthesizer (DDS) for a higher voltage output. In certain exemplary embodiments, a DDS need not be implemented. It will be understood that AC currents and voltages are applied on the surface of the interface (not counting DC bias), but DC voltages may be for the multiplexer 708 and the demultiplexer 712. In some exemplary embodiments, when amplifiers and / or other hardware are used, a DC bias may be applied to the amplifiers that receive the voltage from the demultiplexer, for example as shown in FIG. 7.

[0098] In certain exemplary embodiments, the demultiplexer 712 and the controller 702 may be directly connected to each other. Thus, for example, the path between the demultiplexer 712 and the controller 702 may lack amplifiers and / or hardware bandpass filters and / or other hardware elements. This may be possible due to the low noise (and thus high signal-to-noise ratio or SNR) provided by the demultiplexer 712. In certain exemplary embodiments, the noise in the signal provided by the demultiplexer 712 to the controller 702 is less than 5%. In such cases, the controller 702 may be configured to filter the noise in the signal from the demultiplexer 712 using software (such as including SBPF 706 or simple thresholding) as needed. That is, the noise is so low that it can simply be ignored in certain exemplary embodiments. This may be advantageous, for example, because more potentially useful data may be retained and data may be acquired faster compared to approaches that implement amplifiers and hardware bandpass filters and / or the like. Having fewer components may simplify the overall design. In other words, the controller 702 can have an integrated software bandpass filter 706 that is configured to operate on the signals provided by the demultiplexer 712 if and when necessary. Exemplary measurements made using the hardware approach described herein are described in more detail below in conjunction with Figures 20A-22.

[0099] Leakage can be caused by having many multiplexer channels. In this regard, if the input current exceeds or is very close to the maximum that can be handled on a channel, the excess current is not distributed evenly among the other channels. This can cause problems (e.g., low SNR) during reading of voltage measurements from electrode pairs, because other electrodes may behave inappropriately as injectors during measurements, despite the intention to prevent this and to limit the injectors to different electrode pairs.

[0100] To help address current leakage issues, wiring and multiplexer and demultiplexer design can be used. The wiring used to connect the controller to the multiplexers and demultiplexers can be highly insulated against noise. The resistance of the wire can help promote isolation, and therefore the thickness of the wire, the material used, the homogeneity of the material, etc. can help reduce noise through the wiring.

[0101] The design of the multiplexer and demultiplexer can help reduce the amount of leakage by smart control of the channels and inspection of measurements when leakage occurs. In certain exemplary embodiments, there is a direct connection from the controller to each channel of the multiplexer for current injection. Current is applied and all channels are measured in parallel. The main channel in the multiplexer should have full current if there is no leakage. If the main channel has less current (e.g., 30mA) than the mainstream or any of the other channels have currents that exceed the tolerance range, in certain exemplary embodiments, that level is recorded as the maximum current injection level for later use. Similar wiring can be provided for the demultiplexer, for example, in embodiments where the multiplexer and demultiplexer are separated from each other.

[0102] Referring again to FIG. 7, in certain exemplary embodiments, an amplifier 716 and / or a hardware bandpass filter (HBPF) 718 may be inserted between the demultiplexer 712 and the controller 702. This may be useful when there is a low SNR. The amplifier 716 may be used so that the measured voltage may be amplified with the proposed gain control. The optional HBPF 718 may be used to filter the measured voltage within a certain band range, for example, using a low-pass filter and / or a high-pass filter. It may also be desirable to include the HBPF 718 when a uniform current distribution is not possible (due to high noise in the measurement) and / or when the current cannot be increased due to hardware limitations. Both the amplifier 716 and the HBPF 718 are optional, as indicated by the dashed lines. By omitting these elements, the voltage measurement acquisition speed may be increased and the hardware implementation may be simplified.

[0103] Using algorithms as described in more detail below, the controller 702 may receive input from the demultiplexer 712 to identify an interaction location of an action (e.g., a touch, hover, gesture, and / or other action) performed in association with the interface, distinguish between different actions performed in association with the interface, execute machine learning algorithms to identify actions (such as gestural actions) performed in association with the interface, recognize other actions performed in association with the interface, and / or the like.

[0104] In the approach outlined in FIG. 7, advanced or powerful computers are not required to handle static and gesture-related interactions. This is possible because there is no need to recreate the electric field for the entire substrate. Instead, compatible algorithms can be used for fast identification of touch-related interactions (including multi-touch and gesture-related inputs). Furthermore, performance requirements can be reduced because ML-related techniques do not need to be used at the level of understanding a single interaction (e.g., touch) point, but can instead be used to understand a collection of interactions (e.g., touch points), for example, for gesture-related functions and / or other related functions.

[0105] Several exemplary compatible algorithmic approaches for processing voltage-related signals are now described. It will be understood that these exemplary algorithmic approaches may be used together with the exemplary hardware configurations described above, or may be used separately. That is, signals from the exemplary hardware arrangement of FIG. 7 may be used in conjunction with any one or more of the algorithms described below, or the algorithms described below may process data obtained from other hardware implementations. That is, the algorithmic approaches may not be hardware independent. Although exemplary modifications to the algorithms are described below, it will be understood that these algorithms may be modified to serve different purposes.

[0106] The algorithms may be implemented as computer-executable instructions stored in a memory (e.g., RAM, ROM, HDD, SSD, memory card, firmware, etc.). At least one processor may be configured to execute these computer-executable instructions. The processor may be a general-purpose processor, part of the controller 702 of FIG. 7, an ASIC, and / or the like.

[0107] Although the use of one or more multiplexers and / or demultiplexers is described above, it will be understood that different exemplary embodiments may not require such hardware. That is, in some cases, the electrodes may be directly (or at least more directly) connected to the controller in some cases. This arrangement is possible, especially when the total number of electrodes is smaller. For example, in systems where the desired resolution is lower, fewer electrodes may be provided and input / output signals may be provided from the controller without an intervening multiplexer. For example, in systems where there are two electrodes designed to detect proximity / hovering and touch, the use of a multiplexer may be omitted. This may be useful for robotic skin sensors or other machines that simply detect human presence and / or touch and react accordingly. Example Single and Dual 4-Loop Algorithms

[0108] A first exemplary algorithmic approach may be considered a four-loop algorithm. As will be appreciated from the following description, the four-loop algorithm may be unidirectional or dual-directional. Dual-directional may be useful for larger surfaces and / or high-precision applications. The algorithmic approach may generally be considered to be based on a four-pole readout and is similar to the approaches described above.

[0109] With regard to the four-loop algorithm, FIG. 8 is a flowchart illustrating a first algorithmic approach for identifying interactions (touch-related interactions, Z-coordinate location for hover detection, etc.) in relation to an interface according to certain exemplary embodiments, and FIG. 9 is a diagram illustrating how the flowchart of FIG. 8 may operate in a first phase according to certain exemplary embodiments. The example of FIG. 9 includes 16 electrodes arranged around the perimeter of a substrate supporting a conductive coating. In the example of FIG. 9, the electrodes are arranged equidistantly around all four edges of the substrate. The smaller circles between the numbered electrodes are the assumed locations of measurements (this applies to this figure and other similar figures described in more detail below).

[0110] The approach of Fig. 8 aids in characterizing actions performed in relation to the interface shown generally in Fig. 9. The characterization of the interaction may include the type of interaction with the interface and the location for it. Valid interaction types include static touch actions, multi-touch input actions, various gesture actions, hovering actions, and / or the like. For single and multi-touch input, the location may be X and Y coordinates relative to the substrate. For gestures, the characterization may further include gesture type, where the gesture starts and stops, vector magnitude and / or direction, etc.

[0111] Note that a collection of adjacent electrode pairs is defined such that each electrode is part of two different adjacent electrode pairs in the collection. In the arrangement of Figure 9, the interface is divided into rows and columns of cells, with cells defined by the lines between opposing electrodes. Depending on the arrangement, the number of cells in each row may be equal to one more than the number of horizontally extending electrodes, and the number of cells in each column may be equal to one more than the number of vertically extending electrodes. Note that the "lines" are conceptual in the sense that opposing electrodes are not directly wired to each other, the conductive coating is not patterned, etc.

[0112] As shown in step S802, a current is applied to the first adjacent electrode pair to establish an electric field across the surface of the substrate. In step S804, at least some of the adjacent electrode pairs are used to measure a voltage difference induced on the surface. That is, in certain exemplary embodiments, (i) each adjacent electrode pair other than the first adjacent electrode pair, and (ii) each adjacent electrode pair other than the two adjacent electrode pairs that include the electrodes in the first adjacent electrode pair are used to measure a voltage difference induced on the surface. The voltage may be a voltage drop measured along a line as shown diagrammatically in FIG. 9. Steps S802 and S804 are repeated for each adjacent electrode pair in the collection, for example, as informed by the decision made in step S806. The value obtained in step S804 may be stored in a lookup table or the like (and this storage location may be used in the following algorithmic description when data obtained from the electrodes is stored), and this or other suitable data structure may be used to efficiently determine a maximum function drop, for example, using a MIN function.

[0113] In step S808, it is determined in which of the cells the action was performed. This determination is made based on the measured voltage difference. To facilitate this determination, the voltage difference measured in step S804 may be stored in a look-up table or other structure (e.g., managed by the controller 702). This determination may include, for example, selecting the measurement with the largest voltage difference and determining that the cell bounded by the electrodes used to generate the selected measurement is the cell in which the action was performed. With regard to the former, the total number of measurements used in the first step is the number of electrodes multiplied by (number of electrodes - 3). The measurements may be compared with each other to find the maximum voltage drop, which may be selected for follow-up. For example, a determination may be made as to whether the voltage drop is at least as large as a threshold level. This thresholding approach may be advantageous as it may help ignore spurious signals caused by, for example, electromagnetic interference (EMI), improper or unexpected hovering, passers-by, etc.

[0114] A calculation is performed using the measured voltage difference associated with the determined cell to characterize the action. As shown in step S810, the action is characterized as being one of a static interaction and a gesture input. It should be noted that a determination can also be made as to whether a multi-touch input is provided. The calculation performed to characterize the action includes a reconstruction of the electric field in the determined cell using a subset of electrodes, provided that the action is a static interaction, as shown in step S812. Meanwhile, the calculation performed to characterize the action includes collecting information about a plurality of perturbations to the interface and using a machine learning model in conjunction with the collected information, provided that the action is a gesture input, as shown in step S814.

[0115] In step S812, a reconstruction of the electric field for a portion or "cell" of the interface is created. It is not necessary to reconstruct the electric field for the entire substrate. In certain exemplary embodiments, reconstructions of the electric field adjacent to other cells are not created. Instead, as described above, a subset of electrodes may be used in this reconstruction. The subset of electrodes used for the reconstruction may include at least four electrodes provided in pairs in a common horizontal and / or vertical alignment that bounds the determined cell. For example, if a touch is provided in the central circular area of ​​FIG. 9, the vertical electrode pairs 2-8 and 3-7 may be used in recreating the electric field for the portion for the interface, the horizontal electrode pairs 4-14 and 5-13 may be used in recreating the electric field for the portion for the interface, or all of these pairs may be used. In general, the subset may include at least four electrodes provided in a first common alignment that bounds the determined cell in a first direction, and at least four other electrodes provided in a second common alignment that bounds the determined cell in a second common direction that is orthogonal to the first common direction. In certain exemplary embodiments, the subset of electrodes used in the reconstruction may further include additional electrodes aligned in pairs adjacent to the electrode pairs bounding the determined cell. Thus, following this example of FIG. 9, when only vertical electrode pairs are used, vertical electrode pairs 1-9, 2-8, and 3-7 may be used in the reconstruction.

[0116] The calculations performed to characterize the motion may include a finite element method (FEM) approach. The FEM approach may implement, for example, center of gravity, momentum, and / or minimum triangle calculations, provided that the motion is a static interaction. The FEM approach may be implemented at an algorithmic level and need not be presented as part of the visualization in certain exemplary embodiments. In certain exemplary embodiments, this reconstruction is created for only a portion of the interface. Image processing techniques can then be applied to more precisely determine the location of simple touch-related interactions. FIG. 10 is a diagram illustrating how the flowchart of FIG. 8 may operate in a second phase, according to certain exemplary embodiments. More specifically, FIG. 10 illustrates a minimum triangle approach to determine touch location. These techniques may be used instead of machine learning models when the motion is a static interaction. In FIG. 10, the circles at the corners are intended to indicate electrode locations, while the circles at the center of the edges are where measurements are supposed to be made. In certain exemplary embodiments, the designated electrodes and their respective voltage measurements are used in conjunction with a center of gravity (COG) algorithm to identify the best-fit touch triangle and touch area using repeated measurements. That is, as can be seen from FIG. 10, the connecting lines divide the surface into a series of triangles, and the touch location can be determined with reference to the triangles created. From an electrical perspective, a touch interaction on the surface produces a maximum voltage drop on the nearest electrode, and voltage also drops on other electrodes within the cell area. The COG algorithm in certain exemplary embodiments uses these relative voltage drops between electrodes to define a touch area and distance to the electrodes, which can be used to determine the location of the touch interaction.

[0117] In contrast to this approach for simple static interaction detection, gesture detection may involve the use of machine learning models. Models may be trained based on different interaction types such that different gestures are associated with different models. The training data may include a series of touch coordinates provided. A collection of static interaction points may be collected over time to define a gesture. The time for collection may be bounded by detection of a touch off event and / or the passage of a predetermined time period. That is, a gesture may be defined as all touches received until no touches are detected for a predetermined time period and / or until a timer reaches a predetermined timing. With respect to the former, data regarding multiple detections may be collected using the approach described in relation to step S812. This information may be stored in a buffer. Timestamp information may be associated with a location. A gesture may be considered complete when no further points are detected for a predetermined amount of time or when there is a predetermined time gap between subsequent detections. With respect to the latter, most gestures may be completed within 1-5 seconds depending on the size of the substrate, the complexity of the gesture, and / or the like. This timing can then be used to determine when a gesture is likely to be completed. As training data is received, different models can be developed for different gesture types. In use, information about multiple perturbations can be matched with one of these machine learning models to determine the gesture type.

[0118] Briefly, in certain exemplary embodiments, during use, information for a collection of perturbations can be obtained, the information for the collection of perturbations can be matched with a machine learning model, and gesture inputs provided to the interface can be recognized based on the matching. The collected information about the perturbations can be stored in a buffer or other storage medium, and the matching can be performed using the buffer. The buffer can be wiped after a touch-off event is detected and / or after matching is completed.

[0119] One approach using machine learning models may include defining an XLD contour region in association with information for a collection of perturbations. In association with the defined XLD contour region, an "IS" model may be generated. The "IS" model may be compared to a number of trained XLD "SHOULD" contour models. Training of the model may be done artificially, similar to optical character recognition (OCR) techniques, and / or directly on the surface (e.g., via surface-based training on real objects). Based on the comparison, a determination may be made as to whether there is a match with the trained XLD contour model. Although XLD contour models are described here and elsewhere, it will be understood that other image segmentation, region-based contour models, and / or other techniques may be used in different exemplary embodiments.

[0120] It will be appreciated that the current may be provided essentially continuously throughout the process of Figure 8. It will further be appreciated that steps S802 through S806 are performed continuously during static interaction detection. In some exemplary embodiments, steps S802-S806 may be performed while steps S812 and / or S814 are being performed.

[0121] As mentioned above, the processing loop may be unidirectional or bidirectional. In the former case, steps S802 to S806 may be performed unidirectionally in a loop around the substrate. That is, the processing may sequentially consider electrodes 1-2, 2-3, 3-4, etc. In the latter case, steps S802 to S806 may be performed bidirectionally around the substrate in a first loop and a second loop that is the inverse of the first loop. That is, the processing may consider electrodes 1-2 and 16-15 simultaneously in a first iteration, then electrodes 2-3 and 15-14 simultaneously in a second iteration, etc. It will be understood that unidirectional and bidirectional processing here (and in the algorithmic approaches described below) may or may not define a particular order for activating electrode pairs. In certain exemplary embodiments, unidirectional simply refers to activating each pair in the system such that a current is applied to one electrode and a voltage is received from the other electrode, and bidirectional refers to activating each pair in the system such that a current is applied to each electrode of the pair and a corresponding voltage is received from each electrode of the pair. In other cases where ordering is an issue, a starting electrode pair may be selected and the electrodes may be activated in an order defined relative thereto (e.g., clockwise or counterclockwise therefrom). Exemplary Divider Loop Algorithm

[0122] A second exemplary algorithmic approach can be considered a divider loop algorithm. As mentioned above, the divider loop algorithm can be unidirectional or dual-directional. The divider loop algorithm is advantageous in that it can be used for initial interactivity detection and also for subsequent interaction location tracking.

[0123] Figure 11 is a flow chart illustrating a second algorithmic approach for identifying interactions (such as touch-related interactions) in association with an interface, according to certain exemplary embodiments. Figure 12 is a flow chart illustrating how interactions identified using the technique of Figure 11 can be more precisely identified and / or characterized, according to certain exemplary embodiments, and Figure 13 is a diagram illustrating how the flow chart of Figure 11 can operate in a first phase, according to certain exemplary embodiments.

[0124] Similar to the four-loop algorithmic approach detailed above, the approach of Figure 11 aids in characterizing the operations performed in conjunction with the interface shown generally in Figure 13. The electrodes may be continuously active during the process of Figure 11 in certain exemplary embodiments.

[0125] Also similar to above, in FIG. 13, the interface comprises a substrate supporting a conductive coating and a plurality of electrodes applied around the edge of the substrate. In step S1102, a current is applied from the plurality of electrodes to first and second electrodes, where the first and second electrodes are on opposite edges of the substrate such that a first "conceptual" line connecting the first and second electrodes defines a first and second region of the interface. In step S1104, a voltage difference induced in the first and second regions of the interface as a result of the current application is measured. The measurement may be performed using four electrodes in a particular exemplary embodiment.

[0126] In step S1106, one of the first and second regions of the interface is selected based on the measurements made in step S1104. Depending on the electrode placement, the first and second regions of the interface may be substantially equal in size. The selection may be based on which of the first and second regions has a larger voltage drop. That is, in certain exemplary embodiments, the region associated with the larger voltage drop is selected. That said, in certain exemplary embodiments, a determination may be made as to whether the measured voltage differences are at least substantially equal to each other. For example, a determination may be made as to whether the measured voltage differences differ from each other by no more than a threshold amount. If the two measured voltage differences are within this threshold, a determination may be made that a touch-related event is unlikely to have occurred. In this case, the process may be stopped.

[0127] Here, in step S1108, two of the electrodes from the plurality of electrodes are defined as current injection electrodes. The current injection electrodes used here are different from the first and second electrodes and different from any previously defined current injection electrodes. Furthermore, a second notional line connecting the current injection electrodes defines first and second sub-regions within the selected region and any other previously selected sub-regions. As above, depending on the electrode placement, the first and second sub-regions may be substantially equal in size. This may be true for the first and second sub-regions within each iteration of the loop defined in FIG. 11. In step S1110, a current is supplied to the defined current injection electrodes, and the voltage difference induced in the first and second sub-regions as a result of step S1110 is measured in step S1112. As above, in certain exemplary embodiments, the measurements may be performed using four electrodes.

[0128] In step S1114, one of the first and second sub-regions is selected based on the measurements made in step S1112. As noted above, the selection here may be based on which of the first and second sub-regions has a larger voltage drop. That is, in certain exemplary embodiments, the sub-region associated with the larger voltage drop is selected. Also, as can be seen from the loop in FIG. 11, this selection may be made on an iteration-by-iteration basis.

[0129] Steps S108-S1114 are repeated until a final sub-region of a predetermined size is selected, for example, as determined in step S1116. The predetermined size may be preset in certain exemplary embodiments. In certain exemplary embodiments, the predetermined size may represent a sub-region bounded by two immediately adjacent electrodes on each side. This is the case in the example of FIG. 13, where the final sub-region is bounded by a pair of adjacent electrodes including electrodes 2-3 and 5-13 for the upper and lower boundaries, and electrodes 2-8 and 3-7 for the left and right boundaries.

[0130] In step S1118, a determination of the location of the perturbation relative to the interface is made from within the final sub-region. This can be performed using image processing techniques, such as, for example, center of gravity, momentum, minimum triangle calculation, etc., as described above. The approach outlined in FIG. 12 may also be used in certain exemplary embodiments. That is, as shown in FIG. 12, the final sub-region may be conceptually divided in step S1202 into a grid with a predefined layout of cells. The predefined layout may be selected, for example, from a number of different possible predefined layouts that are pre-stored and accessible by the controller 702. The more cells, the more accurate the detection may be. The layout may be a grid with rows and columns, a honeycomb-like arrangement, and / or the like. For grid and grid-like approaches, the cells may be oriented in different numbers of rows and columns. For example, 3×3, 4×4, 5×5, and / or other arrangements of cells (which may or may not have the same number of rows and columns in different exemplary embodiments) may be used. The number of cells may be increased based on improvements in coating conductivity, the addition of electrodes, etc. The location of the perturbation can be determined with reference to this given applied cell layout.

[0131] As mentioned in step S1204, the determination of the location of the perturbation is performed with respect to one of the cells in the given layout where the perturbation was likely to have occurred. In this regard, it may be considered whether the likely location was within the center cell (see step S1206). If the cell was likely to have been the center cell, the process may end (see step S1208). If the cell where the operation was likely to have been performed is not in the center or at a corner of the given layout, it may further consider one adjacent cell, or if the cell where the operation was likely to have been performed is at one of the corners of the given layout, it may further consider multiple adjacent cells (see step S1210). In the latter case, one or more cells adjacent to the cell where the perturbation was likely to have occurred may be outside the final sub-region. In some exemplary embodiments, if the determined cell is on an edge (such as, but not limited to, a corner), additional cells (including cells outside the sub-region) may be considered. For example, in certain exemplary embodiments, if the predicted cell is on a side edge, additional evaluation methods such as (bicubic) neighborhood comparison of voltage drops may be used to more accurately identify the cell. Thus, it will be appreciated that some example embodiments may perform more accurate cell identification based on extended analysis in the case of border cell identification.

[0132] To help determine which cells are likely to have been perturbed, the measured voltages can be compared to baselines or other data associated with each of the cells in a given layout. That is, each cell in each possible final sub-region can be pre-characterized to have an associated voltage-related measurement that can be used in determining when a touch-related event is likely to have occurred for that respective cell. Pre-characterization may be done using machine learning, using manual calibration, and / or the like. The machine learning approach continues to learn / train as new touch detection values ​​come into the system. Clustering can be used, for example, based on a user forcing results closer to previously trained data, etc. Larger clusters mean less accuracy, and thus cluster size can be linked to the size and / or position of user interface elements, board size, etc. The overall machine learning approach can be highly dynamic. Another example of dynamic training can involve scripted calibration. In scripted calibration, the user can be asked to perform a number of tasks. These tasks may include instructions to touch a particular location for a particular amount of time, instructions to draw a particular shape in a particular area, and / or combinations thereof (e.g., instructions to touch a first location for a first amount of time, instructions to move to a second location for a second amount of time, etc.). The scripts may be stored in a computer-readable storage medium and accessed by a controller. The controller may, for example, retrieve these scripts and control lights and / or speakers to describe where / how the scripted interactions should be performed. In certain exemplary embodiments, the scripts may be updated (e.g., when updated via a network connection to a remote system, through downloads, by component replacement, etc.). In some exemplary embodiments, an authorized user may be able to dynamically train the system to recognize more locations and / or gestures through dynamic training.For example, in certain exemplary embodiments, a user may custom program user interface components and train the system to anticipate interactions in different locations. In some exemplary embodiments, the masks can be dynamically optimized to make more accurate decisions. A more static manual calibration that does not update over time may also be used. A more static approach may involve, for example, accessing pre-stored information and ensuring that the system is pre-calibrated based on known characteristics of the interface itself, expected interactions, etc. By understanding the characteristics of the substrate, a more static rule-based approach may be used to define a range of XY values. It will be appreciated that static and / or dynamic approaches may be used for point detection and / or gesture detection (the latter may be implemented as a collection of point detection operations, as described herein). A static approach is particularly well suited for simple gestures related to interactions used to turn lights on / off, which may otherwise be used to simply flip a switch or use a simulated analog controller to dim lights, adjust speaker volume, control room or vehicle temperature, etc. A dynamic approach may be better for more complex approaches such as, for example, handwriting recognition.

[0133] As alluded to above, different masks may be defined for different areas of the interface, and the decision as to which mask to apply to a given final sub-region may be based on the distance to the electrode. In this regard, for example, the further the touch is from the electrode, the higher the ratio should be. For example, a comparison to a voltage threshold may be made to remove potential spurious signals. The voltages used in pre-characterization and subsequent matching may be the voltages that bound each final sub-region.

[0134] That is, in the example of FIG. 13, electrodes 2-5, 7-8, and 13-14 may be used in pre-characterization and pattern matching, for example, such that each cell in a 3×3 layout is pre-characterized. In use, the voltage drops of the electrodes bounding the final sub-region may be measured, the measured voltage drop ratio of adjacent electrode pairs may be calculated, and cell selection may be performed based on the calculated ratio. Thus, in certain exemplary embodiments, the voltage drop ratio of each electrode may be taken into account, such that cell selection is performed by matching the measured voltage drop ratio with the pre-characterization. In certain exemplary embodiments, the drop ratio may be defined as a percentage, for example, based on the voltage drop due to a touch reference voltage without touch.

[0135] The operation of Figure 13 will now be described to help further illustrate aspects of this exemplary approach. In Figure 13, the interface is conceptually divided in half using a "line" between electrodes 1 and 9. Because the touch is on the left side of the interface as indicated by the circle, the voltage drop into the left half is greater than the right half. In the next iteration, the left half of the interface is conceptually divided in half using a horizontal line "connecting" electrodes 5 and 13. The touch location is above this line, so the next sub-region for comparison is to the left of the 1-9 line and above the 5-13 line, as determined by the voltage drop above the line being greater than the voltage drop below the line.

[0136] This current sub-region is divided in half using the line "connecting" electrodes 2 and 8. The touch location is to the left of this line as determined by the voltage drop above the line being greater than the voltage drop below the line. This current sub-region is divided approximately in half using the line "connecting" electrodes 4 and 14. The touch location is below this line. This current sub-region is divided approximately in half using the line "connecting" electrodes 3 and 7. The touch location is below this line.

[0137] At this point in operation, adjacent electrodes bound the final sub-region; i.e., electrode pair 2-8 is directly adjacent to electrode pair 3-7, electrode pair 4-14 is directly adjacent to electrode pair 5-13, and so on. A check can be performed to verify this final sub-region. Once the final sub-region is reached, a 3x3 grid is applied, taking into account voltage measurements from at least these boundary electrodes (electrodes 3-5 and 7-8). These measurements are compared to the pre-characterization data and it is determined that the final touch location is within the top right cell.

[0138] It will be understood that in certain exemplary embodiments, subsequent divisions are perpendicular to each other. Using the above language, the first and second lines for a first iteration are perpendicular to each other, and the current injection electrodes are selected in successive iterations such that a perpendicular second line is defined in the successive iterations. Different exemplary embodiments may use different partitioning methods. For example, vertical division may be repeated until an active column is determined, and then horizontal division may be repeated until an active row is determined, or vice versa.

[0139] 14A-14B are flow charts illustrating the end-to-end process in more detail. That is, FIGS. 14A-14B are flow charts illustrating further details regarding the unidirectional and bidirectional processing loops applied using the exemplary approach of FIG. 11 according to certain exemplary embodiments. The process begins by defining two starting electrodes that equally divide the surface area for current injection (step S1402). The first four electrodes are defined for a voltage measurement loop for a first half-area detection (step S1404). Two electrodes are applied for current injection for half-area detection. This is done once for a unidirectional loop (step S1406a) and twice for a bidirectional loop (step S1406b). Electrode voltage measurements are collected (steps S1408a and S1408b). As shown in FIG. 14A, if the voltage difference does not indicate touch detection, as determined in step S1410a, the process returns to step S1406a. Similarly, as shown in FIG. 14B, if one or more voltage differences do not indicate a touch detection, as determined in step S1410b, the process returns to step S1406b.

[0140] But if not, in step S1412, the next two electrodes for current injection are defined. In step S1414, the next four electrodes for voltage measurement are defined for the next half-region detection. Current injection is applied for the two electrodes for the next half-region detection (steps S1416a and S1416b). Voltage measurements of the four electrodes are collected (steps S1418a and S1418b). As shown in FIG. 14A, if the voltage difference does not indicate touch detection, as determined in step S1420a, the process returns to step S1406a. Similarly, as shown in FIG. 14B, if one or more voltage differences do not indicate touch detection, as determined in step S1420b, the process returns to step S1406b.

[0141] In step S1422, it is determined whether the final sub-region has been found. If not, the process returns to step S1412. If found, in step S1424, a sub-pixel mode is selected. The sub-pixel mode is associated with a predefined layout that is applied to the final sub-region in step S1426. In step S1428, specific electrodes for current injection are defined, and in step S1430, specific electrodes for voltage measurement are defined. The electric field sub-pixel regions are created using the voltages measured in step S1432 (see steps S1432a and S1432b). In step S1434, sub-pixels in the final grid are identified. The touch location (e.g., XY coordinates) is calculated in step S1436. It will be appreciated that the cells in the final grid are associated with the XY location of the interaction. The XY location may be calculated based on surface conductivity, known cell size, etc. The electrode pairs may be considered to be used "normally" for some EIT-related techniques, in the sense that, for example, the electric field within the final sub-pixel area is recreated and analyzed to determine the XY position of the interaction.

[0142] If gesture recognition is valid, the touch location is added to the collection of touch locations in step S1438. In step S1440, the XLD contour region is expanded. In step S1442, the efficiency of the touch point detection is evaluated to determine whether more touch detection points should be collected (in which case the process returns to step S1410a for FIG. 14A and to step S1410b for FIG. 14B) or not (in which case the process proceeds to step S1444). In this regard, step S1422 serves both to check for errors (e.g., the touch detection cannot be resolved because the voltage difference is below a threshold) and to handle pauses or "gaps" in the input. A pause in input can occur when a person holds a finger steadily in one position for a period of time, and a gap can occur when a person does not provide input for a short period of time (e.g., because the person momentarily removes the finger from the substrate while making a complex shape) but continues to provide touch input after this short period. For error checking, the system can check for null values ​​in the XY position determination and return to step S1422 accordingly. To account for pauses and gaps, the system can consider the timestamp associated with the input. This allows additional touch locations to be collected and used in gesture recognition. If there are no new touch locations streaming, an IS model is generated using the expanded XLD contour region in step S1444. A comparison with the trained XLD contour model is performed in step S1446. A gesture or other interaction type is determined based on this information in step S1448.

[0143] The approaches described above can be used for single-touch, multi-touch, and / or hover-related actions. The ML-related techniques described above can also be applied to data collected using this technique to enable gesture-related functionality.

[0144] Similar to the four-loop algorithmic approach described above, in certain exemplary embodiments, current may be delivered unidirectionally or bidirectionally between the first and second electrodes / current injection electrodes.

[0145] FIG. 15 is a diagram similar to FIG. 13, except that additional electrodes are applied to increase accuracy, according to certain exemplary embodiments. Specifically, in the example of FIG. 15, 24 electrodes are used with a divider loop. Increasing the number of electrodes increases the resolution over existing methods (e.g., the number of measurement points with the divider loop is 1,152 when 3×3 sub-pixelization is used, while the measurement points with the classical 4-pole approach is 24*21=504). FIG. 15 also demonstrates that sub-pixelization can be performed in several stages, for example, when the “final” sub-region is still large. In this regard, as explained above, the final region can be divided into, for example, 3×3, 4×4, or 5×5 cells. However, if the region is large and / or higher resolution is desired, it may not be feasible to go to a larger size (e.g., 6×6, which is the size of the final sub-region shown in FIG. 15). As a result, certain exemplary embodiments can further divide the sub-region defined between adjacent electrodes into smaller areas before applying the mask. In the example of FIG. 15, the region defined by electrode pairs 3-11, 4-10, 6-20, and 7-19 is further divided into four areas to which a 3×3 mask may be applied. That is, this example can be considered to potentially include two 3×3 grids and two 3×3 grids for a total of four sub-pixel areas in the “final” sub-region. It will be understood that the number of sub-pixel areas in the “final” sub-region may be pre-set (e.g., based on the size of the substrate, the position of the electrodes, the desired resolution, mask characteristics, and / or the like) in a particular exemplary embodiment. Thus, having four areas in the “final” sub-region is merely an example, and different exemplary embodiments may include more or fewer areas.

[0146] To determine which of these four areas the mask should be applied to, a divider loop algorithmic approach may again be used, using a subset of electrodes associated with the determined area. Doing so may indicate that the top right area is the area to which the mask should be applied. The process may then proceed as described above. In the example of FIG. 15, the "final" sub-region is divided into four areas, with a vertical and horizontal division at its center. To determine whether the mask should be applied to one of the two top quadrants or one of the two bottom quadrants, the drop values ​​are compared for the 6-20 and 7-19 electrode pairs. Similarly, to determine whether the mask should be applied to one of the two left quadrants or one of the two right quadrants, the drop values ​​are compared for the 3-11 and 4-10 electrode pairs. In certain exemplary embodiments, the drop value measurements may be made at electrodes closer to the "final" sub-region, and current may be injected at electrodes farther from the "final" sub-region. In the example of FIG. 15, this includes measuring voltage drop values ​​at electrodes 6 and 7 based on current injection from electrodes 20 and 19 to determine whether the mask should be applied to the top or bottom quadrant, and measuring voltage drop values ​​at electrodes 3 and 4 based on current injection from electrodes 11 and 10 to determine whether the mask should be applied to the left or right quadrant. In certain exemplary embodiments, measurements may be taken in one direction to determine an area in the "final" sub-region, and measurements may be taken in the other direction to determine the cell in the grid that applies to that area. In certain exemplary embodiments, measurements may be taken in both directions (e.g., based on injection at electrodes 6 and 7 and measurement at electrodes 20 and 19, and injection at electrodes 20 and 19 and measurement at electrodes 6 and 7, for horizontal division, etc.). This "swapping" referred to in FIG. 15 may be useful, for example, as a "double check" of the drop values ​​to improve accuracy in detection, which may be particularly beneficial for large interfaces. Cell identification may be performed in the manner described herein, for example, once areas within the "final" sub-region are identified.Although certain exemplary embodiments refer to "applying" a grid to areas of the "final" subregion, it will be understood that this description refers more broadly to an approach that further identifies which area(s) should be examined. Thus, while a grid may be "applied" to each area, the approach more generally selects one or more of the areas and then, based on that identification, identifies cells where a touch is likely to have occurred. Example Discrete Loop Algorithm

[0147] A third exemplary algorithmic approach may be considered a discrete loop algorithm. As noted above, a discrete loop algorithm may be unidirectional or dual-directional.

[0148] Fig. 16 is a flow chart illustrating a third algorithmic approach for identifying interactions (such as touch-related interactions) in association with an interface, according to certain exemplary embodiments, and Fig. 17 is a diagram illustrating how the flow chart of Fig. 16 may operate in a first phase, according to certain exemplary embodiments. Similar to the four-loop algorithmic approach described in detail above, the approach of Fig. 16 assists in characterizing operations performed in association with a touch-sensitive interface, as shown generally in Fig. 17. Electrodes may be continuously active during the process of Fig. 16, in certain exemplary embodiments.

[0149] Also similar to above, in Fig. 17 the interface comprises a substrate supporting a conductive coating and a plurality of electrodes applied around an edge of the substrate. However, now, as shown in Fig. 17, opposing electrode pairs form a plurality of spaced apart horizontal regions and a plurality of spaced apart vertical regions. In certain exemplary embodiments, the electrode pairs forming the horizontal regions may be substantially vertically centered in their respective horizontal regions and / or the electrode pairs forming the vertical regions may be substantially horizontally centered in their respective vertical regions.

[0150] In step S1602, a stabilization value is measured or calculated for each of the horizontal regions and each of the vertical regions. These stabilization or baseline voltages are values ​​that may be measured when there is no motion provided to the touch sensitive interface. That is, the stabilization or baseline voltages may be measured when the substrate is not being touched. The stabilization value may be an average taken over a predetermined period of time (e.g., voltages measured over 5 seconds, 10 seconds, etc.). This may help any accumulated charge to be dissipated and temporary EMI or other effects to be accounted for. In certain exemplary embodiments, the stabilization value may be preset prior to installation, may be measured once at installation, or may be measured at each system start-up or restart. In certain exemplary embodiments, the stabilization value may be retrieved from the last time a measurement was made. The stabilization values ​​may be associated with different electrodes, and the stabilization values ​​may vary between electrodes in some cases. For example, the variation is possible based on the size of the substrate, the electrode placement (relative to each other and / or to the coating), damage to the substrate and / or coating, differences in the coating caused by the variation, etc. The stabilization value, in certain exemplary embodiments, can be thought of as providing a type of background correction and reducing noise reduction (related to SNR optimization).

[0151] In step S1604, the voltage drop induced in each of the electrodes is measured. In certain exemplary embodiments, measuring the voltage drop may include subtracting a stabilization value for each vertical and horizontal region. Thus, the definition of the active vertical and horizontal regions, described in more detail below, may take into account the baseline voltages set for each vertical and horizontal region.

[0152] Next, in step S1606, a determination is made as to which vertical region is associated with the largest measured voltage drop in the vertical direction. This "candidate vertical drop line" is then checked to determine whether it is in fact an active vertical region, whether an active vertical region is adjacent to it, or whether there is likely no actual perturbation to the interface.

[0153] Therefore, in step S1608, an active vertical drop line is found based on the determined candidate vertical drop line. More specifically, if the difference in voltage drop between the determined vertical region and its adjacent vertical region is within a first threshold, the determined vertical region is defined as an active vertical region. Otherwise, if there are two vertical regions adjacent to the determined vertical region (meaning the candidate vertical drop line is not at the leftmost edge or the rightmost edge), one of the two areas between the determined vertical region and its adjacent vertical region is defined as an active vertical region. In this case, one of the two areas between the determined vertical region and its adjacent vertical region is defined as an active vertical region based on which of these differences in the second voltage drop is outside the first threshold. Specifically, there is a "movement" in the direction of the area with a larger voltage drop. On the other hand, if there is only one vertical region adjacent to the determined vertical region (meaning the candidate vertical drop line is actually at the leftmost edge or the rightmost edge), the area between the determined vertical region and one adjacent vertical region is defined as an active vertical region.

[0154] A check may be made to determine whether a vertical descent line is found. If a descent line is not found, the process may return to step S1604. In this manner, the system simply waits for another possible perturbation event.

[0155] If it is determined that a likely touch event has occurred, essentially the same operations may be performed in the horizontal direction: in step S1610, candidate horizontal drop lines are determined based on the measured voltage drops. In this regard, a determination is made as to which horizontal region is associated with the largest measured voltage drop in the horizontal direction.

[0156] In step S1612, an active horizontal drop line is found based on the determined candidate horizontal drop line. As above, if the voltage drop difference between the determined horizontal area and its adjacent horizontal area is within the second threshold, the determined horizontal area is defined as an active horizontal area. Otherwise, if there are two horizontal areas adjacent to the determined horizontal area (meaning the candidate horizontal drop line is not at the top horizontal line or the bottom horizontal line), one of the two areas between the determined horizontal area and its adjacent areas is defined as an active horizontal area. In this case, one of the two areas between the determined horizontal area and its adjacent horizontal area is defined as an active horizontal area based on which of these differences in the first voltage drop is outside the second threshold. On the other hand, if there is only one horizontal area adjacent to the determined horizontal area (meaning the candidate horizontal drop line is actually at the top horizontal line or the bottom horizontal line), the area between the determined horizontal area and one of its adjacent horizontal areas is defined as an active horizontal area.

[0157] The flow chart of FIG. 16 shows that active vertical regions are determined before active horizontal regions. However, in different exemplary embodiments, this order may be reversed. Also, while "horizontal" and "vertical" are used herein and elsewhere, it should be understood that both herein and elsewhere, a perfectly arranged table with precise rectangular rows and columns need not be used. That is, terms such as "horizontal", "vertical", "table", "grid", "cell", etc. should be understood to allow for any suitable arrangement, including a rectangular table with rectangular cells, an arrangement with a honeycomb or other tessellation of cells, etc.

[0158] It should be noted that the first and second thresholds in the exemplary approach of FIG. 16 may be the same as or different from each other. These threshold parameters may be calculated based on static values ​​extracted during steady state reference measurements when the stabilized values ​​are measured. They may be determined or at least partially influenced by the substrate size, conductivity, total number of electrodes, placement of electrodes (distance between, etc.), etc. In certain exemplary embodiments, the thresholds may be calculated based on touch events and non-touch events. That is, the non-touch measurements may provide the stabilized values ​​as detailed above. The touch generates a voltage, the difference of which may provide the threshold.

[0159] In step S1614, the location of the perturbation to the interface is determined from within the overlap area between the defined active horizontal and vertical regions. As with step S1118 above, this can be performed using image processing techniques, such as, for example, center of gravity, momentum, minimum triangle calculations, etc. Additionally, the approach outlined in FIG. 12 may also be used in certain exemplary embodiments.

[0160] As mentioned above, this discrete loop approach can be implemented unidirectionally or bidirectionally. With regard to the former, for example, current can be applied to electrodes in a unidirectional loop around the substrate, where the loop includes a first phase in which current is applied to electrodes forming vertical regions and a second phase in which current is applied to electrodes forming horizontal regions. With regard to the latter, for example, current can be applied to electrodes in a bidirectional manner around the substrate, where the first and second loops each include a first phase in which current is applied to electrodes forming vertical regions and a second phase in which current is applied to electrodes forming horizontal regions, where the first and second loops are the inverse of the first loop.

[0161] The following pseudocode can be used to implement a related discrete loop approach:

[0162] JPEG2024508163000002.jpg131136

[0163] In this example, there are 5 vertical lines and 5 horizontal lines, so 10 stabilization and drop values ​​are taken. As will be appreciated from the above discussion, these horizontal and vertical regions may be predefined on each side of each opposing electrode pair.

[0164] Using the example of FIG. 17, for example, if the voltage measurement of VL02 is less than the voltage measurement of VL01 (considering the respective stabilization values), it may be determined that the active drop line is VL02. In this case, a determination is made as to whether the touch event likely occurred in VL02, in one of the areas adjacent to VL02 (between VL02 and VL01, or between VL02 and VL03), or not at all. If the difference in voltage drop between VL02 and the neighboring vertical areas VL02 and VL01 is not within a threshold, the active vertical area AVR is not VL02 itself. Thus, a determination is made that the active vertical area AVR is in one of the areas between VL02 and VL01 or between VL02 and VL03. Because the voltage drop is greater to the left of VL02, the area between VL01 and VL02 (i.e., AVR02) is determined to be an active vertical area. For example, assume there is a threshold of 20 mA. If the active line is at 100mA, the right line is at 90mA, and the left line is at 160mA, then the touch location is to the left of the active line and in the area in between.

[0165] These operations are essentially repeated for horizontal rows. In this regard, for example, if the voltage measurement of HL02 is less than that of HL01 (taking into account the respective stabilization values), the active drop line may be determined to be HL02. In this case, a determination is made as to whether the touch event likely occurred in HL02, in one of the areas adjacent to HL02 (between HL02 and HL01, or between HL02 and HL03), or not at all. If the difference in voltage drop between HL02 and the neighboring horizontal areas HL02 and HL01 is not within a threshold, the active horizontal area AVR is not HL02 itself. Thus, a determination is made that the active horizontal area AHR is in one of the areas between HL02 and HL01 or between HL02 and HL03. Because the voltage drop is greater above HL02, the area between HL01 and HL02 (i.e., AHR02) is determined to be the active horizontal area.

[0166] The touch location is found to be at the intersection of AVR02 and AHR02, so a grid can be applied to this sub-region of the interface and processing can proceed as above.

[0167] Instead, for example, if the voltage measurement of VL01 is less than the voltage measurement of VL02 (again considering the respective stabilization values), it is assumed that the active drop line is VL01. In this case, a determination is made as to whether the touch event is likely to have occurred in VL01, in one area adjacent to VL01 (between VL01 and VL02), or not at all. The active vertical area AVR is not VL01 itself, since the difference in voltage drop between VL01 and the neighboring vertical area VL02 is not within the threshold. Thus, a determination is made that the active vertical area AVR is in the area between VL01 and VL02. Because VL01 is at the leftmost edge of the interface, error checking can be simplified. Thus, the area between VL01 and VL02 (i.e., AVR02) is determined to be the active vertical area. A similar explanation applies when the active drop line is HL01 instead of HL02. Again, a grid or other approach can be applied to further define the location of the touch-related interaction. Exemplary Balanced Path Algorithm

[0168] A fourth approach for detecting interactions (e.g., touch, hover, gesture, or other interactions) with an article of manufacture will now be described in connection with Figures 28-40. As noted above, the algorithm can be unidirectional or dual-directional. More specifically, Figures 28-33 demonstrate generally how this approach operates, and Figure 35 is a flow chart summarizing the operation of this exemplary approach.

[0169] As will be appreciated, the article of manufacture includes a conductive region, for example, as defined by the outer box in Figure 28. Of course, the conductive region can take on different sizes, shapes, and dimensions in different exemplary embodiments. It may be provided in association with an interface, for example, the conductive region may be all or part of that interface. Similarly, electrodes may be provided on a display, on a wall or floor, within a defined area of ​​a warehouse or other location, etc., depending on the end use. The electrodes are positioned in electrical contact with the conductive area. In this example of FIG. 28, 16 electrodes are provided around the perimeter of the conductive area (see step S3502 of FIG. 35). It will be appreciated that more or fewer electrodes may be provided in different locations relative to the conductive area, e.g., provided that they are nevertheless electrically connected to the conductive area.

[0170] Horizontal and vertical balanced paths are determined for the conductive region (see step S3504 in FIG. 35). These balanced paths are shown in dashed lines in FIG. 28. The balanced paths may be determined using opposing electrode pairs. For each defined opposing electrode pair, a current is applied to a first electrode in each pair, a first voltage-related value is measured for a second electrode in each pair, a current is applied to a second electrode in each pair, and a second voltage-related value is measured for the first electrode in each pair. A horizontal and vertical balanced path is defined to exist along the conductive region when the first and second voltage-related values ​​are equal.

[0171] In certain exemplary embodiments, the opposing electrode pairs used in determining the horizontal and vertical balance paths may be electrode pairs that are directly opposite each other, and / or at least some electrode pairs that are not directly opposite each other. With respect to the former, in the example of FIG. 28, electrode pairs 1-9, 2-8, 3-7, 4-14, 5-13, 6-12, and others are directly opposite each other. With respect to the latter, electrode pairs that are not directly opposite each other may be used as slave lines in calculating the horizontal and vertical balance paths, for example, to increase the accuracy of determining where the balance paths are located. In this regard, the opposing electrode pairs used in determining the horizontal and vertical balance paths may include opposing electrode pairs that include first and second electrodes that are directly opposite each other, and slave electrode pairs that are not directly opposite each other, where the slave electrode pairs include a first electrode of a mating opposing electrode pair and a third electrode that is different from the second electrode of each mating opposing electrode pair. In this case, the third electrode of each slave electrode pair may be adjacent to the second electrode of each mating opposing electrode pair. However, it will be understood that "directly adjacent" is not required. For example, one standard slave line or a set of standard slave lines could be used.

[0172] Although the balanced paths shown in the example of FIG. 28 are linear, it will be understood that this may not be the case in actual implementations. The balanced paths may depend, for example, on where the electrodes are located relative to the conductive regions, differences in the conductivity of the conductive regions caused by environmental factors (e.g., the device on which the electrodes are located, the operation of other equipment that may cause EMI, etc.), manufacturing variations in the conductive regions and / or electrodes, etc. Thus, even when the electrodes are equidistantly placed around a regular shape (which, as will be clear from the description, is not required for all embodiments), the horizontal and vertical uniform paths may be non-linear and / or may not be in the exact horizontal and vertical centers of the conductive regions. Instead, they may be representable as splines, curves, or other shapes in some cases. FIG. 34 shows, in a schematic manner, different exemplary horizontal balanced paths that may result in different exemplary embodiments. It will be understood that the maximum voltage value is at the source electrode and the minimum voltage value is at the receiver electrode.

[0173] Characterization of voltage values ​​is performed for the area from the balance path to the electrodes on the parallel axis for measurements along the vertical axis (see step S3506 in FIG. 35). That is, (1) for a vertical balance path, a horizontal reference voltage value is characterized for the area between the vertical balance path and the electrodes disposed vertically along the conductive region, and (2) for a horizontal balance path, a vertical reference voltage value is characterized for the area between the horizontal balance path and the electrodes disposed horizontally along the conductive region. Referring to FIG. 28 for a more specific explanation, the vertical balance path extends between electrodes 1 and 9, and the characterization identifies voltage drops as discrete locations such as between lines 1-9 and 2-8, between lines 1-9 and 3-7, etc. It will be appreciated that lines 1-9, 2-8, and 3-7 are oriented vertically, but the voltage drops indicate horizontal locations across the interface. These voltage drops may be used as reference voltages in certain exemplary embodiments.

[0174] In certain exemplary embodiments, the horizontal reference voltage values ​​may be characterized such that zero values ​​are provided adjacent to the vertical balance path and maximum values ​​are provided adjacent to electrodes disposed vertically along the conductive region, and the vertical reference voltage values ​​may be characterized such that zero values ​​are provided adjacent to the horizontal balance path and maximum values ​​are provided adjacent to electrodes disposed horizontally along the conductive region. In some cases, this results in the reference values ​​being reflected with respect to the balance path. However, in cases where the balance path is non-linear, this relationship may not hold. For example, the maximum value may vary from line to line or on both sides of a single given line. In any case, the characterization of the horizontal reference voltage values ​​and the vertical reference voltage values ​​may be performed using interpolation, e.g., a non-zero interpolation approach used to find non-coincident extreme values. It will be appreciated that non-zero order interpolation may be applied between different discrete measurements taken horizontally and vertically for the vertical reference voltage values ​​and the horizontal reference voltage values, respectively.

[0175] Similar to the approach described above, the maximum drop line is found. For example, for each electrode arranged horizontally along the conductive region, a current is applied to the electrode and the voltage drop is measured at the opposing electrode, such that the current is applied and measured for a different vertically opposing electrode pair (see step S3508 in FIG. 35). FIG. 29 shows the measurement made along the vertical drop line. The balance path is still shown for reference. From the different vertically opposing electrode pairs, determine the first electrode pair associated with the maximum voltage drop in the vertical direction (see step S3510 in FIG. 35). For the interaction in the blue circle area shown in FIG. 29, the 2-8 vertical line is identified, as shown in FIG. 30.

[0176] For each electrode positioned vertically along the conductive region, a current is applied to the electrodes and the voltage drop is measured at the opposing electrodes such that current is applied and measured for a different horizontal opposing electrode pair (see step S3512 in FIG. 35). FIG. 31 shows measurements made along a horizontal drop line. The balance path is again shown for reference.

[0177] From the different horizontal opposing electrode pairs, determine the second electrode pair associated with the maximum voltage drop in the horizontal direction (see step S3514 in FIG. 35). For the interaction in the blue circle area shown in FIG. 31, the 4-14 horizontal line is identified, as shown in FIG. 32.

[0178] It will be appreciated that current application and voltage measurement may be performed bidirectionally for electrode pairs. For example, current may be applied to electrode 1 and voltage may be measured at electrode 9, or current may be applied to electrode 9 and voltage may be measured at electrode 1. In certain exemplary embodiments, each electrode will have at least one voltage measurement that may be generated in the opposite direction (and, for example, neighboring or other electrodes may be used if greater accuracy is desired, as described in more detail below). Additionally, it will be appreciated that electrode pairs are searched for (e.g., in steps S3510 and S3514), but the electrode pairs found help define vertical and horizontal drop line regions for additional operations that may use more electrodes than the determined electrodes in the vertical and horizontal directions.

[0179] In practice, the sub-region is defined based on at least the first and second electrode pairs. That is, in certain exemplary embodiments, the above-mentioned measurements identify the line and the general direction of interaction with respect to the line (e.g., left or right of the vertical descent line, and above or below the horizontal descent line). For example, if the measured voltage value is below the reference value associated with the identified descent line, the location is likely to be closer to the respective balance path, whereas if the measured voltage value is higher than the reference value associated with the identified descent line, the location is likely to be further from the respective balance path. This may indicate which adjacent electrode pairs help define the sub-region. In certain exemplary embodiments, the sub-region may be defined using the horizontal balance path and the vertical balance path, but this is not necessary.

[0180] The sub-regions in the example of FIG. 33 are bounded by lines 1-9, 2-8, 4-14, and 5-13. The characterized horizontal reference voltage values ​​are used to calculate a horizontal position of the interaction from within the defined sub-region based on a horizontal distance from a vertical balance path (see step S3516 of FIG. 35). This is the 4-14 based measurement shown in FIG. 33. Similarly, the characterized vertical reference voltage values ​​are used to calculate a vertical position of the interaction from within the defined sub-region based on a vertical distance from a horizontal balance path (see step S3518 of FIG. 35). This is the 2-8 based measurement shown in FIG. 33. It will thus be appreciated that the horizontal position of the interaction is further based on the voltage drop measured using the first electrode pair, and the vertical position of the interaction is further based on the voltage drop measured using the second electrode pair.

[0181] The determined horizontal and vertical positions are output (see step S3520 of FIG. 35). These points can be used to determine interaction details (e.g., x, y, and / or z coordinates of the interaction, time, etc.) as well as interaction type (e.g., touch, hover, gesture, etc.). Data regarding the interaction may be collected and used, for example, for gesture determination and / or analysis, as disclosed herein. It will be appreciated that proximity-based detection may be performed as a variation of gesture and / or hover related functions. As discussed above, different types of interactions can be detected using the algorithmic approaches described here and elsewhere herein.

[0182] It will be appreciated that the algorithm may be executed continuously or on a more on-demand basis in different exemplary embodiments. For example, the sub-process shown and described in connection with steps S3508-S3520 may be executed more or less continuously, as needed. In certain exemplary embodiments, this sub-process may be executed only in response to an initial, coarser detection that a possible interaction has occurred.

[0183] It should be understood that while FIG. 35 illustrates a process that includes device configuration and characterization through detection, explicitly contemplated herein are the concepts of "passive" enabling detection at a later period of time, and "active" configuration and / or characterization with "active" detection and assuming earlier pre-configuration and / or pre-characterization.

[0184] As mentioned above, the slave line may be used for position determination in certain exemplary embodiments. In this regard, the horizontal position of the interaction may be further based on a voltage drop measured using a third electrode pair different from the first electrode pair, and the vertical position of the interaction may be further based on a voltage drop measured using a fourth electrode pair different from the second electrode pair.

[0185] In certain exemplary embodiments, the third electrode pair may include one electrode of the first electrode pair and another electrode not included in the first electrode pair, and the fourth electrode pair may include one electrode of the second electrode pair and another electrode not included in the second electrode pair. In certain exemplary embodiments, the third electrode pair may include one electrode of the first electrode pair and another electrode adjacent to the other electrode of the first electrode pair, and the fourth electrode pair may include one electrode of the second electrode pair and another electrode adjacent to the other electrode of the second electrode pair. In certain exemplary embodiments, none of the electrodes of the first electrode pair are included in the third electrode pair, and none of the electrodes of the second electrode pair are included in the fourth electrode pair. The reference voltage may be pre-characterized for a set of possible third and fourth electrode pairs, for example, as described above. In short, it will be understood that one or more slave lines may be used to improve accuracy, and that the slave lines may, but do not necessarily, include adjacent electrodes and / or adjacent slave lines. For example, a dedicated slave line may be used, all possible slave lines may be used, all or some of the slave lines in a parallel direction may be used, etc.

[0186] Figure 36 is an example showing how a slave line can be used to improve accuracy, according to certain exemplary embodiments. In the example of Figure 36, the vertical balance path is shown to be slightly biased. Thus, if a master measurement is made based on the 4-14 line and a slave line is used along with the 5-13 line (or because the slave is configured as the horizontal balance path or adjacent line), the voltage related values ​​will be slightly different as a result of the bias related pre-characterization.

[0187] FIG. 37 is a variation of an exemplary matrix approach that may be used to improve accuracy, according to certain exemplary embodiments. In this exemplary matrix approach, pre-characterization is performed on neighboring electrodes. For example, in addition to pre-characterization on directly opposed electrode pairs, pre-characterization is completed on 3-8, 2-9, 1-10, 16-11, 2-7, 1-8, 16-9, 15-10, etc. The maximum horizontal and vertical descent lines are still determined and these are taken as master measurements. Then, one or more neighboring electrodes are used for slave line measurements. In the example of FIG. 37, the vertical descent line is 2-8, and since the interaction is likely to be to the left of this line, the neighboring electrode 3 is used as an additional source (e.g., for the 3-8 line, and potentially for the 3-7 line and / or others). Thus, the slave lines in this matrix approach can be used to confirm or refine the position detection.

[0188] As explained above, the balance path changes based on, for example, the placement of the electrodes. In general, the horizontal and vertical balance paths intersect at the "center of gravity" of the electric field generated by the currents supplied to the different electrode pairs. As a result, the center of gravity may be off-centered relative to the conductive region. In general, the center of gravity is biased toward the direction of higher electrode density, for example, as shown in FIG. 38. With this recognition, it is possible to use a system with only three electrodes. The center of gravity can be located, and then, for example, the electric field can be bisected to generate a balance path that intersects at the center of gravity. FIG. 39 is a diagram showing an example three-electrode system with an example center of gravity and an example balance path, which may be similar to certain example embodiments. The system of FIG. 39 can function similarly to the above, for example, AB, AC, BA, BC, CA, and CB lines are used to locate the interaction after pre-characterization has been performed.

[0189] As described herein, this and other approaches can include electrodes provided in different locations than those shown generally in the drawings. These locations include locations around the edges of the conductive region, electrodes provided on the periphery of the conductive region, electrodes provided within or on the conductive region (including possibly the central area), etc. The electrodes may be regularly or irregularly spaced, may be provided in rows and columns or other configurations, or may be provided around only two sides (e.g., opposing or adjacent sides) of the conductive region.

[0190] FIG. 40 is an example arrangement in which electrodes are provided only on opposing edges of the conductive region, according to certain exemplary embodiments. Balanced paths can still be developed and lines of descent can still be determined. For example, measurements can be taken including 3-9, 2-10, etc. to develop a vertical balanced path. The algorithmic approach described in this section can work with this arrangement. Thus, it will be understood that "horizontal and vertical" as used herein need not be used strictly in this approach. Instead, orthogonal or simply separate balanced paths may be used in different exemplary embodiments. For example, there may simply be two different paths on a coordinate system, and the paths may be defined by electrodes.

[0191] In certain exemplary embodiments, the conductive region may include a conductive coating, and the electrodes may be formed in or electrically connected to the conductive coating. In certain exemplary embodiments, the conductive coating may be provided over the entire conductive region. Thus, the conductive region may include a surface on which a conductive coating is formed, which may be, for example, a thin film coating, or may include conductive elements provided in a liquid that is applied onto the surface. Alternatively, or in addition, the conductive region may include a surface in which conductive particles are embedded and / or provided. The conductive surface may be a floor, wall, ceiling / roof and / or the like within a building and / or vehicle, a part of a machine, etc.

[0192] It will be appreciated that this algorithmic approach, as well as others described herein, may be used in connection with a wide variety of systems, for example, where a hardware processor is provided to perform the corresponding operations, where it will also be appreciated that, as with other approaches described herein, non-transitory computer-readable storage media tangibly storing instructions that, when executed by a processor, are configured to perform these operations are also contemplated.

[0193] It will be understood that while certain operations are described as being horizontal and vertical in the various algorithmic approaches described herein, other "reference orientations" may be used for any of the approaches depending, for example, on the geometry of the interface, conductivity, electric fields, etc. Similarly, the order of operations may be varied in the various algorithmic approaches described herein (e.g., horizontally related operations may be performed first and vertically related operations may be performed second, or vice versa). Exemplary Gesture Recognition Related Techniques

[0194] FIG. 18 illustrates generally how points may be collected and used for gesture recognition in certain exemplary embodiments. Surface interactions are monitored continuously using hardware 1802, for example, as described above in connection with FIG. 7. A tracking engine 1804 tracks voltage-related inputs and applies point-related decision-making logic 1806 to determine if and when an interaction has occurred. Point-related calibration 1808 aids in this regard. The tracking engine 1804 may implement some or all of one or more of the algorithmic approaches described above. The point-related calibration 1808 may include information such as, for example, interface characterization data (e.g., size, sheet resistance / conductivity, etc.), mask characterization information (e.g., mask size, drop ratio, etc.), etc.

[0195] The interactions registered by the tracking engine 1804 are provided therefrom as output 1812 to the classifier engine 1810. The output 1812 from the tracking engine 1804 includes position data (e.g., x and y coordinates) and a timestamp associated with the interaction. When the tracking engine 1804 is extended to handle hovering motions, the z coordinate may also be output. Thus, the output 1812 includes at least x, y, and t data.

[0196] The output 1812 is collected and provided to gesture-related decision-making logic 1814, which in certain exemplary embodiments, for example, consults a computed MF model database 1816 to determine if a gesture has been input. That is, an IS model may be generated from the output 1812 from the tracking engine 1804 and then analyzed to determine if it matches any trained models referenced in the model database 1816.

[0197] The output from the classifier engine 1810 is then provided to an end application 1818 for control or other purposes. For example, lights may be turned on / off based on the touch or swipe, their brightness may be adjusted, etc. In addition to lighting control, gestures may be used in smart tables (e.g., to call a waiter), on smart whiteboards (e.g., to draw or clear the screen), and in automotive applications (e.g., to open or close a door or trunk). In some exemplary embodiments, surface touch points may be collected via operation of a first thread (corresponding to the tracking engine 1804) and then registered for analysis and gesture recognition via operation of a second thread (corresponding to the classifier engine 1810). Of course, different exemplary implementations may perform these tasks serially, for example, if parallel processing is not possible. Exemplary Hovering-Related Techniques

[0198] As mentioned above, the techniques described herein can be used to track, recognize, and take action based on hover-related interactions with an interface. Similar electrical principles as those described above apply with respect to this three-dimensional related interaction tracking. That is, hover-related interactions can be made possible by using electromagnetic fields to understand Z-distance to a surface, essentially considering the relationship between voltage drop and distance to a surface.

[0199] 19A-19C help explain how hover-related interactions can be managed according to certain exemplary embodiments. As shown in FIG. 19A, for example, a distance d between the conductive surface of the interface and the hover causes a voltage drop in the z direction (perpendicular to the x and y directions) due to the generated electric field. The amount of voltage drop is detected using hardware described herein. That is, a current (e.g., up to 500 mA, suitable for large interfaces) is applied to the surface of the substrate. A reference voltage is pre-characterized relative to the substrate, for example, for different positions on the substrate. When a hover interaction occurs, a voltage drop is triggered.

[0200] The algorithmic approach described above can be adapted to understand the hover interaction location. With only minor modifications, the distance d to the substrate can be understood. However, it should be noted that the resolution may be distance dependent unless further algorithmic or other modifications are made. The distance resolution is related to the applied electric field. As shown in FIG. 19B, a hover interaction at a distance d1 can be located within a radius r1 on the substrate. However, for a hover interaction at a distance d2 greater than d1, the location is within a radius r2 greater than r1. This is because the voltage drop reflects an inverse curve characteristic of the voltage drop axis versus the surface non-contact distance axis, as shown, for example, in FIG. 19C. However, it should be noted that the distance detection can be adjusted in several ways. For example, properties can be set to determine the distance range used specifically for a particular application. Thus, interactions can be assumed to be within a particular distance to locate them more accurately, and vice versa.

[0201] It has been found that hovering-related functionality may be enabled using the exemplary hardware and software setup described herein for large surfaces. For example, for a surface having a 67 inch diagonal, the hovering / proximity-related sensitivity has been determined to be approximately 1 meter. Sensitivity may be affected by factors such as the size, shape, number of electrodes of the conductive surface, etc. Sensitivity may be increased to approximately 1-2 meters (and possibly more) even for a 67 inch diagonal sensing surface (for example) by providing complementary coatings on the sensing surface and the sensed surface. For example, complementary components may be provided on the sensing surface (e.g., a robot or other machine) and the human to help increase the distance over which hovering / proximity-related measurements may be made. Exemplary Measurements

[0202] 20A-22 help illustrate what types of information can be gathered using the exemplary hardware described herein and how the information can be used in conjunction with the exemplary algorithms described herein. In this regard, FIGS. 20A-20B provide information on how voltage-related measurements can be used in conjunction with the discrete loop algorithmic approach described herein, according to certain exemplary embodiments. As shown in FIG. 20A, for example, 16 electrodes are provided around the surface. When the surface is perturbed, a voltage drop is triggered. Measurements (in millivolts) are provided for each of the 16 electrodes, as well as associated drop values. Measurements are indicated with the prefix "Measurement-" and a suffix of the corresponding electrode. Similarly, drop values ​​are indicated with the prefix "dV-" and a suffix of the corresponding area. The interface in this example is divided into nine active vertical regions (AVR-01, AVR-02, ..., AVR-09) and five active horizontal regions (AHR-01, AHR-02, ..., AHR-05). The highlighting under AVR-02 and AHR-04 indicates that these are the active lines where touch occurred. Figure 20B visualizes the board, highlighting these active lines and showing the overlap areas where touch-related interactions likely occurred.

[0203] FIG. 21 illustrates voltage-related signals collected in association with a simple touch interaction using certain exemplary embodiment hardware, and FIG. 22 illustrates voltage-related signals collected in association with a swipe interaction using certain exemplary embodiment hardware. In these graphs, the x-axis illustrates time and the y-axis illustrates voltage. The colored lines in the graphs represent measurements taken at each of the electrodes. As can be seen in both FIG. 21 and FIG. 22, the voltage measurements are substantially constant until there is a large dip indicating an interaction. The duration of the dip is fairly short. Their profiles are different due to the presence of different interaction types. That is, the dip is substantially U-shaped in FIG. 21, which indicates a single touch-on / touch-off event. In contrast, in FIG. 22, the dip has a sloping trough, indicating a swipe interaction. Example Applications

[0204] The techniques described herein can be used in connection with a wide variety of different end user applications or end uses. These applications can span across a variety of different industries. Furthermore, the hardware architecture and algorithm related teachings disclosed herein can be used in both touch and non-touch related applications, for example, in a number of different interaction and / or detection related applications. At a high level, one can provide more control as a matter of convenience (e.g., in connection with touch sensitive interfaces), can detect motion and / or human presence to be more proactive in terms of actions to be taken (e.g., to avoid obstacles in a setting), can result in more hygienic interactions on shared surfaces when hover interaction and / or gesture features are enabled (e.g., in various health related domains), can gather insights for workplace safety and efficiency (e.g., by collecting data on how individuals in a factory are working), etc. In general, the techniques of certain exemplary embodiments can provide control signals, for example, by interacting with walls, surface tops, dashboards, or other objects in a manner that is more easily accessible than today's buttons, switches, and / or displays, "interactions and movements" can be detected so that actions (manual and / or automatic) can be taken, etc. In health care environments (for example), where infection may be a concern, contactless interaction may be useful in situations involving shared control panels such as those found in elevator interfaces, environments where dirty hands typically contact surfaces such as kitchens and utility areas, etc. Touch sensing related applications may be useful in automotive, aerospace, health care, education, transportation, consumer products, electronics (including appliances), building construction and styling, office environments, information displays, and other industries. Non-touch sensing related applications may be useful as an alternative or supplement for proximity sensors, zone / area detection, human detection, the health care industry, etc. where contactless, touch-like interaction is desirable. The technology may be used in retail stores, public areas (e.g., airports, trade shows, entertainment venues, etc.), hospitals, educational environments, homes, workplaces, etc.

[0205] The uses in these applications and industries are wide and varied. Additional sensors and / or technologies can be used to provide further capabilities for extending to other applications or industries or to further extend the examples described herein. Thus, certain exemplary embodiments may be provided within or as part of a larger sensor ecosystem. For example, certain exemplary embodiment techniques can be augmented with light detectors, IR detectors, thermometers, humidity sensors, as well as various Internet-of-Things (IoT) and / or other sensors, radar, LIDAR, light sensing, and the like. As an example, in automotive and other vehicle applications, the number of features offered can be increased since there is a rich set of sensor technologies used. In this regard, new generation vehicles are controlled with advanced controller technologies to sense the environment for automatic parking, safety, autonomous driving, and the like. As another example, body tracking can be used to determine if a seat belt is engaged, if the driver appears to be asleep, and the like. Using the technology described herein, liquids (including gases, oil, wiper fluid, brake fluid, and the like) can be accurately measured and airbags can be activated to protect the driver and passengers, and the like. An on-board processor can track the driver's movements (e.g., head nodding indicating loss of consciousness, shaking indicating complete loss of consciousness, etc.) and warn of possible danger. This can be considered a gesture tracking problem. ML can be used to adapt configurations related to tracked driver actions for performance.

[0206] With so many integrated technologies offered inside a vehicle, there are many areas where surface sensing techniques can be used. This includes any kind of control function on a small or large surface, including navigation, temperature, audiovisual entertainment, gear shifting, windows, locks, airbag engagement, etc. Some steering wheels are currently tracked using simple but expensive force sensors, and the techniques described herein can be used in place of such technology. For example, the techniques described herein can be used in place of sensors that track hand movements for the purposes of adaptive steering control, to understand how a person is driving in general, to infer when a person has been driving for a long time and may be fatigued, etc. The techniques disclosed herein can be used on windshields to track environmental conditions, damaged areas, etc., for example, to replace existing more expensive sensors while providing the ability to provide full surface control / interactivity.

[0207] There are a variety of different types of hovering-related or other non-touch / contactless applications that can be achieved using different hardware platforms. For example, hovering-related applications may include multi-point distance sensing without XY sensing (e.g., as a replacement for other sensors that cannot be used on large surfaces and / or are very expensive), low-range distance sensing and high-precision XY sensing, medium-range distance sensing and medium-precision XY sensing, high-range distance sensing and low-precision XY sensing, etc. Exemplary Industrial Safety Applications

[0208] The technology described herein is applicable to industrial safety applications. For example, the technology described herein can be used to track equipment (e.g., forklifts, dollies, etc.) and personnel (e.g., using EIT technology on floors, individuals, and / or equipment that may be mobile or stationary, etc.), provide alerts when objects get too close to various degrees of hazard (e.g., "heat processing" and / or "hot" zones, etc.), issue alerts to pinch / injury risks and / or to stop / adjust equipment such as robots or other automated machines to help reduce the possibility of injury, generate alerts when moving equipment (e.g., forklifts) come within (e.g., too close) of personnel and / or when personnel enter an area of ​​potentially moving equipment, provide alerts when objects come within (e.g., too close) of energized and / or de-energized surfaces, etc. Certain exemplary embodiments provide retrofit safety solutions that can be used with existing machines, layouts, and / or the like. Certain exemplary embodiments provide a custom approach to accommodate new equipment, layouts, and / or the like. In both new and retrofit situations, the techniques disclosed herein can be used to supplement or potentially replace existing safety-related systems, for example, by providing "blind area" detection that may exist in or around equipment or access passageways, for example, when items are blocking other sensors (such as laser sensors or other optical imaging means), when there are "blind spots," etc.

[0209] Generally, one or more areas of interest are defined. Such areas may be defined by providing a conductive surface. The surface may be made conductive by having a conductive coating formed thereon (e.g., a thin film or other coating, a tape or other adhesive, paint, etc.) and / or by having a conductive material embedded therein (e.g., in the case of floors, walls, or other surfaces). In certain exemplary embodiments, conductive mats / carpets, pads, decals, and / or the like may be used. The conductive surface may be provided as a "skin" for the robot, cobot, or other machine. It will be understood that the "skin" may be pained, wrapped, or otherwise coated in this and / or other embodiments to form a sensing surface. The skin may be provided on the entire device, only the moving sections, portions of the section that may cause a particular type of injury, etc. It will be understood that "soft goods" may be used for the conductive areas in certain exemplary embodiments, and additionally or alternatively, certain exemplary embodiments may utilize hard materials (e.g., pads, plates, flooring, roofing, and / or the like) that may be particularly suitable for industrial environments. In certain exemplary embodiments, it may be desirable to apply conductive material onto or integrate / embed conductive material into another surface during or after manufacture. In some cases, the sensing surface may be visually perceptible to humans using aesthetic signals or cues to inform people where sensing areas are, and as a result, where sensing areas may be present.

[0210] One or more electrodes are provided on the surface (e.g., formed on, around, or in a conductive surface). In general, more electrodes provide higher sensitivity and potentially better definition. In some cases where simple touch on / off or overall proximity detection is desired, one or two electrodes may be used. In some cases, the range may be expanded, for example, by providing "transponders" on different items. For example, a person may be asked to wear a transponder that transmits a specific electrical signal that can be detected. The controller is programmed to recognize different interaction types with respect to the area of ​​interest. The interaction types may include, for example, hover / proximity, touch, gesture, and / or other actions. They may be predefined and / or machine learned. Different interaction types may be specified for different areas. The controller is also programmed to take different actions, potentially per interaction and / or per area. For example, pause / stop, start / enable start, slow down, raise an alarm, alarm authority, and combinations of these and / or other actions may be programmed.

[0211] Further exemplary use cases are described in more detail below in relation to specific industrial safety applications. In particular, the techniques disclosed herein may be used to provide industrial safety solutions for areas such as, for example, machine guarding, robots and mobile equipment, loading and shipping, access and fencing, and / or other contexts, as discussed in more detail below. It will be appreciated that certain exemplary embodiments may be provided to overlap and / or complement existing solutions. This may be useful, for example, to provide redundancy for certain safety applications. A redundant system implementing the techniques disclosed herein may help ensure safety ratings, for example, by providing fail-safe redundancy in the event that a single system has a point of failure.

[0212] In this regard, the exemplary approaches provided herein can be used in duplex, triplex, etc., for redundant detection of the same event in the same area. In such cases, different systems can function simultaneously and outputs are compared to see if an event has occurred. Alternatively, another system can be activated if an error or malfunction is detected in the operating system (e.g., if an event is not reported within a predetermined time, if a problem is detected due to a circuit not responding, etc.). Additionally or alternatively, redundancy can be provided for different safety system related or other technologies. For example, one safety system may implement laser-based detection of hazards, etc., and the exemplary techniques disclosed herein can be used to augment or provide redundancy for such systems (e.g., in low light conditions or focal lengths where laser-based or other technologies are difficult). In such cases, the systems can function simultaneously or when another system begins to generate invalid or unreliable outputs (and / or stops generating valid or reliable outputs). Exemplary Machine Guard Related Technology

[0213] Safety issues may exist when operators and / or other personnel operate or interact in relatively close proximity to hazardous areas. Certain exemplary embodiments relate to retrofit and / or self-installation solutions, as well as other solutions integrated / embedded during manufacturing / construction. That is, by providing modular accessories that can detect touch, proximity, and / or other interactions, it is possible to detect humans or other objects that may be too close to the machine. The machine may then be controlled to de-energize itself (e.g., stop / pause, slow down, reverse, etc.). FIG. 23A illustrates a machine with a modular detection system based on the technology disclosed herein installed, according to certain exemplary embodiments. In the example of FIG. 23A, a modular accessor may be installed. In this case, a generally rectangular area is defined, as shown in the dashed area with diagonal lines. When a person or object approaches or touches this area, as detected using the proximity, hovering, and / or touch-related features disclosed herein, the machine may be controlled to improve the overall safety situation. To enable this operation, modular components may be installed to the left, right, above, or below the dashed shaded area, or may be installed elsewhere, but an area of ​​interest may be defined that "projects" and therefore protects the dashed shaded area.

[0214] Safety issues may also exist when workers access dangerous spaces or areas. It may be difficult to make large, complex, and / or older machines safer using current technology. The technology disclosed herein can be used to provide custom integration for such machines. FIG. 23B illustrates the disclosed technology used to create a protected area for a machine in a custom-made manner suitable for a specific other machine, according to certain exemplary embodiments. The technology disclosed herein can be used to create custom integration for upgrading an existing system, for example, with one or more multiple protected areas or zones using modular components. The shaded areas surrounded by dashed lines shown in FIG. 23B are areas that may be protected. Different actions may be taken for different areas, for example, according to custom or other programming of the controller. For example, the machine may be controlled to slow down or pause if a human is detected below it, while it may be controlled to stop completely if a human comes within a predetermined proximity of the operating area in front of the machine.

[0215] FIG. 23C illustrates the techniques disclosed herein used for active enabling of a defined area, according to certain exemplary embodiments. In some cases, a machine may be energized or powered unnecessarily. This can waste energy and / or create potentially dangerous conditions when a person leaves a workstation or other unattended area (e.g., because they forgot to turn the machine off, because they evacuated in consideration of an accident, and / or for other reasons). Thus, certain exemplary embodiments can detect when an operator is in or near a defined control area (see dashed shaded area in FIG. 23C). In certain exemplary embodiments, proximity detection can then be used to determine when a person is standing in the defined area, when they are being controlled, when they enter / exit the defined area, when they are alert in the defined area (e.g., collapse, sway, nod their head, etc.), etc. Control functions can include pausing or stopping operation when a person leaves the area, starting or enabling the machine when a person enters the area, etc. The controller may be programmed to perform these and / or other functions in different exemplary embodiments. Exemplary Machine Robotics and Mobile Device Related Technologies

[0216] Certain exemplary embodiments may be used to help protect workers from moving robots and vehicles. This may be particularly advantageous as automation increases. While some automated / autonomous systems are very safe when properly installed and when individuals are properly trained, there may be situations that are more potentially dangerous due to lack of familiarity with such machines. Even with machines that people are familiar with, and even with experienced individuals, there is often a tendency to simply ignore (e.g., walk past) potentially dangerous situations. People tend to take shortcuts by passing near dangerous machines when they think they have experience or feel the risk is low. Some people may walk up to dangerous machines to inspect them, simply because they are curious.

[0217] Some current industrial solutions rely on physical guards (e.g., cages, scaffolding, and / or the like) to protect humans from "heat." Such guards can help keep people away from forceful machines, machines that interact with hazards, and the like. FIG. 24A illustrates a "skin" applied to a robot, according to certain exemplary embodiments. This skin can be provided instead of or in addition to such physical guards. The skin can be integrated into the machine, the physical guards, and / or the like. In the example of FIG. 24A, the skin is shown as a diagonal line within a dashed line and is provided on the robot itself. The operation (e.g., movement) of the machine to which the skin is applied can be slowed, paused, stopped, or otherwise controlled as programmed based on detected proximity or other interaction with a human or other object. For example, if a human unwittingly comes within a predetermined distance of a swinging robot arm (e.g., gets too close), the skin on the robot arm can provide a signal to a controller, which detects the presence of the human and safely stops the arm to help avoid contact between the human and the robot arm. Similarly, the skin of the robotic arm can provide a signal to a controller that detects the presence of a human and takes different actions based on how far the human is from the equipment. For example, a visual and / or audio warning can be triggered when a human comes within a first threshold distance, the machine can be slowed down when a human comes within a second threshold distance, the machine can be stopped when a human comes within a third threshold distance (e.g., within or near touch range and therefore considered too close), etc.

[0218] Sometimes, in some current industrial settings, gaps exist between areas where detectors are provided. For example, cages installed around machines can detect interactions with the fencing itself, leaving a large area where humans, animals, or objects can enter. Sometimes, mobile carts are daisy-chained together, and gaps can exist between different sections of such carts. Other examples exist. The technology disclosed herein can be used to help close these potentially dangerous sensing gaps. This technology can be used to augment and / or replace visioning systems that may leave gaps that can create risks. FIG. 24B illustrates, for example, the technology disclosed herein being used on a mobile cart in a manner that helps close vision gaps (e.g., blind spots), according to certain exemplary embodiments. That is, sensor technology of the type disclosed herein may be provided in the shaded areas shown with dashed lines. These areas can help detect issues such as under the wheels, between areas of the cart (e.g., driving area and cargo area), and / or between carts. The technology disclosed herein can be used as a retrofit solution in some cases. In other cases, it may be provided with the device and gaps that may otherwise exist may be known in advance based, for example, on known use, geometry, and / or other factors related to the device itself.

[0219] Certain exemplary embodiments may also be used in connection with cobots. Cobots can assist humans to perform tasks, for example, on a production line. However, cobot safety systems are still in their infancy, e.g., as cobot technology continues to develop. Sometimes very expensive pressure sensors and the like are incorporated to, for example, give the cobot more human-like programmed capabilities. However, these sensors may be replaced and / or supplemented with the techniques disclosed herein. In this and other industrial safety system related contexts, the proximity detection and even pressure sensing capabilities (simulated or real) enabled by certain exemplary embodiments are advantageous compared to force sensors, since, for example, the system can be trained or otherwise programmed to distinguish between different proximity, hitting a human (which may trigger a mitigation plan, such as moving away from the area, a safety system being activated, etc.), hitting an inanimate object and / or other object, such as a cardboard box (which may not require a change in plan or behavior, for example). For example, the techniques disclosed herein may be used to help control the movement of a cobot, for example, by helping to sense human presence and / or human movement. Such detection may help prevent contact with humans. Machine learning can be employed to help train cobots to understand how humans are likely to use them. For example, the sensing technology of certain exemplary embodiments can be used to help train cobots to follow movements in space, potentially without specialized equipment such as specialized "smart gloves." Instead, natural human movements can be sensed within the field projected by the cobot or in close proximity to the cobot. These movements can then be repeated by the cobot or translated into expected work instructions from a human working with the cobot. In turn, if out-of-the-ordinary movements are detected, the cobot can react accordingly, for example by stopping, pausing, sounding an alarm, to indicate a possible problem with the human collaborator.24C is an example skinned cobot (denoted by dashed lines with diagonal lines) that may implement the techniques disclosed herein for the above and / or other purposes according to certain example configurations, where omnidirectional sensing is possible, as well as other applications and use cases. Exemplary loading and shipping related technologies

[0220] The techniques disclosed herein may be used to provide safer and / or more automated loading / unloading and shipping related functions. For example, pedestrians are often "overlooked" when items are being moved, loaded / unloaded, and / or the like. This may occur in the context of automated guided vehicles (AGVs) that are not properly trained or cannot respond quickly enough, as well as machines operated by humans (e.g., the human may be obscured by the machine and / or the item being moved, causing confusion as the human's movements may be unpredictable, etc.). The techniques disclosed herein may also be useful for "blind area" detection, such as, for example, detection around corners or "penetrating" objects in operator "blind spots," low light or otherwise occluded or obscured areas where other sensors (such as laser-based systems) may not function. FIG. 25A illustrates the techniques disclosed herein implemented on a mobile loading machine, according to certain exemplary embodiments. That is, in this example, the sensing area may be defined as the shaded area inside the dashed line. Having a sensor here can potentially help detect humans or other objects whether they are in a sensing path, whether they are wearing a transponder, etc. Automatic control can take over to turn the machine around, slow or stop the machine, sound an alarm, etc. This technique can be used in conjunction with defined paths, transponders, etc. In general, however, the technique can be used to detect proximity and / or touch.

[0221] With many moving parts (human and machine parts), collisions can occur. For example, when loading / unloading in a shipping area, an object may need to move in and out of a doorway, loading dock, or other area. A crew may be required. However, the machine operator and individual crew members may not know what each other person is doing. This may be especially true when multiple doors, docks, and / or the like are in relatively close proximity (e.g., adjacent) to one another. FIG. 25B illustrates a defined shipping area provided in the floor and sensors defined on the machine, according to certain exemplary embodiments. Sensors and / or transponders may also be provided to humans working in the area to increase detection distance. The controller may be configured to control the machine moving the object to stop if it comes within close range of a protected area on the floor, into a person, and / or the like.

[0222] As perhaps alluded to above, some loading / unloading operations are performed automatically. For example, some automated loading machines move containers in and out of units. These automated loading machines tend to turn off sensors in tight spaces because they function poorly in these conditions. For example, LIDAR systems can be very susceptible to backscattering, visioning systems can function poorly in low light, etc. Thus, there can be gaps in sensing when at least some sensors are turned off. Certain exemplary embodiments employ sensor technology to improve upon these approaches. For example, loading equipment can have embedded sensors and conductive features can be used to guide the equipment in and out of areas. This can work because an electric field can be generated and the machine can be guided (e.g., controlled to steer in different directions) as a function of electrical signal strength / guidance as if there was proximity or gesture detection. 25C illustrates, for example, conductive and / or sensing areas (within the diagonal lines inside the dashed lines) provided outside the shipping area that can be used to guide a suitably equipped automated loader, according to certain exemplary embodiments. Pallet / inventory tracking technology can be complex and expensive, especially if one seeks to use the tracking system for security purposes as well. The approaches disclosed herein can provide either or both services in a potentially more cost-effective and efficient package. Exemplary Access and Fencing Related Technology

[0223] Certain exemplary embodiments provide access and fencing solutions that can help protect workers from potentially dangerous equipment. Such systems, in different exemplary embodiments, can issue alerts, alarms, and / or otherwise modify the operation of the equipment (e.g., by disabling them, controlling their movement, etc.).

[0224] With regard to machine guards, for example, some existing solutions include light curtains, which have known difficulties in being implemented with electric fields, fences that are expensive, inflexible, and known to be often easy to break, and the like. With regard to the latter, some current fencing approaches simply put a barrier in place. These barriers can be cost and space intensive, and may lack the ability to sense and act in a responsive manner. FIG. 26A illustrates a smart fencing solution that may be used in conjunction with or in place of an existing fence, in different exemplary embodiments. In FIG. 26A, only the area across the top of the fence is shown with sensing capabilities. This is because a physical barrier is provided around the other sides. In this case, the techniques disclosed herein may be useful in providing supplemental protection for existing guards, such as, for example, in FIG. 26A, where a side fence is provided through a physical barrier and additional detection capabilities are provided via the top sensing surface. However, it will be understood that some or all of the other parts of the existing fence may be supplemented or even replaced with the sensing technology of certain exemplary embodiments. A "smart fencing" solution may help a fencing system respond to situations in real time. For example, if a person or equipment is approaching a restricted space, an alarm can be generated, a door or gate can be closed, etc. If a person or equipment enters a restricted space, an action can be taken on the equipment within the space (e.g., shut down, pause, slow down, etc.).

[0225] In some cases, compliance status may be compromised by, for example, having cluttered areas. For example, the U.S. Occupational Safety and Health Administration (OSHA) has published regulations in some cases regarding access to different equipment, breakers, and / or the like. FIG. 26B illustrates how the technology disclosed herein can be used to provide real-time compliance-related updates, according to certain exemplary embodiments. That is, it is possible to define areas that need to remain uncluttered and use the technology disclosed herein to determine if there are items in the way of such areas. The geometry of these areas can be custom defined to meet the requirements of regulations, best practices, the specific structure of a building or other area, etc. Additionally, the system can be programmed or learned to sense when there is a temporary obstruction (e.g., a person or equipment legally crossing a route or temporarily standing in one place) and distinguish such a situation from a blockage that is less compliant. In some cases, compliance-based reporting can be automated.

[0226] FIG. 26C illustrates a defined walkway that may benefit from the techniques disclosed herein, according to certain example configurations. People may walk outside of the walkway or other designated safety areas. Walkways are visually useful, but do not alert, alarm, or neutralize hazards when they are crossed. The techniques disclosed herein may be used with active means to detect when a person walks outside the defined area, or when equipment moves (intentionally or accidentally) outside the defined area, and active measures may be taken to alert, alarm, and / or control the functionality of equipment within the facility.

[0227] 26C illustrates a situation for detecting when someone walks outside a defined area. However, it will be appreciated that the techniques disclosed herein can be used to detect when someone walks inside a defined (potentially "no-go") area, across a plane / threshold that delineates the defined (e.g., no-go) area, etc. In other words, in different use cases, the defined area can be an area where people are allowed, an area where people are prohibited, etc. Exemplary Holistic Sensing Related Technologies

[0228] 27 illustrates how the techniques disclosed herein, according to certain exemplary embodiments, can be used to provide a holistic view into an entire operating area. For example, fixed machinery, loaders / trucks, people, materials, items, accessories (including pallets), and other objects all have their own inherent electric fields. These fields can be determined and used to help monitor who and what is "on the floor", where people and things are, whether compliance is being met with regard to keeping access lanes clear, at least rough inventory control, etc. This information can be used to inform transportation / logistics systems, drive information displays, assist with accounting-related and / or audit-related functions, etc. From a holistic perspective, as discussed above, features may be provided related to machine safety (e.g., moving items in connection with conveyors, floor / inventory control, inbound / outbound, and outbound, including interaction points with large "unique" equipment such as corrugators, broadly applicable equipment such as wrappers, etc.; traffic areas including prohibited spaces, pedestrian walkways, emergency personnel location services, etc.; robotics and automation in connection with industrial robots, autonomous mobile vehicles, cobots, etc.; and / or the like.

[0229] As can be appreciated from the above, certain illustrative embodiments enable potentially low fidelity sensing data to be transformed into high value insights and actionable information. For example, surfaces can measure and "feel" phenomena in the physical world and use that data to improve digital and physical processes. This can be accomplished in different illustrative embodiments by leveraging novel algorithms and AI applied to sensing technologies that are integrated into larger product ecosystems, and the like.

[0230] The techniques disclosed herein are advantageous for a number of different reasons. For example, in certain exemplary embodiments: · Conformable in that it can be applied to virtually any surface, regardless of material, shape or size; Contains simple components and architectures and can be implemented using edge computing technologies; · Durable in that it can be designed for implementation to withstand environmental factors such as dust, sun, water, vibration, heat, cold, humidity, and / or the like.

[0231] This approach is advantageous in that bumps and / or minor collisions are less likely to cause the sensor to become misaligned, which facilitates ease of maintenance and complements ease of application (which may be provided by appropriately enabled appliques or mats, epoxy applied areas and / or the like).

[0232] It should be noted that the "skin" related techniques may be used in connection with cobot and non-cobot machines, in industrial safety and / or non-industrial safety applications. Exemplary Technology Platform for Generating and Passively Detecting Changes with Electric Fields

[0233] FIG. 41A is a system block diagram for a sensing system, according to certain exemplary embodiments. The exemplary system of FIG. 41A generates and passively detects changes associated with an electric field. These changes are generally caused by interactions with the electric field, for example, via shunting, constructive or destructive interference, or other electrical effects. The electric field is applied to at least two electrically connected electrodes (e1, e2, ...e in the example of FIG. 41A). n A current is generated for the sensing area 4102 by a conductive surface 4104 having electrodes e1, e2, ... e1 through a control circuit 4106. n 4, and the control circuit 4106 may correspond to some or all of the components shown in and described above in connection with FIG. 7. In short, a current is supplied to electrodes e1, e2, ... e n Interaction with the electric field created in the sensing area 4102 causes a voltage drop that is observable by the control circuit 4106.

[0234] The sensing area 4102 can be defined in X, Y, and / or Z dimensions. "True" omnidirectional sensing is possible; that is, the sensing area 4102 can be defined to be any or all of the area surrounding the system. While certain systems may purport to provide 360-degree sensing, sensors are often limited in that they cannot "see through" the equipment to which they are attached or otherwise connected. For example, 360-degree cameras cannot "see through" the machine to which they are attached, although this is at least theoretically possible. In contrast, the electric field created by certain exemplary embodiments can be present on any side of the machine, even under the machine, even if blocked by the machine itself, because the electric field can project through the machine for a long distance. This can be useful for a mobile device to use it to detect obstacles on any side of it, including potential obstacles on the ground (e.g., spikes, broken glass, etc.), or to guide a vehicle by following a defined pattern while also detecting potential approaching obstacles such as a human that may walk into the defined path. The sensing area 4102 can have a linear or non-linear shape. Similarly, the shape of the sensing area 4102 can be regular or irregular. The size, shape, and arrangement of the sensing area 4102 can be determined based on the particular application. For example, the sensing area 4102 can be defined as a portion of a machine, such as a pinch-pint or arm access area, an area around a mobile machine operated by a human, such as a forklift or fork truck or AGV (e.g., such that detection is possible outside of a particular defined location because the machine is mobile and / or potentially unguarded), on or around the skin of a robot or cobot, a prohibited space, such as in front of an area where access is required, inside or outside a walkway, intersection, or other defined path, inside or outside a control or operator area, inside or outside a control or operator area.The sensing area 4102 may be provided to replace at least the bottom, sides, lower portion, and / or top of a glass, plastic, wire, or other barrier, in which case a glass, plastic, wire, or other barrier may be provided.

[0235] A variety of different materials and / or structures can be used for the conductive surface 4104. For example, the conductive surface 4104 can be a dedicated article, such as a conductive mat, a conductive ceiling or wall panel, a substrate (which may or may not be transparent). In other cases, the conductive surface 4104 can be an applique, such as a film or tape, that includes conductive elements. In still other cases, the conductive surface 4104 can include a sprayed conductive material, such as a paint or thin film or other coating. In still other cases, the conductive surface 4104 can include conductive particles or other elements that are laminated, molded, or embedded into an article, such as a mat, floor, ceiling, or wall panel. The conductive surface 4104 can be integrated with or attached to an article in the sensing area 4102 or proximate the sensing area 4210. For example, the conductive surface 4104 can be attached to a fence or other guard structure, provided in or near a control room or operator area, or connected as a skin on a robot or cobot. The conductive surface 4104 preferably has a sheet resistance in the ranges specified above.

[0236] Field-defining electrodes e1, e2, ... e n can be formed as dedicated attachable components in certain exemplary embodiments. For example, a puck-like electrode may in some cases be (re)positionable on the conductive surface 4104. In other cases, the electric field-defining electrodes e1, e2, ... e n can be formed as wires or wire leads, printed conductors, or other conductive materials. In certain exemplary embodiments, the electric field defining electrodes e1, e2, ... e nare within, around, or otherwise proximate to the sensing area 4102. The electric field defining electrodes may be electrically connected to the conductive surface in any suitable manner. For example, in certain exemplary embodiments, the electrodes may be provided in tape or roll form. The individual electrodes may be in direct contact with the conductive surface (e.g., in a plane that contacts the conductive surface) or may be electrically connected to the conductive surface in other manners (e.g., by connecting with leads from the conductive surface, by having leads from the conductive surface directly contact the conductive surface, etc.).

[0237] In certain exemplary embodiments, the electric field defining electrodes e1, e2, ... e n are provided around at least one periphery of the sensing area 4102. This is useful for guarding applications, i.e. applications incorporating transparent interfaces (e.g. public information displays, touch screen devices, etc.). In different exemplary embodiments, the electric field defining electrodes e1, e2, ... e n is provided within the sensing area 4102, which may be possible when a transparent interface is not required (e.g., when monitoring pinch points on a machine, when provided as a sensor on an AGV, etc.).

[0238] Field-defining electrodes e1, e2, ... e n may be provided on the same plane as each other or on different planes. n may be provided on the same plane as the conductive surface 4104 or on a different plane. In certain exemplary embodiments, the conductive surface 4104 may be provided on a first plane and at least one (and possibly all) of the electric field-defining electrodes may be provided on one or more planes other than the first plane. This arrangement may be, for example, a linear arrangement of electric field-defining electrodes e1, e2, ... e nmay be provided as wires on opposite sides of a door frame or entryway, and the conductive surface 4104 is on the floor, upper casing, or other area. In object and / or person tracking applications, for example, the conductive surface 4104 and the electric field defining electrodes e1, e2, ... e n It may be desirable for the electrodes to be provided on parallel planes (e.g., floor and ceiling) or vertical planes (e.g., ceiling and wall or floor and wall). When an electrode is formed as a wire or wire lead and described as being on a different plane compared to the conductive surface 4104, it will be understood that at least a portion of the electrode (e.g., a distal terminal portion thereof) is provided on the different plane since the electrode is still electrically connected to the conductive surface 4104. In certain exemplary embodiments, the conductive surface and / or the field-defining electrodes may be positioned and / or insulated to control the directionality of the electric field. This may be useful in preventing stray signals from another area (e.g., another side of a wall or outside the detection zone) from triggering unintended or otherwise spurious effects.

[0239] 41A, the control circuitry 4106 operates under the control of a controller 4108 having an operably coupled memory 4110. The memory 4110 includes control logic 4112, a detection algorithm 4114, and an interaction template 4116. The control logic 4112 is a coordinating program logic that assists the controller 4108 and the control circuitry 4106 in cooperating to create an electric field and passively monitor changes therein. That is, the created electric field is caused to function as a passive sensor for the detection system.

[0240] The detection algorithm 4114 assists in monitoring changes with the electric field that may indicate interaction with the electric field. It detects the presence of electrodes e1, e2, ... e n4114 uses data derived from outputs from at least some of the interactions. The interactions have properties including three-dimensional spatial and timing properties; that is, each interaction has a spatial coordinate and a timestamp associated with it, providing x, y, z, and t data. The detection algorithm 4114 may be any one or more of the algorithms mentioned above or other algorithms.

[0241] In some cases, the detection algorithm 4114 alone may generate an output indicative of an interaction. For example, for simple one-touch functions, and for some hovering or proximity actions, the detection algorithm 4114 may generate enough data to enable the system to function. In other cases, a determination may be made as to whether the interaction matches one or more defined interaction templates 4116. As described below, the interaction templates 4116 help distinguish between different interaction types (e.g., different detectable disturbances to the electric field) and / or their characteristics. For example, in situations where it is desirable to detect complex gestures, the presence or absence of a particular type of material, etc., if a determination is made that the detected interaction matches one of the defined templates, or if such a determination is not necessary and the output can be determined solely from the detection algorithm 4114, a corresponding action may or may not be triggered depending on the characteristics of the interaction. For example, in a proximity detection system, the presence of a human or other object may be detected as an interaction or disturbance to the electric field. However, the corresponding template may specify that no action should be taken if the interaction is beyond a threshold distance. Similarly, corresponding templates may specify that different actions should be taken based on different distances or distance thresholds, proximity, location, trajectory, rate of arrival or other speed, type of object detected, whether the object has passed through or entered a defined area, etc.

[0242] Certain exemplary embodiments execute the detection algorithm 4114 to process the template data of the interaction template 4116 without having to reconstruct the entire electric field. In other words, (a) based on data derived from the output from at least some of the electrodes, changes with the electric field can be monitored, which potentially indicate an interaction with the sensing area 4102, and (b) a characteristic of the interaction can be determined without having to reconstruct the entire electric field.

[0243] The control logic 4112 can instruct the controller 4108 based on the output of the interaction template 4116. For example, the controller 4108 can use the interface 4118 to interact with an external system 4120 based on a signal from the control logic 4112 that in turn depends on the output from the interaction template 4116. The external system 4120 can be a vehicle control system (which can cause a corresponding vehicle to slow down, stop, turn, etc.), a display device (which can cause touches or gestures to be registered as inputs and updated automatically), an inventory tracking or management system (which can be updated to indicate receipt or shipment of tracked items), etc. The interface 4118 can be an interface to a control system of a machine used in the loading / unloading operation, which can be controllable in response to disturbances to the electric field during the loading / unloading operation. In this regard, the conductive surface can correspond to one or more pained, marked, or otherwise designated walls of a container or vehicle into which an object is loaded / unloaded. In certain exemplary embodiments, an interface to the scheduling system may be provided, and detection may relate to unplanned and / or unexpected disturbances to the electric field caused by perturbations emanating from humans, other machines, structures, recognizable electronic signals, pests, or debris. In this regard, the scheduling system may indicate where people, items, and / or other objects are supposed to be located (or not located), and deviations from this detected by the system during operation may be detected as errors. The interface 4118 may be any suitable computer-mediated interface. Hardware and / or software components may be included. For example, an application programming interface (API) may be used with a wired or wireless connection to the external system 4120.

[0244] FIG. 41B illustrates an example platform architecture stack 4130 according to certain example embodiments. The example platform architecture stack 4130 of FIG. 41B includes AI and other algorithms 4132. The AI ​​and other algorithms 4132 learn over time to better enable the platform to adapt to different types of behavior and to better understand the user's intent. The AI ​​and other algorithms 4132 may include categorization, recognition, prediction, and other functions. The drivers 4134 are interfaces that control the hardware elements 4136 and allow their functions to be controlled and / or understood by the software, including the AI ​​and other algorithms 4132. The hardware elements 4136 are the physical elements of the platform and include the sensing system itself, physical controls, displays, and / or any other physical components that may be provided (e.g., cameras, motion sensors, haptics, speakers, etc.). The API 4138 enables communication between different software and / or other components. The outputs 4140 include the platform's responses to the user or other parties. Output 4140 may include, for example, sound, lighting, display changes, haptic feedback, and / or the like. Input 4132 in this context refers to user-facing interactive elements, including interaction paradigms for interacting with and managing functionality, content, animations, and / or the like. Horizontal surfaces 4144 and vertical surfaces 4146 are elements within an environment, which may include, for example, tabletops, counters, desks, floors, ceilings, platforms, rugs, mats, walls, windows, doors, panels or separators, lockers, signage, etc.

[0245] FIG. 42 illustrates the contents of an interaction template 4200, according to a particular exemplary embodiment. The interaction template 4200 in the example of FIG. 42 includes a plain human understandable description 4202 of what the particular interaction template 4200 is designed for. For example, the description 4202 may indicate in "plain language" that the particular interaction template 4200 is for detecting a particular kind of gesture made with a hand, a particular type of contaminant that may enter a defined area, etc. The type information 4204 describes the general subject matter of the particular interaction template 4200. For example, the type information 4204 may be hierarchical. For example, at a first level, the type information 4204 may distinguish between gesture, touch, and hover interaction. At a second level of gesture types, further type information 4204 may distinguish between pan or swipe left, pan or swipe right, checkmark, freeform drawing, etc. In certain exemplary embodiments, a taxonomy may be provided for this type information 4204, which in some cases may be a multi-level hierarchical taxonomy.

[0246] Signature / Fingerprint 4206 is a collection of data that defines a particular interaction template. For example, each interaction may be characterized by a series of normalized voltage-related measurements. By identifying the critical components, different interaction "signatures" or "fingerprints" can be classified and associated with different templates. In some cases, the various template signatures or fingerprints may be pre-programmed, for example, before the detection system is shipped, installed, and / or activated. In other cases, the various template signatures or fingerprints may be defined on-site, for example, through a training program, configuration program, or calibration program. In still other cases, the various template signatures or fingerprints may be machine-learned, for example, over time and potentially through actual use of the detection system. In some cases, the specific fingerprints may be generated by an electromagnetic field attenuator that includes electronic devices in the form of specific tags or markers. These tags or markers may cause specific patterns or specific recognizable disturbances to the electric field.

[0247] This may be useful in guiding AGVs to / from an area (e.g., may be useful during loading / unloading operations), signaling the presence of a person or obstacle (e.g., may be useful to assist in signaling to a person or machine that a person or obstacle is present), marking items (may be useful in inventory tracking and / or other scenarios). In general, the recognizable electronic signal may emanate from a placed marker, such as a marker placed on a person, near a defined path, near an area that cannot be interrupted, and / or the like. The marker itself may output a defined, constant or fixed signal, or a pattern or signal that is distinguishable from other signals and recognizable as coming from the placed marker.

[0248] In some cases, different templates may be proposed and require an authorized human to indicate that they are valid. This may be done using a signature or fingerprint for scenarios developed through training, configuration, or calibration programs, or machine-learned, and / or possibly even pre-programmed. A verified flag 4208 is used to signal whether a particular interaction template 4200 is verified in this manner. In some cases, interaction templates that are not verified may not be activated. That is, in some cases, interaction templates that are not verified may not be registered by the detection system.

[0249] Similarly, the active flag 4210 indicates whether a particular interaction template 4200 is eligible to be registered by the detection system when the detection system is in use. Interaction templates that have not been verified may be flagged as invalid. In other cases, templates that are not used due to the circumstances of where or how the detection system is deployed may be flagged as inactive. For example, a detection system that provides a guard against a barrel of peanut butter may have an interaction template activated to detect pests, but may not have a gesture-related interaction template activated. Conversely, a public information display may have a gesture-related interaction template activated, but a template for detecting pests deactivated.

[0250] The priority level 4212 is a numeric or other indicator of the criticality of a particular interaction template 4200. This information may be used, for example, to prioritize which interactions are most important to respond to. For example, in an industrial safety setting, it may be more important to respond to a runaway forklift detected by the system compared to a machine located in a designated walking path. As another example, it may be more important to respond to a touch on a public information display compared to a simultaneous hovering interaction. Thus, it will be appreciated that a criticality level may be assigned to each template, and when multiple interactions are detected, actions may be triggered according to different respective criticality levels. In this regard, multiple interactions may be detected simultaneously and / or in close and / or overlapping temporal proximity to one another. In certain exemplary embodiments, the controller 4108 is configured to determine whether multiple interactions with the sensing area 4102 occur simultaneously and / or in close and / or overlapping temporal proximity to each other, determine whether each of the multiple interactions matches one of the defined templates without reconstructing the entire electric field, and selectively trigger a corresponding action based on one or more of the characteristics of the interactions in response to determining that two or more of the multiple interactions match one of the defined templates. Prioritization of actions for multiple detections can be according to priority level information for the respective templates. In other cases, prioritization can be based on heuristics such as, for example, first-in first-out (FIFO), last-in first-out (LIFO) and / or the like. The memory 4110 may maintain a suitable data structure for handling prioritization, such as, for example, a queue (or priority queue) or stack data structure. Elements in the queue or stack data structure can indicate templates, characteristics of interactions, and aspects of actions.Alternatively, or additionally, the action itself can be enqueued into a queue data structure or pushed onto a stack data structure, from which the control logic 4112 can process the item.

[0251] The action list 4214 specifies which action, if any, should be taken in response to a determination that a particular interaction associated with the interaction template is detected. The action list 4214 can include one or more actions in different exemplary embodiments, which can have conditional logic associated with it. In the case of a touch interaction, the action may simply be to output the coordinates and a timestamp. However, in the case of a guard application, different actions may be specified for different proximities. The conditional logic can specify, for example, that no action should be taken if a person or object is detected beyond a first threshold distance from the guard area, that an audio and / or visual alarm should be sounded if a person or object is detected between a first and a second distance from the guard area, that the machine should be slowed down if a person or object is detected between a second and a third distance from the guard area, and that the machine should be stopped if a person or object is within a fourth threshold distance from the guard area. Conditional logic can also be used to help identify faults in the monitored machine. For example, if a machine is expected to move into a particular area, produce a particular output, move particular items, open and close, remain unblocked, provide throughput, etc., the detection system can detect variations from the specified conditions and the conditional logic can indicate if a problem has occurred.

[0252] Different actions may be specified for different templates, and these actions may be programmable to reflect conditional or other logic. Different programmable actions may include, depending on the situation, lights, signs, and / or sirens, slowing down an object or equipment (such as a robotic arm, forklift, or AGV), shutting down an object or equipment (such as a conveyor), de-energizing an object or equipment, initiating a phone call, Short Message Service (SMS), email, or other electronically mediated message or alert, moving an object or equipment relative to a defined area so that the object or equipment is moved into or out of the defined area or to a specified location (e.g., from an area where access is required, such as an electrical panel), etc.

[0253] In certain exemplary embodiments, the template may simply be the signature / fingerprint 4206. In certain exemplary embodiments, the template may include links to other data structures that contain, for example, information related to actions or action lists. In other words, as an example, in certain exemplary embodiments, the action lists for a given interaction template may be provided in a different data structure that is linked to or otherwise identified by the given interaction template.

[0254] In certain exemplary embodiments, mappings may be maintained between different actions to be triggered in a variety of different use cases and different interactions. For example, a "gesture vocabulary" may be defined to cover different functions such as, for example, moving an interface device (such as a mouse cursor) left / right / up / down, selecting a particular application or function, dragging, activating, invoking, opening a menu or submenu, single / double click, etc.

[0255] Some use cases and use case concepts have been described above, and further use cases are described in further sections below. The following subsections enumerate different applications and use cases for which the above-mentioned system can be adapted. That is, a technology platform including a definable sensing area, conductive surface, electric field defining electrodes, algorithms, interaction definitions, etc. can be used to provide the following elements and / or other elements. It will be understood that detection can be pre-characterized via calibration and / or machine learning. Apart from that, the electric field definition, sensing technology, approach to trigger events, etc. can remain the same. Exemplary Object Tracking Related System

[0256] There are various ways in which objects, including inanimate objects and / or humans, can be tracked using the passive sensing techniques of certain exemplary embodiments. Figures 43A-43B are two examples in that they show different object tracking related systems incorporating sensing systems, according to certain exemplary embodiments. In Figure 43A, for example, boxes B1, B2, ...B n are stacked on a mat Mi from which they can be moved by humans or machines such as a forklift. In this example, boxes B1, B2, ... B ncan move to mat M2 behind the track T. A first sensing system 4302a and a first transceiver 4304a are connected to mat M1, and a second sensing system 4302b and a second transceiver 4304b are connected to mat M2. The first sensing system 4302a of mat M1 helps to project a first electric field F1, and similarly the second sensing system 4302b of mat M2 helps to project a second electric field F2. Mats M1 and M2 may be configured as conductive surfaces, and electric field defining electrodes may be provided in, around or adjacent to those conductive surfaces. The electric fields F1 and F2 correspond to different sensing areas. Thus, the sensing areas here are provided adjacent to mats M1 and M2. The mats M1 and M2 have interfaces 4304a and 4304b connected to them, which may include wired or wireless connections to external systems, such as an inventory management system 4306 present in a network location (such as a cloud computing infrastructure). The sensing systems 4302a and 4302b are configured to detect when a box is moved to / from there, i.e., boxes B1, B2, ...B n The movement of causes a change to the electric fields F1 and F2, which can be detected by the sensing systems 4302a and 4302b. This can be achieved by removing the electric fields F1 and F2 from the boxes B1, B2, ...B nmay have some inherent electric field modifying properties (e.g., may have some electrical conductivity) or may be provided with tags or markers therein and / or thereon that disrupt the electric fields F1 and F2 in an understandable manner. In certain exemplary embodiments, the sensing systems 4302a and 4302b may be configured to distinguish different types of objects detected in different sensing areas by their different inherent electric field modifying properties and / or different tags or markers attached thereto. Packaging materials may also be used for this purpose. Upon such detection, the inventory management system 4306 is updated. For example, status data held in a non-volatile memory location of the inventory management system 4306 may initially indicate that the box B1 is on or near the mat M1. Once the box B1 is picked up and moved, the status data is updated to indicate that it is being moved. If the person or machine involved in the movement of the box B1 is recognizable (e.g., by a characteristic pattern emanating from the box B1 as an inherent electric field or with the aid of a marker that emits a special output), this information may also be stored. Once the box B1 is placed on or near the mat M2, the status data may be updated again. In this manner, inventory tracking and / or management can be performed in a warehouse or other location, with audit trails provided at different steps. If a box is lost, status data can be consulted. In general, the sensing systems 4302a and 4302b can send electronic messages to the inventory management system 4306 related to the movement of objects.

[0257] It will be appreciated that the box may, for example, be moved from mat M2 to mat M1 in different scenarios. It will also be appreciated that different sensing systems may be used to help guide the machine into or around the track, for example as described in detail above. This may be useful, for example, when using AGVs, etc. While this example has been described in the context of a mat onto / from which objects may be moved, it will be appreciated that different examples may use different conductive surfaces that the object does not need to contact (e.g., a ceiling area, a floor area, a fenced or otherwise defined area, etc.). Similarly, while this example has been described in the context of a box, different types of objects may be trackable.

[0258] In certain exemplary embodiments, the sensing area may include a conduit, and the detectable event may relate to a blockage, a clog, an expected inflow, an unexpected inflow, and / or the presence of a contaminant within the conduit.

[0259] In Fig. 43B, the "objects" being tracked are, for example, humans in a warehouse or other environment. In Fig. 43B, the ceiling has a conductive surface C. Electric field-defining electrodes are electrically connected to surface C and may be provided on the ceiling, walls perpendicular to the ceiling, the floor, and / or other locations. A sensing system 4302c in this environment, with the help of surface C and the electric field-defining electrodes, detects the positions of humans H1, H2, ...H1, H2, ...H3, H4, H5, H6, H7, H8, H9, H10, H11, H12, H13, H14, H15, H16, H17, H18, H19, H20, H21, H22, H23, H24, H25, H26, H27, H28, H29, H30, H31, H32, H33, H34, H35, H36, H37, H38, H39, H40, H41, H42, H43, H44, H45, H46, H47, H48, H49, H50, H51, H52, H53, H54, H55, H56, H57, H58, H59, H60, H61, H62, H63, H64, H65, H66, H67, H68, H70, H71, H72, H73, H74, H75, H76, H77, H78, H79, H80, H81, H82, H83, H84, H85, H86, H87, H88, H90, H91, H92, H93, H94, H95, H96, H97, H98, H99, H100, H10 n Humans H1, H2, ...H n The presence and movement of different people H1, H2, ...H3 creates changes in the electric field F3 that the sensing system 4302c can monitor. n To help locate and / or distinguish between humans H1, H2, ... H n contains markers or tags t1, t2, ... t nmay be provided. As with other markers or tags described herein, these markers or tags may be wearable, portable, or otherwise provided and may generate their own characteristic signal or interfere with the electric field in a known manner. This includes generating a constant and fixed signal or providing a distinguishable pattern of signals. The sensing system 4302c may use the interface 4304c to report events to an external system, such as, for example, a human resource (HR) management system 4308 and / or a location tracking system 4310. The HR management system 4308 may be used to supplement or replace time cards, track breaks and / or productivity, etc. The location tracking system 4310 may track the time of people H1, H2, ...H1, H2, ...H3, H4, H5, H6, H7, H8, H9, H10, H11, H12, H13, H14, H15, H16, H17, H18, H19, H20, H21, H22, H23, H24, H25, H26, H27, H28, H29, H30, H31, H32, H33, H34, H35, H36, H37, H38, H38, H39, H40, H41, H42, H43, H44, H45, H46, H47, H48, H49, H50, H51, H52, H53, H54, H55, H56, H57, H58, H59, H60, H61, H62, H63, H64, H65, H66, H67, H68, H69, H70, H71, H72, H73, H74, H75, H76, H77, H78, H79, H80, H81, H82, H83, H84, H8 n The location and / or identity of the people, vehicles, and other personnel may be tracked. Thus, if there is an accident (for example), the location tracking system 4310 may be consulted to see who was in the environment and where they were last located. Tracking this type of data may allow the impact of an accident to be more easily assessed and recovery options to be initiated more quickly and precisely. As discussed above, the arrangement of FIG. 43B is useful in a wide variety of environments including, for example, warehouses, shipping facilities, fulfillment centers, airports, mines, and / or the like. Exemplary Home Appliance Related Systems

[0260] The disclosed technology can also be used with a variety of different consumer electronics-related solutions, e.g., to interface with such devices. For example, it can provide gesture or other control of televisions, CD / DVD players, streamers, A / V receivers, projectors, video game systems, and / or the like. Essentially, a sensing area is created by the consumer electronics device or through electronics connected to the consumer electronics device. The sensing area monitors human movements and interprets them to provide input to the consumer electronics device. In this regard, the consumer electronics device can maintain a mapping between different possible interactions (e.g., such as the templates disclosed above) and commands appropriate for the device being controlled. These mappings can include commands to initiate and terminate recognition of the different interactions, as well as the control-related interactions themselves.

[0261] 44A-44D are examples illustrating how the techniques disclosed herein may be used in connection with a consumer electronics device, according to certain exemplary embodiments. In FIG. 44A, a consumer electronics device 4402 includes a sensing system 4404. The sensing system 4404 is configured to project an electric field 4406 that is a sensing area with which a user interacts. As a general matter, motion into the passively projected sensing area may be detected and misinterpreted as a control signal, for example, during the normal course of shifting or stretching while watching television, while eating popcorn, while a dog jumps on / off the couch, while a child runs in front of the projection screen, etc. Thus, certain exemplary embodiments may monitor for certain interactions signaling that a subsequent control type interaction is accepted. In the example of FIG. 44A, a user holding a wand W moves the wand W into a star shape. This star shape is unique and unlikely to be misdetected due to one of the factors described above and / or other factors. Upon detecting this shape, the system may transition into a mode of accepting user-based control type input.

[0262] User-based control type inputs may be specific to the particular device being controlled and / or application being interacted with. For example, Fig. 44B shows an upward moving wand that may be used to turn up the volume, change the television channel, adjust the frequency tuned by the radio, etc. Up and down movements may be used in this regard. Moving in a left / right direction may rewind or fast forward, with multiple left or multiple right movements rewinding or fast forwarding at a faster pace. Pressing towards the display may signal a selection, an X may signal a cancel or "back" action, etc.

[0263] As mentioned above, the consumer electronics device 4402 may be a video game system. FIG. 44C shows a human holding a wand W as if it were a baseball bat, such as might be provided in a baseball game. For example, the swing of the bat can be detected. In other cases, the position and / or orientation of the wand W can be known or tracked and used as input for the video game. In some cases, complex movements can be used to signal a reload of a gun (e.g., inserting a new clip), putting down a backpack, picking up an item in the virtual world, etc. Because the video game system is detecting changes in the electric field, there is no need to provide a visible sensor bar, infrared candle, or other detector outside of the device 4402 itself. That is, a sensing system 4404 having a conductive surface and electric field-defining electrodes can be provided in or on the device 4402, for example on a front grill or bezel.

[0264] In FIG. 44D, a defined movement of the wand W (in this case a circular movement) ends a defined period of interaction with the home electronic device 4402.

[0265] The use of the wand W can be beneficial because tags or markers can be used to facilitate detection by causing the wand W to output a recognizable signal, or can have inherent properties that are recognizable as particular disturbances to the electric field. Although certain exemplary embodiments have been described with respect to wands, different objects can be used in different cases. In some cases, simple human gestures without separate objects can be used for control. Exemplary Workplace-Related Systems

[0266] Certain exemplary embodiments may be used for a variety of workplace-related interactions, such as in a home or other office, in a school or other educational environment, or anywhere else where work is performed using a workstation or the like. Types of workplace-related interactions that may be facilitated by certain exemplary embodiments include, for example: Workstation arrival and "sign of life" features. The sensing system can detect when a user arrives in a defined space in the vicinity of a workstation. The sensing system can then send a signal to power on or wake up a monitor, computer, or other device. Humans interact with the electric field in a characteristically recognizable way, and the system can monitor the electric field within a defined area in the vicinity of the workstation (e.g., in a cubicle, in an office, etc.). Thus, the system can ignore passersby based on distance or speed (e.g., because they are too far from the workstation or moving too fast to stop). When it is detected that a user is arriving at the workstation, the system can interface with the computer or other device, check if the device is off or asleep, and then send a control signal via the appropriate interface. Gesture-based unlocking and powering on of a workstation. The sensing system can detect when a user initiates a gesture and unlock and / or power on the workstation. For example, based on the principles of the previous point, the sensing system can detect when a user approaches a workstation and then performs a defined motion, such as turning a key to start a car or lock a door, waving a hand, etc. In an exemplary embodiment, for example, if a user performs a "secret," "reserved," or otherwise "customized" gesture and enters a password, etc., a form of two-factor authentication (2FA) can be provided. Another form of authentication can be provided by having the user possess a tag or marker that disrupts an electric field in a way that is unique to the particular user. The computer system can interface with a security system that links the gesture and the tag or marker together to provide such authentication. Activating peripheral devices or tools (such as keyboards, mice, power mats, and / or the like) via hover interactions. In some cases, peripheral devices and tools may go into hibernation mode, especially when they are battery powered. However, the detection system of certain exemplary embodiments may wake up these peripheral devices or tools based on, for example, detected hovering motion. Initiating a shortcut action. Many computer-based functions are repetitive in nature. For example, standard text is often provided in everything from letters to patent applications, it is not uncommon to toggle back and forth between applications when performing a bank reconciliation, and many people want to present or refresh specific websites throughout the day (e.g., to check news feeds, stock prices, sports scores, etc.). Certain exemplary embodiments include user-defined mappings of gestures to specific actions. Then, when such actions are detected, the sensing system can initiate a corresponding action. Shortcut actions may also include "typing" frequently used words or phrases, auto-correcting spelling mistakes, auto-completing words or phrases, etc. Focusing lights or cameras on an area. In certain exemplary embodiments, it is possible to turn lights on / off or dim them, focus lights in a space, adjust cameras used for video conferencing, telemedicine, and / or other applications. This can be accomplished using different defined gestures. Pointing can, for example, focus lights, cameras, etc. Similarly, pointing can cause video conferencing or other software to share a screen (or a different screen), highlight an area, etc. Raise / lower the desk. Similar to turning lights on / off or dimming them, the desk can be raised or lowered using similar gesture controls. Automatically locking the workstation when you walk away. In parallel with unlocking the workstation, a sensing system can detect when the user walks away from the workstation and then lock the screen, put the screen to sleep, or take other actions.

[0267] The sensing system of certain exemplary embodiments may also provide a form of "macro-workplace awareness." For example, the sensing system may detect objects that may be present in the work environment (which may be useful in determining when supplies are low, when a hallway is blocked, when a lunchroom is crowded, etc.) and / or may recognize tasks and provide data to a task analysis computer-based platform. Recognizing tasks allows data to be provided to the task analysis computer-based platform to assist in recognizing which tasks are performed most frequently, identifying which tasks take the most time, identifying which tasks can be optimized (e.g., through training to perform better moves, to move materials to different locations, etc.), and / or the like. Task analysis may also be used to recognize intent and provide tailored productivity analysis. It may also be possible to recognize individuals within a space based on patterns of past movements, activities, time, used items, etc., etc. Additionally, task analysis may be used to identify periods of interactivity and / or productivity, head nodding at a workstation, etc.

[0268] With respect to use and interaction with content, gesture controls can be provided to achieve functionality including, for example, viewing and / or accessing files for review, sorting, sizing, sharing, editing, and / or the like. Gesture controls can also be provided to achieve presentation-related functions such as, for example, moving to the next / previous slide, going to sleep, playing embedded or other videos, etc. For video conferencing, gesture controls can be provided to start and end a conference as well as manage or adjust settings such as, for example, changing the volume, muting, pan / tilt / zoom controls, etc.

[0269] Other possible interaction types involve interfacing with elements in a room, for example, as shown in Figures 45A-45C. In Figure 45A, for example, a first sensing field 4502a is created via a first sensing system 4504a proximate to a first office wall W1, and a second sensing field 4502b is created via a second sensing system 4504b proximate to a second office wall W2. A first user may be able to control lighting by gesturing or actually sliding down the first office wall W1 using one finger. A second user may gesture upwards with their entire hand proximate to the second office wall W2, for example, to retract the shade 4506. In different exemplary embodiments, based on gestures and / or other interactions with the created fields, different office effects may be provided that are related to technology components including lighting, projectors, video conferencing, and / or the like.

[0270] Figure 45B is similar to Figure 45A, except that in free-write mode, where written text is displayed on glass or other surface, it shows a user touching (or possibly gesturing) adjacent to wall W1, and tracking of the user's finger can be used as if the user were using a dry erase marker on the whiteboard.

[0271] FIG. 45C is similar to FIG. 45A, except that it shows the user making gestures to advance slides in presentation mode.

[0272] 45D-45F show how glass or other transparent surfaces, including workstations and / or conference room equipment, can benefit from the technology disclosed herein. The surface can react and / or shift based on an attachment that triggers a different state (e.g., a marker triggering a whiteboard function, a keyboard triggering a display, etc.). In FIG. 45D, a keyboard can be placed on the desk to enable typing functionality, gestures can be used to unlock functionality for displaying on a far wall, etc. The form of the surface can change based on the context, including use between, for example, communal, private, available, transparent, etc. Similarly, interactions can be defined to personalize colors, patterns, feedback, etc. In FIG. 45E-45F, the meeting has not started so the surface is transparent and in FIG. 45F it becomes opaque. This can be achieved using switchable glass based on polymer-dispersed liquid crystal (PDLC), polymer-assembled liquid crystal (PALC), electrochromic, organic light emitting diode (OLED), and / or other technologies. The surface can display different items using projection or other imaging.

[0273] Certain exemplary embodiments may implement sensor functionality, including, for example, a sensing system that acts as a sensor on the ceiling of a room. The sensing system in such a case may sense a person walking in or out of the room (e.g., to turn lights on / off or indicate to an office scheduling system that the room is in use, etc.), detect whether a person is in the room (e.g., to consult a reservation system to determine whether the room is in use beyond a scheduled period of time, etc.), and determine where a person is in the room (e.g., to adjust interior and / or exterior blinds to block the sun based on the time of day or to provide privacy, etc.). Certain exemplary embodiments may detect presence and optionally location in the room instead of a motion sensor. The techniques of certain exemplary embodiments may be advantageous compared to motion sensors, which may indicate false absence (e.g., when a person is stationary or outside a viewing area for a period of time) as well as false detection (e.g., caused by the movement of a spider or other pest in front of the sensing system). The sensing system of certain exemplary embodiments does not need to rely on motion, and detections (electric field disturbances) associated with pests such as spiders or mice can be easily distinguished from detections (electric field disturbances) caused by humans.

[0274] It will be appreciated that the technology disclosed herein can be used to monitor use, sense and track activity for a variety of other purposes. For example, the technology can be used for wellness / health purposes (e.g., understanding emotions, understanding if people are fainting or coughing, moving around a sufficient amount of space to have good mental health and blood circulation, etc.), space optimization purposes (e.g., understanding which areas of an office or other structure are not being used or which other areas may be overworked to inform, e.g., how to better organize leasing activities, tools, tasks, or other items, etc.), service and housekeeping purposes (e.g., indicating which areas need to be cleaned and how severely they are potentially based, e.g., on the number of people present throughout the day, the duration of stay of those present, etc.). Data collected by the sensing system regarding use and activity sensing can be transmitted to a management system where the data can be interpreted for purposes and transmitted, e.g., to people who need to know (e.g., cleaning or service companies, sanitation providers, etc.).

[0275] Although certain exemplary embodiments have been described in connection with office walls, it will be appreciated that the techniques disclosed herein may be used with a variety of different dividers including, for example, cubicle walls, movable partitions in conference rooms, etc. Indeed, the techniques disclosed herein may be used to trigger movement of a movable partition, for example, via gesture or other control near a designated surface. Exemplary Public Information Display Related Systems

[0276] The sensor technology described herein may be used in connection with a wide variety of different public information display related systems. The technology disclosed herein may be particularly useful in large multi-touch or multi-interaction applications. In this regard, one way in which the technology disclosed herein may be used is in connection with providing a variety of different types of more interactive digital signage. For example, a public information display may be provided in a storefront. In such a case, the technology disclosed herein may be used to provide an interactive window display or "smart window." A person "window shopping" may use touch, hovering, or other functionality to interact with the display to, for example, browse a product catalog, cause information about a selected product to be displayed, view a simulation of the selected or otherwise displayed product to show different prices, colors, sizes, and / or the like, check product availability or inventory, place an order, etc. In a sense, the technology disclosed herein may be used in conjunction with these public information displays to provide a more "live" and interactive window shopping experience.

[0277] The sensor technology described herein may be used in connection with maps and / or wayfinding in various locations, such as, for example, subway or train stations, airports, shopping malls, public areas, casinos, hotels, office buildings, resorts, and / or the like. Touch, hovering, and / or other controls may be used to interact with the map, make selections, provide other input (e.g., store categories, virtual keyboards, etc.) and / or the like, etc.

[0278] In restaurant type applications, a kiosk may be provided for ordering food and / or drinks. The sensor technology disclosed herein may be used in conjunction with a digital display provided, for example, via an LCD or other display, to browse the menu, make selections, etc. Also, in restaurant type applications, a more "static" paper or other menu may be stacked or otherwise provided on a table, bar top, etc., and the technology disclosed herein may be used to detect a selection on the table. The location of the selection may be mapped to a menu item and reported to a server, kitchen, bartender, or others. Similarly, bar games such as trivia may be played on the table, for example, by having a person select an entry from a list, "write" an answer, etc.

[0279] The technology disclosed herein may also be used for, for example, ticketing kiosks where one can purchase subway, train, or plane tickets, entertainment venues such as movie theaters and musicals, and the like.

[0280] The technology disclosed herein may also be used in merchandisers, including, for example, shelves, refrigerators, and freezers. In the latter case, a sensing system may be used to detect an approaching person or a person's hand, for example to trigger a defrost system in anticipation of a door opening. If the sensing system determines that a person is interacting with a particular product (e.g., picking it up, looking at it, or spending a lot of time with it), a display connected to the sensing system may provide information about the product. For example, once it is determined which product is being interacted with, a database may be consulted to retrieve and display nutritional facts, stock / sell-by dates, suggested recipes, recommendations for other foods or beverages to pair with the item, etc.

[0281] In still other applications, the techniques disclosed herein may be used in connection with museum displays, scenic overlooks or observation decks, and the like. The functionality disclosed in U.S. Provisional Application No. 62 / 786,679, filed December 31, 2018 (and WO 2020 / 141446) may be provided in such contexts, for example, with interactive engagement mediated by the techniques disclosed herein. Parallax-related solutions disclosed in U.S. Provisional Application No. 62 / 786,679, filed December 31, 2018 (and WO 2020 / 141446) may be implemented in connection with certain exemplary embodiments.

[0282] In an educational environment, for example, one or more children, a teacher, and / or other people may interact with an object, such as an oversized map, a model, etc. Different touch or non-touch interactions are detected by a sensing system and reported to a controller, which may, for example, cause some action to be taken (e.g., causing information to be displayed on a display device, making a sound, rotating an orrery, etc.). Exemplary Vehicle-Related System

[0283] The sensor technology described herein may be used in connection with a wide variety of different vehicle-related systems. For example, sensing areas may be located inside and / or outside the vehicle. Inside the vehicle, the sensing areas may be used to control vehicle systems, such as heating / cooling, radio and / or other media, window and / or sunroom control, defroster, etc. Interior detection may also relate to whether the driver is falling asleep or has fallen asleep (e.g., rocking, nodding head, hands off the wheel, etc.), the presence of a person, pet, or other object in the back seat or trunk, etc. Outside the vehicle, the sensing technology may be used to assist with parallel parking, prevent unsafe lane changes, provide blind spot detection, and detect on-going or suddenly appearing "obstacles" such as a child running in front of the car or a shopping cart moving through a grocery store parking lot.

[0284] 46A-B are examples of how the sensing techniques disclosed herein can be used in different vehicle-related systems, according to certain exemplary embodiments. In FIG. 46A, a first electric field 4602a is defined with respect to a dashboard D via a first sensing system 4604a. The first sensing system 4604a is tied to a control system of the vehicle. Commands are formulated such that different actions and interactions can be associated with different vehicle-related functions. In FIG. 46A, for example, a passenger can make a downward gesture near a window W to lower the window W, and a driver can gesture to twist a knob to have the radio change stations.

[0285] In FIG. 46B, a second electric field 4602b is defined in all directions around the car via a second sensing system 4604b. As the car moves, the presence of a child in front of the car and / or a light pole behind the car can be detected. If the driver gets too close to the child or if the child suddenly turns towards the car, the second sensing system 4604b can interface with the vehicle's control system to slow down and potentially stop the vehicle. Proximity-based detection can be used for this purpose, for example, with multiple thresholds defined. The thresholds may be defined in terms of absolute distance and / or as a function of speed and expected braking distance (e.g., such that a longer "run" is provided for faster moving vehicles, assuming braking distance is constant, etc.). Similarly, the driver can be warned before backing up into a light pole, for example by a beep, a visual warning, and / or by the car slowing down or stopping completely.

[0286] Although certain exemplary embodiments have been described in connection with automobiles, it will be understood that the techniques disclosed herein may be used in other vehicles including, for example, trucks, motorcycles, boats, airplanes, lawn mowers or tractors, trains, and / or other machinery. Exemplary Unattended Retail and Sales Related System

[0287] Exemplary techniques disclosed herein may be used in unattended retail and sales-related systems. For example, in a vending machine, rather than having to key in an alphanumeric combination to make a selection, one may simply touch a selection. A sensing system may be disposed within the vending machine to monitor touch or other interactions. For example, a touch may indicate a selection and trigger a visual confirmation (e.g., lighting around the selection), and a further touch or gesture (e.g., a checkmark) may confirm the selection. As discussed above, the same or similar approaches may be used at kiosks, tables, or bars, etc., where food and / or drinks are ordered. The ability to have touch and / or gesture control may make physical keypads and associated hardware obsolete or redundant, thereby allowing for an expanded product display area.

[0288] In certain exemplary embodiments, a first sensing system may be provided in association with a merchandiser, store shelf, or the like. In such cases, the first sensing system may detect when an item has been removed (e.g., particularly if the item has a characteristic pattern such as would be expected in aluminum, tin, and other canned goods). The first sensing system may track the removal of the item and communicate with an inventory management system, for example, for restocking, reordering, and / or other purposes. In certain exemplary embodiments, a second sensing system may be provided in association with a grocery store cart, hand basket, or the like. In such cases, the second sensing system may track the unique items placed in the cart for checkout automation, theft deterrence, and / or other purposes.

[0289] Similarly, a sensing system may be provided in connection with a hotel minibar or the like. In such a case, the system may determine when a particular item has been removed from the refrigerator. The sensing system may be tied to a billing system to automatically bill the guest for the purchase. Similarly, the sensing system may send a message to room service or the like indicating, for example, that the minibar or the like needs to be restocked.

[0290] The technology disclosed herein may also be used in casinos and other gaming environments. For example, a person may interact with a slot machine using gestures to pull a lever or press a button. A user may also interact with a player tracking module (PTM) to order drinks, perform room service, hail a car, etc. Additionally, the technology disclosed herein may be used in connection with table games. For example, a display may be used to show cards for, for example, poker, blackjack, or other card games. Gesture functionality may be provided to flip (rotate wrist), discard (flick), "hit" (double tap), or perform other actions on cards. The display may be mounted under a table or other surface, and sensing systems may be provided at each of a number of defined seats around the table. In another exemplary embodiment, a more static or fixed physical table. In such cases, the technology disclosed herein may track the tossing of cards, such as in pai gow, the rolling of virtual dice, such as in dice games such as craps, etc. Comparison with existing touch sensing technologies

[0291] The exemplary approach described herein has several advantages over existing techniques, including existing EIT-based techniques. The following table summarizes some advantages that are made possible through the implementation of the techniques disclosed herein. It will be understood that some or all of these and / or other advantages may be present depending on the implementation. At least some such advantageous aspects may naturally derive from the implementation of the techniques disclosed herein. However, it will be understood that the claims are illustrative of the invention and that these advantages should not be read automatically and completely therein. [Table 1] TIFF2024508163000004.tif213167 TIFF2024508163000005.tif101169

[0292] In certain exemplary embodiments, the algorithmic approaches detailed above can be used individually. In different exemplary embodiments, they can be combined to increase accuracy and repeatability. The divider loop approach has been found to be the fastest of the first three described approaches. It can be used to track touch location based on the activated electrodes in each division (hence the term "divider loop"). Thus, not all electrodes need to be used all the time. For example, initially, only six electrodes are used for the first touch, etc. That is, in certain exemplary embodiments, in the first measurement, the two middle electrodes are used to apply current, then the left and right electrode pairs are used to measure voltage to determine which area to select as the new active area (e.g., test the left half further or test the right half further), and this process continues after the new half is selected. This algorithm has advantages such as lower energy consumption, longer life, etc.

[0293] The discrete loop algorithmic approach works differently compared to the divider loop algorithmic approach because in the discrete loop algorithmic approach all electrodes are active and measurements can be transferred for tracking purposes that may be useful in gesture detection. The sensitivity and accuracy of the discrete loop algorithmic approach may be higher than the divider loop discrete loop algorithmic approach, but the speed may be slower.

[0294] The 4-loop algorithmic approach is similar in meaning to the discrete loop, where all electrodes are active for continuous measurements. This algorithm is typically the slowest compared to the other two algorithms. Since the 4-loop does not necessarily require region segmentation, it does not have to use sub-pixelation, but can instead use centroiding or other image processing approaches, which can be very fast. Furthermore, it may be the simplest to implement in conjunction with repositionable electrodes or electrodes that can be positioned at the time of delivery, complex substrate shapes and / or topographies, etc.

[0295] Some exemplary embodiments using machine learning algorithms are described as using XLD contours and / or paths in conjunction with the "IS" and "SHOULD" models, and cluster-based approaches are also described. It will be appreciated that any suitable ML technique may be used in different exemplary embodiments. For example, models constructed by regression, nearest neighbor (NN), k-nearest neighbor (KNN), convolutional neural network (CNN), support-vector machine (SVM), decision tree, and / or other techniques may be used in different exemplary embodiments. Decision trees are particularly useful for understanding simple feature requirements and neural networks are particularly useful for understanding complex feature requirements, although as noted above, any ML model approach may be used in different exemplary embodiments.

[0296] It will be understood that in different exemplary embodiments, more or fewer electrodes may be provided, in different exemplary embodiments, electrodes may be provided along only some edges (e.g., only the top and bottom edges, only the left and right edges, etc.), the spacing between electrodes may be uniform or not uniform in different exemplary embodiments, etc. Having electrodes along a partial subset of edges, such as only opposing edges of a substrate, may be advantageous, especially for large interfaces. The algorithmic approaches described above may be adjusted to account for this different arrangement. In an exemplary embodiment where a non-uniform electrode arrangement is present, more current may be applied to the surface to help address the non-uniformity, compared to an arrangement where uniform spacing is present. In such situations, different calibration, stabilization, and / or training data may be collected.

[0297] Certain exemplary embodiments have been described in connection with electrodes regularly spaced and / or oriented around the periphery of the substrate, e.g., on all four sides of the substrate. Symmetric electrode locations with equal distances have the advantage of reducing algorithmic complexity. However, it will be appreciated that in different exemplary embodiments, asymmetric electrode locations with irregular spacing, and optionally with electrodes on only a partial subset of the sides of the substrate, may be used. Calibration for such asymmetric electrode positioning may be more complex, but can be handled by algorithmic adaptation, stabilization and calculation of grid characteristic values, ML model training, etc. after such electrode placement is established.

[0298] In this regard, it will be appreciated that understanding touch and tracking touch location is made possible by reading electrode values ​​and using them in the algorithms described herein. One task that makes this possible is to measure the voltages of the electrodes (according to techniques specific to the algorithms described above) and convert these values ​​to at least one touch location. With symmetrical (equidistant) electrode placement, the conversion of measurements to touch location can be based on distance, which is equal between all electrodes and therefore can be simple to track, especially if a pre-calibration is performed based on surface size, number of electrodes, etc., to obtain the coefficients used in the conversion function. This can be more complicated for asymmetrical electrode locations, but is still possible.

[0299] It will be appreciated that the segmentation approach outlined in relation to FIG. 4 may be used in relation to certain exemplary embodiments. That is, in certain exemplary embodiments, each electrode may be connected to a multiplexer for task management (e.g., to apply a current or measure a voltage). Furthermore, tasks may be performed for different predefined, potentially overlapping sub-regions of the interface. In certain exemplary embodiments, any one or more of the algorithms described above may be used for each sub-region. The segmentation approach may be advantageous when it is desired to increase the speed of recognizing interactions, which may be particularly important for large interfaces. Certain exemplary embodiments may take advantage of processors having multiple processing cores, for example. In this regard, the overlapping segmentation approach may be easily implemented using the four-loop algorithmic approach outlined above. However, other algorithmic approaches may need to be modified, for example, to custom define regions for the divider loop algorithm and the discrete loop algorithm, as will be appreciated by those skilled in the art. Thus, an approach such as that described in FIG. 4 may be used in certain exemplary embodiments to increase system speed and / or accuracy, for example, by using an architecture having multiple processing cores, etc.

[0300] Although certain exemplary embodiments have been described in relation to human movement, it will be understood that the same or similar techniques may be applied in relation to animals. For example, it may be desirable to detect the movement of livestock on a farm (e.g., in an open environment, in a pen, in a processing facility, etc.), service or companion animals in any environment, etc. This may be useful for tracking and inventory control, trying to avoid accidents in areas where animals may encounter machinery, etc.

[0301] Certain exemplary embodiments have been described as including one sensing system per sensing area. However, in different exemplary embodiments, multiple sensing systems may be provided for a single sensing area. These multiple sensing systems for a single sensing area may project differently configured electric fields, for example, perpendicular to one another and / or in a manner that otherwise spans a desired space. The ability to use multiple sensing systems for a single area may be beneficial in providing increased accuracy and / or precision in 3D or other spaces.

[0302] Although certain exemplary embodiments have been described in relation to EIT techniques, other tomography techniques (including ECT techniques), for example based on surface conductivity, may be used in different exemplary embodiments. In such embodiments, the need to create a complete image of the substrate, similar to that described above, is alleviated. It will be understood that the techniques disclosed herein may generate an electric field and monitor changes to the electric field. However, monitoring the changes and determining "interaction" with the electric field need not depend on a complete reconstruction of the (changed) electric field. Thus, tomography techniques according to certain exemplary embodiments should be understood to create an electric field, but should not be deemed to require a subsequent (re)construction of the entire electric field, unless specifically stated and / or claimed.

[0303] Although certain exemplary embodiments have been described with reference to glass substrates, different materials may be used for the conductive surface of the technology described herein. Plastic and other transparent materials may be used, as well as non-transparent materials. It will be understood that different exemplary applications may use transparent or non-transparent conductive surfaces. When non-transparent surfaces are used, or where aesthetics are less of a concern (e.g., less than transparent touch applications), the electrodes may be positioned in the center or other areas of the surface, rather than restricting their positioning to peripheral areas.

[0304] It will be understood that as used herein, a gesture can refer to a provided movement that is either a contact gesture or a non-contact gesture. In a touch system, for example, a gesture can be a contact gesture or a hover-related gesture. In different use cases, gestures that can be detected can be intentional or unintentional, and the term "gesture" can be used to cover either or both scenarios.

[0305] While the present invention has been described in connection with what are presently considered to be practical and preferred embodiments, it is to be understood that the invention is not limited to the disclosed embodiments and / or deposition techniques, but rather is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. 1. A detection system comprising: A conductive surface; a plurality of electric field defining electrodes electrically connected to the conductive surface, the surface and the electrodes cooperating to create an electric field within a sensing area, the created electric field acting as a passive sensor for the detection system; a controller operatively coupled to a memory, the memory storing a plurality of defined templates corresponding to respective detection system events for the sensing area, the templates having corresponding actions defined therefor; The controller comprises: monitoring changes associated with the electric field based on data derived from outputs from at least some of the electrodes, the derived data being indicative of interactions with the sensing area, the interactions having characteristics including three-dimensional spatial and timing characteristics; determining whether the interaction matches one of the defined templates without reconstructing the entire electric field; selectively triggering the corresponding action based on one or more of the characteristics of the interaction in response to determining that the interaction matches one of the defined templates; A detection system configured to execute program logic to perform operations including:

2. 1. An industrial safety system for guarding a defined area, comprising: A conductive surface; a plurality of electric field defining electrodes electrically connected to the conductive surface, the surface and the electrodes cooperating to create an electric field about the defined area, the created electric field acting as a passive sensor for the defined area; a controller operatively coupled to a memory, the memory storing a plurality of defined templates corresponding to respective industrial safety system safety events associated with the defined area, the templates having corresponding actions defined therefor; The controller comprises: monitoring changes associated with the electric field based on data derived from outputs from at least some of the electrodes, the derived data being indicative of interactions with the defined area, the interactions having characteristics including three-dimensional spatial and timing characteristics; determining whether the interaction matches one of the defined templates without reconstructing the entire electric field; selectively triggering the corresponding action based on one or more of the characteristics of the interaction in response to determining that the interaction matches one of the defined templates; and configured to execute program logic to perform operations including:

3. 1. A detection system for use with a machine, comprising: a conductive surface provided on the machine; a plurality of electric field defining electrodes provided on the machine and electrically connected to the conductive surface, the surface and the electrodes cooperating to create an electric field relative to the machine, the created electric field acting as a passive sensor for the detection system; a controller operatively coupled to a memory, the memory storing a plurality of defined templates corresponding to respective detected system events associated with the machine, the templates having corresponding actions defined therefor; The controller comprises: monitoring changes associated with the electric field based on data derived from output from at least some of the electrodes, the derived data being indicative of an interaction with the machine, the interaction having characteristics including three-dimensional spatial and timing characteristics; determining whether the interaction matches one of the defined templates without reconstructing the entire electric field; selectively triggering the corresponding action based on one or more of the characteristics of the interaction in response to determining that the interaction matches one of the defined templates; A detection system configured to execute program logic to perform operations including:

4. The system of any of claims 1 to 3, wherein the conductive surface is a mat, an applique, or a doorway.

5. The conductive surface may be a sprayed conductive material, a thin film or other coating, an applied conductive tape, and / or Conductive particles laminated, molded or embedded therein; The system according to any one of claims 1 to 3, comprising:

6. The system of claim 2 , wherein the electric field defining electrodes are within or around the defined area.

7. the electric field defining electrodes are provided around at least one periphery of the defined area; The system of claim 2 , wherein the conductive surface is transparent or the defined area is an opening.

8. The system of claim 2 , wherein the electric field-defining electrodes are provided around at least a portion of a perimeter of the defined area.

9. The system of any preceding claim, wherein the defined template is structured to distinguish between different detectable disturbances to the electric field.

10. the defined template is structured to distinguish between different detectable disturbances to the electric field; The system of claim 2 , wherein the detectable disturbance includes a human or object passing through the defined area, coming within one or more predetermined threshold distances of the defined area, and / or entering the defined area.

11. The system of claim 10 , wherein the object is any contaminant that affects the electric field, the contaminant being a pest, a metal particle, a solid waste, and / or a liquid.

12. The system of claim 9 , wherein the detectable disturbance comprises an electromagnetic field attenuator.

13. The action is: activating lights, signs, and / or sirens; slowing down objects or equipment; shutting down objects or equipment; de-energizing objects or equipment; sending phone calls, SMS, emails, or other electronically mediated messages or alerts; moving an object or device relative to the defined area such that the object or device moves into or out of the defined area or to a designated location; The system of any one of claims 1 to 3, wherein the programmable actions are selected from a set of programmable actions comprising:

14. The controller: determining whether multiple interactions with the defined area occur in close and / or overlapping temporal proximity to one another; determining whether each of these interactions matches one of the defined templates without reconstructing the entire electric field; in response to determining that two or more of the plurality of interactions match one of the defined templates, selectively triggering the corresponding action based on one or more of the characteristics of the interactions; The system of claim 2 , further configured to perform further operations including:

15. The controller: determining whether multiple interactions with the machine occur in close and / or overlapping temporal proximity to one another; determining whether each of these interactions matches one of the defined templates without reconstructing the entire electric field; in response to determining that two or more of the plurality of interactions match one of the defined templates, selectively triggering the corresponding action based on one or more of the characteristics of the interactions; The system of claim 3 , further configured to perform further operations including:

16. 16. The system of claim 14 or 15, wherein the templates have different severity levels assigned to them, and the actions are triggered according to the different respective severity levels.

17. 20. The system of claim 16, wherein actions are processed according to a priority queue based on different severity levels.

18. The system of claim 2 , wherein a glass, plastic, wire, or other barrier is provided.

19. The system of claim 2 , wherein the defined area represents at least a bottom, side, lower portion, and / or top portion of a glass, plastic, wire, or other barrier.

20. 3. The system of claim 2, wherein the defined area is within or outside a portion of a machine, a prohibited space, a walkway, an intersection, or other defined path, and / or within or outside a control area or operator area.

21. The defined area is inside or outside a control or operator area; 21. The system of claim 20, wherein at least one of the templates determines whether a human is present and / or alert in the control or operator area, and if the human is not present and / or alert, the corresponding action is to deactivate a machine.

22. A system according to any preceding claim, wherein the conductive surface is provided on a first plane and at least one of the electric field-defining electrodes is provided on a plane other than the first plane.

23. The system of claim 1 , wherein at least some of the templates are machine-learned.

24. The system of claim 2 , wherein at least some of the templates are machine learned while the safety system is guarding the defined area.

25. The system of claim 3 , wherein the conductive surface and / or the electric field defining electrodes are provided on a mobile machine.

26. 26. The system of claim 25, wherein the machine is an autonomous guided vehicle.

27. 26. The system of claim 25, wherein the machine is operated by a human and at least one of the detection system events relates to sensing in a blind spot or area of ​​the human operator in an otherwise obstructed environment.

28. The system of claim 6 , wherein the conductive surface and / or the electric field defining electrodes are provided to a robot or cobot.

29. The system of claim 3 , wherein the conductive surface is a skin of the robot or cobot.

30. The system of claim 1 , wherein the conductive surface and / or the electric field defining electrodes are provided on a mobile machine and the detection system is operably coupled to a control system of the machine.

31. The system of claim 1 , wherein at least some of the detected system events are associated with unplanned and / or unexpected disturbances to the electric field.

32. 32. The system of claim 31, wherein the unplanned and / or unexpected disturbance is caused by a recognizable electronic signal emanating from a positioned marker.

33. The system of claim 1 , wherein the defined area is at least a portion of a conveyor.

34. The system of claim 1 , wherein at least some of the events relate to identifying stored energy.

35. The system of claim 1 , wherein the conductive surface and / or the electric field defining electrodes are provided on a machine used for loading / unloading operations.

36. The system of claim 3 , wherein markers or conductive paint are provided in an area from which objects are moved during loading / unloading operations to disrupt the electric field.

37. 37. The system of claim 36, further comprising an interface to a control system of the machine, the control system of the machine being controllable in response to disturbances to the electric field during load / unload operations.

38. A system according to any one of claims 1 to 3, provided in duplicate for redundant detection of the same event.

39. A system according to any preceding claim, wherein separate safety systems are provided which implement different sensing technologies.

40. A system according to any one of claims 1 to 3, provided as a redundant detection system, the controller being configured to switch over to the primary detection system in case of a detected malfunction.

41. 1. An object tracking system, comprising: A conductive surface; a plurality of electric field defining electrodes electrically connected to the conductive surface, the surface and the electrodes cooperating to create an electric field in a sensing area, the created electric field acting as a passive sensor for the sensing area; a controller operatively coupled to a memory, the memory storing a plurality of defined templates corresponding to object tracking system events for the sensing area, the templates having corresponding actions defined therefor; The controller comprises: monitoring changes associated with the electric field based on data derived from outputs from at least some of the electrodes, the derived data being indicative of interactions with the sensing area, the interactions having characteristics including three-dimensional spatial and timing characteristics; determining whether the interaction matches one of the defined templates without reconstructing the entire electric field; selectively triggering the corresponding action based on one or more of the characteristics of the interaction in response to determining that the interaction matches one of the defined templates; 16. An object tracking system configured to execute program logic to perform operations including:

42. 42. The system of claim 41, wherein the conductive surface is provided in the sensing area.

43. 42. The system of claim 41, wherein an object is movable into / out of the sensing area, and movement of an object into / out of the sensing area causes a change in the electric field.

44. 42. The system of claim 41, wherein movement of an object into or out of the sensing area causes a change in the electric field due to inherent electric field modifying properties of the object itself.

45. 42. The system of claim 41, wherein a tag or marker applied to an object moved into or out of the sensing area causes a change in the electric field.

46. 42. The system of claim 41, wherein the controller is configured to detect the object moving into / out of the sensing area based on changes to the electric field caused by the object and / or a tag or marker applied thereto by monitoring changes with the electric field.

47. 47. The system of claim 46, wherein the conductive surface is a mat and the object is moveable to / from the mat.

48. 42. The system of claim 41, wherein the controller is configured to distinguish between different types of objects detected within the sensing area.

49. 47. The system of claim 46, wherein the conductive surface is provided in, or as part of, a floor and / or a ceiling.

50. 42. The system of claim 41, wherein the sensing area includes a conduit and the object tracking system event relates to a blockage, a clog, an expected flow, an unexpected flow, and / or a contaminant in the conduit.

51. 42. The system of claim 41, further comprising an interface to an inventory tracking and / or management system, wherein the controller is configured to detect movement of an object and send an associated electronic message to the inventory tracking and / or management system.

52. 42. The system of claim 41, wherein the detectable object is a human.

53. 53. The system of claim 52, wherein the controller is further configured to track how many people are present within the sensing area and, optionally, where they are located.

54. 1. A method of using a detection system, comprising: creating an electric field within a sensing area, the created electric field acting as a passive sensor for the detection system, the electric field being created using a conductive surface and a number of field-defining electrodes electrically connected to the conductive surface; having a memory, the memory storing a plurality of defined templates corresponding to respective detection system events for the sensing area, the templates having corresponding actions defined therefor; monitoring changes associated with the electric field based on data derived from outputs from at least some of the electrodes, the derived data being indicative of interactions with the sensing area, the interactions having characteristics including three-dimensional spatial and timing characteristics; determining whether the interaction matches one of the defined templates without reconstructing the entire electric field; selectively triggering the corresponding action based on one or more of the characteristics of the interaction in response to determining that the interaction matches one of the defined templates; A method comprising:

55. 55. A non-transitory computer readable storage medium storing instructions that, when executed, perform the method of claim 54.

Images