Touch sensing in non-capacitive touch mode

Abstract

In some implementations, a method may involve detecting that touch sensing using a capacitive touch sensor system is inoperable in a given user context. Touch sensing using the capacitive touch sensor is disabled, and touch sensing using an alternative sensor system is enabled. The alternative sensor system may include a fingerprint sensor system, an optical sensor system, and a force sensor system. Touch sensing using the alternative sensor system may be limited to specified portions in the graphical user interface to achieve an improved user experience. In some implementations, the specified portion may be co-extensive with an active area of the fingerprint sensor system, and a user interface control may be arranged into an array of tiles.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application No. 17 / 806,233, filed on June 9, 2022, entitled “TOUCH SENSING IN NON-CAPACITIVE TOUCH MODES,” which is hereby incorporated by reference. Technical Field

[0003] The present disclosure generally relates to touch screen devices and related methods, including but not limited to touch screen devices and methods operating in modes using fingerprint sensor systems, optical sensor systems, force sensor systems, and other sensor systems for detecting touch. Background Art

[0004] Touch screen devices are often characterized by multiple devices. Many touch screen devices, including smartphones, use capacitive touch screen technology to register user input on the touch screen device. Capacitive touch screen technology has largely replaced resistive touch screens due to industrial design, durability, and performance considerations. Generally, capacitive touch screens require "bare-hand" contact to sense touch because the change in capacitance on the touch screen is caused by a small charge attracted to the fingertip at the point of contact. This can cause problems when a user is operating a touch screen device underwater or when the user is wearing protective gloves. Summary of the invention

[0005] The systems, methods and devices of the disclosure each have several innovative aspects, no single one of which is solely responsible for the desirable attributes disclosed herein.

[0006] One innovative aspect of the subject matter described in the present disclosure may be implemented in a device. The device may include a display, a capacitive touch sensor system, and one or more of the following: a fingerprint sensor system, an optical sensor system, or a force sensor system. The device may also include: a control system configured to communicate with the capacitive touch sensor system, the fingerprint sensor system, and the display, wherein the control system is further configured to: determine that a user context of the device is incompatible with touch sensing for the capacitive touch sensor system, and present a graphical user interface on the display of the device, wherein the graphical user interface includes a first portion, the first portion configured to sense touch using one or more of the following: the fingerprint sensor system, the optical sensor system, or the force sensor system.

[0007] According to some implementations, the first portion includes an area of ​​the display that is coextensive with an active area of ​​the fingerprint sensor system. In some implementations, the graphical user interface also includes a second portion, wherein the second portion of the graphical user interface is configured to display a video or image. In some implementations, the control system configured to determine that the user context is incompatible with touch sensing of the capacitive touch sensor system is configured to determine that the user context includes one or more of the following: (i) the display of the device is covered or submerged in water, (ii) the user is wearing gloves, or (iii) the user is using a prosthetic or robotic hand.

[0008] According to some specific implementations, the control system is further configured to: use the fingerprint sensor system to obtain the position of the touch input on the graphical user interface. In some specific implementations, the control system configured to use the fingerprint sensor system to obtain the position of the touch input on the graphical user interface is configured to: divide the first portion of the graphical user interface into a tile array corresponding to the active area of ​​the fingerprint sensor system; use the fingerprint sensor system to perform a scan of one or more tiles in the tile array to identify the user's finger; and associate the user's finger with one of the tiles in the first portion of the graphical user interface to obtain the position of the touch input. In some specific implementations, the control system configured to use the fingerprint sensor system to obtain the position of the touch input on the graphical user interface is configured to: use one or more pressure or force sensors associated with the force sensor system or fingerprint sensor system to detect the applied force on the display; use the fingerprint sensor system to perform a scan to identify the user's finger; and associate the user's finger with at least one tile in the tile array in the first portion of the graphical user interface to obtain the position of the touch input, wherein the tile array divides the first portion into a grid.

[0009] According to some specific implementations, the fingerprint sensor system includes an ultrasonic fingerprint sensor system. The control system configured to use the fingerprint sensor system to obtain the position of the touch input on the graphical user interface may be configured to: cause an ultrasonic transmitter or transceiver in the ultrasonic fingerprint sensor system to generate an ultrasonic wave; receive reflections of the ultrasonic wave at a piezoelectric receiver array to obtain image data identifying a stylus or pen in contact with a surface of the device; and associate the stylus or pen with a position in the first portion of the graphical user interface to obtain the position of the touch input using the image data. In some specific implementations, the control system configured to use the fingerprint sensor system to obtain the position of the touch input on the graphical user interface may be configured to: receive an indication of mechanical deformation at the piezoelectric receiver array of the ultrasonic fingerprint sensor system to obtain image data identifying a stylus or pen in contact with a surface of the device; and associate the stylus or pen with a position in the first portion of the graphical user interface to obtain the position of the touch input using the image data.

[0010] According to some specific implementations, the control system is further configured to: use the force sensor system to obtain the position of the touch input on the graphical user interface, wherein the control system configured to use the force sensor system to obtain the position of the touch input is configured to: detect an applied force from an object on the display; obtain multiple force measurements from multiple force sensors associated with the force sensor system; each of the multiple force sensors is positioned at a different position of the device; and estimate the position of the object in the first portion of the graphical user interface based on the multiple force measurements to obtain the position of the touch input.

[0011] Other innovative aspects of the subject matter described in the present disclosure can be implemented in a method. The method may include: determining, using a control system, that a user context of a touch screen device is incompatible with touch sensing by a capacitive touch sensor system of the touch screen device; and presenting, using the control system, a graphical user interface on a display of the touch screen device, wherein the graphical user interface includes a first portion configured to sense touch using one or more of: a fingerprint sensor system, an optical sensor system, or a force sensor system of the touch screen device.

[0012] According to some implementations, the first portion of the graphical user interface includes an area of ​​the display that is coextensive with an active area of ​​the fingerprint sensor system. In some implementations, the graphical user interface also includes a second portion, wherein the second portion of the graphical user interface is configured to display a video or image. In some implementations, the user context that is incompatible with touch sensing of the capacitive touch sensor system includes one or more of the following: (i) the display of the touch screen device is covered or submerged in water, (ii) the user is wearing gloves, or (iii) the user is using a prosthetic or robotic hand.

[0013] According to some specific implementations, the method further includes: using the control system, using the fingerprint sensor system to obtain the position of the touch input on the graphical user interface. In some specific implementations, using the fingerprint sensor system to obtain the position of the touch input on the graphical user interface includes: dividing the first portion of the graphical user interface into a tile array corresponding to the active area of ​​the fingerprint sensor system; using the fingerprint sensor system to perform a scan of one or more tiles in the tile array to identify a user's finger; and associating the user's finger with one of the tiles in the first portion of the graphical user interface to obtain the position of the touch input. In some specific implementations, using the fingerprint sensor system to obtain the position of the touch input on the graphical user interface includes: using one or more pressure or force sensors associated with the force sensor system or fingerprint sensor system to detect an applied force on the display; using the fingerprint sensor system to perform a scan to identify a user's finger; and associating the user's finger with at least one tile in the tile array in the first portion of the graphical user interface to obtain the position of the touch input, wherein the tile array divides the first portion into a grid.

[0014] According to some specific implementations, the fingerprint sensor system includes an ultrasonic fingerprint sensor system. In some specific implementations, using the fingerprint sensor system to obtain the position of the touch input on the graphical user interface includes: causing an ultrasonic transmitter or transceiver in the ultrasonic fingerprint sensor system to generate an ultrasonic wave; receiving a reflection of the ultrasonic wave at a piezoelectric receiver array to obtain image data identifying a stylus or pen in contact with the surface of the touch screen device; and associating the stylus or pen with a position in the first portion of the graphical user interface to obtain the position of the touch input using the image data. In some specific implementations, using the fingerprint sensor system to obtain the position of the touch input on the graphical user interface includes: receiving an indication of mechanical deformation at the piezoelectric receiver array of the ultrasonic fingerprint sensor system to obtain image data identifying a stylus or pen in contact with the surface of the touch screen device; and associating the stylus or pen with a position in the first portion of the graphical user interface to obtain the position of the touch input using the image data.

[0015] According to some specific implementations, the method further includes: using the optical sensor system to obtain a position of the touch input on the graphical user interface.

[0016] According to some specific implementations, the force sensor system is used to obtain a position of a touch input on the graphical user interface. Using the force sensor system to obtain the position of the touch input may include: detecting an applied force from an object on the display; obtaining a plurality of force measurements from a plurality of force sensors associated with the force sensor system; each of the plurality of force sensors is positioned at a different position of the touch screen device; and estimating the position of the object in the first portion of the graphical user interface based on the plurality of force measurements to obtain the position of the touch input.

[0017] Some or all of the operations, functions, and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Therefore, some innovative aspects of the subject matter described in this disclosure may be implemented in one or more non-transitory media having software stored thereon.

[0018] For example, the software may include instructions for controlling one or more devices to perform a method. In some implementations, the method may involve: determining that a user context of a touch screen device is incompatible with touch sensing of a capacitive touch sensor system of the touch screen device; and presenting a graphical user interface on a display of the touch screen device, wherein the graphical user interface includes a first portion configured to sense touch using one or more of: a fingerprint sensor system, an optical sensor system, or a force sensor system of the touch screen device.

[0019] According to some embodiments, the user context that is incompatible with touch sensing of the capacitive touch sensor system includes one or more of the following: (i) the display of the touch screen device is covered or submerged in water, (ii) the user is wearing gloves, or (iii) the user is using a prosthetic or robotic hand.

[0020] In some implementations, the method may further involve: using the fingerprint sensor system to obtain a position of the touch input on the graphical user interface. In some implementations, the method may further involve: using the force sensor system to obtain a position of the touch input on the graphical user interface. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Details of one or more specific implementations of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the specification, drawings, and claims. Note that the relative dimensions in the drawings may not be drawn to scale. The same reference numerals and names in the various drawings indicate the same elements.

[0022] Figure 1 is a block diagram illustrating example components of an apparatus according to some disclosed implementations.

[0023] Figure 2 A flow diagram is shown that provides an example of operations for detecting user context operating in a non-capacitive touch mode according to some disclosed implementations.

[0024] Figure 3 A flow chart is shown that provides an example of operations for operating in a non-capacitive touch mode according to some disclosed implementations.

[0025] Figure 4A An example ultrasound image of a touch screen device submerged in water is shown according to some implementations.

[0026] Figure 4B An example ultrasound image is shown with water droplets on the surface of a touch screen device according to some implementations.

[0027] Figure 5A An example of a device with a graphical user interface (GUI) in a configuration that limits a touch-sensitive area of ​​an application to a portion of the GUI is shown according to some implementations.

[0028] Figure 5B An example of a device with a GUI in a configuration that limits a touch-sensitive area of ​​a music player application to a portion of the GUI is shown according to some implementations.

[0029] Fig. 6A An example of a device with a GUI in a configuration with touch sensing in a capacitive touch sensing mode enabled is shown.

[0030] Figure 6B An example of a device with a GUI in a configuration with touch sensing in a non-capacitive touch sensing mode enabled is shown according to some implementations.

[0031] Figure 6C An example of a device with a GUI in a configuration with touch sensing in a non-capacitive touch sensing mode enabled for an application is shown according to some implementations.

[0032] Fig.6D An example of a device with a GUI in a configuration with touch sensing in a non-capacitive touch sensing mode enabled for a camera is shown according to some implementations.

[0033] Fig. 6E An example of a device with a GUI in a configuration with touch sensing in a non-capacitive touch sensing mode enabled for a music player application is shown according to some implementations.

[0034] Fig. 7A A cross-sectional schematic diagram of an example fingerprint sensor system configured for touch detection using mechanical deformation according to some implementations is shown.

[0035] Figure 7B A cross-sectional schematic diagram of an example fingerprint sensor system configured for touch detection using ultrasonic reflection is shown according to some implementations.

[0036] Figure 8 A schematic block diagram of an example force sensor system configured for touch detection using multiple force sensors according to some implementations is shown. DETAILED DESCRIPTION

[0037] For the purpose of describing the innovative aspects of the present disclosure, the following description relates to certain specific implementations. However, one of ordinary skill in the art will readily recognize that the teachings herein may be applied in many different ways. The described specific implementations may be implemented in any device, apparatus, or system that includes a touch sensing system as disclosed herein. In addition, it is contemplated that the described specific implementations may be included in or associated with various electronic devices, such as, but not limited to: mobile phones, multimedia Internet-enabled cellular phones, mobile television receivers, wireless devices, smart phones, smart cards, wearable devices such as bracelets, armbands, wristbands, rings, headbands, patches, etc., devices, personal data assistants (PDAs), wireless email receivers, handheld or portable computers, netbooks, notebooks, smartbooks, tablet computers, printers, copiers, scanners, fax machines, global positioning system (GPS) receivers / navigators, cameras, digital multimedia players (such as MP3 players), camcorders, game consoles, watches, clocks, calculators, television monitors, flat panel displays, electronic reading devices (e.g., e-readers), mobile health devices, computer monitors, automotive displays (including odometer and speedometer displays, etc.), cockpit controls and / or displays, camera view displays (such as displays for rear-view cameras in vehicles), electronic photographs, electronic billboards or signs, projectors, architectural structures, microwave ovens, refrigerators, stereo systems, cassette recorders or players, DVD players, CD players, VCRs, radios, portable memory chips, washing machines, dryers, washer / dryers, parking meters, packaging (such as in electromechanical systems (EMS) applications, including microelectromechanical systems (MEMS) applications, as well as non-EMS applications), aesthetic structures (such as image displays on a piece of jewelry or clothing), and various EMS devices. The teachings herein may also be used in applications such as, but not limited to, electronic switching devices, radio frequency filters, sensors, accelerometers, gyroscopes, motion sensing devices, magnetometers, inertial components for consumer electronic devices, components for consumer electronic products, steering wheels or other automotive components, varactor diodes, liquid crystal devices, electrophoretic devices, drive schemes, manufacturing processes, and electronic test equipment, etc. Therefore, these teachings are not intended to be limited to the specific implementations depicted in the figures, but rather have broad applicability, as will be apparent to one of ordinary skill in the art.

[0038] More and more devices have adopted touch screens, which enable users to interact with graphical user interfaces via touch as input. Users can rely on touch screen input as an alternative to physical buttons, keyboards and other input devices. Many touch screen devices operate using capacitive touch screen technology. Typically, a capacitive touch screen device includes an insulator (e.g., glass) coated with a transparent conductor (e.g., indium tin oxide). Because the human body can act as an electrical conductor, touching the touch screen causes distortion of the electrostatic field of the touch screen. The distortion can be measured as a change in capacitance, which in turn can be used to determine the contact position on the touch screen.

[0039] There are various user scenarios that can render a capacitive touch screen device inoperable. One situation occurs when the touch screen device is submerged underwater. In such instances, the touch screen device is overwhelmed by the capacitance change and essentially acts as if a giant finger uniformly covers the touch screen, thereby hindering touch detection. Even for devices that claim to be water-resistant or waterproof, touch detection can deteriorate or not work when such devices are covered with water. Another situation occurs when a user operates a touch screen device using gloves. Typically, the material of the gloves is an electrical insulator that insulates the user's fingers. This prevents the capacitive touch screen from detecting the conductivity of the fingertips through the gloves. Therefore, when the user is wearing gloves, the capacitive touch screen may be inoperable. This is a problem for users who wear gloves in cold weather conditions and for users who wear gloves to protect their hands in certain occupations or recreational activities. Similarly, another situation occurs when a user is operating a touch screen device with a prosthesis. Many prosthetic hands are made of non-conductive materials, making it impossible for the prosthetic hand to interact with capacitive touch screen technology.

[0040] Some of the disclosed methods, devices, and apparatuses relate to operating a touch screen device in a non-capacitive touch mode using an alternative sensor system (such as a fingerprint sensor system, an optical sensor system, or a force sensor system). The touch screen device may detect a user context that is incompatible with touch sensing of the device's touch sensor system (e.g., a capacitive touch sensor system). Upon detecting such a user context, the touch screen device may enter a non-capacitive touch mode, wherein a designated portion of a graphical user interface is configured to sense touch using an alternative sensor system to the capacitive touch sensor system. In some implementations, the designated portion may enable touch sensing using a fingerprint sensor system such as an ultrasonic fingerprint sensor system. In some examples, the fingerprint sensor system may perform a scan that identifies a user's finger on an active tile and then associates the active tile with a location on the graphical user interface. In some examples, the fingerprint sensor system receives reflections of an ultrasonic wave to identify an object (e.g., a stylus) to locate the location of the object, or the fingerprint sensor system may receive an indication of a mechanical deformation caused by an object (e.g., a stylus) to locate the location of the object. In some implementations, the designated portion may enable touch sensing using an optical sensor system to generate an image to determine the location of a user input. In some implementations, the designated portion can enable touch sensing using a force sensor system employing multiple force sensors to determine the location of a user input.

[0041] Specific implementations of the subject matter described in the present disclosure may be implemented to achieve one or more of the following potential advantages. The user experience is enhanced by enabling touch sensing and recognition in underwater environments, environments where the user is wearing gloves (e.g., cold weather), situations where the user has a prosthetic or robotic hand, or other environments that may normally make touch sensing inoperable. This allows the user to continue to operate their smartphone or other device while swimming, participating in water activities, wearing gloves, using a prosthetic or robotic hand, etc. Functionality is improved by designating a portion of a graphical user interface for touch sensing that corresponds to an active area of ​​an alternative sensor system. Less power is wasted by disabling the capacitive touch sensor system and enabling touch operation only in a designated portion corresponding to an active area of ​​an alternative sensor system. When a fingerprint sensor system is used for touch sensing, power can be saved by reducing the use of a fingerprint sensor transmitter during scanning and reducing the use of fingerprint image processing. Touch recognition is also improved by dividing a designated portion of a graphical user interface into an array of tiles to assist in associating user input with the location of the user input.

[0042] Figure 11 is a block diagram illustrating example components of an apparatus according to some disclosed implementations. In this example, apparatus 101 includes fingerprint sensor system 102, touch sensor system 103, interface system 104, control system 106, and display system 110. Some implementations may include memory system 108, optical sensor system 112, force sensor system 114, and / or gesture sensor system 116.

[0043] According to some examples, the fingerprint sensor system 102 may be or may include an ultrasonic fingerprint sensor. Alternatively or additionally, in some implementations, the fingerprint sensor system 102 may be or may include another type of fingerprint sensor, such as an optical fingerprint sensor, a photoacoustic fingerprint sensor, and the like. In some examples, an ultrasonic version of the fingerprint sensor system 102 may include an ultrasonic receiver and a separate ultrasonic transmitter. In some such examples, the ultrasonic transmitter may include an ultrasonic plane wave generator. However, various examples of ultrasonic fingerprint sensors are disclosed herein, some of which may include a separate ultrasonic transmitter, and some of which may not. For example, in some implementations, the fingerprint sensor system 102 may include a piezoelectric receiver layer, such as a polyvinylidene fluoride PVDF polymer layer or a polyvinylidene fluoride-trifluoroethylene (PVDF-TrFE) copolymer layer. In some implementations, a separate piezoelectric layer may be used as an ultrasonic transmitter. In some implementations, a single piezoelectric layer may be used as both a transmitter and a receiver. In some examples, the fingerprint sensor system 102 may include an array of ultrasonic transducer elements, such as an array of piezoelectric micromachined ultrasonic transducers (PMUTs), an array of capacitive micromachined ultrasonic transducers (CMUTs), etc. In some such examples, PMUT elements in a single-layer array of PMUTs or CMUT elements in a single-layer array of CMUTs may be used as ultrasonic transmitters and ultrasonic receivers.

[0044] The data received from the fingerprint sensor system 102 may sometimes be referred to herein as "fingerprint sensor data," "fingerprint image data," etc., although the data will typically be received from the fingerprint sensor system in the form of electrical signals. Thus, such image data may not necessarily be perceptible to humans as an image without additional processing.

[0045] The touch sensor system 103 may be or may include a capacitive touch sensor system, such as a surface capacitive touch sensor system or a projected capacitive touch sensor system. The touch sensor system 103 may be or may include other types of touch sensor systems, such as a resistive touch sensor system, a surface acoustic wave touch sensor system, an infrared touch sensor system, or other suitable types of touch sensor systems. Depending on the user context (e.g., underwater), some of the aforementioned touch sensor systems may make touch recognition inoperable. In some specific implementations, the area of ​​the touch sensor system 103 may extend over most or all of the display portion of the display system 110.

[0046] In some examples, the interface system 104 may include a wireless interface system. In some implementations, the interface system 104 may include a user interface system, one or more network interfaces, one or more interfaces between the control system 106 and the fingerprint sensor system 102, one or more interfaces between the control system 106 and the touch sensor system 103, one or more interfaces between the control system 106 and the memory system 108, one or more interfaces between the control system 106 and the display system 110, one or more interfaces between the control system 106 and the optical sensor system 112, one or more interfaces between the control system 106 and the force sensor system 114, one or more interfaces between the control system 106 and the gesture sensor system 116, and / or one or more interfaces between the control system 106 and one or more external device interfaces (e.g., ports or application processors).

[0047] The interface system 104 may be configured to provide communication between components of the apparatus 101 (which may include wired or wireless communication, electrical communication, radio communication, etc.). In some such examples, the interface system 104 may be configured to provide communication between the control system 106 and the fingerprint sensor system 102. According to some such examples, the interface system 104 may couple at least a portion of the control system 106 to the fingerprint sensor system 102, and the interface system 104 may couple at least a portion of the control system 106 to the touch sensor system 103, for example, via a conductive material (e.g., via a conductive metal wire or trace). According to some examples, the interface system 104 may be configured to provide communication between the apparatus 101 and other devices and / or humans. In some such examples, the interface system 104 may include one or more user interfaces. In some examples, the interface system 104 may include one or more network interfaces and / or one or more external device interfaces (such as one or more universal serial bus (USB) interfaces or serial peripheral interfaces (SPI)).

[0048] The control system 106 may include one or more general purpose single-chip or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or a combination thereof. According to some examples, the control system 106 may also include one or more memory devices, such as one or more random access memory (RAM) devices, read-only memory (ROM) devices, etc. In this example, the control system 106 is configured to communicate with and control the display system 110. In a specific implementation in which the device 101 includes a fingerprint sensor system 102, the control system 106 is configured to communicate with and control the fingerprint sensor system 102. In a specific implementation in which the device 101 includes a touch sensor system 103, the control system 106 is configured to communicate with and control the touch sensor system 103. In a specific implementation in which the device 101 includes a memory system 108 separate from the control system 106, the control system 106 may also be configured to communicate with the memory system 108. In an implementation where the device 101 includes an optical sensor system 112, the control system 106 is configured to communicate with and control the optical sensor system 112. In an implementation where the device 101 includes a force sensor system 114, the control system 106 is configured to communicate with and control the force sensor system 114. According to some examples, the control system 106 may include one or more dedicated components configured to control the fingerprint sensor system 102, the touch sensor system 103, the memory system 108, the display system 110, the optical sensor system 112, and / or the force sensor system 114.

[0049] Some examples of device 101 may include dedicated components configured to control at least a portion of fingerprint sensor system 102 (and / or to process fingerprint image data received from fingerprint sensor system 102). Figure 1106 as separate components, in some implementations, at least a portion of the control system 106 and at least a portion of the fingerprint sensor system 102 may be co-located. For example, in some implementations, one or more components of the fingerprint sensor system 102 may reside on an integrated circuit or "chip" of the control system 106. According to some implementations, the functionality of the control system 106 may be divided between one or more controllers or processors, such as between a dedicated sensor controller and an application processor of a device (also referred to herein as a "host" processor), such as a host processor of a mobile device. In some such implementations, at least a portion of the host processor may be configured for fingerprint image data processing, determining whether currently acquired fingerprint image data matches previously acquired fingerprint image data (such as fingerprint image data acquired during an enrollment process), and the like.

[0050] In some examples, the memory system 108 may include one or more memory devices, such as one or more RAM devices, ROM devices, etc. In some specific implementations, the memory system 108 may include one or more computer-readable media, storage media, and / or storage media. Computer-readable media include both computer storage media and communication media, including any media that can transfer a computer program from one place to another. The storage medium can be any available medium that can be accessed by a computer. In some examples, the memory system 108 may include one or more non-transient media. As an example and not limitation, non-transient media may include RAM, ROM, electrically erasable programmable read-only memory (EEPROM), compact disk ROM (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage device, or any other medium that can be used to store desired program code in the form of instructions or data structures and can be accessed by a computer.

[0051] In some examples, device 101 includes display system 110, which may include one or more displays. In some examples, display system 110 may be a light emitting diode (LED) display, such as an organic light emitting diode (OLED) display, or may include a light emitting diode (LED) display, such as an organic light emitting diode (OLED) display. In some such examples, display system 110 may include layers that may be collectively referred to as a "display stack."

[0052] In some implementations, the device 101 may include an optical sensor system 112. The optical sensor system 112 may include one or more arrays of optical sensor pixels. The one or more optical sensor pixel arrays may include one or more active pixel sensor arrays, which may include a complementary metal oxide semiconductor (CMOS) sensor, a charge coupled device (CCD) image sensor, a charge injection device (CID) sensor, or any other sensor capable of detecting changes in light. The optical sensor system 112 may be configured to capture images using one or more optical sensors (e.g., a camera).

[0053] In some implementations, the device 101 may include a force sensor system 114. The force sensor system 114 may include one or more force sensors. In some cases, the one or more force sensors may be multiple force sensors positioned at different locations of the device 101 below the display of the display system 110. The force sensor may be a piezoresistive sensor, a capacitive sensor, a thin film sensor (e.g., a polymer-based thin film sensor), or other type of suitable force sensor. If the force sensor is a piezoresistive sensor, the piezoresistive sensor may include silicon, metal, polysilicon, and / or glass. In some implementations, one or more force sensors may be mechanically coupled to the fingerprint sensor system 102. In some implementations, one or more force sensors may be separate from the fingerprint sensor system 102. One or more force sensors of the force sensor system 114 may be coupled to a portion of the control system 106.

[0054] In some implementations, device 101 can include gesture sensor system 116. Gesture sensor system 116 can be or can include an ultrasonic gesture sensor system, an optical gesture sensor system, or any other suitable type of gesture sensor system.

[0055] The device 101 can be used in a variety of different scenarios, some examples of which are disclosed herein. For example, in some implementations, a mobile device may include at least a portion of the device 101. In some implementations, a wearable device may include at least a portion of the device 101. The wearable device may be, for example, a bracelet, an armband, a wristband, a ring, a headband, or a patch. In some implementations, the control system 106 may reside in more than one device. For example, a portion of the control system 106 may reside in a wearable device, and another portion of the control system 106 may reside in another device such as a mobile device (e.g., a smart phone). In some such examples, the interface system 104 may also reside in more than one device.

[0056] Figure 2A flowchart is shown that provides an example of operations for detecting a user context operating in a non-capacitive touch mode according to some disclosed implementations. In some examples, process 200 can be at least partially performed by the methods described herein. Figure 1 The control system 106 is executed. As with other processes disclosed herein, Figure 2 The operations outlined in the process 200 may include different, more, or fewer blocks than those indicated. In addition, the blocks of the process 200 disclosed herein are not necessarily executed in the order indicated. In some specific implementations, one or more blocks may be executed concurrently.

[0057] At box 210 of process 200, the control system of the device may detect a constant pressure change. The device may be or may include a touch screen device such as a smart phone. The device may be equipped with one or more pressure sensors and / or one or more force sensors. The pressure and / or force sensor may detect a pressure change applied to the device. The detection of pressure changes may be implemented using only one force sensor or only one pressure sensor, because the pressure and / or force sensor does not need to estimate the position for touch sensing at this stage. Once the pressure or force applied to the device exceeds a threshold amount, the control system may initiate a determination of whether to enter a non-capacitive touch mode. The pressure change may be caused by a finger tapping the screen, by a water droplet, by an object pressed against the screen, or by any other external force or environmental force. The pressure change may indicate a user context that will make touch sensing using a capacitive touch sensor system inoperable.

[0058] In some implementations, the control system can be configured to detect changes in environmental conditions (such as humidity, temperature, ambient light, etc.). Detecting such changes in environmental conditions is performed as an alternative or in addition to detecting changes in constant pressure. For example, the control system can detect changes in humidity, which can indicate that the device is in a wet condition (e.g., submerged in water). The control system can detect changes in temperature, which can indicate that the device is in cold weather, where a user may be more likely to wear gloves. The control system can detect changes in ambient light, which can indicate that an object is approaching the device or that the device is underwater.

[0059] In the event that a change in pressure or other specified environmental condition is detected at block 210, process 200 may proceed to block 220. In the event that a change in pressure or other specified environmental condition is not detected, process 200 may proceed to block 230. At block 230, the device remains in the capacitive touch mode for touch sensing.

[0060] At block 220 of process 200, the control system of the device may identify the source of the change in pressure or environmental conditions. The identification of the source of the change may occur by identifying an object approaching the display of the device. At block 210, the identification of the object may be triggered by a change in pressure or a specified environmental condition. In some implementations, the identification of the object may be performed by scanning one or more areas of the display using a fingerprint sensor system or an optical sensor system. The fingerprint sensor system or the optical sensor system may be configured to scan a local or partial area of ​​the display to obtain data (e.g., image data) about an object approaching the display. The data may be used to classify the type of object approaching the display of the device. For example, the fingerprint sensor system may be an ultrasonic fingerprint sensor system configured to generate ultrasonic waves. The reflection of the ultrasonic waves may provide unique signal characteristics that may be used to identify the type of object. Different materials may reflect back different acoustic properties because different materials behave differently in terms of how the sound waves are reflected back. The acoustic properties may be analyzed to identify the object approaching the display of the device. This may be done by analyzing the sound waves reflected back at different frequencies. Using the acoustic properties, the control system may identify the type of object approaching the display, such as whether the object is solid or liquid, or the type of material from which the object is made. Thus, the control system can detect whether the device is underwater, whether the user is wearing gloves, and whether the user is using a prosthetic or robotic hand.

[0061] In some implementations, the detected pressure or change in the specified environmental condition at block 210 may alternatively trigger the presentation of a pop-up, window, or other interface by which the user may select a user context corresponding to the non-capacitive touch mode or directly select the non-capacitive touch mode. For example, the user may determine that the device is underwater, the user is wearing gloves, the user is using a prosthetic hand, the user is using a robotic hand, or a user context that otherwise renders capacitive touch sensing inoperable. Thus, the detected pressure or change in the specified environmental condition may trigger the user to initiate an opportunity for the device to enter the non-capacitive touch mode at block 240 or remain in the capacitive touch mode at block 230.

[0062] After determining at box 210 or box 220 that the device is compatible with touch sensing using a capacitive touch sensor system, at box 230, the control system may enable touch sensing using a capacitive touch mode. In the capacitive touch mode, the capacitive touch sensor may receive user input that provides a touch location corresponding to an x, y coordinate of a touch sensor coordinate system. At box 250, using these coordinates, a touch driver of the control system acts on the given coordinates. However, if it is determined at boxes 210 and 220 that the device is not compatible with touch sensing using a capacitive touch sensor system, then at box 240, the control system may enable touch sensing using a non-capacitive touch mode. In the non-capacitive touch mode, a fingerprint sensor system, an optical sensor system, a force sensor system, or other sensor system may be used to receive user input to provide a touch location. In some examples, the touch location may correspond to a valid tile in an array of tiles in a designated portion of a graphical user interface. At box 250, using the valid tile, a touch driver coupled to the control system acts on the valid tile.

[0063] Figure 3 A flowchart is shown that provides an example of operations for operating in a non-capacitive touch mode according to some disclosed embodiments. In some examples, process 300 can be at least partially performed by the methods described herein. Figure 1 The control system 106 is executed. As with other processes disclosed herein, Figure 3 The operations outlined in the process 300 may include different, more, or fewer blocks than indicated. Additionally, the blocks of the process 300 disclosed herein are not necessarily executed in the order indicated. In some implementations, one or more blocks may be executed concurrently.

[0064] At block 310 of process 300, the control system determines that the user context of the touch screen device is incompatible with touch sensing of the touch sensor system. In some implementations, the touch sensor system is a capacitive touch sensor system. Depending on the user context in which the touch screen device is operating, the touch sensor system may be rendered inoperable or substantially inoperable.

[0065] As used herein, "user context" generally describes the environment in which a user is located, the user's current activities, and / or the situation in which the user is operating. The terms "context," "user context," and "user environment" are used in this disclosure and may be used interchangeably. To list some non-limiting examples, user context may refer to the physical environment in which a user is located, the user's physical activities, the user's clothing, the user's transportation patterns, and / or the user's physiological condition.

[0066] In some implementations, the determination can be made through a user-initiated action. The control system can receive user input indicating that the user context is incompatible with touch sensing of the touch sensor system, or at least indicating that the user wants to enter a "special" or "non-capacitive" touch mode.

[0067] In some specific implementations, the determination can be made in an automatic manner by the control system. The control system can determine that the environmental condition has changed and initiate a scan using a fingerprint sensor system or an optical sensor system to obtain image data about an object approaching the touch screen device, wherein the image data can be used to identify the type of object approaching the touch screen device. When the force applied to the touch screen device exceeds a threshold amount, the pressure sensor or force sensor can detect a change in pressure. Alternatively, another environmental sensor such as a temperature sensor, a humidity sensor, an optical sensor, etc. can detect a change in a specified environmental condition. A change in pressure or a specified environmental condition can indicate a change in user context. This can trigger the control system of the touch screen device to identify the source of the change in pressure or specified environmental condition. In some instances, the source can be identified by performing a scan (e.g., a low-resolution scan) of certain activation areas of the fingerprint sensor system or the optical sensor system to obtain image data about an object approaching the touch screen device. The image data can be processed to determine whether the type of object approaching the touch screen device makes the touch sensor system inoperable or substantially inoperable. Examples of user contexts that will make the touch sensor system inoperable or substantially inoperable include, but are not limited to: (i) the touch screen device is covered or submerged in water, (ii) the user is wearing gloves, and (iii) the user is using a prosthetic or robotic hand. Thus, the image data may be processed to determine whether the state of the object is solid or liquid, whether the material of the object is electrically insulating, whether the object is water, and the like.

[0068] As an example, an ultrasonic fingerprint sensor system may be used to capture an ultrasonic image. The properties of the ultrasonic image may be analyzed to ascertain information about the user context, including information about the material of objects approaching the touch screen device, the relative size and shape of the objects, and the relative proximity of the objects. Figure 4A An example ultrasound image of a touch screen device submerged in water is shown according to some implementations.Having a touch screen device covered or submerged in water will substantially disrupt the operation of touch sensing in a capacitive touch sensor system. Figure 4B An example ultrasound image with water droplets on the surface of a touch screen device according to some specific implementations is shown. Covering the touch screen device with only a few water droplets does not substantially interfere with the operation of touch sensing in a capacitive touch sensor system. Figure 4A and Figure 4BAs shown in the ultrasound image in , the touch screen device can distinguish between situations, such as distinguishing between having a few water drops on the device and having the device submerged in water, which would render touch sensing in a capacitive touch sensor system inoperable or substantially inoperable.

[0069] Back to Figure 3 At block 320 of process 300, a graphical user interface (GUI) is presented by the control system on a display of the touch screen device. The graphical user interface includes a first portion that is configured to sense touch using one or more of: a fingerprint sensor system, an optical sensor system, or a force sensor system of the touch screen device. The first portion may occupy certain areas of the display area of ​​the touch screen device, or the first portion may occupy all or substantially all of the display area of ​​the touch screen device. The first portion may be a designated portion of the graphical user interface that can be used for touch recognition without the assistance of a capacitive touch sensor system.

[0070] In some implementations, the graphical user interface includes a first portion and a second portion, where the second portion can be a visible area for displaying an image or video. The graphical user interface can be divided between a non-touch-sensitive area (the second portion) and a touch-sensitive area (the first portion). In some implementations, the first portion can occupy half, more than half, less than half, or some other fraction of the display area, and the second portion can occupy the remainder of the display area.

[0071] In some implementations, the first portion occupies an area of ​​the display area corresponding to the active area of ​​the fingerprint sensor system. Thus, the fingerprint sensor system may be coextensive with the first portion. The active area of ​​the fingerprint sensor system may be the area where the fingerprint sensor pixel array resides. As an example, a 4 mm×9 mm or 8 mm×8 mm fingerprint sensor may represent the active area of ​​a fingerprint sensor system in some touch screen devices, which may represent a small portion of the display area. Touch sensing using the fingerprint sensor system may be limited to a small area of ​​the display area. However, as the active area of ​​the fingerprint sensor system increases, the touch-sensitive area in the non-capacitive touch mode also increases.

[0072] Depending on the application, the graphical user interface of the application may be reused or reformatted in its presentation to accommodate the touch-sensitive controls within the active area of ​​the fingerprint sensor system. In some cases, the graphical user interface may present the application and its controls in a window or frame coextensive with the active area of ​​the fingerprint sensor system. In some other cases, the graphical user interface may present only the touch-sensitive controls of the application in a window or frame coextensive with the active area of ​​the fingerprint sensor system.

[0073] Figure 5AAn example of a device with a graphical user interface in a configuration that restricts applications to specified portions of the graphical user interface is shown according to some implementations. Figure 5B An example of an apparatus having a graphical user interface in a configuration that limits touch-sensitive controls of a music player application to only a specified portion of the graphical user interface according to some implementations is shown. As with other disclosed implementations, Figure 5A and Figure 5B The proportions, numbers, arrangements, and types of elements shown are presented for illustrative purposes only. Other implementations of device 500 may have different numbers, arrangements, and / or types of elements. Device 500 includes a fingerprint sensor system 510 and a display system 520. In some implementations, the fingerprint sensor system 510 includes an ultrasonic fingerprint sensor system. In other implementations, the fingerprint sensor system 510 includes an optical fingerprint sensor system, a photoacoustic fingerprint sensor system, etc. Device 500 may also include a touch sensor system and a control system. The touch sensor system may include a capacitive touch sensor system coextensive with the display system 520. In Figure 5A and Figure 5B , the fingerprint sensor system 510 has an active area shown with dashed lines outlined by a dashed rectangle. The active area of ​​the fingerprint sensor system 510 resides below the display system 520. The active area of ​​the fingerprint sensor system has an array of fingerprint sensor pixels.

[0074] exist Figure 5A , a graphical user interface 530a presented on a display system 520 includes a designated portion 540a. The designated portion 540a is coextensive with the active area of ​​the fingerprint sensor system 510. The designated portion 540a occupies most of the lower half of the graphical user interface 530a. The application and its interface controls are confined to the designated portion 540a, wherein the designated portion 540a is configured to sense touch using the fingerprint sensor system 510. In other words, the user is able to interact via touch with the application and its touch-sensitive controls in the designated portion 540a of the graphical user interface 530a. Figure 5A As shown, the application is an Internet browser. The remainder of the graphical user interface 530a shows a home screen with various application icons in the background. The designated portion 540b is coextensive with the active area of ​​the fingerprint sensor system 510, which is smaller than Figure 5A540a. The designated portion 540b occupies a small portion of the lower half of the graphical user interface 530b. The touch-sensitive controls of the music player application are limited to the designated portion 540b, wherein the designated portion 540b is configured to sense touch using the fingerprint sensor system 510. In other words, the user can interact with the music player via touch with the touch-sensitive controls in the designated portion 540b of the graphical user interface 530b. The rest of the graphical user interface 530b shows an image associated with the music player application. As an example, a user interface button may be displayed in the designated portion 540b, while information about album art, artists, titles, and lyrics may be displayed in the rest of the graphical user interface 530b. Similarly, with respect to a camera application, a user interface button such as an image capture button may be displayed in the designated portion 540b, while viewfinder information or images may be displayed in the rest of the graphical user interface 530b.

[0075] Back to Figure 3 The process 300 of disabling touch sensing using the touch sensor system and enabling touch sensing using the fingerprint sensor system, the optical sensor system, and / or the force sensor system in a designated portion of the graphical user interface after determining that the user context is incompatible with touch sensing using the touch sensor system. Operating the touch screen device without the assistance of the touch sensor system for touch sensing may be referred to as touch sensing in a "special" touch mode or a "non-capacitive" touch mode. Alternatively, this may also be referred to as touch sensing in an "underwater" touch mode, a "glove" touch mode, or a "prosthetic" touch mode.

[0076] Regardless of whether the designated portion occupies the entirety or only a small portion of the graphical user interface, touch sensing using a fingerprint sensor system, an optical sensor system, and / or a force sensor system may not be as accurate as touch sensing using a capacitive touch sensor system. For example, the touch resolution of the touch position accurately located using a fingerprint sensor system may not be as accurate as the touch resolution of the touch position accurately located using a capacitive touch sensor system. In some specific implementations, when operating in a non-capacitive touch mode, the graphical user interface may present larger user interface controls, such as larger user interface buttons and icons. This provides improved usability and user satisfaction for registering user input with low touch resolution. In addition or in an alternative, the graphical user interface may present a grid or array of MxN tiles for receiving user input when operating in a non-capacitive touch mode. When operating at a low touch resolution, each tile in the tile may define a sufficiently large space for receiving touch input. This also provides improved usability and user satisfaction for registering user input with a low touch resolution.

[0077] Fig. 6AAn example of a device having a graphical user interface in a configuration with touch sensing in a capacitive touch sensing mode enabled is shown. Figure 6B An example of a device with a GUI in a configuration with touch sensing in a non-capacitive touch sensing mode enabled is shown according to some implementations. Figure 6C An example of a device with a GUI in a configuration with touch sensing in a non-capacitive touch sensing mode enabled for an application is shown according to some implementations. Fig.6D An example of a device with a GUI in a configuration with touch sensing in a non-capacitive touch sensing mode enabled for a camera is shown according to some implementations. Fig. 6E An example of a device having a GUI in a configuration with touch sensing in a non-capacitive touch sensing mode enabled for a music player application according to some implementations is shown. As with other disclosed implementations, FIG. 6A to FIG. 6E The proportions, numbers, arrangements, and types of elements illustrated are presented for illustrative purposes only. Other implementations of the device 600 may have different numbers, arrangements, and / or types of elements. The device 600 includes a fingerprint sensor system 610 and a display system 620. In some implementations, the fingerprint sensor system 610 includes an ultrasonic fingerprint sensor system. In other implementations, the fingerprint sensor system 610 includes an optical fingerprint sensor system, a photoacoustic fingerprint sensor system, etc. The device 600 may also include a touch sensor system and a control system. The touch sensor system may include a capacitive touch sensor system that is coextensive with the display system 620. In FIG. 6A to FIG. 6E , the fingerprint sensor system 610 has an active area shown as a dashed line outlined by a grid. The active area of ​​the fingerprint sensor system 610 resides below the display system 620. The active area of ​​the fingerprint sensor system has an array of fingerprint sensor pixels. The active area of ​​the fingerprint sensor system is comparable to FIG. 6A to FIG. 6E The indicated ones may be larger or smaller, or may be the same size but in a different location.

[0078] refer to Fig. 6A , a graphical user interface 630a presented on a display system 620 includes a plurality of user interface controls 640a in a "capacitive" touch mode or a "normal" touch mode. In the capacitive touch mode or the normal mode, a user can interact with the graphical user interface 630a via touch input using a touch sensor system such as a capacitive touch sensor system. The plurality of user interface controls 640a can be arranged as icons on the graphical user interface 630a. The icons are neither enlarged nor arranged in a grid or array of tiles.

[0079] On the contrary, FIG. 6B to FIG. 6EThe graphical user interface 630b, 630c, 630d, 630e presented on the display system 620 includes a plurality of user interface controls 640b, 640c, 640d, 640e in a "non-capacitive" touch mode or a "special" touch mode. In the non-capacitive touch mode or the special mode, a user can interact with the graphical user interface 630b, 630c, 630d, 630e via touch input using a fingerprint sensor system 610 (such as an ultrasonic fingerprint sensor system). However, it will be understood that in the non-capacitive touch mode or the special mode, a user can interact with the graphical user interface via touch input using an optical sensor system, a force sensor system, or other alternative sensor system. The plurality of user interface controls 640b, 640c, 640d, 640e can be arranged as icons, virtual buttons, sliders, or other controls on the graphical user interface 630b, 630c, 630d, 630e.

[0080] A plurality of user interface controls 640b, 640c, 640d, 640e may be arranged within a grid or array of tiles 650b, 650c, 650d, 650e, respectively. The tile arrays 650b, 650c, 650d, 650e may occupy an area coextensive with the active area of ​​the fingerprint sensor system 610. As discussed below, when a touch or force is applied to the display system 620, the fingerprint sensor pixels corresponding to one or more tiles 650b may be scanned to identify the user's finger and determine the touch location. In these examples, the graphical user interface 630b is first divided into the tile array 650b, and then a scan is performed to determine the touch location. Alternatively, when a touch or force is applied to the display system 620, the fingerprint sensor pixels are scanned or at least partially scanned to identify the user's finger, and then the user's finger is associated with a particular tile in the tile array 650b to determine the touch location. In such an example, a scan is performed first, and then the graphical user interface 630 is divided into the tile array 650b to determine the touch locations.

[0081] like Figure 6B As illustrated, the plurality of user interface controls 640b are enlarged icons on the main page of the graphical user interface 630b. Fig. 6A A plurality of user interface controls 640a in Figure 6B The plurality of user interface controls 640b in FIG. 6 are larger and arranged in a tile array 650b. Figure 6C As illustrated, a plurality of user interface controls 640c occupy the lower half of the graphical user interface 630c. The lower half has an area designated for sensing touch, while the upper half has an area designated for displaying images, videos, or other information. The user interface controls 640c may be icons arranged in the lower half of the tile array 650d. Fig.6DAs illustrated, the plurality of user interface controls 640d are virtual buttons for a camera application located to the right of the graphical user interface 630d. The remainder of the graphical user interface 630d displays a viewfinder or image for the camera application. The plurality of user interface controls 640d may be virtual buttons arranged in a tile array 650d. Fig. 6E As illustrated, the plurality of user interface controls 640e are virtual buttons and sliders of a music player application located on the right side of the graphical user interface 630e. The rest of the graphical user interface 630e displays information of the music player application. The plurality of user interface controls 640e can be virtual buttons and sliders arranged in the tile array 650e.

[0082] Return to Figure 3 , at box 330 of process 300, the control system uses a fingerprint sensor system, an optical sensor system, or a force sensor system to obtain a location of the touch input on the graphical user interface. As discussed above, at least a first portion of the graphical user interface is configured to sense touch using a fingerprint sensor system, an optical sensor system, or a force sensor system. One or more pressure or force sensors can detect a touch or force applied to a display of the touch screen device to determine that a touch input is received on the graphical user interface. One or more pressure or force sensors can be associated with a fingerprint sensor system, a force sensor system, or other sensor system. When the touch input is detected, the control system can use one of several methods to obtain the location of the touch input.

[0083] In some implementations, a fingerprint sensor system can be used to obtain the location of the touch input. The fingerprint sensor system can perform a scan of one or more areas of the first portion to identify an object (e.g., a user's finger). The scanned object can be associated with coordinates in the first portion or with a valid tile in the tile array to obtain the location of the touch input.

[0084] The fingerprint sensor system may be an ultrasonic fingerprint sensor system. The ultrasonic fingerprint sensor system may include an ultrasonic transmitter or transceiver configured to generate ultrasonic waves. The ultrasonic fingerprint sensor system may include an ultrasonic receiver or transceiver configured to receive reflections of ultrasonic waves, wherein the ultrasonic receiver may include a piezoelectric receiver array. In some cases, the piezoelectric receiver array may be divided into several sections so that some or all of the sections are scanned for ultrasonic imaging. In some specific implementations, the control system may scan all areas of the piezoelectric receiver array. In some specific implementations, the control system may select a scanning area of ​​the piezoelectric receiver array. For example, different areas may be scanned until the user's finger is identified in one of the scanned areas.

[0085] In one example, one or more fingerprint sensor pixels are scanned until a user's finger is identified. The scan may be a full scan or a partial scan of an active area of ​​the fingerprint sensor system. The first portion of the graphical user interface may be divided into an MxN array of tiles. The fingerprint sensor pixel that scans the user's finger may be associated with an active tile in the MxN array of tiles. The location of the active tile provides the location of the touch input.

[0086] In another example, the first portion of the graphical user interface can be divided into an MxN array of tiles. The fingerprint sensor system performs a scan of one or more tiles until the user's finger is identified. The user's finger is associated with a valid tile in the MxN array of tiles, and the location of the valid tile provides the location of the touch input.

[0087] In some implementations, the control system scans one or more regions of the fingerprint sensor system to acquire image data. The scan may be initiated by a stylus, pen, or other object in contact with a display of a touch screen device. The acquired image data may correspond to signals generated by a piezoelectric receiver array in response to an ultrasonic signal and / or mechanical deformation caused by an object in contact with a surface (such as a cover glass of a touch screen device). In some instances, the object is a stylus. The control system may be configured to detect a double pattern or a ring pattern in the acquired image data. Using the acquired image data, the control system may be configured to detect the positioning of the object on the surface of the touch screen device to provide a location of a touch input.

[0088] Fig. 7A A cross-sectional schematic diagram of an example fingerprint sensor system configured for touch detection using mechanical deformation according to some implementations is shown. Figure 7B A cross-sectional schematic diagram of an example fingerprint sensor system configured for touch detection using ultrasonic reflection according to some implementations is shown. Fig. 7A and Figure 7B, stylus 701 is in contact with cover glass 703 of device 700. Display 711 is located below cover glass 703. Display 711 may include an LED display, an OLED display, or other suitable display. Device 700 includes a thin film transistor (TFT) substrate 714 having a piezoelectric receiver array, in this example, the piezoelectric receiver array may be an ultrasonic sensor array. In some specific implementations, adhesive 713 couples TFT substrate 714 to display 711. Piezoelectric layer 715 is coupled to TFT substrate 714. Piezoelectric layer 715 may include one or more ferroelectric polymers, such as polyvinylidene fluoride (PVDF) or polyvinylidene fluoride-trifluoroethylene (PVDF-TrFE) copolymers. Examples of PVDF copolymers include 60:40 (molar percentage) PVDF-TrFE, 70:30 PVDF-TrFE, 80:20 PVDF-TrFE, and 90:10 PVDR-TrFE. Alternatively or additionally, the piezoelectric layer 715 may include one or more other piezoelectric materials, such as polyvinylidene chloride (PVDC) homopolymer or copolymer, polytetrafluoroethylene (PTFE) homopolymer or copolymer, diisopropylammonium bromide (DIPAB), aluminum nitride (AlN) and / or lead zirconate titanate (PZT). The device 700 includes an electrode layer 717 adjacent to the piezoelectric layer 715 and a passivation layer 719 adjacent to the electrode layer 717. In this example, at least a portion of the control system 706 is configured to be electrically connected to the electrode layer 717, the piezoelectric layer 715, and the TFT substrate 714 via a flexible printed circuit 721.

[0089] exist Fig. 7A , stylus 701 moves in the direction indicated by arrow 705. Acoustic waves 707 generated by the interaction of stylus 701 with cover glass 703 propagate in the direction of arrow 709 (and in other directions). Device 700 detects the mechanical deformation caused by stylus 701. Such "passive" implementations can be energy efficient because an active transmitter, such as an ultrasonic transmitter, does not have to be used to track stylus 701. Acoustic waves 707 caused by the mechanical deformation provide a signal to TFT substrate 714 to acquire image data, which can be analyzed to acquire the position of stylus 701. In some implementations, the acquired image data can indicate the position of stylus 701 in a dual pattern.

[0090] exist Figure 7B, stylus 701 is in contact with cover glass 703. Ultrasonic waves are generated by piezoelectric layer 715 in the ultrasonic transmitter. The generated ultrasonic waves are indicated by arrow 725 pointing upward. One or more segments of electrode layer 717 may receive a voltage (e.g., a burst tone) to cause piezoelectric layer 715 to generate ultrasonic waves. Reflection of the ultrasonic waves caused by stylus 701 is received by TFT substrate 714. The reflected ultrasonic waves are indicated by arrow 729 pointing downward. The reflected ultrasonic waves provide a signal to TFT substrate 714 to acquire image data, which can be analyzed to acquire the position of stylus 701. In some implementations, the acquired image data may indicate the position of stylus 701 in a ring pattern.

[0091] Back to Figure 3 In some implementations of process 300, an optical sensor system can be used to obtain the location of the touch input. The optical image system can include an optical sensor pixel array that scans an image of an object (e.g., a user's finger) in contact with a cover glass or display of a touch screen device. The scanned image can be associated with the location of the touch input in the first portion of the graphical user interface.

[0092] In some implementations, a force sensor system may be used to obtain the location of a touch input. The force sensor system may include multiple force sensors positioned at different locations on a touch screen device. When a force is applied at a specific location on the touch screen device, each of the multiple force sensors may obtain a force measurement. The force measurement at each of the different locations of the touch screen device may be used to calculate the location of the maximum force. The location of the maximum force corresponds to the location of the object applying the force. The force measurements from the multiple force sensors are used to triangulate the touch location. Therefore, the location of the object applying the force may be estimated based on multiple force measurements at different locations.

[0093] Figure 8 A schematic block diagram of an example force sensor system configured for touch detection using multiple force sensors according to some specific implementations is shown. Device 800 includes force sensor system 810 and display system 820. Force sensor system 810 may include multiple force sensors 830. Device 800 may also include a touch sensor system and a control system. The touch sensor system may include a capacitive touch sensor system coextensive with display system 620. Figure 8, graphical user interface 840 includes designated portion 850 divided into array of tiles 860. Tiles 860 are numbered 1-12 in designated portion 850. Designated portion 850 overlaps force sensor system 810 and serves as a touch-sensitive area of ​​graphical user interface 840. Force sensors 830 may be located at each corner under tiles numbered 1, 3, 10, and 12 of designated portion 850. When an object exerts a force on display system 820, force measurements obtained by force sensors 830 may be used to calculate the location of the applied force. The location of the applied force may correspond to one of tiles 860. As shown in FIG. Figure 8 As shown, the location of the applied force is block number 5.

[0094] Return to Figure 3 One or more touch drivers of the touch screen device may receive the location of the touch input from the control system. The one or more touch drivers may perform operations based on the location of the touch input.

[0095] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items (including single members). As an example, "at least one of a, b, or c" is intended to cover: a, b, c, ab, ac, bc, and abc.

[0096] The various illustrative logical components, logic blocks, modules, circuits, and algorithmic processes described in conjunction with the specific implementations disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. The interchangeability of hardware and software has been generally described in terms of functionality and illustrated in the various illustrative components, blocks, modules, circuits, and processes described above. Whether such functionality is implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0097] The hardware and data processing apparatus for implementing the various illustrative logic pieces, logic blocks, modules and circuits described in conjunction with the various aspects disclosed herein may be implemented or executed using a general purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. In some implementations, specific processes and methods may be performed by circuits specific to a given function.

[0098] In one or more aspects, the functions described may be implemented in hardware, digital electronic circuitry, computer software, firmware (including the structures disclosed in this specification and their structural equivalents), or in any combination thereof. Specific implementations of the subject matter described in this specification may also be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by a data processing apparatus or for controlling the operation of a data processing apparatus.

[0099] If implemented in software, each function may be stored as one or more instructions or codes on a computer-readable medium such as a non-transient medium or sent via a computer-readable medium. The process of the method or algorithm disclosed herein may be implemented in a processor executable software module that may reside on a computer-readable medium. Computer-readable media include both computer storage media and communication media, including any medium that can transfer a computer program from one place to another. Storage media may be any available medium that can be accessed by a computer. As an example and not limitation, non-transient media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection may be appropriately referred to as a computer-readable medium. Disks and optical disks as used herein include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks, and blue-ray disks, wherein disks generally reproduce data magnetically, and optical disks reproduce data optically with lasers. The above combination should also be included in the scope of computer-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination or set of codes and instructions on a machine-readable medium and a computer-readable medium, which may be incorporated into a computer program product.

[0100] Various modifications to the specific implementations described in the present disclosure may be apparent to those of ordinary skill in the art, and the universal principles defined herein may be applied to other specific implementations without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the specific implementations shown herein, but to conform to the widest scope consistent with the claims, principles, and novel features disclosed herein. The word "exemplary", if any, is used specifically herein to mean "used as an example, instance, or illustration". Any specific implementation described herein as "exemplary" is not necessarily to be construed as superior or superior to other specific implementations.

[0101] Certain features described in this specification in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations, either individually or in any suitable subcombination. Furthermore, while features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from a claimed combination may in some cases be removed from the combination, and a claimed combination may be directed to subcombinations or variations of subcombinations.

[0102] Similarly, although the operations are depicted in a specific order in the figure, this should not be understood as requiring such operations to be performed in the specific order shown or in a sequential order, or to perform all the illustrated operations to achieve the desired result. Under certain circumstances, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the specific implementation described above should not be understood as requiring such separation in all specific implementations, and it should be understood that the program components and the system can generally be integrated together in a single software product or packaged into multiple software products. In addition, other specific implementations are within the scope of the following claims. In some cases, the actions narrated in the claims can be performed in different orders and still reach the desired result.

[0103] It should be understood that unless the features in any particular described implementation are explicitly identified as incompatible with each other, or the surrounding context implies that they are mutually exclusive and not easily combined in a complementary and / or supporting sense, the overall concept and concept of the present disclosure may be selectively combined with the specific features of those complementary implementations to provide one or more comprehensive but slightly different technical solutions. Therefore, it should also be understood that the above description is given only as an example and may be modified in detail within the scope of the present disclosure.

Claims

1. A device comprising: monitor; Capacitive touch sensor system; One or more of the following: a fingerprint sensor system, an optical sensor system, or a force sensor system; and a control system configured to communicate with the capacitive touch sensor system, the fingerprint sensor system, and the display, wherein the control system is further configured to: determining that a user context of the device is incompatible with touch sensing of the touch sensor system; and A graphical user interface is presented on the display of the device, wherein the graphical user interface includes a first portion configured for sensing touch using one or more of: the fingerprint sensor system, the optical sensor system, or the force sensor system.

2. The device of claim 1, wherein the first portion comprises a region of the display that is coextensive with an active area of ​​the fingerprint sensor system. 3 . The apparatus of claim 1 , wherein the graphical user interface further comprises a second portion, wherein the second portion of the graphical user interface is configured to display a video or an image.

4. The apparatus of claim 1 , wherein the control system configured to determine that the user context is incompatible with touch sensing of the capacitive touch sensor system is configured to determine that the user context includes one or more of the following: (i) the display of the apparatus is covered or submerged in water, (ii) the user is wearing gloves, or (iii) the user is using a prosthetic or robotic hand.

5. The apparatus of claim 1 , wherein the control system configured to determine that the user context is incompatible with touch sensing of the capacitive touch sensor system is configured to: identifying an object in proximity to the device; and A determination is made that the object is incompatible with touch sensing by the capacitive touch sensor system.

6. The apparatus of claim 5, wherein the control system configured to identify the object in proximity to the apparatus is configured to: detecting a change in pressure on the device using one or more pressure or force sensors associated with the force sensor system or fingerprint sensor system, wherein the change in pressure indicates that the object is approaching the device; scanning an image of the object proximate to the device using the optical sensor system or the fingerprint sensor system; as well as The object is classified based on the image.

7. The apparatus of claim 1, wherein the control system is further configured to: The fingerprint sensor system is used to obtain a location of a touch input on the graphical user interface.

8. The apparatus of claim 7, wherein the control system configured to use the fingerprint sensor system to obtain the location of the touch input on the graphical user interface is configured to: dividing the first portion of the graphical user interface into an array of tiles corresponding to an active area of ​​the fingerprint sensor system; as well as performing a scan of one or more tiles in the array of tiles to identify a user's finger using the fingerprint sensor system; as well as The user's finger is associated with one of the tiles in the first portion of the graphical user interface to obtain the location of the touch input.

9. The apparatus of claim 7, wherein the control system configured to use the fingerprint sensor system to obtain the location of the touch input on the graphical user interface is configured to: detecting an applied force on the display using one or more pressure or force sensors associated with the force sensor system or fingerprint sensor system; performing a scan using the fingerprint sensor system to identify a user's finger; as well as The user's finger is associated with at least one tile in an array of tiles in the first portion of the graphical user interface to obtain the location of the touch input, wherein the array of tiles divides the first portion into a grid.

10. The apparatus of claim 7, wherein the fingerprint sensor system comprises an ultrasonic fingerprint sensor system.

11. The apparatus of claim 10, wherein the control system configured to use the fingerprint sensor system to obtain the location of the touch input on the graphical user interface is configured to: causing an ultrasonic transmitter or transceiver in the ultrasonic fingerprint sensor system to generate ultrasonic waves; receiving reflections of the ultrasonic waves at a piezoelectric receiver array to acquire image data identifying a stylus or pen in contact with a surface of the device; and The stylus or pen is associated with a location in the first portion of the graphical user interface to obtain the location of the touch input using the image data.

12. The apparatus of claim 10, wherein the control system configured to use the fingerprint sensor system to obtain the location of the touch input on the graphical user interface is configured to: receiving an indication of mechanical deformation at a piezoelectric receiver array of the ultrasonic fingerprint sensor system to acquire image data identifying a stylus or pen in contact with a surface of the device; and The stylus or pen is associated with a location in the first portion of the graphical user interface to obtain the location of the touch input using the image data.

13. The apparatus of claim 1, wherein the control system is further configured to: Using the force sensor system to obtain a position of a touch input on the graphical user interface, wherein the control system configured to use the force sensor system to obtain the position of the touch input is configured to: detecting an applied force from an object on the display; obtaining a plurality of force measurements from a plurality of force sensors associated with the force sensor system, each force sensor of the plurality of force sensors being positioned at a different location on the device; as well as The position of the object in the first portion of the graphical user interface is estimated based on the plurality of force measurements to obtain a location of the touch input.

14. A method comprising: determining, using the control system, that a user context of a touch screen device is incompatible with touch sensing by a capacitive touch sensor system of the touch screen device; as well as The control system is used to present a graphical user interface on a display of the touch screen device, wherein the graphical user interface includes a first portion configured to sense touch using one or more of: a fingerprint sensor system, an optical sensor system, or a force sensor system of the touch screen device.

15. The method of claim 14, wherein the first portion of the graphical user interface comprises an area of ​​the display that is coextensive with an active area of ​​the fingerprint sensor system.

16. The method of claim 14, wherein the graphical user interface further comprises a second portion, wherein the second portion of the graphical user interface is configured to display a video or an image.

17. The method of claim 14, wherein the user context that is incompatible with touch sensing of the capacitive touch sensor system includes one or more of the following: (i) the display of the touch screen device is covered or submerged in water, (ii) the user is wearing gloves, or (iii) the user is using a prosthetic or robotic hand.

18. The method of claim 14, wherein determining that the user context of the touch screen device is incompatible with touch sensing of the capacitive touch sensor system comprises: identifying an object in proximity to the touch screen device; as well as A determination is made that the object is incompatible with touch sensing by the capacitive touch sensor system of the touch screen device.

19. The method of claim 18, wherein identifying the object in proximity to the touch screen device comprises: detecting a change in pressure on the touch screen device using one or more pressure or force sensors associated with the force sensor system or the fingerprint sensor system, wherein the change in pressure indicates that the object is approaching the touch screen device; scanning an image of the object in proximity to the touch screen device using the optical sensor system or the fingerprint sensor system; as well as The object is classified based on the image.

20. The method of claim 14, further comprising: Using the control system, the fingerprint sensor system is used to obtain a location of a touch input on the graphical user interface.

21. The method of claim 20, wherein using the fingerprint sensor system to obtain the location of the touch input on the graphical user interface comprises: dividing the first portion of the graphical user interface into an array of tiles corresponding to an active area of ​​the fingerprint sensor system; performing a scan of one or more tiles in the array of tiles to identify a user's finger using the fingerprint sensor system; as well as The user's finger is associated with one of the tiles in the first portion of the graphical user interface to obtain the location of the touch input.

22. The method of claim 20, wherein using the fingerprint sensor system to obtain the location of the touch input on the graphical user interface comprises: detecting an applied force on the display using one or more pressure or force sensors associated with the force sensor system or fingerprint sensor system; performing a scan using the fingerprint sensor system to identify a user's finger; as well as The user's finger is associated with at least one tile in an array of tiles in the first portion of the graphical user interface to obtain the location of the touch input, wherein the array of tiles divides the first portion into a grid.

23. The method of claim 20, wherein the fingerprint sensor system comprises an ultrasonic fingerprint sensor system.

24. The method of claim 23, wherein using the fingerprint sensor system to obtain the location of the touch input on the graphical user interface comprises: causing an ultrasonic transmitter or transceiver in the ultrasonic fingerprint sensor system to generate ultrasonic waves; receiving reflections of the ultrasonic wave at a piezoelectric receiver array to acquire image data identifying a stylus or pen in contact with a surface of the touch screen device; as well as The stylus or pen is associated with a location in the first portion of the graphical user interface to obtain the location of the touch input using the image data.

25. The method of claim 23, wherein using the fingerprint sensor system to obtain the location of the touch input on the graphical user interface comprises: receiving an indication of mechanical deformation at a piezoelectric receiver array of the ultrasonic fingerprint sensor system to acquire image data identifying a stylus or pen in contact with a surface of the touch screen device; as well as The stylus or pen is associated with a location in the first portion of the graphical user interface to obtain the location of the touch input using the image data.

26. The method of claim 14, further comprising: The optical sensor system is used to obtain a location of a touch input on the graphical user interface.

27. The method of claim 14, further comprising: Using the force sensor system to obtain a location of a touch input on the graphical user interface, wherein using the force sensor system to obtain the location of the touch input comprises: detecting an applied force from an object on the display; obtaining a plurality of force measurements from a plurality of force sensors associated with the force sensor system, each force sensor of the plurality of force sensors being positioned at a different location on the touch screen device; and The position of the object in the first portion of the graphical user interface is estimated based on the plurality of force measurements to obtain a location of the touch input.

28. One or more non-transitory media having software recorded thereon, the software comprising instructions for controlling one or more devices to perform a method comprising: determining that a user context of a touch screen device is incompatible with touch sensing of a capacitive touch sensor system of the touch screen device; as well as A graphical user interface is presented on a display of the touch screen device, wherein the graphical user interface includes a first portion configured to sense touch using one or more of: a fingerprint sensor system, an optical sensor system, or a force sensor system of the touch screen device.

29. One or more non-transitory media according to claim 28, wherein the user context that is incompatible with touch sensing of the capacitive touch sensor system includes one or more of the following: (i) the display of the touch screen device is covered or immersed in water, (ii) the user is wearing gloves, or (iii) the user is using a prosthetic or robotic hand.

30. The one or more non-transitory media of claim 28, the method further comprising: The fingerprint sensor system is used to obtain a location of a touch input on the graphical user interface.

Images