Cover Detection

Abstract

The present invention relates to a device and method for detecting a cover covering at least a portion of an electronic device, the device including at least one audio transducer adapted to transmit and receive acoustic signals. defining at least one characteristic of the interrogation signal; transmitting an interrogation signal within a predetermined frequency range having at least one characteristic using at least one of the transducers; receiving a signal with at least one audio transducer; Analyzing a characteristic of the received signal and a characteristic of the transmitted signal to detect a selected number of differences therebetween; Comparing the difference between the transmitted signal and the received signal to a set of predetermined differences associated with an object covering at least one of the transducers in the device; Includes.

Description

[Technical field]

[0001] The present teachings relate to a method for detecting a cover of an electronic device. [Background technology]

[0002] Electronic devices such as mobile phones and smart watches are typically equipped with proximity sensors, which are used to detect the proximity of an object, such as a part of the user's body, using ultrasound reflected from that object, as described, for example, in US Patent Application Publication No. 2017 / 329431. In a mobile phone, the main function of such a proximity sensor is to detect when a user brings the device close to the ear during a call, in which case the touch screen of the mobile phone is disabled or switched off to prevent false touch events due to the user's ear or another body part unintentionally contacting at least one of the screens of the mobile device. Since the touch screen is typically not used while the user is on a call and holds the device close to the head or next to the ear, the touch screen controller is turned off or can go into a low power mode to save power. Also, the screen lighting of the device is typically turned off to save power. Typically, an infrared ("IR") proximity sensor is used, but an ultrasonic proximity sensor can also be used.

[0003] In many mobile devices, there are additional requirements for cover detection or pocket mode, as described in Norwegian Patent Application No. 20211143. Cover detection is a short-range proximity detection aimed at determining whether to enable or disable the screen of the mobile device. Close-range detection means detecting whether an object is closer than a few centimetres from the screen of the device, or more precisely from the cover detection sensor of the device, usually including within a few millimetres or even touching the device. Therefore, one important objective of the cover detection system is to prevent the activation of a screen that should remain off. Doing so can prevent unintended touch events or operations that result from the unintended activation of the screen of the electronic device. Similarly, cover detection needs to be reliable in that the user can quickly access the screen whenever the user wishes to operate the device. It should be noted that the user experience may be affected if the screen is not enabled early when the user wishes to access the device. It is therefore desirable for the cover detection system to be able to reliably distinguish between events that should enable the screen and events that should disable or leave the screen asleep. Furthermore, the cover detection needs to respond quickly for a better user experience.

[0004] Some conventional proximity detection methods have performance issues because they are too slow to achieve a smooth user experience in cover detection, and achieving responsive and reliable cover detection, especially for relatively stationary objects, can be difficult for some sensor technologies.

[0005] Ultrasonic signals are used for proximity sensors in electronic devices (e.g. smartphones), but ultrasound is not an ideal technology in situations where the sensor components are covered by a completely static object near the transducer. This would be the case if a smartphone was placed on a table with its screen facing up and a piece of paper was placed on top of the smartphone covering the screen and the proximity sensor.

[0006] In protective cases available on the market, detection of the flip cover covering the screen is performed using, for example, a touch screen sensor of the mobile device, or the part of the flip cover used to cover the screen may be provided with a dedicated sensor, such as a Near Field Communication ("NFC") sensor or a magnetic sensor, which triggers a flip case detection mode when the screen part of the protective case comes within a predetermined distance of the screen of the mobile device. Protective cases that require a dedicated sensor for cover detection may become more expensive and less attractive to customers. Summary of the Invention

[0007] As stated above, the object of the present invention is to provide a solution that makes use of transducers already present in electronic devices. This is obtained as set forth in the appended claims.

[0008] The invention is therefore suitable for use when the same acoustic transducer is used in a device used for both speaking and listening, as well as for traditional proximity detection to detect objects within a certain range of the device. The invention is based on the fact that an acoustic signal emitted behind a cover will result in a different signal at the receiver depending on whether it propagates freely in the environment or is trapped between the device and the cover or object, and it mainly works when the wavelength of the acoustic signal is longer than the distance from the transducer to the cover. Detection is seen as a pressure wave propagating from the transmitter to the receiver, or as a pressure rise around the transducer. If the same transducer is used to transmit and receive the signal, the cover may be detected as a change in acoustic impedance sensed near the transducer.

[0009] Specifically, experiments have shown that cover increases the amplitude of the received signal, but as described below, the analysis may be configured to adapt to specific situations, such as distinguishing between or using different covers, such as those with absorbing or reflective or interfering structures. The analysis needs to be defined to distinguish between direct propagation from the transmitter to the receiver in the absence of cover, and propagation trapped between the cover and the device. This can be achieved by a learning procedure performed by the user and a user interface that defines different situations based on exemplified use (e.g., registration when the device is covered, registration when the mobile phone is held close to the ear, etc.).

[0010] In fact, the invention can operate by transmitting an acoustic signal and detecting the resulting change at the receiver, either at the same moment of transmission or a short time after transmission, before a reflection is expected. The received signal can be compared to previously registered signals characterizing the signal propagating along an uncovered device surface, and such signals can be filtered. The analysis can also compare the received signal to previous instances where the device surface is covered, so as to recognize certain signal characteristics.

[0011] According to one embodiment, the interrogation signal may be in the audible range with a frequency or frequency chirp below 5 kHz depending on the circumstances. It is also possible to combine an audible signal with ultrasound, for example, to improve resolution if the device is adapted to measure non-stationary, moving objects, such as detected using an IMU (Inertial Measurement Unit) in the device, when the objects are at a greater distance from the device. Additionally, the device may only use ultrasound if no ambient sound is detected.

[0012] If an audio use case is running simultaneously on the electronic device, the interrogation signal may be selected from the output of the audio use case to avoid generating a specific interrogation signal for the cover detection method. This requires that the audio use case signal is made available to the cover detection method and can be analyzed to find a suitable interrogation signal within the existing audio output. This audio loopback is described in detail in Norwegian Patent Application No. 20211333. If the ambient noise is above a certain threshold, the audio output can be modified with an additional interrogation signal or the amplitude of a portion of the audio output signal that is ideally suitable as an interrogation signal can be amplified to create an interrogation signal that is identifiable taking into account the ambient noise in the first acoustic signal. It is also possible to analyze the ambient noise, for example using neural networks or machine learning, to predict noise patterns and transmit an interrogation signal during periods with sufficient ambient noise.

[0013] If the device is a personal device (e.g., a smartphone or smartwatch) and the user receives a notification or alarm sound played on a speaker, or other sound including vibration, the cover detection method may be enabled to piggyback on the sound being played and use that sound as an interrogation signal to see if the receiver detects if the received signal matches a previously detected signal, or to store characteristics that characterize the cover being in place.

[0014] As described in Norwegian Patent Application No. 20211143, in a scenario where the cover detection method is triggered only when other device sensors indicate that the cover detection state needs to be updated or detected by a user action (e.g., pressing the power button, touching the screen, lifting the phone, etc.), if the phone is not already configured to play a specific sound in such a situation, the cover detection method may play a short, configurable notification sound that is acceptable and even selectable for the user. The notification sound available on the electronic device should take into account the current ambient noise level and the acoustic characteristics of the cover detection module and include a set of interrogation signals suitable for cover detection.

[0015] The interrogation signal for a particular device may be selected via a calibration process that instructs the user to perform a series of actions while the cover detection method is active, such as covering the device with at least one, but potentially different, type of readily available object, or removing the object covering the device when instructed to do so, or the device may be placed in a pocket or bag. As the user provides ground truth of the recordings associated with this particular device, the machine learning model may be improved using an incremental ML training process of at least one neural network used for the cover detection method to ensure that the performance of the method is optimized for the target device.

[0016] Some user actions that dictate the use of cover detection are those that essentially move the mobile device significantly. One example is the raise-to-wake feature of modern smartphones, where the state of cover detection determines whether a biometric authentication mechanism, which usually includes at least one camera, should be activated. In these situations, acoustic cover detection mechanisms can use additional interrogation signals in the ultrasonic frequency range and also perform echo analysis of the received ultrasound.

[0017] According to one embodiment, the device may be set to listen for background noise or sounds from the surroundings and then configured to adapt the interrogation signal to a frequency band with lower noise levels when the environment is quiet and the user does not desire audible sounds from the device.

[0018] Thus, the interrogation signal may be scaled up and down based on a prediction of both the current level and future possible changes in the ambient noise level picked up while the interrogation signal is played by the cover detection method. This prediction can be based on statistical methods, standard ML-based training in different environments, or the latest incremental training of the target device in its current location.

[0019] According to one embodiment, the first audio transducer and the second audio transducer may be the same transducer capable of transmitting the interrogation signal and receiving the first acoustic signal and echo, or the first transducer may be a microphone and the second audio transducer may be a speaker of the electronic device.

[0020] According to one embodiment, the characteristic of the interrogation signal is the amplitude of the interrogation signal. Alternatively or additionally, the characteristic of the interrogation signal is the transmission length or time period during which the interrogation signal is transmitted. Thus, either the amplitude and / or the time period of the transmitted interrogation signal are determined from the analysis of the first signal. In this way, the interrogation signal can be adapted or one or more suitable characteristics of the interrogation signal can be calculated depending on the prevailing situation in which the electronic device is placed. In this way, the cover detection method can make it possible to find a suitable interrogation signal such that the audibility of the interrogation signal is at least minimized or eliminated for a typical human user. Furthermore, the power consumption may be optimized according to the prevailing conditions. The characteristics of the interrogation signal may be selected depending on the particular application and may include the phase, frequency, and / or amplitude of the interrogation signal, as well as a frequency chirp or other encoding.

[0021] According to one embodiment, the characteristic of the interrogation signal may be selected by the processing unit based on an analysis of noise detected from the environment, for example by monitoring background noise. A frequency of the interrogation signal may then be selected such that the audibility at the selected frequency is estimated to be low compared to the audibility of the interrogation signal at another frequency. In this way, the processing unit may analyze the frequency spectrum of the first signal and determine the frequency of the interrogation signal. The processing unit may be able to analyze the power of a particular frequency or a particular group of frequencies of the first signal to find either a suitable frequency or at least one characteristic of the interrogation signal.

[0022] By choosing to use a lower amplitude of the interrogation signal, the processing unit may use a longer interrogation signal to improve cover detection. Vice versa, i.e., the processing unit may shorten the transmission length and increase the amplitude of the interrogation signal. The processing unit may select the characteristic between successive transmission cycles of the interrogation signal or may dynamically adapt the characteristic during continued transmission of the interrogation signal.

[0023] Thus, in accordance with the proposed teachings, reliable coverage detection can be achieved while minimizing the audibility of the interrogation signal to a typical user, and the interrogation signal can be selected depending on the prevailing noise conditions, so this synergy is advantageous for essentially simultaneously adapting the interrogation signal depending not only on audibility but also on the acoustic noise surrounding the device.

[0024] From yet another perspective, the present teachings can provide a computer software product for performing any method step described herein. Thus, the present teachings also relate to computer readable program code having specific capabilities for performing any method step described herein. In other words, the present teachings also relate to a non-transitory computer readable medium that stores a program for causing an electronic device to perform any method step described herein. [Brief description of the drawings]

[0025] Exemplary embodiments will now be described with reference to the accompanying drawings, which may not necessarily be drawn to scale without affecting the generality scope of the present teachings. [Figure 1] 1 is a front perspective view of an electronic device having a proximity detection system. [Diagram 2] 1 is a side perspective view of an electronic device having a proximity detection system. [Diagram 3] FIG. 2 shows a circuit diagram of a device according to the invention. [Figure 4] FIG. 2 illustrates an exemplary process according to the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] 1 is an exemplary front perspective view of an electronic device 100, shown here as a mobile phone or smartphone. The mobile phone 100 has a screen 101 for displaying content and interacting with the device 100. Located above the top edge 110 of the screen 101 are earphones 102 and a proximity sensor 106. The proximity sensor 106 is preferably constituted by a transceiver.

[0027] In this disclosure, the terms top, bottom, left, and right are used in a non-limiting relative sense for ease of understanding. Furthermore, the location of each component, such as the proximity sensor 106, is shown by way of example only. Those skilled in the art will understand that such sensors or earbuds 102 can be placed in different locations compared to those in this example without affecting the scope or generality of the present disclosure.

[0028] As will be appreciated, the earphones 102 include speakers that are used to output acoustic signals, such as the audio of a phone call. In some phones, the same speakers in the earphones 102 may also be used to output ultrasonic signals, for example for ultrasonic-based user interaction. Some devices may have dedicated ultrasonic transducers for touchless interaction. The screen 101 may include not only a display for displaying content, such as photos and videos, but also a touch screen sensor for touch-based user interaction. In accordance with the present invention, the proximity sensor 106 is an acoustic detection-based sensor, or other type of sensor suitable for proximity detection. The proximity sensor 106 has a field of view ("FoV"), which is a three-dimensional envelope or space around the sensor 106 within which the sensor 106 can reliably detect a proximity event, e.g., a proximity event defined when an object or cover is too close to use acoustic reflection for detection. Detection of a proximity event is used, for example, to turn off the touch screen and display (or screen 101) of the device 100 to prevent undesired touch screen operations. Such undesired touch screen operations may occur when a user places earphone 102 in contact with or close proximity to the ear and the touch screen is not disabled. Detection of a proximity event using proximity sensor 105 is used to disable the touch screen so that undesired touch screen operations can be prevented.

[0029] Also shown in Figure 1 are a pair of speakers 105 and a microphone 103. Some phones may include one or more other speakers, e.g., 105, used for hands-free operation and / or audio playback. Such other speakers 105 may be different from the earphones 102, and such speakers 105 may be larger than the earphone speakers. In the illustrated example, the left speaker 105a and the right speaker 105b may be used for stereophonic playback. The microphone 103 on the underside of the phone 100 is typically used for phone calls, especially when the user holds the earphones 102 close to the ears. The microphone 103 may also be used for hands-free operation and other audio capture or recording.

[0030] Some phones may include one or more additional microphones, such as a microphone 104 on the top side of the phone. This additional microphone may be used, for example, for spatial sound capture or other purposes. Multiple microphones may be provided on either the top and / or bottom surface or the sides of the device 100. In some devices, multiple microphones and / or multiple speakers may be used for ultrasonic interaction with the device 100. Multiple receiver-transmitter combinations may be used to extract ambient spatial information. In some cases, such an ultrasonic sensing arrangement realized by a speaker and microphone can remove the requirement for a dedicated proximity sensor 106. Thus, in such cases, the device 100 may not have a separate proximity sensor 106.

[0031] 1, it can be seen that the proximity sensor 106 requires valuable space on the screen side of the device. Thus, in most cases, a bezel 102 may be required to accommodate a component such as a dedicated proximity sensor 106. In the absence of such a component, the screen 101 may extend toward the edge of the device, thereby allowing most or all of the screen side of the device 100 to be utilized as display area, rather than wasting space in the dead area of ​​the bezel 102.

[0032] 2 shows a side perspective view of the device 100. The FoV 205 of the proximity sensing system is shown extending divergently from the proximity sensor 106 along the axis 206 such that the cross-sectional area of ​​the FoV 205 in a plane perpendicular to the axis 206 increases with distance from the proximity sensor 106 along the axis 206. When the device is uncovered, the acoustic signal propagates through space above the transmitter 106 and the receiver 105 can detect the reflection to detect the proximity of a person or object as described in Norwegian Patent No. 20211333.

[0033] As shown in FIG. 2, a cover 207 covers the device, similar to a screen, at a distance 250 from the surface 101. For example, this distance would be zero if the device was placed on a table with the transducers 105, 106 facing down, or if a protective cover was used, or if it was simply placed in a pocket. In situations where the device is a phone held to the ear, with the ear and head forming the cover, there may be a distance 250 between the transmitter and the cover. Typically, this distance is much less than the wavelength of the acoustic signal, so the acoustic signal is partially constrained between the cover and the device, affecting the signal received at the receiver differently than a signal reflected from a more distant object. The signal can be represented as a waveguide for acoustic or pressure waves propagating in the plane between the cover and the device, and may be affected by structures that introduce interferences and reflections within the waveguide.

[0034] As shown in Fig. 3, the device 101 according to the invention includes a processing unit 309 for processing the signal received by the microphone 105. To perform an analysis of the received signal, the signal transmitted by the speaker 106 needs to be known. This can be achieved in the speaker driver circuit 306 by transmitting a signal with known characteristics to the speaker 106. However, as the speaker may be a multi-purpose transducer, there may be limitations such as influencing the transmitted signal. Therefore, as described in the above mentioned Norwegian patent no. 20211333, a reference signal may be directly communicated from the speaker 106 to the processor 309 to provide a direct comparison between the transmitted and received signals. Since the comparison can be made directly, different types of audio signals may be used as interrogation signals, including alarms, music, etc.

[0035] As mentioned above, according to another embodiment of the invention, the receiver or microphone 105 can listen independently of transmission to provide background noise or sound monitoring, for example as part of a voice-activated system. The processor may then instruct the speaker driver circuit to analyze the background signal and select a signal transmission in a frequency range or amplitude.

[0036] The processing unit is a computer or data processor, such as a microprocessor or microcontroller. The processing unit may be a combination of different hardware components or modules. In some cases, the processing unit may essentially be a virtual machine running on the processor. Also, in some cases, the processing unit may include at least one machine learning module to improve accuracy of measurements across use cases of the electronic device. The processing unit may further include an artificial intelligence ("AI") module.

[0037] As shown in Figure 4, the method according to the present invention starts at step 401, where an interrogation signal is defined and transmitted via the speaker or the like 106. The interrogation signal has a known characteristic, which may be predetermined or transmitted directly from the speaker 106 or acoustic driver circuit 306. The characteristic may be a defined frequency or part of the signal such as a chirp, or a known piece of music or sound.

[0038] During the same time period, and possibly also immediately following transmission, the signal is received (402) at at least one receiver, such as an audio transducer.

[0039] The difference between the transmitted and received signals can then be analyzed (403). The time delay between the transmission and reception of the signal characteristics may be indicative of whether the transducer-containing device is covered, whether it is amplitude independent, and / or whether it is frequency dependent. Thus, the received signal may be analyzed to find a selected number of differences, which number depends on the circumstances.

[0040] The detected differences are then compared (404) to a set of predetermined differences stored in computer memory within processor 309. As shown, this comparison and its results may be added to the storage (405) by machine learning or other analytical means to update the set of predetermined differences.

[0041] If the comparison indicates that the received data indicates that the device is covered, the processor provides a set of instructions appropriate to the situation, such as deactivating the display or touch surface (406), or, if the comparison indicates that the set of instructions activates the display and touch-sensitive surface (407). These instructions may be set by the user.

[0042] In summary, the present invention relates to a device or method for detecting a cover covering at least a part of an electronic device including at least one audio transducer adapted to transmit and receive acoustic signals, the method comprising: defining at least one characteristic of the interrogation signal; transmitting an interrogation signal within a predetermined frequency range having at least one characteristic using at least one of the transducers; receiving a receive signal with at least one audio transducer; Analyzing a characteristic of the received signal and a characteristic of the transmitted signal to detect a selected number of differences therebetween; Comparing the difference between the transmitted signal and the received signal to a set of predetermined differences associated with an object covering at least one of the transducers in the device; Includes.

[0043] The electronic device may be a mobile phone or a stationary or laptop personal computer and is capable of detecting whether an acoustic transducer is covered.

[0044] As a first step, the receiver may be used to detect acoustic signals from the environment and then the transmit signal may be adjusted to avoid interference with the signals from the environment. The analysis of the acoustic signals from the environment may include providing an ambient noise level as a function of frequency in a selected frequency range corresponding to the frequency range of the interrogation signal. The characteristic may be selected in response to the ambient noise level, or in response to the ambient noise level as a function of frequency in the selected frequency range, such that the transmit level is a predetermined amount less than the ambient noise level.

[0045] The interrogation signal is preferably in the audible range with a frequency from below 1 kHz to 20 kHz, and in some cases is at least partially constituted by an audio signal such as a music track. Alternatively, the interrogation signal is pulsed, the interval of which depends on the ambient noise level or other measurement activity such as device movement, or is constituted by a frequency chirp within a predetermined frequency range. The acoustic transmitter and receiver may be separate transducers or may consist of one transducer capable of operating in both modes.

[0046] According to one embodiment of the present invention, the comparison between the transmitted and received signals is based on pre-stored comparisons, which are based on neural networks or machine learning of previously measured, pre-processed, or raw microphone data.

[0047] The present invention also relates to an electronic device having a touch-sensitive surface and / or a display surface, comprising an acoustic transmitter and an acoustic receiver. The acoustic transmitter is connected to a speaker driver circuit and applies a known signal to the speaker to transmit an acoustic interrogation signal having a selected characteristic. The acoustic receiver is connected to a processor that receives and processes the acoustic signal. The processor is connected to at least the driver circuit or the speaker to receive selected information related to the characteristic of the transmitted interrogation signal. The processor is configured to detect a selected number of differences based on an analysis of the transmitted and received information. The differences between the transmitted and received signals may then be compared to a set of predefined differences associated with a particular object covering at least one of the transducers in the device. In this manner, objects may also be classified and recognized.

[0048] Preferably, the processor includes a computer memory device containing a number of variations of the selected difference associated with the presence of an object covering the device, the selected difference being pre-processed based on previous cover detections, for example using a machine learning or neural network processing module.

Claims

1. 1. A method for detecting a cover covering at least a portion of an electronic device including at least one audio transducer adapted to transmit and receive acoustic signals, comprising: - defining at least one characteristic of the interrogation signal; - transmitting, using at least one of said audio transducers, said interrogation signal within a predetermined frequency range having at least one of said characteristics; receiving a received signal with at least one of said audio transducers; - analyzing characteristics of the received signal and characteristics of the transmitted signal to detect a selected number of differences therebetween; comparing the difference between the transmitted signal and the received signal to a set of predefined differences associated with an object covering at least one of the audio transducers in the electronic device; A method comprising:

2. The method of claim 1 including an initial step of detecting acoustic signals from an environment and adjusting the transmitted signal to avoid interference with the acoustic signals from the environment.

3. 3. The method of claim 2, wherein the analysis of the acoustic signals from the environment includes providing an ambient noise level as a function of frequency in a selected frequency range corresponding to a frequency range of the interrogation signal.

4. The method of claim 2 , wherein the characteristics are selected as a function of an ambient noise level such that a transmission level is a predetermined amount less than the ambient noise level.

5. The method of claim 3 , wherein the characteristics are selected in response to an ambient noise level as a function of frequency in the selected frequency range.

6. 10. The method of claim 1, wherein the interrogation signal is in the audible range from below 1 kHz to 20 kHz.

7. The method of claim 6 , wherein the interrogation signal is an audio signal, such as a music track or a notification sound.

8. 10. The method of claim 1, wherein the interrogation signal is pulsed, the interval of which depends on the ambient noise level or other measurement activities such as movement of the electronic device.

9. 2. The method of claim 1, wherein the interrogation signal comprises a frequency chirp within the predetermined frequency range.

10. The method of claim 1 , wherein the transmitter and receiver are constituted by the same transducer.

11. 2. The method of claim 1, wherein the comparison between the transmitted signal and the received signal is based on a pre-stored comparison, the pre-stored comparison being based on a neural network or machine learning of previously measured, pre-processed, or raw microphone data.

12. 1. An electronic device having a touch-sensitive surface and / or a display surface, the electronic device including an acoustic transmitter and an acoustic receiver, the acoustic transmitter connected to a speaker driver circuit and transmitting a known signal to a speaker to transmit an acoustic interrogation signal, the acoustic receiver connected to a processor for receiving and processing the acoustic signal, the processor connected to the speaker driver circuit or a speaker to receive selected information related to a characteristic of the transmitted acoustic interrogation signal, the processor configured to analyze a characteristic of the received signal and a characteristic of the transmitted transmitted signal and detect a selected number of differences therebetween, the differences between the transmitted signal and the received signal being compared to a set of predetermined differences related to an object covering at least one of the transducers in the electronic device.

13. 13. The electronic device of claim 12, wherein the processor includes a computer memory device containing a number of changes in the selected difference associated with the presence of an object over the electronic device.

14. The electronic device of claim 13 , wherein the selected differences are pre-processed based on previous cover detections, for example using a machine learning or neural network processing module.

15. 13. The electronic device of claim 12, wherein the interrogation signal is in the range of less than 1 kHz to 20 kHz.

16. The electronic device of claim 12 , wherein the interrogation signal is less than 5 kHz.

17. The electronic device of claim 12 , wherein the acoustic transmitter and the acoustic receiver are constituted by the same transducer.

Images