System and method for controlling access

Abstract

The invention relates to a method and system for initiating a communication sequence between two devices, wherein both devices include corresponding communication units, and wherein at least one of the devices comprises at least one sensor for registering a short physical contact between the device and an object The sensor is connected to a processing unit and is configured to transmit a signal to the processing unit. The processing unit is configured to recognize a signature in the sensor signal being associated with a tapping of the device toward a surface, and the processor, at the detection of the signature, is configured to initiate the communication between the devices by transmitting a request.

Description

[0001] SYSTEM AND METHOD FOR CONTROLLING ACCESS

[0002] The present invention is related to a system and method for initiating access or communication between two or more devices.

[0003] Communication between electronic devices has become very common, for example in public areas such as stores, coffee shops, hotels and restaurants in order to provide WiFi access to their customers as discussed in WO2022 / 189140, NO20221243 and NO20221245. These are aimed at limiting the access to a system or between electronic devices using a combination of acoustic signals, which indicates the distance between the devices and may transmit access codes, and WiFi for initializing and maintaining communication. In WO2011042748 and NO20221245 solutions are described where for example connection is made to screens or local communication units where the relative positions may be relevant.

[0004] All of the solutions above require a degree of using active signals in order to detect the presence of other devices in the vicinity, user interface through a screen and / or establishing connection initializing the communication. This increases the power consumption of the devices, especially when several devices are located in the vicinity which may be requesting access.

[0005] The present invention is aimed at solving the problems mentioned above using a very simple and energy efficient method, device and system for initializing a communication. This is obtained as defined in the accompanying claims.

[0006] According to the invention the communication may be initialized using a simple movement and a passive sensor, thus reducing the overall power consumption in the device before trigging the communication. The invention will be discussed below with reference to the accompanying drawings, illustrating the invention by way of examples.

[0007] Figure 1 illustrates the system according to the invention.

[0008] Figure 2 illustrates the signal received at the microphone according to the invention.

[0009] Figure 1 illustrates a laptop computer 1 with a body part 2 including a keyboard 3. The laptop also includes at least one acoustic transducer units 5,6, in the drawing including at least one speaker 5 and at least one microphone 6 in the figure being constituted by two transmitters, such as a speaker 5, and one receiver, such as a microphone 6. As is well known in the art the acoustic transducer unit may work both as transmitter and receiver. The laptop in the drawing also has additional sensors 8 such as camera or acoustic transducers that can be used for which may be used as proximity sensors. Vibration and motion sensors (not shown) may also be used to detect a tapping 14 against or between the devices.

[0010] The at least one speaker and / or microphone may be constituted by standard units of a type allowing use outside the audible range, preferably in the ultrasound range in the 20-25kHz range. It is also possible to use transducers being suitable for both transmitting and receiving ultrasound signals in which case the microphone 6 may be omitted.

[0011] In figure 1 a mobile device 11 is also shown. As mobile devices 11 also include acoustic transducers 12,13 for transmitting and receivers the mobile device may also be capable of detecting a tapping 14 between devices 1 ,11 , either initiating a communication between the devices or initiating a listening sequence waiting for an initiation signal from the other device.

[0012] The present invention is using either the internal audio system or at least one embedded sensor 15 (e.g. vibration sensor, Inertia Measuring Unit (IMU), acceleration sensor, magnetometer, motion detection sensor, fusion sensors based on a combination of other sensor, etc) in an electronic device to detect that the device itself is either being tapped against another object or is being tapped by another object. In this context, tapping means one or more consecutive taps. Identifying the tapping regardless of initiator will be used to initiate a communication phase from the primary device to another secondary device which may or may not be the object that was tapping or being tapped by the electronic device. In some situations, the electronic device may use sensor fusion between concurrent data points from multiple embedded sensors to identify the tap in more reliable way. Combining information from the audio system with concurrent data from one or more other sensors such as movement sensors, infrared sensors or radio frequency sensors in a processing element using signal processing or machine learning through a Deep Neural Network (DNN) based on Convolutional Neural Network (CNN) and / or Recurrent Neural Network (RNN) or a combination of both is also a viable approach that could also improve the reliability of the detection. Thus, as an example, the system might learn to differentiate between tapping with an inactive device, like furniture, and a device emitting a radio (such as WiFi) signal, or when a mobile device is actively tapped and moved against a stationary laptop or the opposite situation, which could represent different situations.

[0013] The communication phase may either be establishment of a communication path between the two devices where data may be exchanged for a period once the communication channel has been established. In some cases, the communication relationship is already established, and the communication phase can commence immediately. Alternatively, the communication phase may be a preconfigured signal to perform specific actions such transferring a conference call (i.e. Teams, Zoom, WebEx, etc) from the secondary device to the primary device or vice versa or transfer a specific application with its current context (e.g. the most recently edited document, song currently being played, etc) from one device to another. According to one embodiment of the invention, the different signal from different sensors may be compared to investigate how the tapping affects the measurements. For example, the comparison between the frequency specter of the resulting sound signal with the response of a movement sensor such as an IMU, accelerometer and gyro may indicate the risk of using denoise in the movement sensor from certain sounds or frequencies from a sound signal. Any such interference may then be reduced by filtering out such frequencies from an emitted sound signal (e.g. denoising).

[0014] The requested action triggered by the device tapping should preferably be configured by the user. The configuration can also depend on the nature of the tapping itself (e.g number of taps, timing between taps, pre-defined multi-tap sequence, strength of tap(s), location of tap, etc) to allow different taps to mean different things. It could also be based on the orientation and movement of the electronic devices when the tapping occurs provided the devices can detect its own orientation (i.e. 6 degrees of freedom, screen angle sensor, hinge sensor, hall sensor, etc). The primary device and secondary device can, if available, exchange information about their orientation at the time of the tapping process or it can relay the information during the subsequent communication phase as an implicit or explicit part of the protocol.

[0015] The primary and secondary device may be in active state where all systems are enabled or in a sleep state where at least the relevant embedded sensors which may include the audio system is still operational. In some systems, only the microphones can be enabled in low-power sleep states. In other systems, only the speakers can be enabled in these low-power sleep states. In other systems, both microphones and speakers can be enabled. It is important to understand the power management and power state in the electronic devices which are used for this innovation. In some scenarios, the power consumption of the different subsystem may dictate the solution that is chosen for the tap detection. Although the tapping is normally done by a user with another electronic device such as a smartwatch, mobile phone or tablet, there are scenarios where the tapping can be done by a production robot or a mechanical device suitable for the purpose of tapping.

[0016] As discussed, the audio system may include at least one microphone and at least one speaker. Electronic devices with multiple microphones and multiple speakers may provide more data points making the detection process using the audio system more reliable and robust.

[0017] The electronic devices can be passive, that is, only the microphones are enabled during the tap process. Alternatively, one or both electronic devices can in addition to the enabled microphones actively emit a continuous or duty- cycled acoustic signal by enabling at least one speaker too. The acoustic output signal emitted by the electronic device may be a sine, a chirp, a coded signal or a consist of one or more sine frequencies ranging from infrasound to audible frequencies and ultrasound frequencies. The signal may even be a combination of the signals mentioned above. The acoustic signal should ideally not be audible but use frequencies outside of the audible band. If audible sound must be used, the signal can be masked by mixing the coded signal with a suitable audible sound. When the tap occurs, the microphone(s) will receive an impulse due to frequency components of the tap sound but also from vibrations in the microphone caused by the physical displacement of the device due to the tap or vibrations through the device and its mechanical components.

[0018] In figure 2 a typical acoustic signature 20 of one tap is illustrated. This may depend on the materials and mass of the two devices and may as such include a possibility to recognize the devices from the sound signature. As the signature also depends on the applied force and where on the devices the tapping is performed it may also indicate the identity of the user. The electronic device may use both the acoustic signature of the tap process itself and in some cases also analyze the acoustic echoes from the user tapping one object against the other and align the information in time from the echos of the user while tapping with the acoustic tap signature to increase reliability and robustness of the detection. As discussed above the present invention may also use a second sensor 15 (e.g. MEMS sensor) configured to provide a signal 21 also being related to the tapping process, adding additional information to the processing and thus more reliable solutions and additional analysis, such as direction and amount of movements, vibration sensitivity and frequency range of additional sensor, etc.

[0019] The accuracy and interpretation of the signature may be chosen depending on the situation, taking into account the expected variation in the tapping process. If several users are expected to request access using it the accepted deviation will be largen than when is only related to one user, in which case the user may be asked to repeat the tapping a number of times the first time the system is use, thus registering the signature and expected deviation.

[0020] In one embodiment, the user is tapping the screen of an electronic device with his / her mobile phone to initiate the setup of a connection between the two electronic devices through any viable communication channel that includes wired or wireless technology either directly between the devices or via a suitable device such as a networking router or wireless access point. The communication channel may be secure utilizing authentication and encryption schemes. The primary electronic device may be any type of electronic device including computers, smartwatch, laptops, tablet, radios, TVs, speaker system or mobile phones, NFC-devices, tags, etc. Similarly, once the user taps the electronic device with his / her mobile phone after the communication channel has been established, the secondary device may send its own information about the tap (e.g. acoustic data, IMU data, or data from other device sensors such as vibration sensor, magnetometer sensor, etc) to the primary device to allow it to compare the information to improve the robustness. The acoustic signature of the received microphone signal caused by the device tapping can be identified through Deep Neural Network inference based on preprocessing of the raw microphone data into ML features based on for instance amplitude changes, wavelets, FFT analysis etc.

[0021] The electronic device can use data extracted from the user during either an initial calibration step or setup phase to learn the acoustic signature of the tapping made the user and his / her device. The acoustic signature can be used to update the original DNN used to identify the tapping done by the user. The DNN update can be done on the device or by utilizing updates done in a cloudbased service. If the user is changing either the primary or secondary device, the calibration step or setup phase must be performed again.

[0022] External noise and concurrent acoustic playback from the electronic device itself can make it harder to correctly identify the acoustic signature of user tap. To differentiate external noise and concurrent audio playback from the electronic device itself of the, the electronic device should monitor the acoustic playback and use it to remove the acoustic playback from the raw microphone data to make it easier to identify the acoustic signature of the tapping.

[0023] Removing the concurrent acoustic playback can be done using normal signal processing techniques or by using one or more DNNs to do the extraction. The extraction DNN can be combined with the identification DNN if it makes the processing more efficient

[0024] The electronic device may include not only an audio system but other sensors that may be used to detect a tap done by the device, or a tap done to the device.

[0025] In another embodiment, both devices are electronic devices, and both can detect tapping. In this case, when the primary device initiates the process to set up the communication path due to its own tap detection. The information can be transmitted to the secondary device using acoustic, wired or wireless technologies such as WiFi and Bluetooth Low Energy (BLE). When receiving the tap information, the secondary device can use its own independent detection capabilities and corresponding timing information to accept or reject the communication set up from the primary device. In some embodiments, the tap may only trigger the communication phase if the primary device orientation or device set up are according to the use-case specification. If the secondary device did not detect a tap or either both devices or either one of them was not oriented according to the specification for the use-case in question, it could reject the communication setup immediately and return a suitable error code to the primary device providing context as to why the communication setup was rejected.

[0026] Even though this invention describes a scenario with only two devices, it is possible to use the invention in similar fashion even when there are more electronic devices involved in the process describe in detail above.

[0027] In one embodiment, the primary device enables the microphone to capture data with a predefined sampling rate (e.g. 16 KHz or higher). The primary device will monitor its own concurrent playback if any and process the received microphone data using a DNN specifically designed to separate the acoustic playback from the rest of the acoustic signals The secondary device will enable IMU sensors and monitor them to detect when the secondary device taps the primary device. This processing should preferably be done as background processing using a low-power sensor system in the secondary device. Once the primary device detects the acoustic signature of the tap, it should initiate the setup of the communication channel by connecting to the secondary device. If the secondary device receives the connect request within a configurable time period T of its own tap detection, the secondary device will accept the connect request and continue establishing the communication channel. It is also possible for the secondary device to continuously capture sensor data including IMU data in the background into a finite buffer but postpone the processing of the data to detect the tap until the primary device has indicated that a tap occurred at time T1 . In this scenario, the sensor data buffer must be large enough for the secondary device to retrieve enough historical sensor data to be enabled to do the tap detection for the targeted time window given the event information received from the primary device. In this case, the secondary device can reduce tap processing cycles and thereby its own power consumption.

[0028] In some scenarios based on capabilities or design choices, it is clear which device is the primary device and which device is the secondary device. However, there are scenarios where both devices may be equally equipped to act as primary and secondary device. In such scenarios, the first device to initiate the setup of the communication channel may act as a primary device and the other device may continue as a secondary device. There is a multitude of different ways to select primary and secondary device in this case but they are not important for this invention.

[0029] The established communication channel between the primary or secondary devices can be used for several different use-cases including but not limited to:

[0030] • Screen sharing

[0031] • File sharing

[0032] • Transferring calls from primary device to secondary device or vice versa.

[0033] • Secure communication channel VoIP or AV calls.

[0034] • Transferring applications with or without current context from one device to another

[0035] Initiate pairing of a peripheral Bluetooth device to a central device or other pairing or connect sequences

[0036] In another embodiment, the user will tap his / her smart speaker with his / her smart phone to transfer the song that is currently being played on the smart speaker to his / her mobile phone to make sure that the song continues to be played on the mobile phone or the wireless air pods connected to the phone. Similarly, the song can be transferred from the user’s smart phone to the smart speaker or Smart TV if the user taps the secondary device with his / her smart phone.

[0037] In another embodiment, the primary or secondary device may run other sensors concurrently such as Human Presence Detection or Occupancy-sensors based on Time-Of-Flight sensors, camera vision, IR sensors, wireless technologies, radar solutions, UWB solutions or ultrasound-based echo sensors. These sensors can be used to gate the tap detection on the electronic device. Unless the presence sensor indicates that someone is in front of the device and using it, the tap detection is disabled. Once humane presence is detected, the tap detection will be enabled again. It is also possible to only allow tap detection if the devices are in acceptable power states for instance logged into the computer or the TV is turned on.

[0038] If both devices are synchronized using a high-precision clock e.g. Precision Time Protocol or similar, which should ensure microsecond resolution, the timestamp(s) of the tap(s) can be exchanged during the communication phase to make sure that the taps happened at the same time to allow the secondary device to accept the communication request from the primary device. If the timestamps differ, the secondary device may reject the communication setup returning an appropriate error code.

[0039] Gating the tap detection could in some cases save processing and power consumption. If the device that is used for tapping needs to be handheld during the tapping, the tap detection can be suspended or disabled if the device is not moving or being handheld. Until the device starts to move, tap detection will not take place. In some devices, low power hardware-based motion detectors may be used suspend the tap detection while the device is not moving. Once the device starts moving again, the tap detection process can resume. The gating can be used to temporarily stop or at least significantly reduce the sampling of the sensors that are used for tap detection. One example is that the IMU sampling rate can be reduced while the device is not handheld but is stationary. Other one-shot or on-change sensors detecting significant motion can be used to detect when the continuous sampling of the IMU sensors need to resume. This scheme allows the device to limit its power consumption when the device cannot tap the other device.

[0040] Another possibility is to use machine learning to create a DNN that will recognize how a user is handling his / her device to gate the tap detection unless the user is actively trying to tap the other device. This scheme requires a base model based on ML training from a large data collection before the user gets the device. The DNN model can be updated on-device or in a cloud-server as the user starts using the device for tapping. Once the DNN model has been updated with enough relevant date from the user (i.e. left-hand, right-hand, etc), the gating of the tap detection can be extended. It is also possible to use information from the environment around the electronic devices to gate the tap detection. This is possible if the electronic device can communicate and share environment information before tapping process. If the electronic device has GPS sensors, tapping can be gated if the devices are not close to each other. Similarly, the tapping can be gated if the devices are connected to wireless networks that are not close to each other. As an example, Bluetooth Low Energy has a scheme where the BLE connection is resumed when the devices are in proximity again after being out of reach from each other using BLE. In this scenario, the tap detection would only be active on both devices when the BLE connection is available. In BLE, tapping can also be gated when the RSSI value (i.e. signal strength) of the received messages is low indicating that the other device is not close enough to tap the device.

[0041] When tapping the phone against the laptop screen, sound will propagate both through the air and the laptop itself to the microphones. The received sound on the microphone will be a short impulse, varying somewhat between the phone, phone casing, the laptop, materials, etc. An example is shown in the figure above, where a phone with a hard cover was tapped against the laptop screen. The figure shows the microphone data from one single tap.

[0042] To summarize the present invention relates to a method and system for initiating a communication sequence between two devices, wherein both devices include corresponding communication units, and wherein at least one of the devices comprises a sensor for registering a short physical contact between the device and an object and generating a signal related to the registration. The sensor is connected to a processing unit, the processing unit being configured to recognize a sensor signature being associated with a tapping of the device toward a surface, wherein processor at the detection of the recognized signature is configured to initiate the communication between the devices by transmitting a request.

[0043] The sensor may be a microphone and the sensor signature is the acoustic sound signature of the tapping of the device toward the object. The sound signal will not only reflect the tapping incident but also the surface materials and applied force. The acoustic signature may then be is the recognized signature between two known devices, e.g. the materials of the devices. As discussed above other sensors may also be contemplated, both in combination with and as alternative to microphones. For example, inertia measuring units, accelerometers etc. The combination of sensors may provide improved measurements, for example by registering the acceleration or inertia of a device at the same time as the sound, thus avoiding false positive detections when similar sounds occur in the surroundings. Using two or more microphones may also improve the signal quality, and also be used to detect where on the device the tapping has been performed, for example by using the phased differences between detection at two different sensors. The location may then be used to interpret the tapping, for example by defining different areas starting different processes or being interpreted as different requests. If a computer registers a tapping close to the screen it could be interpreted as a request to share screens and close to the keyboard as a request to share the keyboard, etc. Similar solutions can also be defined using other sensors or combinations of sensors in one or more of the devices, for example registering the direction of a tapping movement using an IMU or accelerometer in one of the devices in addition to a registered sound or vibration from the tapping.

[0044] The object may be a device also including the ability to register and possibly recognizing the tapping signature. When two objects are tapped against each other they may also have different sensors. As an example, if one of the devices, for example a computer, has microphones, and the other device includes an accelerometer the devices register different signatures but can together confirm a tap and initiate a process.

[0045] One of said devices may thus include a second sensor configured to measure a characteristic related to the tapping, such as the movement of the device. The processing unit may then be adapted to analyze the information related to the sensors, for example the frequency response a sensor, and be configured to adapt the communication signal to the frequency response in order to avoid interference in the second sensor. Each device may include its own processing unit and, if connected, be configured to communicate the result of the analysis, or a sensor in on device may be configured to communicate the sensor signal to the other device.

[0046] The request transmitted after having detected a tap may be an acoustic signal configured to be recognized by a second device or an electromagnetic signal such as WiFi, Bluetooth or Infrared.

Claims

Claims1 . Method for initiating a communication sequence between two devices, wherein both devices include corresponding communication units, and wherein at least one of the devices comprises at least one sensor for registering a short physical contact between the device and an object, the sensor being connected to a processing unit and being configured to transmit a signal to the processing unit, the processing unit being configured to recognize a signature in the sensor signal being associated with a tapping of the device toward a surface, wherein processor, at the detection of the signature, is configured to initiate the communication between the devices by transmitting a request.

2. Method according to claim 1 , wherein at least one of the at least one sensor is a microphone and the sensor signature is the acoustic signature of the tapping of the device toward the object.

3. Method according to claim 2, where in the acoustic signature is the recognized signature between two known devices, e.g. the materials of the devices.

4. Method according to claim 1 , wherein the object is a second device also being configured to recognize a corresponding sensor signature.

5. Method according to claim 1 , wherein the transmitted request is an acoustic signal configured to be recognized by a second device.

6. Method according to claim 1 , wherein the transmitted request is an electromagnetic signal such as WiFi, Bluetooth or Infrared.

7. Method according to claim 1 , wherein at least one of the devices includesa second sensor being configured to measure a characteristic related to the tapping, especially a movement.

8. System for initiating a communication sequence between two devices, wherein both devices include corresponding communication units, and wherein at least one of the devices comprises at least one sensor for registering a short physical contact between the device and an object, the sensor being connected to a processing unit for providing a sensor signal to the processing unit, the processing unit being configured to recognize a signature in the sensor signal being associated with a tapping of the device toward a surface, wherein processor, at the detection of the signature, is configured to initiate the communication between the devices by transmitting a request.

9. System according to claim 8, wherein at least one of the at least one sensor is a microphone and the sensor signature is the acoustic signature of the tapping of the device toward the object.

10. System according to claim 9, where in the acoustic signature is the recognized signature between two known devices, e.g. the materials of the devices.11 . System according to claim 8, wherein the object is a second device also being configured to recognize a corresponding sensor signature.

12. System according to claim 8, wherein the transmitted request is an acoustic signal configured to be recognized by a second device.

13. System according to claim 8, wherein the transmitted request is an electromagnetic signal such as WiFi, Bluetooth or Infrared.

14. System according to claim 8, wherein one of said devices includes a second sensor configured to measure a characteristic related to the tapping, such as the movement of the device.

15. System according to claim 14, wherein the system includes a processing element adapted to analyze the information related to the sensors, for example the frequency response of the second sensor, the system being configured to adapt the communication signal to the frequency response in order to avoid interference in the second sensor.

16. System according to claim 8, wherein the at least one sensor includes an inertia measuring unit.

Images