Activity monitoring for electronic devices

The integration of IMU data with ultrasound in electronic devices addresses power consumption and measurement quality issues by enabling continuous and accurate proximity and cover detection with reduced power usage.

JP2024540810AInactive Publication Date: 2024-11-06ELLIPTIC LAB AS
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2024518449
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-09-23
Filing Date
2022-09-21
Publication Date
2024-11-06
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing electronic devices face challenges in maintaining measurement quality and reducing power consumption due to interruptions and high power consumption during ultrasound-based proximity and cover detection, especially when sensors are not functioning optimally.

Method used

A system that combines inertial measurement units (IMU) with ultrasound data to maintain measurement quality and reduce power consumption by using low-power units during inactive periods and enhancing responsiveness during active periods, allowing sensor data to be continuously collected and processed.

Benefits of technology

The system ensures continuous and accurate detection of device proximity and cover states with reduced power consumption by leveraging IMU data during ultrasound inactive periods, minimizing latency and optimizing sensor fusion for efficient power management.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024540810000001_ABST
    Figure 2024540810000001_ABST
Patent Text Reader

Abstract

The present invention relates to a method and an electronic device comprising at least two measurement units, a first measurement unit being an active measurement unit for transmitting and receiving signals related to a selected first parameter in a sequence including an inactive period, and a second measurement unit configured to essentially continuously measure a predefined second parameter, the power consumption of the second measurement unit being lower than the power consumption of the first measurement unit. The first parameter relates to a proximity measurement which involves measuring the proximity of an object near the device, and the second parameter relates to measuring the activity of the device. The device includes a processing unit coupled to the measurement unit, the processing unit configured to register deviations of the second parameter and to change the order of the first measurements upon a given deviation.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to electronic devices configured to monitor and control the use of the electronic device. [Background technology]

[0002] In electronic devices, including mobile phones and similar devices described in WO 2020246897A1, US 2020158556, WO 20210456628, US 2020026342, and US 2008 / 0051165, measurements such as ultrasonic measurements are used to detect the proximity, presence, and gestures of a user near the device, for example for user input to the system. In electronic devices with screens, ultrasonic measurements are also used to detect if an object is so close that it covers at least one of the screens and the user cannot visually interact with the device. If this cover detection feature is enabled, the device will not turn on the screen as long as the screen is covered. Cover detection sensors are designed to save power by keeping the screen off when it cannot be used. Cover detection sensors can report the distance to an object covering the screen, but are typically binary sensors with detectable cover detection states of "covered" and "uncovered." Proximity, occupancy, gesture and cover detection sensors can use multiple input streams, but they are tightly coupled to the ultrasonic input stream. However, because measurements require a transmit-receive sequence, there are pauses between measurements, limiting the amount of information that can be detected. This means that detection of sudden changes, such as moving the phone away from the listening position, may not occur in time. Also, active measurements consume power, so active periods must be minimized. In US Patent Application 2020158556, the power level of the ultrasonic signal is reduced to reduce the power consumption of the device, but this reduces the sensitivity and quality of the measurements. Summary of the Invention [Problem to be solved by the invention]

[0003] It is therefore an object of the present invention to provide a solution that improves the performance of a device whilst limiting its power consumption. This is achieved as set forth in the accompanying claims. [Means for solving the problem]

[0004] The invention provides a solution even when at least one of the sensors in the electronic device is not operational, and the invention uses a low-power unit to increase the activity of the measurement, consuming power as needed, maintaining the quality of the measurement while having low power consumption during inactive periods and responsiveness during active periods.

[0005] In a preferred embodiment, both sensor data (such as an inertial measurement unit (IMU)) and ultrasonic data are used to detect objects near or covering the screen of the device. Detection is tightly coupled with ultrasonic input and output data. With the present invention, the measurement system of the device can run as long as either or both of the IMU and ultrasonic data streams are running. If the ultrasonic stream is not active, the sensor estimates are updated only based on the available IMU data and vice versa. For example, the time to receive the first sensor data from an acceleration sensor is shorter than the time to receive the first audio sample. Therefore, utilizing the acceleration sensor data during the start-up of the audio system is beneficial to the performance of the proximity sensor or the cover detection sensor. The sensor data can also be continuously stored in memory by a low-power processor. Here, the sensor data is collected from all sensors in the system (such as a sensor hub) and provided to the proximity sensor or the cover detection sensor at start-up. This allows the proximity sensor or the cover detection sensor to use the sensor's pre-start data.

[0006] In another embodiment using sensor data (e.g., inertial measurement unit (IMU), hinge sensor, etc.), ultrasonic data is used for occupancy sensors in personal computers (e.g., laptops). These occupancy sensors are used to wake up the laptop when a user approaches and lock the device for security reasons when the user leaves. By combining the sensor data with ultrasonic input / output data, the occupancy sensor can use sensor fusion to analyze the current situation and improve sensing and potentially reduce power consumption. One example is detecting that a user is carrying a device and is clearly near the device. In such a situation, an ultrasonic signal may not be needed for the occupancy sensor as long as the device is being carried. This creates a discontinuity in the ultrasonic input / output signal. Another example is a system in which a hinge sensor provides the current angle of the laptop screen, providing important sensor fusion information to the occupancy sensor to improve performance for all angles of the screen. [Brief description of the drawings]

[0007] The invention will now be described with reference to the accompanying drawings, which show, by way of example, the invention. [Figure 1a] FIG. 2 illustrates a situation in which proximity detection is activated. [Figure 1b] FIG. 1b shows a different situation from FIG. 1a in which proximity detection is active. [Figure 2a] FIG. 1 illustrates prior art cover detection. [Figure 2b] FIG. 1 illustrates prior art cover detection. [Figure 2c] FIG. 1 illustrates a solution according to the invention with sensor activity in a context involving an IMU and an acoustic sensor. [Figure 2d]FIG. 2c illustrates a solution according to the invention with sensor activity in a different context than in FIG. 2c, including an IMU and an acoustic sensor. [Figure 2e] 2c and 2d show a solution according to the invention with sensor activity in different situations, including an IMU and an acoustic sensor. [Figure 2f] 2c-2e show a solution according to the invention with sensor activity in different situations including an IMU and an acoustic sensor. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Fig. 1 shows a situation where a mobile phone 1 is in use and is held up to the ear of a user 6. A processing unit 4 may be connected to a number of sensors, such as a proximity or cover detection sensor 2 and a movement / IMU sensor 3 according to a preferred embodiment of the invention. The proximity sensor 2 may detect this situation and the processing unit may then shut down certain activities, such as some functions related to the touchscreen 5, preferably using an ultrasonic signal 8. At the same time, the movement / IMU sensor 3 registers a certain amount of movement. This movement may be analyzed by the processing unit 4 and may indicate a change in use, for example that the device has been moved away from the user or even if the proximity sensor 2 is not active, which may indicate that the touchscreen 5 or the proximity sensor 2 should be activated.

[0009] As mentioned above, a device such as a mobile phone may include multiple different sensors, such as a light sensor. If the light sensor detects a significant amount of ambient light, possibly compared to a recent measurement, while the ultrasonic-based proximity or cover detection sensor 2 detects an object covering the screen, the processing unit 4 may determine that the ultrasonic sensor 2 is blocked and override the proximity or cover detection signal. Alternatively, the light sensor data may be fed directly to the proximity or cover detection sensor or may be included in a detection algorithm within the sensor or processing unit, including machine learning based sensor fusion as described in WO2020246897A1 mentioned above.

[0010] Some electronic devices may include a foldable screen, where at least a screen sensor (such as a contact sensor, a magnetic sensor, a hinge sensor, a Hall sensor, etc.) detects when the screen is unfolded. The sensor data may include angle information about the foldable screen. If the sensors detect that the screen is at least partially unfolded, an ultrasonic-based proximity or cover detection sensor 2 may be activated. If these sensors detect an object covering the screen, the processing unit 4 may determine that the ultrasonic sensor 2 is blocked and override the proximity or cover detection signal. Alternatively, data from at least one screen sensor may be directly fed to the proximity or cover detection sensor along with other types of sensor data (such as an IMU) or may be included in a detection algorithm in the sensor or processing unit, including machine learning based sensor fusion as described in WO2020246897A1 mentioned above.

[0011] Other electronic devices include hardware-based motion sensors that are used to control the operation of a low-power sensor hub. Even though the sensor hub is a low-power processing element, as long as the low-power hardware-based motion sensor indicates that the device is completely stationary, the sensor hub is turned off to save power. In this case, events for sensors such as the IMU sensor in the sensor hub are not reported because the sensor values ​​do not change. When the hardware-based sensor indicates that the device is not completely stationary, the sensor hub is turned on. This turns on all sensors, such as IMU sensors, that the user or system has requested to be active. This scheme is about reducing power consumption, since the hardware-based sensors consume less power than, for example, a sensor hub processor with an IMU sensor enabled. Although it uses a chain of sensors to operate, it can utilize the sensor setup described herein.

[0012] FIG. 1b shows a device with pocket mode enabled. Pocket mode is a feature found in many modern smartphones that uses a cover detection sensor to detect when a cover 7 is placed on the surface of the smartphone. This is a feature that is used in a variety of use cases. Essentially, one or more sensors 2 are used to determine whether the screen, touch sensitivity, or other active units should be at least partially turned off when, for example, the surface of the smartphone is covered, e.g., when it is face down on a table, in a bag, or in the user's pocket. Typically, the cover detection sensor 2 does not operate when the device is being used by the user, i.e., when the screen is on and the user is visually interacting with the smartphone. Typically, the cover detection sensor 2 is a one-off sensor that is automatically disabled when the first sensor event is reported. The purpose of this sensor is to prevent the screen from being turned on when it is covered and the user does not intend to use or visually interact with the smartphone or its screen.

[0013] Prior art is shown in Figures 2a and 2b. In Figure 2a, the cover detection CD is continuously active, typically based on IR detection. In this scenario, the cover detection sensor CD is not a momentary sensor, but continuously reports changes in the cover detection state. An example of a use case with continuous reporting is an always-on clock displayed on an unlit screen. This allows the user to check the time without having to fully turn on the screen, reducing overall power consumption. Ideally, a smartphone should only display the clock when the screen is uncovered, i.e. when the user can see the screen.

[0014] Figure 2b shows a more advanced system, where the movement sensor IMU registers certain activity and then triggers cover detection, which is repeated each time movement is registered.

[0015] More specifically, as shown in FIG. 2b, the cover detection sensor is triggered by the processing unit 5 based on a user action, such as raising the smartphone to the user's face (raise-to-wake), touching the screen 5, or pressing the power button, as detected by the IMU 3. When any of these triggers are detected, the device briefly activates the cover detection to determine whether the smartphone screen is covered. If the screen is not covered, an appropriate action is taken, such as turning on the screen or turning on any login mechanism, including a biometric login mechanism. If the screen is covered, as shown in FIG. 2b, the trigger event is ignored since the screen is covered, and the system waits for the next signal or detected action from the IMU. Some smartphones turn on the screen for a few seconds to allow another type of user action (such as a swipe up or a button press) to override the detection of an object covering the screen. In any case, when triggered, the cover detection sensor 2 is typically only active for a short period of time (i.e., a few seconds).

[0016] Considering the limited battery capacity of modern smartphones, the power consumption in pocket mode use cases based on ultrasonic cover detection sensors, or sensor fusion of ultrasonic cover detection sensors with other suitable sensors, may be too high for always-on scenarios.

[0017] According to the present invention, the pocket mode use case may be realized using acoustic sensors such as ultrasonic cover detection sensors 2 or by configuring a fusion sensor as described in WO2020246897A1, where a processing unit 4 combines sensor information from the ultrasonic cover detection sensors 2 with any number of other sensors including IMU sensors 3 (accelerometers, gyroscopes, etc.) and light sensors, at least some of which are configured to continuously monitor interactions with the device and have low power consumption compared to active ultrasonic sensors 2.

[0018] The objective of the invention is to reduce the power consumption or wear of the acoustic elements that transmit and receive ultrasound by triggering the cover detection sensor 2 based on pattern recognition used on signals from a number of sensors in the device, possibly based on machine learning, in a fusion sensor running on a low-power sensor hub in a processing unit 4, and at the same time monitoring and sampling the measured use of the device using a passive measurement unit. The processing of the neural network module embedded in the pattern recognition may be offloaded from the sensor hub to an optimized neural network processing core, if the power consumption of the neural network processing in that core is acceptable in terms of overall power consumption. The machine learning process may be of any available type using available tools (such as TensorFlow) and must be able to sample and store information from the various units and sensors in the device, possibly with a user interface that provides user feedback to find typical usage patterns. The data must be tagged with ground truth, either manually or automatically, before the neural network training process takes place on-device or off-device. This process may also take into account concerns such as energy consumption limits and adequate user feedback during training. The machine learning process may also be provided in an external system that can update the software of the device, for example through a firmware update. These updates may be made explicitly by the user or through an over-the-air solution.

[0019] The present invention can use various sensors, including IMU sensors, light sensors, motion sensors, etc., to model and detect smartphone usage patterns, i.e., situations where the screen cover detection state changes enough to warrant a cover detection state update (e.g., removing the device from a pocket, lifting it to activate, placing the device in a pocket, placing the device on a table, walking while holding the smartphone in the hand or in a pocket, etc.). When a pocket mode use case is initiated, sensor information is requested from the device's sensors, possibly including cached information from before the cover detection sensor was activated. All information, including current information, is used to determine the cover detection state. Reducing the latency of sensor events is another advantage when the pocket mode use case is executed only when pattern recognition suggests a possible change in the device's pocket mode state. Since most modern smartphones have an acoustic system wake-up time on the order of 50 ms to 150 ms, a cover detection sensor based on the scheme described herein can respond immediately (i.e., in milliseconds) without waking up the acoustic system as part of a complete transient cover detection sensor cycle.

[0020] As mentioned above, another aspect of the present invention is bridging ultrasonic discontinuities. As shown in FIG. 2c, the ultrasonic system signal 12, including the transmitter US TX and receiver US RX, may have periods 10 during which no actionable information is detected due to unmanageable interference in the frequency band used by the ultrasonic system from other ultrasonic transmitting devices, or destructive intermodulation due to concurrent acoustic use cases, or unmanageable and random gain fluctuations due to speaker protection algorithms varying the ultrasonic output signal, or concurrent acoustic use cases playing at high amplitudes that leave no headroom for mixing of ultrasonic signals, or latency on the order of 50 ms to 250 ms from the activation of the sensor by the system until the ultrasonic input / output path is ready to play and capture the ultrasonic waves. In some cases, the ultrasonic sensor may transmit a non-continuous ultrasonic signal as a burst signal. This creates a discontinuity in the ultrasonic output and ultrasonic input signals. Another similar problem that causes discontinuities in the ultrasonic signal and thereby prevents normal sonication is changing the acoustic system due to a change in the use case. Some of the most popular applications on modern smartphones switch audio input and / or output from one audio device to another based on user actions. Some applications use different audio elements depending on whether the user is holding the smartphone in front of the body or against the ear. The selected audio device can change at any time if the user moves the smartphone again or explicitly causes the smartphone to use speaker mode, headset mode, or handset mode.

[0021] In some platforms with a shared back-end digital acoustic interface (DAI), switching audio from a particular component (e.g., speaker) to another (e.g., earphone receiver) requires other simultaneous audio use cases, such as the ultrasonic output signal used by a sonic sensor, to be stopped when the audio stream is switched from one audio output device to another and resumed after a short pause of tens to hundreds of milliseconds. Because ultrasonic sensors require a stimulus signal to generate echoes from nearby objects, the ultrasonic process is stopped or interrupted and restarted or resumed, respectively, during this process. As a result, ultrasonic use cases (e.g., proximity sensor, gesture sensor, presence sensor, cover detection sensor, etc.) do not have the relevant information to make a decision when this discontinuity occurs.

[0022] By monitoring activity (keyboard usage, touchscreen usage, hinge sensor events, etc.), it is also possible to simply reduce the ultrasonic signal power to conserve power.

[0023] As shown in FIG. 2d, using the present invention, inputs 11 from other sensors such as IMU information may be obtained (when the smartphone is stationary, in a pocket, raised to the head, removed from the head, etc.) and a sensor event may be provided based on one or more other sensor inputs even while the ultrasound is stopped. When the ultrasound stream resumes, the present invention seamlessly accepts data from all sensors 11 and ultrasound signals 12 and uses the combined information (e.g., potentially including sensor data cached during the ultrasound interruption) to provide an accurate sensor event, and through machine learning algorithms implemented as neural networks, can also recognize and use patterns of information from different sensors to estimate device usage during periods when only a portion of the sensors are active.

[0024] In figures 2e and 2f similar situations are shown where the ultrasonic signal 12 is interrupted: in figure 2e the device is therefore unable to obtain ultrasonic information for a short period of time, whereas in figure 2f the device is able to recognise changes in the use of the device due to the additional information and adapt to them or activate ultrasonic sensors depending on the registered circumstances.

[0025] Thus, the signal from a second measurement unit, such as an IMU 3, may include information about the activity of the device both before and during ultrasonic sensor activity, and during interruptions or cessation of ultrasonic sensor activity providing periods without ultrasonic data 10.

[0026] In summary, the invention relates to a method and an electronic device comprising at least two measurement units. The first measurement unit is an active measurement unit that transmits and receives signals related to a selected first parameter in a sequence including at least one active transmission and reception period that provides a signal related to the proximity of an object, such as a cover, to at least a part of the device, such as a screen. The sequence will thus have inactive and active periods. The second measurement unit is configured to essentially continuously measure a predetermined second parameter. This is preferably a passive measurement related to the usage and activity of the device or the device surroundings, such as sound or illuminance, and preferably has a substantially lower power consumption than the first measurement unit.

[0027] The device includes a processing unit coupled to the measurement units and configured to register deviations in the second parameter and, at a predetermined deviation, alter the order of the initial measurements by activating the first measurement unit. The registration of the second parameter may be performed continuously or at a rate sufficient to provide essentially continuous monitoring of the second parameter relative to a reasonably expected rate of change of the parameter. The processing unit may configure a low power hub for the sensors to limit power consumption of the device and may be configured to combine measurements from all sensors to provide a continuous status of activity usage of the device.

[0028] Preferably, the first measurement unit is an ultrasonic measurement unit configured to detect an object, such as a cover or a user, in the vicinity of the device. The ultrasonic measurement is performed with any available sensor type capable of detecting an object near the sensor or covering the sensor. For ultrasonic detection, special transducers or transducers already installed in the device may be used, such as speakers and / or microphones suitable for operation in the ultrasonic near-field, for example in the range of 20 kHz to 24 kHz.

[0029] Preferably, the second measurement unit is an inertial measurement unit (IMU) configured to measure the movement of the device, but may also include a light sensor to measure ambient light, a keyboard / mouse, or possibly a touch sensor, e.g., a screen of the device or a button or switch on the device, and may be selected depending on which available measurement unit consumes the lowest power. When using a touchpad or screen, interactions with the device surface may be measured and its movement patterns analyzed.

[0030] The processing unit of the device may be configured to activate the first measurement unit when the deviation exceeds a predefined limit, such as to allow some minor movement before the active sensor is activated. The threshold may be set at device initialization or may be adaptively set based on previous measurements and machine learning, such as by determining how often the cover detection state actually changes based on when the sensor is activated. In this way, the deviation may be based on a predefined measurement pattern, such as a measurement pattern of device movement while powered on, where the predefined pattern may be based on previous movements registered as typical movements of the device through analysis.

Claims

1. 1. An electronic device including at least two measurement units, including a first measurement unit and a second measurement unit, the first measurement unit being an active measurement unit configured to transmit and receive signals related to a selected first parameter in a sequence including an inactive period, and the second measurement unit being configured to essentially continuously measure a predetermined second parameter, the power consumption of the second measurement unit being lower than the power consumption of the first measurement unit; the first parameter relates to a proximity measurement, which comprises measuring the proximity of an object near the electronic device, and the second parameter relates to a measurement of activity of the electronic device; the electronic device includes a processing unit coupled to the first measurement unit and the second measurement unit, the processing unit being configured to register deviations of the second parameter and to change the order of initial measurements by activating the first measurement unit upon a predetermined deviation. Electronic devices.

2. The electronic device of claim 1 , wherein the first measurement unit is an ultrasonic measurement unit configured to detect an object, such as a cover or a user, in the vicinity of the electronic device.

3. The electronic device of claim 1 , wherein the second measurement unit is an inertial measurement unit or a motion sensor configured to measure a movement of the electronic device.

4. The electronic device of claim 1 , wherein the second measurement unit is a touch sensor unit that measures interactions on a surface of the electronic device.

5. The electronic device of claim 1 , wherein the second measurement unit is a light sensor that registers changes in ambient light.

6. The electronic device of claim 1 , configured to activate the first measurement unit when the deviation exceeds a predetermined limit value.

7. 2. The electronic device of claim 1, wherein the second measurement unit is a keyboard and / or a computer mouse connected to the electronic device.

8. 2. The electronic device of claim 1, wherein the deviation is based on a predetermined measurement pattern including a measurement pattern of movement of the electronic device while the smartphone is powered on and while the user is handling the smartphone and while the user is not handling the smartphone (in a pocket, a bag, on a table).

9. The electronic device of claim 8 , wherein the predetermined measurement pattern is based on previous movements of the electronic device that have been registered through analysis as typical movements of the electronic device.

10. 2. The electronic device of claim 1, wherein an analysis is performed based on machine learning from previous measurements from the first measurement unit and the second measurement unit, and the predetermined deviation is processed with a neural network based on the analysis.

11. 1. A method for controlling a device including at least two measurement units, including a first measurement unit and a second measurement unit, comprising: the first measurement unit being an active measurement unit configured to transmit and receive signals related to a selected first parameter in a sequence including an inactive period, the measurement being related to a proximity measurement including measuring the proximity of an object; the second measurement unit is configured to essentially continuously measure a predetermined second parameter; storing and registering deviations of the second parameter and, at a given deviation, changing the order of the initial measurements by activating the first measurement unit, method.

Citation Information

Patent Citations

  • Gesture detection management for electronic devices

    JP2016502721A

  • Radar correspondence sensor fusion

    JP2020101560A

  • Method for power saving based on motion sensor and mobile terminal using the same

    KR1020100086141A

  • Method for controlling display using sensor data and electronic device thereof

    US20170244827A1

  • Setting polling interval of open / closed position sensor for a movable portion of an electronic device based on activity state of the electronic device

    US20200379538A1