Stable Virtual Sensor

Abstract

The present invention relates to an electronic device and a corresponding method, wherein the electronic device includes at least two sensors. The at least two sensors communicate with a virtual sensor including a processing unit adapted to monitor usage of the device based on inputs from the sensors. The at least two sensors include an activity sensor that senses a user's activity with respect to the device. The processing unit is configured to receive and analyze inputs from the sensors during usage and to provide and store a model representative of typical usage of the device. Furthermore, the device is further configured to compare new measurements received from the sensors with the model and provide a signal if the new measurements from the sensors deviate by a predefined limit value from measurements predicted by the model.

Description

[Technical field]

[0001] The present invention relates to electronic devices having at least one virtual sensor, and more particularly to activity sensors, such as acoustic and inertial sensors, that measure user activity in relation to the electronic device, including gesture recognition, presence detection, activity detection, and positioning. [Background technology]

[0002] Electronic devices are increasingly equipped with physical sensors to provide desired functionality to users. Some electronic devices also utilize virtual sensors, i.e. software sensors that combine information (e.g., sensor events, time series data) from at least one physical sensor and create a new sensor that provides potentially new or improved sensor output based on customer requirements. In most cases, virtual sensors utilize sensor fusion techniques, such as those described in U.S. Patent Application Publication No. 2018 / 0306609 or WO 2020 / 246897, which combine distinct sensors of different types in a "smart home" configuration, to combine information from one or more core sensors to create an entirely new sensor. This provides a cost-effective alternative to specialized hardware sensors in electrical devices that combine the use of acoustic transducers and other sensors to improve the operation of the device. The core sensor is most often a hardware sensor, but can be a virtual sensor that provides a core functionality primarily based on software (e.g., a virtual gyroscope based on acceleration sensor data). In some cases, an aspect of sensor fusion of virtual sensors may be used to create sensors that are not available or even feasible as hardware sensors due to design complexity, size, weight, or cost.

[0003] Virtual sensors provide new options for electronic devices, simplifying design, reducing costs, and eliminating dependency on the supply chain of hardware sensors. Because virtual sensors are software sensors, software updates, including over-the-air updates, are possible even after users acquire the electronic device, allowing the performance of the virtual sensors to be improved. Summary of the Invention [Problem to be solved by the invention]

[0004] A drawback of virtual sensors based on sensor fusion is the situation where at least one of the input sensors is intermittently or permanently defective and produces incorrect sensor event outputs, which can lead to erroneous behavior and cause the virtual sensor to fail. [Means for solving the problem]

[0005] It is therefore an object of the present invention to provide a device and a method suitable for processing defective or unexpected measurements of at least one sensor that is part of a virtual sensor. This is obtained as described in the appended claims.

[0006] Thus, the present invention is able to detect and correct erroneous operation of hardware sensors in a device and monitor not only the usage of the device but also, in some cases, the surrounding environment of the device. Since the virtual sensor is a software sensor, detecting defects in the sensor inputs to the sensor fusion unit can be done in software by verifying all sensor inputs based on the principles described above. In the prior art, various mechanisms based on machine learning or specific neural networks can be used to detect defective sensors, for example in autonomous driving systems, as described, for example, in "Multi-sensor Fault Detection, Identification, Isolation and Health forecasting for Autonomous Vehicles" by Saeid Safavi et al., Sensors 2021, 21, 2547, https: / / doi.org / 10.3390 / s21072547, but in this case the impact of a defective sensor can be severe.

[0007] Virtual sensors can use software or hardware-based neural networks to identify faulty sensors, improve stability, or use other machine learning techniques. Machine learning can be performed in a number of ways, for example based on storing and analyzing typical usage, possibly with feedback from the user through a suitable user interface. Such examples are described in Norwegian Patent Application No. 20210589. Machine learning can thus detect deviations from typical usage, for example based on correlations between different sensor inputs during use, and report the deviations, either letting the user investigate or tailoring the analysis to inform the device that the detected deviations are considered within the range of expected typical usage.

[0008] The invention is based on an electronic device in which a virtual sensor can detect the usage of the device based on different hardware sensors, providing smart sensors that can be used to control, for example, the power output and energy consumption of the device and the information presented to the user, for example in a device where the information displayed on the device depends on the usage as described in Norwegian Patent No. 345005, or on the proximity of the user using ultrasonic signals in combination with an inertial measurement unit as described in Norwegian Patent No. 345509. [Brief description of the drawings]

[0009] The invention will now be described in more detail with reference to the accompanying drawings, which show, by way of example, the invention. [Figure 1] FIG. 1 illustrates an embodiment of the present invention incorporated into a mobile phone. [Diagram 2] FIG. 1 illustrates a virtual sensor system including multiple sensors. [Diagram 3] FIG. 1 illustrates a preferred embodiment of the process according to the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Fig. 1 shows an example of a device 3, such as a mobile phone, including the present invention. Here, the device includes a sound generator 6 connected via a speaker interface 5 to an acoustic output device, such as a speaker 2, an ultrasonic transducer, or a piezoelectric element. The speaker 2 may be connected to a processing module 9, which is also connected via an acoustic interface 4 to other sensors, such as one or more microphones 1 or ultrasonic transducers, and additional sensors 8, such as inertial (IMU) sensors, radar, UWB sensors, pressure sensors, light sensors, touch controllers, etc. The processing module 9 is also included in a virtual sensor unit 11, which utilizes signals from the different sensors to provide a virtual sensor.

[0011] The device may use acoustic signals via a speaker and microphone for communication purposes as a mobile phone, but may also be configured as a virtual sensor to detect the presence of a user 10 .

[0012] FIG. 2 shows a virtual sensor unit 11 in which a first virtual sensor 11a is connected to a number of hardware sensors 1, 2, 8, 12a-12c. Also, a second virtual sensor 11b is connected to additional hardware sensors 12d, 12e. Here, the second virtual sensor may report to the first virtual sensor as an independent sensor. As an example, two ultrasonic sensors may be used and the second virtual sensor may report the direction of the received ultrasonic waves.

[0013] As mentioned above, the virtual sensors 11a, 11b may be vulnerable to errors in the hardware sensors 12a-12e, which may result in device defects. If the virtual sensors do not detect the faulty sensor inputs, the sensor output from the virtual sensors may be incorrect unless the virtual sensors are designed to handle the faulty sensor inputs. The stability of the virtual sensors is also important if the output of the virtual sensors is critical and may cause physical harm.

[0014] More specifically, the invention may be provided with a normal model of operation that is pre-programmed, stored during normal operation, or stored in a learning sequence or period. The device may be configured with an interface that allows it to respond, for example, to verify that a registered deviation is within normal usage conditions or to verify whether it represents an error in operation. The learning sequence or pre-programmed data may include a simulation of turning off the transducer to register how the virtual sensor would react to an error and to allow the reaction to be recognized.

[0015] 3 shows a method according to the invention for evaluating signals from sensors 12a, 12b etc. The signals from sensors 12a, 12b are sent to a processor for comparing measurements 21 with a predefined model 22 representing the typical behavior of the sensors. For example, an inertial sensor records the movement of a mobile phone towards the user's ear and a proximity sensor records the approach of the ear to the mobile phone.

[0016] The device is then configured to register 23 whether the measurements comply with the model. If the measurements are within normal operating ranges, the system goes back to waiting for the next measurements 21, and the measurements may be fed back into the model 22 to update and improve the model based on the new measurements.

[0017] Any deviations from the model 22 are registered 23 and a fault or error is indicated. Deviations from the measurements are compared to additional models 25, 26 representing different types of errors, in order to detect the type of error, including a clogged microphone etc. This allows for improved diagnosis to be presented to the user and / or manufacturer 26.

[0018] In this way, the invention can detect, for example, whether an ultrasonic transmitter is defective because the received signal, based on the stored model, is different from the signal received under conditions corresponding to the current situation by other transducers such as an acoustic receiver, an IMU, an optical sensor, etc. For example, if the transmitter becomes defective during proximity detection, the acoustic receiver will not detect the presence of a user. However, if the IMU detects the device moving, it is highly likely that the user is present. This may indicate that the transmitter and / or both receivers are defective. Similarly, if the device knows that the speaker has transmitted an acoustic signal, but only one of the two microphones receives a reflected signal, this may indicate that the other microphone is defective.

[0019] Acoustic virtual sensors can typically detect when the amplitude, pulse rate, frequency content, signal length, etc. of an acoustic signal are not as expected. This can be more difficult with other sensors, such as IMU sensors, touch panel sensors, and SAR sensors. Fortunately, some sensors include internal error detection mechanisms to identify problems with the sensor operation and can report a corresponding error (e.g., open failure) to the virtual sensor. The virtual sensor may repeatedly attempt to open the sensor to see if the defect is intermittent. Other sensors can report specific error values ​​(e.g., non-numeric values) that the virtual sensor can identify. If a sensor reports one or more values ​​(e.g., acceleration sensor) that are outside of a certain range of values, the virtual sensor can detect that. If a sensor reports one or more values ​​(e.g., expected acoustic amplitude, expected acoustic probe signal, etc.) that are outside of an expected range of values, the virtual sensor can potentially detect that. If a sensor reports multiple values ​​(e.g., acceleration sensor, gyroscope) that are somehow correlated, and at least one of them is defective, the virtual sensor can potentially detect that. The virtual sensor can detect when two or more sensors provide partially overlapping information (e.g., accelerometer and gyroscope, accelerometer, gyroscope, acoustic signal, etc.) and at least one of the values ​​of any sensor is inconsistent with the other values. When two or more sensors provide information that should be correlated, the virtual sensor can detect situations where the information is not correlated.

[0020] If the defective sensor is critical to the performance of the virtual sensor, the virtual sensor may report its own error to the system. The importance of a particular sensor may vary, and in some cases the virtual sensor may continue to provide useful sensor output to the system. If performance with the remaining set of sensors is acceptable, the virtual sensor may need to disable the sensor to reduce power consumption. If a defective sensor does not detect a problem with its own output data, it is possible that the sensor data for that sensor is defective and offsetting the virtual sensor output. Even if the offset in the virtual sensor output is not easily detectable by the user, it may degrade performance with the virtual sensor, and in some cases, degrade the overall performance of the virtual sensor and all functions that rely on it.

[0021] ML training can be used to establish rules for what are acceptable sensor values ​​for a particular sensor. This information can be extracted from the ML model and serves as a partially trusted source of information for when values ​​from a particular sensor appear to be defective. If at least two sensors (e.g., an accelerometer and a gyroscope) have conflicting information, the system must either discard both sensors or decide which sensor to trust. This can be done by the user through a calibration process provided by the device vendor. In such cases, the virtual sensor must either immediately switch to using a different neural network based on the sensor that appears to be working properly for its own detection process, or modify the target sensor inputs so that the neural network effectively ignores the defective sensor. Ignoring a sensor can be done in the ML training process by artificially training the neural network with input data from at least one defective sensor.

[0022] Training of the machine learning or neural network may be performed by recognizing correlated measurements during normal operation, e.g., in situations where the user has not indicated via an interface that a problem has occurred. By continuously sampling and evaluating measurements, a model of typical usage patterns may be built. As new measurements are sampled, they may be implemented into the model by the machine learning algorithm if they are within the range defined by the existing model, or may trigger an error indication if the new measurement deviates significantly from the model. If an error is detected, the measurements may be compared to different error-based models to detect the cause of the error, e.g., a defective sensor.

[0023] In some cases, a set of sensors are part of the same hardware package or are used in a set of different virtual sensors, so if one of them is defective, the other sensors in this package may also be suspect.

[0024] The virtual sensor should automatically periodically try to use a potentially faulty sensor. At some point, the virtual sensor may give up on using the sensor and not enable it to reduce power consumption. In that case, the virtual sensor should either switch to a separate neural network or insert a tuned ML feature into the neural network to indicate the absence of a valid input.

[0025] Virtual sensors based on acoustic data processing, such as those described in the literature cited above, often utilize sensor fusion to improve overall performance. Such virtual sensors may experience issues with the audio component that may affect the virtual sensor (e.g., speaker issues, damaged microphones, clogged microphones, etc.). The core use case for a virtual sensor typically determines the sensors that are used as inputs to the sensor fusion portion of the virtual sensor. In most cases, the sensor inputs are time-series data with frequencies above 1 Hz, but some sensors are on-change sensors where the sensor changes between different states, or where the frequency of the sensor does not make much sense. The most common sensors that use sensor fusion as part of an acoustic virtual sensor include touch sensors, SAR sensors, ambient light sensors, hall sensors, hinge sensors, accelerometer gyroscopes, 6DoF sensors, 3DoF sensors, rotational sensors, and translational sensors.

[0026] As mentioned above, the problem with sensor fusion in these acoustic sensors is that the loss of one of the sensor inputs, including acoustic data due to at least one faulty sensor, acoustic receiver and / or acoustic transmitter, can adversely affect the performance of the virtual sensor. This is especially true when the virtual sensor actively uses machine learning and neural networks, where the sensor information is either used directly as machine learning features or after some pre-processing to extract key features of the input. The virtual sensor may use one or more neural networks and a processing chain with additional processing steps before and after each neural network. The neural networks may be run in parallel or serially depending on the sensor inputs and the particular use case.

[0027] To properly handle faulty sensor inputs, the virtual sensor must include a mechanism to detect if any of the sensor inputs are faulty or suspected to be faulty. Detecting faulty sensors in the detection phase, using either software processes to validate the sensor inputs as described above, or more advanced schemes based on machine learning or neural networks, is one viable option. A suitable machine learning network can be used to identify faulty sensors by looking for inconsistent combinations of sensor inputs. One option is for the virtual sensor to include pre-processing based on a separate neural network or software processes to detect faulty sensor inputs before feeding the sensor inputs of interest to the associated set of neural networks used in the virtual sensor.

[0028] Although it is possible to perform the detection phase as a pre-processing before starting the virtual sensor processing, it may not fit into the platform's sensor framework. After all, it is a requirement that all other inputs used by the virtual sensor are available to the detection process. For practical purposes, it is less complicated to include the detection phase as a separate pre-processing step as part of the virtual sensor processing. Integrating a similar functionality into the core part of the neural network used by the virtual sensor may be more complicated without providing any obvious advantages.

[0029] If an input is tagged as defective, it needs to be handled accordingly in the virtual sensor. There are several options for handling defective sensors. The defective sensor input of interest can be modified to provide a neutral input value that the neural network can recognize, and for practical purposes, can be treated as unimportant for the determination of the sensor output. If the virtual sensor and potentially its neural network are stable enough to provide a correct sensor output even when one of the sensor inputs is defective, it may be possible to ignore the defective sensor input. Another option is to exclude the input from the neural network set. Changing to a different neural network set that does not use the sensor input of interest is also an option for handling defective sensors. Setting up separate neural network sets for different combinations of valid sensor inputs is a surefire way to solve this problem. Ideally, the defective sensor inputs can be handled during the neural network training process. One can either create a large model where any number of faulty sensor inputs are automatically handled by the neural network by predicting the faulty sensor inputs and taking them into account during the ML training process, or create many smaller models (i.e., one model per faulty sensor or combination of faulty sensors) where the faulty sensors are excluded from the neural network processing. It is also possible to inject faulty sensor data during ML training so that the neural network can take the faulty sensors into account. Since the use of faulty sensor data in ML training and the subsequent neural network is difficult to predict, another viable option for virtual sensors is to use a small set of models where the faulty sensors are not included in the neural network processing.

[0030] Faulty sensors can be easily simulated during ML learning by simply excluding the sensor data of the target sensor from the training process. The number of different models that need to be created to address all sensor problems depends on the number of sensor inputs used in the sensor fusion part of the virtual sensor.

[0031] If a virtual sensor detects that at least one of its core sensors is defective but is still able to function beyond well-defined requirements, it preferably reports an error indicating that the sensor is defective. If a virtual sensor cannot provide a usable sensor output when the target sensor is defective, it should report an error to the system itself when a user tries to open it. This report should indicate which sensor is defective. Based on the error report, the system can initiate either a replacement process for the component or electronic device, or well-documented instructions for a repair process that may be performed by the user or the device's supplier or vendor. For example, a clogged microphone hole can be cleaned, possibly using a special tool or a suction device (e.g., a vacuum cleaner).

[0032] In summary, the present invention relates to an electronic device and a corresponding method, where the electronic device comprises at least two sensors in communication, e.g. via wireless or electronic connections or as a software module, with a virtual sensor comprising a processing unit adapted to monitor the usage of the device based on inputs from the sensors.

[0033] The at least two sensors include an activity sensor, such as a proximity sensor and / or an inertial (IMU) sensor, that senses user activity with respect to the device. The processing unit is configured to receive and analyze input from the sensors during use and provide and store a model representative of typical use of the device. Additionally, the device is further configured to compare new measurements received from the sensors with the model and provide a signal if the new measurements from the sensors deviate by a predefined limit from measurements predicted by the model. The model may be based on statistical information from previous usage, factory settings, and / or redefined based on user interactions and activities.

[0034] The model may include a typical situation in which at least one of the sensors is defective, and the processor of the device may be configured to recognize this situation and to ignore the signal from the defective sensor.

[0035] The two sensors may include a proximity sensor that detects whether a user is near the device. Typically, the proximity sensor includes at least one acoustic transmitter and at least one acoustic receiver configured to detect the presence of a user, including user activity such as a gesture, in proximity to the device, preferably defined within a distance threshold. Here, the processing unit is configured to detect defects in at least one of the transducer and receiver if at least some of the measurements deviate too much from the model.

[0036] Other types of sensors may be considered, such as inertial (IMU) sensors, cameras, radar, UWB sensors, pressure sensors, light sensors, and touch controllers.

[0037] Preferably, the device includes a user interface configured to communicate the detected defects to a user and / or indicate whether the signals from the virtual sensors are within normal usage or device performance. The processing unit may be configured to update the model based on instructions from the user.

[0038] The model is generated based on machine learning algorithms that detect correlations between previous measurements from the sensors and is dynamically updated with new measurements if the measurements are within limits determined by the model. In some cases, the limits may be the same as the limits that define the deviation or may be set separately. A defect may be reported if a new measurement from at least one of the sensors falls outside the determined limits.

[0039] The present invention also relates to a method for assessing the performance of an electronic device, where the device includes at least two sensors providing measurements at a predetermined rate, such as a proximity sensor and an inertial measurement sensor, that detect a user's presence, including activity, such as a gesture, associated with the device, and the sensors are connected to or in communication with a virtual sensor that includes a processing unit adapted to monitor usage of the device based on input from the sensors. The method includes: receiving measurements from a sensor; Comparing the measurements with a predefined model representative of a typical use of the device; providing a signal if a new measurement from the sensor deviates by a predefined limit from the measurement predicted by the model; Includes.

[0040] The method may also include the step of the processor ignoring a signal from a faulty sensor if the measurements of one sensor deviate from measurements predicted by the model in the corresponding situation.

[0041] The method may also include comparing the measurements to a set of predefined models each indicative of a selected sensor defect if the measurements deviate from an expected measurement.

[0042] The model may be generated based on machine learning algorithms that detect correlations between previous measurements from the sensors and may be dynamically updated with new measurements if the measurements are within limits determined by the model. A fault may be reported if a new measurement from at least one of the sensors falls outside the determined limits. A user or further analysis may also consider whether the new measurement should be defined as being within the model of normal usage of the device.

Claims

1. an electronic device including at least two sensors in communication with a virtual sensor including a processing unit adapted to monitor usage of the electronic device based on inputs from the sensors; at least two of the sensors include an activity sensor for sensing user activity with respect to the electronic device, the processing unit being configured to receive and analyze input from the sensors during use and to provide and store a model representative of typical usage of the electronic device; and further configured to compare new measurements received from the sensor with the model and provide a signal if the new measurements from the sensor deviate from measurements predicted by the model by a predetermined limit. Electronic devices.

2. The electronic device of claim 1 , wherein the model includes a situation in which at least one of the sensors is defective, and the electronic device is configured to ignore signals from the defective sensor.

3. 2. The electronic device of claim 1, wherein the two sensors include a proximity sensor that detects whether a user is near the electronic device, the proximity sensor including at least one acoustic transmitter and at least one acoustic receiver configured to detect the presence of a user near the electronic device, and the processing unit is configured to detect defects in at least one of the transducer and the receiver.

4. The electronic device of claim 1 , wherein the electronic device includes at least one sensor of an inertial (IMU) sensor, a camera, a radar, a UWB sensor, a pressure sensor, a light sensor, and a touch controller.

5. 2. The electronic device of claim 1, wherein the electronic device includes a user interface configured to communicate detected defects to a user or indicate whether a signal from the virtual sensor is within a range of normal usage or performance of the electronic device, and the processing unit configured to update the model based on instructions from the user.

6. 10. The electronic device of claim 1, wherein the model is generated based on a machine learning algorithm that detects correlations between previous measurements from the sensor, and the model is dynamically updated with new measurements if the measurements are within limits determined by the model.

7. The electronic device of claim 6 , wherein a fault is reported if a new measurement from at least one of the sensors falls outside the determined limits.

8. A method for assessing a performance of an electronic device, the electronic device comprising at least two sensors providing measurements at a predetermined rate, the at least two sensors detecting a user associated with the electronic device, such as a proximity sensor and an inertial measurement sensor, the sensors being connected to a virtual sensor comprising a processing unit adapted to monitor usage of the electronic device based on inputs from the sensors; receiving measurements from said sensors; - comparing said measurements with a predefined model representative of typical usage of said electronic device; providing a signal if the new measurement from the sensor deviates by a predetermined limit from the measurement predicted by the model; A method comprising:

9. 9. The method of claim 8, comprising ignoring a signal from a faulty sensor if the measurement of that sensor deviates from the measurement predicted by the model in a corresponding situation.

10. 9. The method of claim 8, further comprising the step of comparing the measurements to a set of predefined models each indicative of a selected sensor defect if the measurements deviate from the predicted measurements.

11. the electronic device includes a user interface configured to receive an indication of a defect from the user; updating the model upon receipt of a defect report; The method according to claim 8.

12. 9. The method of claim 8, wherein the model is generated based on a machine learning algorithm that detects correlations between previous measurements from the sensor, and the model is dynamically updated with new measurements if the measurements are within limits determined by the model.

13. The method of claim 12 , wherein a fault is reported if the new measurement from at least one of the sensors falls outside the determined limits.

Images