Method and device for obtaining user state data

Abstract

Method and device for obtaining user state data A computer-implemented method for obtaining user state data and underlying signal data in a wearable device, the wearable device including a main processor unit and a low-power processor unit, the method including: accessing an indicator of a device sub-state; if the device sub-state is an unknown device sub-state, turning on the main processor unit and proceeding, and acquiring and processing a first plurality of sensor signals in the main processor unit to obtain the user state data; if the device sub-state is a known device sub-state, turning off the main processor unit and proceeding, and acquiring and processing a second sensor signal used for the known device sub-state in the low-power processor unit to obtain the user state data.

Description

[Technical Field]

[0001] FIELD OF THE INVENTION The present invention relates to acquiring user state data and underlying signal data, and in particular to a computer-implemented method of acquiring user state data in a wearable device and a wearable device for acquiring user state data. [Background technology]

[0002] Background of the Invention Modern wearable technology can capture data about a user's mood, movements, and well-being. Such technology allows users to track and understand their own actions and habits. Wearable devices that track physical activity can also help users understand how much exercise they get on a daily basis and identify patterns in their activity levels. This is often useful for setting and achieving fitness goals, as well as identifying potential health concerns that may need to be addressed. Wearable technology can also be used to collect data for research purposes. Studies might use wearable devices to track the movements and well-being of a group of participants over a period of time to better understand the relationship between physical activity and mental health. This type of research can lead to a deeper understanding of how certain behaviors and habits affect our overall well-being, which in turn can inform the development of new interventions and treatments.

[0003] Wearable technology can also be used to support personal healthcare. Wearable devices that track a user's sleep patterns and heart rate can be used to alert healthcare providers to any potential problems or changes in the patient's health. This type of real-time monitoring can allow healthcare providers to intervene earlier, potentially preventing more serious problems from developing.

[0004] A user's emotional state or mood may be inferred from, for example, biometric user data and may be significantly influenced by other seemingly unrelated bodily processes and activities. However, it is only through monitoring the user over an extended, continuous period of time to collect a comprehensive set of diverse yet consistent signals that reliable correlations may be identified that may be used to objectively measure user data.

[0005] To find reliable correlations, the collected data must remain unbiased by conscious influences; that is, the collection instrument must be small, lightweight, and unobtrusive enough that subjects quickly become unaware of its presence—effectively a "fit-and-forget" approach. Successfully collecting such datasets presents potential ethical issues related to the unintentional collection of information that may be considered private and / or embarrassing by the subjects of the measurements.

[0006] Taken together, the above factors present significant technical challenges that cannot be addressed by existing technologies. Today, clinical-grade biometric and kinematic data can be collected over long periods of time, but only through the use of relatively large, fixed equipment (e.g., in hospitals, laboratories, clinics, or gyms). Conversely, wearable devices or apparatuses (e.g., watches, bands, patches) can collect non-clinical but continuous biometric and kinematic data outside of fixed locations. However, such wearable devices are hampered by the inevitable rapid depletion of available on-device power and data storage capacity. In either type of scenario, the measurement instrument must not be considered unobtrusive enough to be forgotten. Therefore, existing technologies do not consider providing additional privacy measures. The inventors have therefore come to realise that there is a need for techniques and devices for non-invasively acquiring user state (eg, biometric and kinematic) data. Summary of the Invention

[0007] SUMMARY OF THE INVENTION The invention is defined in the independent claims, the content of which is set out below. Further features are set out in the dependent claims.

[0008] According to one aspect of the present invention, a computer-implemented method for acquiring user state data in a wearable device is provided. The wearable device or apparatus includes a main (high-power) processor unit and a low-power processor unit. The processor units, also referred to as controllers or nodes, may be on separate chips or may be part of a multi-core chip. Regardless of the configuration, the processor units may be divided into a main processor unit for high-power processing and a low-power processor unit for low-power processing.

[0009] The method for obtaining user state data includes accessing an indicator of a device sub-state. Optionally, a low-power processor unit accesses the indicator. Optionally, the main processor unit generates the indicator (if the main processor unit is able to successfully classify the device sub-state). The device sub-state includes a predefined operating mode that sets operating parameters of the wearable device. Illustratively, the device sub-state includes an indication of which sensors are active, the device's memory capabilities, and the device's battery capabilities.

[0010] If the device sub-state is an unknown (and / or undefined) device sub-state, the method includes turning on the main processor unit and proceeding. The method then includes (in the main processor unit) acquiring and processing a first plurality of sensor signals to acquire user state data (i.e., data indicative of the user's state). The first plurality of sensor signals are acquired from a first plurality of sensors. The first plurality of sensor signals are used for any unknown device sub-state. The first plurality of sensors may correspond to all sensors available to the wearable device, or any subgroup thereof.

[0011] If the device sub-state is a known (and / or defined) device sub-state, the method includes turning off the high-power processor unit and proceeding. The method then includes (in the low-power processor unit) acquiring and processing a second sensor signal, which the known device sub-state is used to acquire user-state data. That is, the known device sub-state indicates a second sensor signal to be acquired, where the second sensor signal partially forms the user-state data. The second sensor signal is acquired from a second plurality of sensors (or alternatively, a single sensor). In some cases, the second sensor signal may be in the form of a second plurality of sensor signals. In other cases (by way of illustration, while the user is in deep sleep), the second sensor signal may be a single sensor signal.

[0012] Optionally, the second sensor signal includes fewer sensor signals than the first plurality of sensor signals. Alternatively, if it is inherently "noisy," the second sensor signal may include the same number of sensor signals as the first plurality of sensor signals. Of course, where a single sensor is configured to output multiple sensor signals (e.g., a three-axis accelerometer configured to output three signals, one for each orthogonal axis), the absolute number of sensors used in the first plurality of sensor signals may be the same as the absolute number of sensors used in the second sensor signal, with only the number of sensor signals differing (e.g., a three-axis accelerometer may be configured to output only the z-component of acceleration). Of course, sensors in the second plurality of sensors may be present in the first plurality of sensors, and vice versa.

[0013] Separating functionality between the main processor unit and the low-power processor unit based on the device sub-state ensures that a wearable device performing the method provides continuous monitoring of a user without unnecessary data acquisition. That is, if the device sub-state includes a device profile associated with a user in a particular state for which it is desirable to acquire specific biometric and kinematic data, the method may direct activation of only the required sensors (second plurality of sensors). Conversely, if the device sub-state is unknown, it may be desirable to capture more data, for example, all potentially relevant data (even if only a subset of this data is later determined to be relevant to the final classified device sub-state). Thus, the method may involve activation of all available sensors (first plurality of sensors).

[0014] By having the main processor unit turn on and proceed only as needed (when the device sub-state is unknown), unnecessary consumption of battery power is avoided. That is, the main, high-power processor unit may perform power-intensive processing, e.g., to perform classification of acquired data, and facilitate power-intensive data capture from more or all available sensors. When such power-intensive processing and data capture is not required, it has the ability to hand off to a low-power processor unit, which does not need to perform the same power-intensive processing and only controls data capture from a subset of relevant sensors (second plurality of sensors).

[0015] By reducing unnecessary drain on battery power, the method may enable a significant reduction in the form factor of wearable devices, which in turn may enable a "wear it and forget it" approach to data collection, which is particularly advantageous for continuously acquiring clinical-grade biometric and kinematic data that was previously only obtainable using cumbersome hardware. Moreover, by associating known device sub-states with specific subsets of sensor signals, the method provides a measure for preserving user privacy. Illustratively, if the device sub-state indicates that user location data will not be collected (e.g., if the user is predicted to be near their home or workplace), the method may withhold data collection. Similarly, a skilled reader may envision private user activities (as indicated by the device sub-state) for which withholding data collection is desirable.

[0016] Optionally, the method may also include having the main processor unit process the first plurality of sensor signals using at least one pre-trained machine learning (ML) model and output a user state classification and / or user state result. In this manner, the method may also direct the high-powered main processor unit to execute ML models, which may provide previously unknown state classifications. The user state result may illustratively be summarized as a comprehensive classification of user characteristics that takes into account user activity, user physiological characteristics, environmental characteristics, and device characteristics.

[0017] Optionally, the method may cause the main processor unit to process the first plurality of sensor signals by using an initial pre-trained ML model and then subsequent pre-trained ML models. In this manner, the main processor unit may pass the first plurality of sensor signals through the initial trained machine learning model to output a coarse classification of the user state according to a predetermined substate. The main processor unit may then pass the coarse classification of the user state and at least some of the first plurality of sensor signals (and / or data derived therefrom) through subsequent trained machine learning models to output a result of the user state. The coarse classification may be sufficient to represent the current state, and thus subsequent classification may not be necessary, thus avoiding the wearable from wasting unnecessary resources in performing multiple classifications. Illustratively, if a confidence threshold for the classification exceeds a predetermined value, subsequent classification may not be necessary.

[0018] This process may involve storing the first plurality of sensor signals in a buffer for access using the initial trained model and / or subsequent trained models. Following the coarse classification, the buffer may be cleared to avoid unnecessary storage consumption.

[0019] Optionally, the method may also have the main processor unit determine the reliability of the user state, the outcome of the user state, or both. If the confidence in this classification is below a predetermined threshold, the main processor unit may pass the first plurality of sensor signals to the memory of the wearable device for further investigation (at a later time). The main processor unit may then continue to acquire the first plurality of sensor signals. That is, if classification of the user state or outcome of the state is not feasible, the method may also have the wearable device store data from the first plurality of sensors and use it for later investigation. The subsequent investigation may be performed on the device if the device has additional data that increases the confidence in the classification, or may otherwise be performed by an external processing device.

[0020] Optionally, the method may also have the main processor unit determine the reliability of the user state, the outcome of the user state, or both. If the confidence of this classification is below a predetermined threshold, the main processor unit may register or record the user state and / or the outcome of the user state as a new, unknown user state and / or the outcome of a new, unknown state. This approach provides resilience to unknown device substates (and their resulting classifications). Illustratively, if one or more states are unknown or have low classification confidence, the system may still have sufficient confidence to classify the outcome of the state based on other known states. If too many of the underlying user states are unknown or the classification of one underlying user state has too low confidence, the resulting substate may be tagged as unknown.

[0021] In this way, the wearable device may avoid the need to activate energy-intensive classification routines when it encounters the same (or similar) features in the data registered as a new state. This approach to state classification and processing of unknown states may also significantly reduce power consumption and optimize storage capacity usage by the wearable device, both of which are important operating parameters for long-duration wearable electronic measurement devices.

[0022] Optionally, the method may also cause the main processor unit to update adaptive parameters of any trained machine learning model following the user state classification and / or output of the user state results, thus improving the accuracy of the user state classification during the next classification iteration.

[0023] Optionally, the method may also cause the low-power processor unit to detect whether there is an exception when processing the second sensor signal (from the second plurality of sensors). An exception, in this context, is any deviation from an expected value for the device's current state, thereby potentially indicating a change in state. Following detection of the exception, the method may also cause the low-power processor unit to initialize the main processor unit. That is, the main processor unit is turned on (from an off or standby state). The method may then cause the low-power processor unit to pass the second sensor signal for processing in the main processor unit (e.g., pass the signal to a buffer for access by the main processor unit). The main processor unit may then process the second sensor signal. In this manner, the main processor unit's power-hungry processing capabilities may be used to classify unclassified data. Optionally, the main processor unit may supplement the second sensor signal with additional data obtained from the first plurality of sensors (some or all of the first plurality of sensor data).

[0024] Optionally, the method may also have the low-power processor unit detect an exception by first passing at least a portion of the second sensor signal (and / or data derived therefrom) through an anomaly detection algorithm. This enables the low-power processor unit to detect deviations from predetermined or pre-stored expectations for the current (known) device sub-state. The low-power processor unit then detects an exception when the confidence level of the exception detection algorithm's output exceeds a predefined threshold. That is, the low-power processor unit detects an exception when the confidence level of the exception detection algorithm is higher than the threshold, meaning that the exception detection algorithm is confident that it has detected an exception compared to what the algorithm expected for that given state. The operating parameters and thresholds for exception detection may be different for each state, since in each state, a subset of parameters is considered more important than others.

[0025] Optionally, the method may also cause the low power processor unit to update adaptive parameters of the exception detection algorithm following detection of the exception, thus improving the accuracy of exception detection during subsequent iterations of data acquisition.

[0026] Optionally, the method may also cause the wearable device to discard at least a portion of the user state data following acquisition of the user state data. The discarded portion may be a portion of the user state data that is deemed unrelated to known device sub-states. The relevance of the user state data to known device sub-states may be pre-configured for each device sub-state. As an example, it may be desirable to know when a user is in a particular state (e.g., sleeping), which may be indicated by a lack of movement from the sensor signal; therefore, movement data needs to be collected, but the same data does not need to be retained (for classification purposes).

[0027] Optionally, the method may turn on the main processor unit when the device sub-state indicator is accessed. The main processor unit may then remain on if the device sub-state is unknown. Similarly, the main processor unit may be turned off if the device sub-state is known. The method thus ensures that the main processor unit is always on at the start of each cycle of user state data acquisition.

[0028] Optionally, the method may cause the wearable device to obtain the sensor signal (i.e., the first plurality of sensor signals and / or the second sensor signal), at least in part, from any or all of the following: Biosensors for detecting the user's biosignals (ECG sensors, EEG, EMG, EOG, fNIRS, photoplethysmography sensors, skin potential sensors, temperature, chemical composition, bioimpedance, skin moisture, spectroscopic sensors, image sensors, etc.); environmental sensors for sensing local environmental properties (such as ambient temperature, air pressure, acoustics, gas composition and optical properties, humidity, EMF radiation, etc.); Communications (such as Bluetooth receivers, WiFi, LORA, Zigbee, NFC, etc.), and Kinematic sensors (such as accelerometers, magnetometers, gyroscopes, motion sensors, and / or tilt switches).

[0029] Optionally, the user state data acquired by the wearable device may be the result of the user state. The result of the user state may be determined according to any or all of the following sub-states: User activity substate, the user's biological (or physiological) sub-state, environmental substates, and Device substate.

[0030] Optionally, the user state result may include an emotional profile of the user. The emotional profile may be a classification of the user's current emotional state (i.e., the user's emotional state at the time of classification). The emotional profile may be included in the user state result as a tabular collection of marked and scored emotional descriptors or in any other suitable format. As an example, the emotional profile may be provided as scores for pre-defined emotional descriptors (such as "excited," "angry," "stressed," "sad," "bored," "tired," "calm," "happy," etc.). Optionally, the user's emotional profile may be provided as output from at least one pre-trained ML model (such as a pre-trained ML model used to classify the user state and / or the user state result). The use of the emotional profile allows for simultaneous classification of the user's mood during an activity.

[0031] Optionally, the method may also cause the wearable device to transmit the user state data (and / or any data derived therefrom) to an external processing device. The wearable device may transmit the data using a communications subsystem. The external processing device may then further process the data as needed. This is advantageous in that the wearable device may not be able to accurately classify the user state, and the increased processing power provided by an exemplary external processing device (e.g., a PC) may provide more focused classification capabilities. Furthermore, if the wearable device is unable to classify the user state because it has not encountered this state before (and thus the classification will necessarily be inaccurate), transmission to the external processing device may enable model generation (e.g., training a new ML model) that takes this new user state into account, and then installing the new model on the wearable for subsequent classification of this state. The specific aspects of the transmitted data will be determined based on the end use of the wearable device and may include high-level parameters and state results, such as user state and classification, or lower-level data, such as raw data from sensors (particularly useful when on-device classification is not possible).

[0032] Optionally, the method may also have the wearable device obtain signals (i.e., the first plurality of sensor signals and / or the second sensor signal) at least in part from an external device. In this manner, classification is not limited with respect to sensors on the device. Rather, the functionality of the wearable device may be expanded to consider additional sensors. By way of illustration, if a user is at a gym, Bluetooth-enabled gym equipment (beacons) may support classification of gym-related user states (e.g., indoor swimming).

[0033] An embodiment of another aspect of the present invention includes a wearable device having a main processor unit, a low-power processor unit, and a memory, wherein the main processor unit and the low-power processor unit may be configured to perform steps of a method for obtaining user state data, as variously described above.

[0034] Another aspect of the present invention includes a system including a wearable device and an external device. The wearable device includes a main processor unit, a low-power processor unit, and a memory; the main processor unit and the low-power processor unit may be configured to perform steps of a method for acquiring user state data, as variously described above. The external device includes transmitting means configured to transmit a sensor signal to the wearable device. The external device may include receiving means configured to receive a transmission from the wearable device. Illustratively, the receiving means may be configured to receive a set value and / or a start signal.

[0035] An aspect of another aspect of the present invention includes a computer program configured, when executed by a wearable device, to cause the wearable device to perform steps of a method for obtaining user state data.

[0036] Another aspect of the present invention includes a computer-readable medium storing a computer program configured, when executed by a wearable device, to cause the wearable device to perform the steps of a method for acquiring user state data. The computer-readable medium may be non-transitory.

[0037] The invention may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or combinations of them. The invention may also be implemented as a computer program or computer program product, i.e., a computer program tangibly embodied in a non-transitory information carrier (e.g., in a machine-readable storage device or a propagated signal) for execution by or controlling the operation of one or more hardware modules. The computer program may be in the form of a stand-alone program, a part of a computer program, or one or more computer programs, and may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, such as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a data processing environment.

[0038] The firmware and / or software of the wearable device may be programmed using a variety of standard methods, such as wired connections, wireless and wireless network connections, etc. Specific test settings and / or usage parameters may be programmed via a secondary device (such as a PC, tablet, via a web interface, etc.) connected to the wearable device via these methods. The present invention has been described with reference to specific embodiments. Other embodiments are within the scope of the following claims. For example, the steps of the invention can be performed in a different order and still achieve desirable results.

[0039] In summary, a method implemented within a wearable device containing an integrated sensor array (e.g., biological, kinematic, and environmental) would perform classification (e.g., using embedded artificial intelligence (AI)) in a manner that efficiently manages power consumption and data storage (and in turn improves privacy measures). This would enable reliable detection of, and changes in, the wearer's emotional state, mood, health, and well-being. [Brief explanation of the drawings]

[0040] BRIEF DESCRIPTION OF THE DRAWINGS Reference is made, by way of example only, to the accompanying drawings in which:

[0041] [Figure 1] FIG. 1 is a flowchart of a general method for acquiring user state data in a wearable device. [Figure 2] FIG. 2 is a flowchart of a detailed method for acquiring user data in a wearable device. [Figure 3] Figure 3 is an example time series of primary raw sensor data obtained from a 3-axis accelerometer. [Figure 4] FIG. 4 is a flow chart of an example two-stage classification process. [Figure 5] FIG. 5 is a hierarchical representation of the results of a user state and the underlying substates. [Figure 6] FIG. 6 is a first example of a user's emotional profile, which represents the results of the user's state. [Figure 7] FIG. 7 is a second example of a user's emotional profile, showing the results of the user state. [Figure 8] FIG. 8 is a flow chart of an example exception detection process. [Figure 9] FIG. 9 is an example time series of activity sub-state classification and environmental sub-state classification performed by a wearable device. [Figure 10] FIG. 10 is a first example scenario of a power consumption comparison between wearable devices with and without adaptive thresholds and unknown state registration. [Figure 11] FIG. 11 is a second example scenario of a power consumption comparison between wearable devices with and without adaptive thresholds and unknown state registration. [Figure 12] FIG. 12 is a schematic diagram of the underlying electronic blocks of the wearable device. [Figure 13]FIG. 13 is a schematic diagram of a communication subsystem unit, i.e., a Bluetooth Low Energy (BLE) module. [Figure 14a] FIG. 14a is a depiction of the upper portion of an example wearable device. [Figure 14b] FIG. 14b is a depiction of the lower portion of an example wearable device. [Figure 15] FIG. 15 is a set of line drawings of the appearance of an example wearable device. [Figure 16] FIG. 16 is a schematic diagram of a computing device in which hardware aspects of the present invention may be embodied. DETAILED DESCRIPTION OF THE INVENTION

[0042] Detailed Description The methods and devices described herein support the collection and processing of multiple clinical-grade biological, biosignal, kinematic, and environmental data over long periods of time. The devices may be suitable for home use, as well as in laboratory or clinical settings.

[0043] The use of on-device state classification (e.g., using artificial intelligence (AI)) enables autonomous decision-making within small, self-driving wearable devices. By using dynamic activity classification, the technology may also achieve any or all of the following: 1) Minimizing the use of stored electrical energy to enable continuous collection of selected relevant biosignals and environmental data; 2) reducing and compressing collected signal data, storing key extracted features, and spontaneously editing private activities; 3) optimizing sensor activation and measurement frequency based on classification / prediction of user activity; 4) Withholding data collection during private / sensitive activities to preserve user privacy; and 5) Combining these to enable long-term (e.g., up to six weeks) unmanned research based on commercially available battery, data storage, and sensor technology.

[0044] Commercial applications of the present invention include health, well-being, mood, fitness, and emotion monitoring. Other aspects include personalized sports training, health monitoring, medical diagnostics, and clinical trial monitoring.

[0045] 1 is a detailed flowchart of a method for acquiring user state data in a wearable device. The wearable device includes at least a main processor unit and a low-power processor unit. At S1, the method accesses an indicator of a device sub-state. If the sub-state is unknown, the method proceeds to S2, where the main processor unit is turned on. The method then acquires and processes (in the main processor unit) a first plurality of sensor signals to acquire user state data. If the device sub-state is a known device sub-state, the method proceeds to S3 with the main processor unit off. The method then (in the low power processor unit) acquires and processes a second plurality of predefined sensor signals, which are specific to the known device sub-state, to acquire user state data.

[0046] FIG. 2 is a flowchart of an example of a data acquisition method in a wearable device. At S10, the wearable device assesses whether a device sub-state is currently known. Specifically, the assessment is performed by the low-power processor unit, which is the only processor unit active at this time. The device sub-state corresponds to a predefined operating mode that sets the operating parameters of the wearable device. These parameters include: 1) Activation and deactivation states of sensors and communication units (e.g., BLE and Wi-Fi components) contained within the wearable device; 2) the sampling rate of activated sensors; 3) Retention policy, defining the subset of captured data that should be stored; 4) Announcement schedule of communication units; These include (but are not limited to):

[0047] These parameters are determined based on the importance and expected quality of each signal in the particular classified scenario that applies to the user during that state. When the device is turned on for the first time (when there is no information to inform any state classification), the wearable device may be configured to determine that the device sub-state is currently unknown.

[0048] If the wearable device determines that the device sub-state is not currently known, the process continues via step S20. At S20, the wearable device powers on a high-power processor unit. The main processor unit is initialized and begins the state classification process. In this example, the main processor unit is referred to as an "AI core / MCU" (microcontroller unit), although the skilled reader will understand that, in general, the main processor unit need not be specifically configured for AI processing, nor need it be the only core of the processing unit.

[0049] At S30, the wearable device powers on all sensors (or at least a sufficient number of sensors to enable state classification). The wearable device initializes the sensors and causes them to operate (e.g., acquire sensor data) at a predefined maximum output rate. Of course, the predefined output rate may vary from sensor to sensor.

[0050] At S40, all sensors (including communication units) provide primary raw data, which the wearable device stores in an internal buffer. If there is previously acquired data in the buffer, the wearable device appends the newly acquired primary data to the previously acquired data.

[0051] Figure 3 shows an example of primary raw sensor data obtained from a three-axis accelerometer in a wearable device. The data includes a time series of each orthogonal axis of the accelerometer (i.e., x-axis acceleration 30, y-axis acceleration 32, and z-axis acceleration 34) measured in milli-Gs. As can be seen from the plot, the user's acceleration on all three axes is initially constant, but then fluctuates at intermittent intervals (notably from approximately 14:12:10 onwards).

[0052] Returning to Figure 2, at S50, the wearable device (e.g., its main processor unit) runs a classification model on the acquired primary data to determine the state of the device and provide a resulting assessment of the user state. The wearable device performs secondary signal calculations and feature extraction steps on the primary raw data stored in a buffer. The wearable device stores these derived data sets along with the primary raw data in an internal buffer for use by the classifier.

[0053] An example classification model may include two separate stages of classification. First, the wearable device may apply separate pre-trained classification models to the data in the buffer to classify parameters such as the user's activity, location, and environment. Then, using the results of the initial stage, the wearable device may perform a second layer of classification using the data in the internal buffer as input parameters to determine the device state and provide a resulting assessment of the user state. Figure 4 is a detailed flowchart of an example two-stage classification process.

[0054] At S51, the wearable device processes the primary data collected from the sensors using processing techniques relevant to each data stream to calculate secondary values, such as heart rate, posture, respiratory rate, blood oxygen level, etc. Additionally, the wearable device may use sensor fusion techniques to combine sensor streams and generate additional secondary outputs. For example, the wearable device may calculate posture from an IMU via sensor fusion, and the wearable device may calculate RR intervals based on an ECG signal.

[0055] At S52, the wearable device extracts a subset of features from the primary and / or secondary streams to aid in classification. These features are typically created by applying mathematical or statistical operations (e.g., maximum, minimum, logarithm, etc.) to the data sets. They do not need to have an underlying physical meaning established for each, as they are solely used by the AI ​​algorithm to classify the user state outcome.

[0056] At S53, the wearable device stores the output from S51 and S52 (i.e., the primary and / or secondary data and subsets thereof) in a buffer to be processed by the classifier.

[0057] At S54, the wearable device performs a first-layer classification. During this stage, illustratively, a pre-trained classification model based on machine learning techniques (e.g., CNN, convolutional neural network) is applied to the data stored in the buffer to determine the resulting substates of the user state. The classification model may be pre-trained using a conventional AI framework (e.g., Tensorflow or Keras). The wearable device stores the results of this first-layer classification in a buffer. As an example of first-tier classification, a wearable device may perform human activity state classification, such as that outlined in the 2018 study by Jordao et al., "Human Activity Recognition Based on Wearable Sensor Data: A Standardization of the State-of-the-Art."

[0058] Figure 5 is a hierarchical overview of a user state outcome formed from activity sub-states, physiological characteristics (or physiological sub-states), environmental sub-states, and device sub-states. The wearable device calculates the state outcome (using an AI algorithm) based on the sub-state classification and characteristics including the user's activity, happiness, emotional profile, and environmental related characteristics. An example of a user state outcome aspect (emotional profile) is shown in Figure 6, where the user may be described as "working at a desk (seated) in a comfortable indoor environment, feeling very happy and excited."

[0059] An activity sub-state is a classification of the activity the user is currently performing. It is derived (using AI algorithms) based on sensor outputs and secondary data sets (kinematic, environmental, bio-signals, etc.). Examples of activity sub-states include walking, eating, swimming, brushing teeth, desk work (sitting), desk work (standing), showering, sleeping, gym (running), outdoor activity (running), cycling, car (passenger), car (driver), etc.

[0060] Physiological characteristics (or physiological sub-states) include classifications of directly and indirectly (primary, secondary) sensed physiological parameters of a user. Examples include heart rate, R-R intervals, skin temperature, electrodermal activity, etc.

[0061] An environmental sub-state is a classification of the current environment the user is in, based on environmental sensors. It may include both the type and state of the environment (as characteristics of the environment). Example classifications of environmental sub-states include: outdoors (low temperature, low humidity, and high levels of particulate matter and / or pollutants in the air); outdoors (high temperature, high humidity, and low levels of particulate matter and / or pollutants in the air); indoors (average temperature, high humidity, and low levels of particulate matter and / or pollutants in the air), etc.

[0062] A device substate is an indication of the current state of the device based on predefined settings and usage levels of the wearable device's resources (e.g., battery and memory). This substate also includes the currently active measurement profile (e.g., which sensors are active and their data acquisition characteristics). This substate is often set by the outcome of the state to optimize device operation (e.g., memory, power, and contextual relevance of the measurement profile). A device substate may also include information about the type of test (e.g., continuous, event-activated, delayed, sleep study, daytime activity-specific). Wearable devices may be used for sports training, health monitoring, medical diagnosis, and / or clinical trial monitoring. The active test type may control which in-device sensors are active or inactive.

[0063] Returning to Figure 4, at S55, the wearable device assesses the confidence level(s) of the output(s) of the first layer classification. The wearable device may compare the confidence level(s) to a predefined threshold for each classification category (i.e., each user sub-state). If the sub-state classification is successful, the wearable device may move to the next stage of processing. If the classification is unsuccessful, the wearable device terminates the classification procedure.

[0064] At S56, if necessary, the wearable device updates dynamic thresholds and models based on the results of the first-tier classification. This helps ensure that the system is responsive to each user's individual characteristics and can "learn" and adapt to each scenario. For example, the wearable device may update the resting heart rate threshold for a particular user following classification of the user activity sub-state as "sleep." The need for dynamic updates depends on the particular model used in the first-tier classification.

[0065] At S57, the wearable device applies a second layer of classification based on the pre-trained NN classifier, and the output of the first layer of classification, the primary data, the secondary data, and the extracted features stored in the buffer are input into the second layer of classification to determine a resulting user state. As an example, the classification might determine the user activity sub-state as "desk work," the environmental sub-state as "hot and humid office," and the physiological sub-state as indicating an elevated heart rate and high stress indicators (illustratively based on heart rate variability). A second-tier classification might then use all this information to classify the user state result as "very stressed and agitated while working (sitting) at a desk in an uncomfortable indoor environment." Of course, the state result may be stored as a tabular collection of marked and scored sub-states or in any other suitable format. This may also be translated or displayed using text or a graphical representation (e.g., on an external processing device). Examples of graphical representations of user state results (emotional profiles, among others) are shown in FIGS. 6 and 7.

[0066] At S58, the wearable device assesses the confidence level(s) of the output(s) of the second layer classification. The wearable device may compare the confidence level(s) to a predefined threshold for each classification category (i.e., each user sub-state). If the sub-state classification is successful, the wearable device may move to the next stage of processing. If the classification is unsuccessful, the wearable device terminates the classification procedure.

[0067] At S59, if necessary, the wearable device updates the dynamic thresholds and models based on the results of the second layer classification, a process that may occur as described above with respect to S56.

[0068] Returning to Figure 2, at S60 the wearable device assesses the success of the classification at a threshold confidence level. If the threshold is met, the user state outcome is classified and the wearable device proceeds to the next stage of the process (S70).

[0069] If the classification is unsuccessful, the wearable device stores all raw data in memory and clears the buffer for further future investigation at S80, i.e., in this scenario, the wearable device is uncertain about the outcome of the user's current state, so it stores all data and may investigate it later. If the classification is successful, then at S70 the wearable device stores the data in memory. If the classification of a state outcome is successful, only the subset of data used in the classification that is relevant to the state outcome is stored. By way of illustration, if the user is classified as sedentary (among other classifications), the wearable device need not store, for example, high frequency sampled ECG data. Data deemed unnecessary to store is discarded; in this way, the memory requirements of the wearable device are kept to a minimum.

[0070] Following successful classification, the wearable device is equipped with a pre-configured profile (aspects of the device sub-state) to determine which portions of the data to store. As an illustration, for all classifications where the activity sub-state is classified as "sedentary", the associated pre-configured profile may indicate that the wearable device does not need to store high frequency ECG data.

[0071] If necessary, the wearable device may update the classification thresholds and classification model parameters. In some scenarios, the classification model will confidently classify a sub-state, but one or more parameters may be outside predefined boundaries or above or below thresholds. In these scenarios, these boundaries and / or thresholds are updated to adapt to the new conditions and boundaries. This helps the wearable device adapt to underlying physiological differences in the user, the way the user performs an activity, and / or the way the user reacts to various events or stimuli. For example, an activity may be classified as running, but the user may be running too fast compared to a predefined boundary for acceleration. Thus, in this case, the device may update the acceleration boundary to reflect this difference.

[0072] Preferably, the wearable device stores raw, unclassified data with a label or labels indicating characteristics of the data signal. In this manner, if the wearable device is unable to classify the outcome of a user state in the future, but the underlying data features are similar to those of a previously unclassified state, the wearable device may quickly correlate similar data from multiple, different unclassified classifications. Furthermore, in this manner, the wearable device may compile multiple data sets from an unclassifiable state, and if the data sets can be reduced without losing significant information, the wearable may perform such reduction. For example, if the wearable device obtains multiple unclassified data sets that share similar data features from all sensors, but data from an IMU sensor indicates little or no movement from the user, the wearable may delete these data sets without affecting the final classification of this state.

[0073] Following successful classification of the user state results and storing the selected subset of data according to that subset, the wearable device returns to assessment if the device sub-state is now known at S10.

[0074] If the wearable device determines that the device sub-state is now known, processing continues via step S90, where the wearable device powers off the high-power processor unit, thus transferring measurement and communication responsibilities to the low-power processor unit. At S100, the wearable device activates and initializes (and deactivates, if necessary) the sensors and communication modules required for the current device sub-state, based on the predefined profile specified by the device sub-state.

[0075] At S110, the wearable device acquires primary raw data from all activated sensors and communication devices. The wearable device stores these data in an internal buffer. If previous data exists in the buffer, the wearable device appends the newly acquired data to the previous data.

[0076] At S120, the wearable device performs secondary signal calculations and feature extraction steps on the primary raw data. The wearable device stores these derived data sets along with the primary raw data in an internal buffer. The wearable device applies an exception detection algorithm to the data in the buffer to detect any deviations from expected values ​​for the device's current state, thereby potentially indicating a change in state. In turn, a change in state may require, for example, a change in an active sensor to obtain enough data to characterize the result of the user state. Figure 8 is a detailed flowchart of an example of an exception detection process.

[0077] At S121, the secondary processor unit collects the primary raw data. The wearable device converts this measurement data into accurate and uniform values ​​using standard techniques such as unit conversion and calibration.

[0078] At S122, the secondary processor unit calculates secondary values ​​based on the output of the transformed primary raw data. These values ​​are calculated for each sensor (or combination of sensors) based on the properties of the particular sensor. For example, the secondary processor unit calculates attitude from the IMU via sensor fusion and calculates RR intervals based on the ECG signal.

[0079] At S123, the wearable device stores the outputs of S121 and S122 in a buffer to be processed by the exception detection algorithm.

[0080] At S124, the wearable device calculates the exception markers using algorithms that require only low processing power, such as standard statistical methods. For example, the secondary processor unit calculates moving averages, window ranges, standard deviations, etc.

[0081] At S125, the wearable device compares the exception markers to predefined acceptable thresholds for the current state. These thresholds may be static (e.g., the user's acceleration exceeds a predefined absolute value) or adaptive (e.g., the user's acceleration exceeds a historical 90th percentile value). If necessary, additional measurements are taken to confirm the exception. Illustratively, depending on the confidence level of the algorithm, ad hoc measurements from additional sensors (not originally used in the state) may be required. As one example, if the user's posture cannot be accurately determined using a 3-axis accelerometer, the wearable device may activate a 6-axis or 9-axis IMU solely to resolve this uncertainty. As another example, depending on the confidence level of the algorithm, measurements from active sensors may be taken at an increased measurement rate.

[0082] If the exception algorithm determines that the acquired data is an exception (with respect to the current user's device sub-state), then at S126 the wearable device sets a flag indicating that an exception has been detected. If the exception algorithm determines that the acquired data is not an exception (e.g., the exception marker is within an acceptable threshold range), the wearable device updates any dynamic or adaptive thresholds, if necessary, at S127. The wearable device then proceeds to the next processing stage.

[0083] Returning to FIG. 2, at S130, the wearable device compares the confidence of the output from the exception detection algorithm to a threshold to determine whether an exception has been detected.

[0084] If no exception is detected, the wearable device stores in memory a subset of the raw data relevant to the current device sub-state at S140, and the buffer is cleared. In addition, the associated adaptive parameters of the exception detection algorithm are updated with the new data (if necessary). The wearable device then returns to S110 and continues acquiring primary raw data from all activated sensors and communication devices. If an exception is detected (as indicated, illustratively, using the flag from S126), the wearable device stores all captured data in a buffer for further investigation at S150. The wearable device then proceeds to S20 and S30, etc., where the wearable device powers on the high-power processor unit and all sensors. In this manner, the buffered exceptional data may be further investigated using the high-power processor unit.

[0085] At this point, the high-power processor unit is activated; if this is the first time the high-power processor unit has encountered a particular sub-state combination, the high-power processor unit may be configured to begin collecting additional data (from some or all of the first plurality of sensors). This may be achieved through a higher sampling rate or obtaining higher-resolution data from the sensors. If the collected data provides enough information for the high-power processor unit to successfully classify a state outcome, the wearable device may record in memory the characteristics of the initial combination of sub-states that resulted in the unknown state. The wearable device may then associate that combination with a final state outcome, which will be determined after further data collection. The wearable device then updates relevant exception-detection thresholds and adaptation parameters to avoid future misclassification of the appropriate sub-state as an unknown state. This increases the proportion of time spent in a low-power configuration (because the high-power processor unit and additional sensors will not be re-enabled) and also reduces storage requirements (because unnecessary additional / high-resolution data is recorded). However, if collecting additional information does not result in a condition classification, the wearable device may record all of the additional acquired data in memory for later processing.

[0086] Notably, the wearable device may be configured to store in memory an initial set of substates (whether unknown, known, or with low classification confidence) and their characteristics (e.g., acceleration data as a high result, motion signature) along with the acquired raw data. This configuration may then be tagged with a temporary tag as the state result. This plays an important role in optimizing power consumption and reducing the storage capacity required for the data. The combined characteristics of this temporary tag and its associated subset may then be effectively used by a high-powered processor unit as a known state result to identify the same characteristics in future encounters, thereby avoiding the storage of unnecessary additional raw data and the resource consumption associated with the use of additional sensors. This eliminates the power and storage capacity burden associated with unknown state results in the prior art.

[0087] To illustrate the exception detection mechanism, consider the following two example scenarios. In the first example scenario, the current device sub-state is known and indicates a sleep measurement profile, and the user's activity sub-state is known to be "Sleep." Note that other sub-state information, such as the environmental sub-state, is not relevant for the purposes of this example scenario. During a measurement cycle, markers calculated based on the low-power accelerometer indicate an increase in the user's kinematic activity, which is outside of an acceptable threshold for the current state. This acceptable threshold is pre-defined based on previous measurements. An example of these markers is a windowed moving average of the resulting linear acceleration amplitude. As a result, the wearable device raises a flag indicating an exception. This potentially indicates that the user is awake, and control is passed to the wearable's primary processor unit and sensors for further investigation.

[0088] In a second example scenario, the current device state is known and indicates a general measurement profile, the activity substate is known to be "sedentary," and the emotional profile (determined from the user state results) is known to be roughly neutral. Again, note that other substate information, such as the environmental substate, is not relevant for the purposes of this example scenario. During a measurement cycle, primary and secondary biosignals indicate changes in monitored markers that are outside of threshold, while other markers, such as kinematic signals and environmental parameters, do not indicate exceptions. Examples of these biosignal markers include short-term heart rate variability (HRV), heart rate variability, pulse transit time (PTT) change, GSR change, and skin temperature change. These values ​​may be compared to static baseline thresholds for the specified state or adaptive thresholds (e.g., skin temperature). Thus, these exceptions may indicate a change in the user's emotional state and will be flagged accordingly. In this manner, a main processor unit with greater processing power than the secondary processor unit may investigate and classify new states.

[0089] Figure 9 is a timeline of state classification performed by a wearable device. The top box shows the transitions between activity sub-states over a 24-hour period. The bottom box shows the transitions between environmental sub-states over the same 24-hour period. As an illustration, at start-up (time zero), the wearable device classifies the wearer as being in an activity sub-state indicated by scale 7, and an environmental sub-state indicated by scale 3. Table 1 below provides a summary of the relevant activity and environmental sub-states for some of this example.

[0090] [Table 1]

[0091] That is, just before 7 hours after the wearable device was powered on, the user's activity sub-state was reclassified from "Sleep" (scale 7) to an unknown state (scale 14). The environmental sub-state remains unchanged at this point ("Indoors (medium, humid, good air)", scale 3). As noted above, since the user's activity sub-state is unknown at this point, the wearable device decides to power on its high-power processor unit and acquire data from all sensors so that it can later classify this activity sub-state.

[0092] At 8.15 hours after the wearable device was turned on, the environmental sub-state was reclassified from "Indoor (medium, humid, good air)" (scale 3) to "Outdoor (warm, humid, good air)".

[0093] At 8.90 hours after the wearable device is powered on, the user's activity sub-state is reclassified to a second unknown state (scale 15). At 12.00 hours after the wearable device is powered on, the user's activity sub-state is again reclassified to this second unknown state. As described above, in this case the wearable device has determined that the characteristics of the data are similar to the characteristics that it was previously unable to classify. The skilled reader will appreciate that the specific described sub-states are examples and that additional and / or more detailed sub-states are possible. Additionally, physiological sub-states and device sub-states (or aspects thereof) may be similarly plotted / tabulated. For example, an indication of which sensors are active for a particular activity sub-state (aspect of a device sub-state) may be similarly plotted / tabulated.

[0094] To understand the performance of a wearable device when faced with unknown substates, consider the following two example scenarios. In the first example scenario, consider a user using an ergonomic rocking chair that they use when using a PC. In this example scenario, this activity is not recorded as a known state in the device's database. However, sitting in an existing stationary chair is recorded as a pre-defined activity sub-state of the device (with associated device sub-states). This scenario illustrates how the system handles exceptions caused by differences in individual behavior / situations without asserting the consequences of new sub-states / states.

[0095] Once the user sits down, an exception is detected by the secondary processor unit (e.g., an exception related to the activity sub-state before walking to the chair or standing up). This results in the high-power processor unit waking up, initializing / adjusting the associated sensing regime, and acquiring additional information (from environmental sensors, auxiliary devices, IMU, etc.). Based on this data, the resulting user state may be classified as "sitting calmly and using PC."

[0096] After a few minutes, the user may begin to rock back and forth in their chair; this movement is not within the expected range (e.g., of accelerometer data) for the currently classified activity sub-state and therefore triggers an exception, which results in data being collected by a high-powered processor unit.

[0097] Following collection of additional data, the substate is again classified as "sitting" (e.g., based on IMU and other position data, and based on no significant changes in other substate and sensor data). The high-power processor unit therefore updates its motion threshold for this state to accommodate the additional motion caused by rocking in the chair. This allows the secondary processor unit to avoid invoking exception detection routines based on the limited accelerometer data collected in this mode. This, in turn, results in power savings by avoiding the need to turn on auxiliary sensors (e.g., a 9-axis IMU) to reconfirm the substate.

[0098] Figure 10 shows a comparison of the power consumption of a wearable device over a 30-minute period in this first example scenario with and without the implementation of adaptive technology through user state data acquisition. This figure illustrates how the wearable device handles detected exceptions caused by differences in individual behaviors / situations without registering new substate / state results. The power consumption values ​​are based on the average power consumption of the entire wearable device in each specific substate.

[0099] The activity sub-state changes from walking to sitting at 2 minutes, which results in the detection of an exception. This exception detection triggers the wake-up of the high-power processor unit. The high-power processor unit then immediately classifies the sub-state as sitting and returns the device to a low-power consumption operating mode. The sitting user then begins rocking in their chair at approximately 14 minutes, which results in the detection of another exception. As described above, the high-power processor unit adjusts the sitting activity sub-state threshold to accommodate this movement upon further encounters, such as at approximately 27 minutes. Upon this second encounter of the activity sub-state, the techniques herein avoid detecting the exception and allow the system to continue at a lower power consumption setting (e.g., fewer active sensors), resulting in overall power savings and reduced data storage requirements.

[0100] In a second example scenario, a user regularly uses a deep tissue massage gun after working out at the gym. In this example scenario, this activity is not recorded as a known state in the device's database. This scenario illustrates how the system can create new state results (via temporary tagging of an unknown state, as described above) and use the new state results to reduce power consumption and optimize data storage capacity when that state is subsequently encountered.

[0101] When a user is at the gym and performing their daily exercise, the device typically classifies the activity based on biosignals (ECG, heart rate, etc.) as well as other sensor parameters. However, when the user finishes their exercise and first begins using the massage gun, the wearable device will encounter an unknown activity sub-state because the vibrations (and related derived parameters, such as frequency of movement) caused by the massage gun do not match the expected characteristics of the end-of-exercise state. This will prompt a high-powered processor unit to wake up and begin collecting additional data from other sensors and launch more power-consuming classification algorithms. However, the combination of sub-states and / or the confidence of their classifications will result in an unknown state.

[0102] As described above, the high-power processor unit then stores in memory all raw data related to the new, unknown user state or state outcome and assigns a temporary tag to the current combination of sub-states (e.g., one indicating low-effort activity based on biosignals despite significant movement detected by the activity measurement system and being in a gym). This data also contains information about derived data and its characteristics (e.g., movement frequency, movement amplitude, body posture) determined from the sensor output (used as input to the classification algorithm). The wearable device will then use this information in any subsequent activities in which the massage gun is used. In these activities, when an exception is detected by the secondary processor unit, the high-power processor unit may be activated as usual. However, following the collection of additional information, the high-power processor unit may then register the state outcome using the temporary tag assigned to the unknown state on the first encounter. The wearable device may then enter a low-power state without having to store all raw data and without having to remain active until a known state is detected. This significantly reduces the power and storage requirements of the device.

[0103] FIG. 11 shows a comparison of power consumption over 30 minutes of a wearable device in this second example scenario, depending on whether or not the adaptive technology is implemented in a manner that obtains user state data.

[0104] As the user changes activities, the wearable device classifies the new activity and enters / implements its sub-state configuration. However, the first use of the massage gun around the 11-minute mark results in a combination of sensor data that falls outside the parameters of known activity sub-states. Therefore, the system with dynamic classification assigns a temporary tag to the unknown state, after which the unknown state becomes a known state, and the system enters low-power mode. In contrast, a similar system without dynamic classification would remain in high-power mode (the unknown state persists) until it detects a known state (e.g., the sitting activity sub-state at approximately 17 minutes). In addition, the system with dynamic classification quickly identifies the second encounter with massage gun use at 25 minutes and enters the associated low-power mode, while a system without dynamic unknown state classification would remain in high-power mode through all encounters with unknown states. Therefore, using dynamic temporary unknown state classification results in significant power and storage savings during typical user activities.

[0105] 12 is a schematic diagram of the functional electronic blocks underlying the example wearable device used in the processing outlined above. Broadly, the electronic components may be categorized into four major subsystems: power management, logic and memory, sensing, and communications. The skilled reader will appreciate that additional functional subsystems and units (or alternatively, a subset of the exemplary subsystems and units described) may be incorporated into the wearable device without loss of generality.

[0106] Power management subsystem A has three separate units that are responsible for battery protection, charging, and regulation.

[0107] The battery protection unit A1 is responsible for protecting the battery against leakage current, overcurrent, and high / low voltage.

[0108] The multi-rail power supply unit A2 is responsible for providing the necessary power to the high-power processor unit and other subsystems. It does this by converting the voltage provided by the battery into voltage rails (levels) suitable for each component. In addition, this subsystem has the ability to disable certain rails to save power if required by the processor.

[0109] The charging circuit unit A3 is responsible for safely charging the device's battery. Any known device battery technology (e.g., alkaline, Li-ion, or LiPo batteries) may be used. Rechargeable battery technology (e.g., Li-ion) has the advantage that the device does not need to be disposed of after the battery is depleted. The power management subsystem A may be complemented by means for charging rechargeable batteries, such as motion-driven charging means or solar charging means.

[0110] The logic and memory subsystem B is connected to the power management subsystem A via connection (bus or interface) A4 to control the voltage rails and sense the battery level, charge state, and current battery consumption. The Logic and Memory Subsystem B is responsible for executing the embedded code / algorithms, acquiring data from sensors, and storing the collected data in memory. This subsystem may be functionally divided into two main units: a Processor Unit B1 and a Memory Unit B2.

[0111] The processor unit B1 includes at least two separate processor units: a main or high-power processor unit and a secondary or low-power processor unit. These processor units may be on separate chips or may be part of a multi-core chip. Regardless of the configuration, they may be divided into two different types: high-power processor units and low-power processor units.

[0112] The high-power processor unit (main or primary core, main controller) B1.1 has significantly higher processing power than the low-power processor unit B1.2. The high-power processor unit consumes more power than the low-power processor unit. The high processing power of this processor unit makes it suitable for running complex embedded AI classification algorithms or tasks that typically have a high processing load (e.g., requiring high-speed data transfer). However, the high power consumption of this processor unit makes it impractical to run them continuously.

[0113] The low-power processor unit (low-power secondary core, low-power controller) B1.2 has significantly lower power consumption compared to the main processor unit B1.1, which makes it suitable for tasks with lower processing loads (e.g., acquiring sensor data or detecting exceptions) as well as tasks that need to run continuously. Both processor units are connected to each other via interrupt routines and a dedicated data bus B1.3, which allows handover of operations between processor units during operational state transitions.

[0114] Processor unit B1 primarily uses memory unit B2 to store raw and processed data (e.g., collected from sensors) and operational data (e.g., errors and logs). In addition, processor unit B1 (low-power processor unit B1.2) uses memory unit B2 to temporarily store (i.e., buffer) data from sensors for further processing or classification by main processor unit B1.1.

[0115] The volatile memory unit B2.1 of the memory unit B2 is used by the processor unit to temporarily store data during operations.

[0116] The short-term memory unit B2.2 is used for short-term storage of data sets (larger than the volatile memory capacity), mainly during the initial stage of data collection before the data is binarized / classified or during the handover of data from the low-power processor unit B1.2 to the high-power processor unit B1.1. The long-term storage unit B2.3 is the main storage unit of the device and is used to store data and logs for further processing and analysis (by an external processing device, eg a PC).

[0117] A high-speed data bus B3 connects the processor unit B1 and the memory unit B2. The logic / memory sense data bus, interrupt, and control channel B4 connects the logic and memory subsystem B and the sense subsystem C. It is used to collect measurements (for storage and processing) and to configure and control the sensors. The logic / memory communication data bus and control channel B5 connects the logic and memory subsystem B and the communication subsystem D. It is used during data transfer (e.g., data transfer between devices or data transfer between a device and a PC).

[0118] The sensing subsystem C includes various sensors used to collect specific motion, environmental, biosignal, and activity parameters.

[0119] The ECG (electrocardiogram) sensor unit C1 is responsible for measuring cardiac signals via ECG contact electrodes. The PPG (Photoplethysmography) sensor unit C2 is responsible for measuring the heart rate via optical methods and may be synchronized with the ECG sensor unit C1. A non-contact infrared (IR) based temperature sensor unit C3 is used to measure the skin surface temperature.

[0120] The contact temperature sensor unit C4 is used to measure the skin / body temperature. The GSR (Galvanic Skin Response) sensor unit C5 is used to monitor changes in sweat gland activity via measuring changes in the electrical conductivity of the skin. The ambient atmospheric pressure sensor unit C6 and ambient temperature sensor unit C7 are used to monitor the environmental characteristics (air pressure and temperature, respectively) in the vicinity of the user.

[0121] A 9-axis IMU unit C8, including a high-performance 3-axis accelerometer, 3-axis magnetometer, and 3-axis gyroscope, is used to measure detailed kinematic parameters. In addition, this unit may include an embedded low-power processor unit, separate from the main logic subsystem B1, for running sensor fusion algorithms (e.g., posture calculation) and light activity classification (e.g., running, walking, standing). In this way, the power consumption of the wearable device (e.g., incurred through unnecessary activation of the high-power processor unit B1.1) may be reduced.

[0122] The 3-axis accelerometer unit C9 is used to provide less detailed kinematic information (compared to the information provided by the 9-axis IMU C8) during operating conditions when IMU data is not required. Of course, the skilled reader will understand that the wearable device may additionally or alternatively include accelerometers configured to sense acceleration data using fewer or more axes (illustratively, a 2-axis accelerometer and / or a 6-axis accelerometer). The motion switch unit C10 is an ultra-low power (nanowatt range) motion detection sensing system that is used to detect movement when other motion detection systems (IMU C8, accelerometer C9) are not operational (and their activation and associated energy consumption would be excessive beyond what is necessary).

[0123] The skin moisture sensing unit C11 is a sensing system used to measure relative skin moisture. The Environmental Air Analysis Unit C12 measures specific gases, chemicals, or particles in the air (e.g., CO2, NO x , alcohol, etc.

[0124] The communication subsystem D is responsible for transferring data between devices and to / from control devices (e.g., PCs, tablets). Illustratively, a wearable device may also transmit stored data to an external source for examining the stored data (e.g., to improve a pre-trained classification model on the device or to train a new classification model based on acquired exceptional data). Additionally, a wearable device may acquire sensor signals from a sensor unit that is not part of the sensing subsystem C on the device but instead has IoT capabilities.

[0125] The Wi-Fi unit D1 provides the device with a standard Wi-Fi connection, which is used for high-speed data transfer and device setup. The BLE (Bluetooth Low Energy) unit D2 provides low-power communication between the device and a PC / tablet for data transfer and setup. The NFC (Near Field Communication) unit D3 is used for device pairing, device identification, and data transfer authorization. The USB (Universal Serial Bus) D4 capability is used for large data transfers and device setup.

[0126] 13 is a functional schematic diagram of an exemplary BLE unit D2. As described above, a wearable device may also include a BLE module / capability. The BLE unit has an embedded low-power processor unit that handles all overhead related to the BLE protocol. This allows the BLE unit to operate independently of the main processor unit(s) when needed (e.g., in a scenario where both the main and secondary processor units are in sleep / low-power mode).

[0127] Data transfer is one of the main functions of the BLE unit: the stored data may be transferred to a BLE-enabled device such as a PC, smartphone, or tablet for further investigation and processing. The BLE unit may also be used to configure the wearable device. Illustratively, the BLE unit may be used to accommodate firmware upgrades, during sensor calibration, etc. If the wearable device is to be used to acquire user status data for, e.g., medical or research testing, the BLE unit may be used to set test parameters such as test start time, test duration, and active and inactive sensors.

[0128] If the wearable device is configured to acquire sensor signals from external devices, such external devices within the platform may also be equipped with similar BLE units, enabling communication between devices within the platform. Examples of such communication include downloading data from the external device to the wearable device for future transmission from the wearable device to another external device for survey processing or for use by the wearable device's on-device classification algorithm(s). Illustratively, BLE interactions with external devices may be used to synchronize and correlate device exposure with user behavior and user reactions. As an example, a wearable device used in combination with an external treadmill with BLE communication capabilities may enable use of such gym equipment to be linked and correlated with a user's emotional state before and after a workout session.

[0129] In a particular device sub-state, or when the state of the wearable device needs to be determined (for example, when the device is first powered on), the BLE unit may scan the RF environment for other Bluetooth® devices in the vicinity. This allows the wearable device to discover and connect to other devices (active BLE-using beacons) in use within the platform.

[0130] Additionally, on-device pre-trained machine learning classification models may assist in classifying the state using the type of external device discovered (third-party / platform) and an estimate of the proximity of the external device (e.g., based on signal strength). A simplified example of this includes discovering active BLE-enabled gym equipment; when a wearable device senses higher than normal user activity levels and senses other fitness devices in the immediate vicinity, it may increase the confidence in the classification that indicates the user is exercising.

[0131] In addition to scanning for BLE external devices (beacons), a wearable device may also announce its own beacon for other platform devices. In this way, the wearable device itself may form part of a broader platform and may also contribute information to the classification of other wearable devices. That is, when a first wearable device (worn by a first user) announces its presence to a second wearable device (worn by a second user), the classification of the second wearable device may take into account that the first user is in close proximity; thus, the second wearable device may determine that the second user is likely in a social environment. Optionally, this functionality may only exist for / within specific device sub-states. For example, a wearable device may announce its own beacon in all device sub-states where the device does not need to use power to conserve power; the beacon may be muted in inactive device sub-states, such as when the wearable device believes the user is sleeping or otherwise dormant.

[0132] The BLE unit may additionally or alternatively be used to send notifications or communicate with a user's BLE device (e.g., a mobile phone) to provide useful information about the user's activity / status and to capture user feedback or interaction with the device. While the above discusses BLE communication capabilities of a wearable device, the skilled reader will appreciate that this functionality may also be provided with other means or protocols of communication. For example, acquisition of sensor signals from an external device may be performed using any suitable communication protocol, such as ZigBee, Li-Fi, NFC, RFID, LTE, 5G, etc.

[0133] 14a and 14b are a pair of three-dimensional renderings of the exterior of a wearable device. The former rendering shows a low-lying profile of the device, suitable for a "wear-it-and-forget" approach, where the user simply attaches the device to their body (not shown) and, illustratively, the device may remain there until sufficient data has been acquired or the battery supply on the device is depleted. The latter rendering shows the skin-facing side of the wearable, including a pair of ECG electrode snap points and a skin-facing sensor.

[0134] FIG. 15 is a set of line drawings depicting the exterior of an example wearable device 130 (the same wearable as shown in FIGS. 10a and 10b). The exterior shell 132 is a plastic shell designed to house the electronics and protect them from damage while providing comfort for the user. The vent 134 provides a small hole in the shell 132 to allow access to environmental sensors (housed in a suitably sealed chamber within the device). The skin-facing sensors 136 include both contact and non-contact biosignal sensors (e.g., sensors for skin temperature, PPG, IR-based skin temperature, and skin hydration). The ECG electrode snap points 138 are gold-plated ECG electrode snap connectors compatible with standard ECG electrodes.

[0135] 16 is a block diagram of a computing device, such as the computer aspect of a wearable device, that embodies the present invention and that may be used to implement an embodiment method of obtaining user state data on a wearable device. The computing device includes a processor 993 and memory 994. Optionally, the computing device also includes a network interface 997 for communicating with other computing devices (e.g., other computing devices of the invention embodiments). For example, an embodiment may consist of a network of such computing devices. Optionally, the computing devices also include one or more input mechanisms, such as a GUI 996, and one or more display units 995, such as screens or monitors. The components may be interconnected via a bus 992.

[0136] Memory 994 may include a computer-readable medium, which term may refer to a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches, and servers) configured to carry computer-executable instructions or have data structures stored thereon. Computer-executable instructions may include, for example, instructions and data that can be accessed by a general-purpose computer, a special-purpose computer, or a special-purpose processing device (e.g., one or more processors) and cause it to perform one or more functions or operations. Thus, the term "computer-readable storage medium" may also include any medium that can store, encode, or transmit a set of instructions for execution by a machine, causing the machine to perform one or more of the methods of this disclosure. Thus, the term "computer-readable storage medium" may be interpreted to include, but is not limited to, solid-state memory, optical media, and magnetic media. By way of example, and not limitation, such computer-readable media may include non-transitory computer-readable storage media including random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory devices (e.g., solid-state memory devices).

[0137] The processor 993 is configured to control the computing device and perform processing operations, for example, by executing code stored in memory to perform various functions of the subsystems and units (particularly the main processor unit and low-power processor unit) described herein and in the claims. The memory 994 stores data that is read and written by the processor 993. As referred to herein, a processor may include one or more general-purpose processing devices, such as a microprocessor, a central processing unit, or the like. A processor may also include a complex instruction set computer (CISC) microprocessor, a reduced instruction set computer (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor that performs other instruction sets or a processor that performs a combination of instruction sets. A processor may also include one or more special-purpose processing devices, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the like. In one or more embodiments, the processor is configured to execute instructions to perform the operations and steps discussed herein.

[0138] A display unit 997 may display a representation of data stored by the computing device and may also display screens that allow interaction between a user and programs and data stored on the computing device. An input mechanism 996 may also allow a user to input data and instructions into the computing device.

[0139] The network interface (network I / F) 997 is connected to a network such as the Internet and can be connected to other such computer devices via the network. The network I / F 997 may control input and output of data to and from other external devices and / or external processing devices via the network. Other peripheral devices such as a microphone, speaker, visual alarm means, etc. may also be included in the computer device.

[0140] The main processor unit (high-power processor unit B1.1 in FIG. 12 ) may be a processor 993 (or multiple processors) that executes processing instructions (programs) stored on memory 994 and exchanges data via a network I / F 997 or via an input mechanism 996. Among other things, the processor 993 receives configuration data from an external device and / or an external processing device via the network I / F, configures the wearable device to capture user state data, and executes processing instructions to perform user state classification, as in S20-S80 of FIG. 2 . Furthermore, the processor 993 may also execute processing instructions to store user state data on a connected storage unit and / or transmit user state data via the network I / F 997 to an external processing device for further research, to improve the classification algorithm, or to develop a new classification algorithm.

[0141] The low-power processor unit (B1.2 in FIG. 12 ) may be a processor 993 (or multiple processors) that executes processing instructions (programs) stored on memory 994 and exchanges data via network I / F 997 or via input mechanism 996. Among other things, processor 993 executes processing instructions to receive configuration data from an external device and / or external processing device via the network I / F, configure the wearable device to capture user state data, and obtain user state data following successful state classification, as in S90-S150 in FIG. 2 . Furthermore, processor 993 may also execute processing instructions to store user state data on a connected storage unit and / or transmit user state data via network I / F 997 to an external processing device for further research, to improve the classification algorithm, or to develop a new classification algorithm.

[0142] A method embodying the present invention may be performed on a computing device such as that shown in Figure 16. Such a computing device need not have all of the components shown in Figure 16, but may consist of a subset of those components. A method embodying the present invention may be performed by a single computing device in communication with one or more data storage servers over a network. The computing device may also be the data storage device itself, which stores user state data.

[0143] Methods embodying the present invention may be performed by a plurality of computing devices operating in cooperation with one another, one or more of which may be data storage servers that store at least a portion of the user state data.

Claims

1. 1. A computer-implemented method for obtaining user state data in a wearable device including a main processor unit and a low-power processor unit, the method comprising: Accessing device sub-state indicators Including, If the device sub-state is an unknown device sub-state, turning on the main processor unit and proceeding to acquire and process the first plurality of sensor signals in the main processor unit and acquire user state data; and If the device sub-state is a known device sub-state, proceeding by turning off the main processor unit, acquiring and processing in the low-power processor unit a second sensor signal used for the known device sub-state, and acquiring user state data. The computer-implemented method.

2. The method of claim 1 , wherein the second plurality of sensor signals includes fewer sensor signals than the first plurality of sensor signals.

3. 3. The method of claim 1 or claim 2, wherein the main processor unit processes the first plurality of sensor signals and optionally uses at least one trained machine learning model to output a user state classification and / or a user state result.

4. The main processor unit is: passing the first plurality of sensor signals through an initial trained machine learning model to output a classification of a user state according to a predefined sub-state; passing the user state classification and the first plurality of sensor signals through a subsequent trained machine learning model to output a user state result; The method of claim 3 , further comprising processing the first plurality of sensor signals by:

5. below: Determining the reliability of the user state and / or the results of the user state; and If the confidence is below the threshold, passing the first plurality of sensor signals to a memory of the wearable device for further investigation and continuing to acquire the first plurality of sensor signals; 5. The method of claim 3 or claim 4, further comprising:

6. below: Determining the reliability of the user state and / or the results of the user state; and if the reliability is below a threshold, registering the user state and / or user state result as a new user state and / or new state result; The method of any one of claims 3 to 5, further comprising:

7. 7. The method according to any one of claims 3 to 6, wherein the main processor unit updates adaptive parameters of at least one trained machine learning model following the classification of the user state and / or the output of the user state results.

8. A low-power processor unit Detecting whether there is an exception when processing the second sensor signal; Initializing the main processor unit following the detection of an exception; and then passing the second sensor signals for processing in a main processor unit, where the processing in the main processor unit optionally also acquires and uses the first plurality of sensor signals; The method according to any one of claims 1 to 7.

9. Exception detection is as follows: passing the second sensor signal through an anomaly detection algorithm to detect deviations from expected values ​​of the current device sub-state; and Detecting an exception when the confidence level of the output of the exception detection algorithm exceeds a predefined threshold; 9. The method of claim 8, comprising:

10. 10. The method of claim 8 or claim 9, wherein the low-power processor unit updates adaptive parameters of the exception detection algorithm following detection of an exception.

11. 11. The method of claim 1, further comprising, subsequent to obtaining the user state data, discarding at least a portion of the user state data, the portion of the user state data being unrelated to known device sub-states, wherein the association between the user data and the known device sub-states is pre-set for each device sub-state.

12. When the device sub-state indicator is accessed, the main processor unit is turned on, and the method is as follows: If the device substate is unknown, leave the main processor unit on; and Turning off the main processor unit if the device substate is known; The method of any one of claims 1 to 11, further comprising:

13. The first plurality of sensor signals and / or the second sensor signal are: a biosensor for sensing a user's biosignal; environmental sensors for sensing local environmental characteristics; communicating sensors; and motion sensors, 13. The method of any one of claims 1 to 12, wherein the method is at least partially obtained from at least one of:

14. The user state data is: User activity substate; the user's biometric substate; environmental substates; and Device substate, The method according to any one of claims 1 to 13, wherein the user state is determined according to at least one, preferably all of:

15. The method of claim 14 , wherein the user state results include an emotional profile of the user.

16. The method of any one of claims 1 to 15, further comprising transmitting user state data to an external processing device by a communication subsystem.

17. The method of any one of claims 1 to 16, wherein the first plurality of sensor signals and / or the second sensor signal are at least partly obtained from an external device.

18. A wearable device having a main processor unit, a low-power processor unit, and a memory, wherein the main processor unit and the low-power processor unit are configured to perform the steps of the method according to any one of claims 1 to 17.

19. A computer program comprising instructions for causing a wearable device according to claim 18 to carry out the steps of the method according to any one of claims 1 to 17.

20. 20. A computer readable medium storing a computer program according to claim 19.

Images