RF-SENSION-ASSISTED USER ACTION PREDICTION

Abstract

Sensor data streamed from a variety of personal electronic devices worn by a user is received at a hub. An initial analysis of the sensor data is performed using first data fusion to determine instantaneous aspects of the user's movement. A second analysis of these instantaneous aspects is performed using second data fusion to determine behavioral aspects of the user's movement over time. An actionable result is generated based on the second analysis. One or more operations are then performed based on this actionable result.

Description

TECHNICAL AREA

[0001] Aspects of the disclosure generally relate to user interaction prediction supported by radio frequency (RF) sensing. BACKGROUND

[0002] The use of personal wearable electronics is increasing. At any given time, a person may possess multiple electronic devices, such as headphones, a smartwatch, a telephone, a continuous glucose monitor or other vital signs monitor, and / or shoes equipped with sensors. SUMMARY

[0003] In one or more illustrative examples, a procedure for performing data pooling and analysis involves receiving, through a hub, sensor data streamed from a variety of personal electronic devices worn by a user; performing an initial analysis of the sensor data using an initial data fusion to determine instantaneous aspects of the user's movement; performing a second analysis of the instantaneous aspects using a second data fusion to determine behavioral aspects of the user's movement over time; determining an actionable result according to the second analysis; and performing one or more operations based on the actionable result.

[0004] In one or more illustrative examples, the sensor data includes data from an inertial measurement unit (IMU) of one or more IMU sensors of the multitude of personal electronic devices.

[0005] In one or more illustrative examples, the IMU data includes acceleration and / or velocity information relating to a user's movement.

[0006] In one or more illustrative examples, the sensor data includes radio frequency (RF) channel information data from one or more RF transmitters of the multitude of personal electronic devices.

[0007] In one or more illustrative examples, the RF channel information data specify the distances between pairs of the multitude of personal electronic devices and channel state information (CSI) that is representative of the environment between the multitude of personal electronic devices.

[0008] In one or more illustrative examples, the current aspects include one or more user characteristics, attitudes, actions and / or activities.

[0009] In one or more illustrative examples, the behavioral aspects include one or more user intent determination and / or person identification.

[0010] In one or more illustrative examples, the actionable result involves authorizing the user to access a facility or location based on user intent determination and person identification.

[0011] In one or more illustrative examples, the multitude of personal electronic devices include one or more headphones, a biometric device, a smartwatch and / or a mobile phone.

[0012] In one or more illustrative examples, the method further includes: identifying, by the hub, the multitude of personal electronic devices based on advertising messages sent by the respective personal electronic devices; receiving, by the hub, device information messages from the multitude of personal electronic devices, wherein the device information messages specify device-specific interfaces and capabilities of the respective personal electronic devices; and sending, by the hub, configurations to the multitude of personal electronic devices, wherein the configurations specify a cadence for receiving sensor data and / or information about which elements of the sensor data are to be provided to the hub.

[0013] In one or more illustrative examples, the procedure further includes broadcasting, by the hub, a facility query message broadcast requesting that the plurality of personal electronic devices send facility sensor information messages to the hub; receiving, by the hub, the facility sensor information messages requested by the plurality of personal electronic devices; and continuing to receive periodic facility sensor information messages from the plurality of personal electronic devices.

[0014] In one or more illustrative examples, the multitude of personal electronic devices delays the sending of periodic device sensor information messages when there is a conflict with protocol messages being sent or received by the multitude of personal electronic devices.

[0015] In one or more illustrative examples, a system for performing data pooling and analysis includes a multitude of personal electronic devices worn by a user; and a hub device configured to: receive sensor data streamed from the multitude of personal electronic devices; perform an initial analysis of the sensor data using an initial data fusion to determine instantaneous aspects of the user's movement; perform a second analysis of the instantaneous aspects using a second data fusion to determine behavioral aspects of the user's movement over time; determine an actionable result based on the second analysis; and perform one or more operations based on the actionable result.

[0016] In one or more illustrative examples, the sensor data includes data from an inertial measurement unit (IMU) of one or more IMU sensors of the multitude of personal electronic devices.

[0017] In one or more illustrative examples, the IMU data includes acceleration and / or velocity information relating to a user's movement.

[0018] In one or more illustrative examples, the sensor data includes radio frequency (RF) channel information data from one or more RF transmitters of the multitude of personal electronic devices.

[0019] In one or more illustrative examples, the RF channel information data specify the distances between pairs of the multitude of personal electronic devices and channel state information (CSI) that is representative of the environment between the multitude of personal electronic devices.

[0020] In one or more illustrative examples, the current aspects include one or more user characteristics, attitudes, actions and / or activities.

[0021] In one or more illustrative examples, the behavioral aspects include one or more user intent determination and / or person identification.

[0022] In one or more illustrative examples, the actionable result involves authorizing the user to access a facility or location based on user intent determination and person identification.

[0023] In one or more illustrative examples, the multitude of personal electronic devices include one or more headphones, a biometric device, a smartwatch and / or a mobile phone.

[0024] In one or more illustrative examples, the hub is further configured to identify, through the hub, the multitude of personal electronic devices based on advertising messages sent by the respective personal electronic devices; to receive, through the hub, device information messages from the multitude of personal electronic devices, wherein the device information messages specify device-specific interfaces and capabilities of the respective personal electronic devices; and to send, through the hub, configurations to the multitude of personal electronic devices, wherein the configurations specify a cadence for receiving sensor data and / or information about which elements of sensor data should be provided to the hub.

[0025] In one or more illustrative examples, the hub is further configured to broadcast, through the hub, a facility query message broadcast requesting that the multitude of personal electronic devices send facility sensor information messages to the hub; receive, through the hub, the facility sensor information messages requested by the multitude of personal electronic devices; and continue to receive periodic facility sensor information messages from the multitude of personal electronic devices.

[0026] In one or more illustrative examples, the multitude of personal electronic devices delays the sending of periodic device sensor information messages when there is a conflict with protocol messages being sent or received by the multitude of personal electronic devices.

[0027] In one or more illustrative examples, one or more non-volatile, machine-readable media contain instructions for performing data pooling and analysis which, when executed by a system comprising a hub device and a variety of personal electronic devices, cause the system to perform operations to: capture sensor data streamed from a variety of personal electronic devices worn by a user; perform an initial analysis of the sensor data using an initial data fusion to determine instantaneous aspects of the user's movement; perform a second analysis of the instantaneous aspects using a second data fusion to determine behavioral aspects of the user's movement over time; and determine an actionable outcome based on the second analysis.and performing one or more operations based on the actionable outcome. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 illustrates an exemplary sensor hardware system for use in performing the data pooling and data analysis discussed here; Fig. 2A illustrates an exemplary detail of the components of the personal electronic equipment; Fig. 2B illustrates an exemplary detail of the hub's components; Fig. 3A illustrates an exemplary system architecture in which the personal electronic devices stream information from their sensors to an aggregator; Fig. Figure 3B illustrates an exemplary system architecture in which the personal electronic devices stream information from their sensors to a portable hub; Fig. 3C illustrates an exemplary system architecture in which the personal electronic devices each stream information from their sensors to an external hub for processing; Fig. Figure 4 illustrates an example data flow for the operation of the sensor hardware system; Fig. 5A illustrates an example block diagram of data intelligence; Fig. 5B illustrates an example block diagram of data intelligence, with a first fusion performed on the facility and a second fusion performed on the hub; Fig. 5C illustrates an exemplary block diagram of data intelligence, with the first and second fusions performed on the hub; Fig. 5D illustrates an example block diagram of the data intelligence being performed on the hub; Fig. 5E illustrates an exemplary block diagram of data intelligence showing channel state information, with a first fusion performed on the facility and a second fusion performed on the hub; Fig. Figure 5F illustrates an exemplary block diagram of the data intelligence, showing channel state information, with the first and second fusions being performed on the hub; Fig. 5G illustrates an exemplary block diagram of data intelligence, showing channel state information being carried out on the hub; Fig. Figure 6 illustrates an exemplary detail of the components of the fusion block; Fig. Figure 7 illustrates an exemplary sensor transmission plan for a large number of sensors in the system; Fig. Figure 8 illustrates an example message format between personal electronic devices and / or the hub; Fig. Figure 9 illustrates an example of a facility information message; Fig. Figure 10 illustrates an example setup capability query message; Fig. Figure 11 illustrates an example setup message; Fig. Figure 12 illustrates an example of a setup sensor information message; Fig. Figure 13 illustrates an example of personal electronic devices that send facility information messages to the hub; Fig. Figure 14 illustrates an example of the hub sending the setup capability query messages to the personal electronic facilities; Fig. Figure 15 illustrates an example of the facility operating as a hub that broadcasts a facility query message; and Fig. Figure 16 illustrates an example of an ongoing transmission of the activated facility sensor information messages from the personal electronic facilities to the hub. DETAILED DESCRIPTION

[0028] As required, detailed embodiments of the present invention are disclosed herein; however, it is understood that the disclosed embodiments are merely exemplary of the invention, which can be implemented in various and alternative forms. The figures are not necessarily to scale; some features may be enlarged or reduced to show details of certain components. Therefore, the specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching a person skilled in the art to use the present invention in various ways.

[0029] A user can wear multiple personal electronic devices. Each device can be designed to perform a specific function (or set of functions). For example, headphones can be used to receive audio from connected source devices and / or as an input microphone. A smartwatch can be used for biomarker monitoring, interface with a mobile phone, activity detection, and so on. A glucose monitor can monitor blood sugar levels and periodically report them back to a central device. A wearable fitness tracker can count steps or calories burned during an activity.

[0030] These devices have the ability to communicate with other devices to which they are connected. Headphones can provide audio from a connected source device, smartwatches or fitness trackers can provide time and activity data, continuous glucose monitors can report blood glucose levels, and a multi-device wearable posture tracking system can track the posture of the wearer when instructed. To perform their respective functions, each device can implement various sensing modalities, such as an inertial measurement unit (IMU), a pressure sensor, a temperature sensor, a short-range wireless radio, a global navigation satellite system (GNSS) receiver, or a satellite communication module.

[0031] Aspects of the disclosure relate to pooling and analyzing the data generated by the facility from the user's facilities. A fusion of RF channel / communication information (including received energy / signal strength, channel state information, channel impulse response, angle of incidence, central-peripheral range, etc.) with IMU data can be performed to achieve improved activity detection.

[0032] The information available from these devices, such as channel status information (CSI) from the radios, raw IMU outputs, devices to which they are assigned, and so on, can be used to gain more insights into a user's behavior, in addition to monitoring what a single device is intended to do. Collectively, the information from the sensing and communication hardware present in the devices provides rich information about the person wearing them, their actions, and their environment. This can be leveraged using machine learning at the device level and / or centrally. Utilizing the insights derived in this way from a network of such personal devices can enable additional use cases beyond the individual target use cases of each device.The various data points, when combined, can be transformed into actionable output using machine learning techniques, either at the individual device or at a hub (e.g., a facility where aggregated data from individual devices is available for further processing). The hub can be another device on the person or a central, off-body device with which the other devices can communicate, such as a phone, laptop, tablet, desktop computer, cloud server, etc.

[0033] Fig. Figure 1 illustrates an exemplary sensor hardware system 100 for use in performing the data pooling and data analysis discussed here. The sensor hardware system 100 can include spatially distributed personal electronic devices 102 on or near a person to determine various characteristics, actions, and activities of that person. The sensor hardware system 100 can further include a separate hub device 104 that communicates with the personal electronic devices 102 worn by the user.

[0034] The personal electronic devices 102 can be placed directly on the human body (e.g., with or without a housing) or can be embedded in objects worn on the body, such as clothing, shoes, etc. As shown, the user wears a personal headphone device 102A on their head. The user also has a personal chest device 102B located on their chest. The user also has electronic wrist devices 102C, 102D on each wrist and electronic ankle devices 102E, 102F on each ankle. It should be noted that these sensors 202 are merely examples, and more, fewer, or differently arranged personal electronic devices 102 can be used.

[0035] Fig. Figure 2A illustrates an exemplary detail of the components of the personal electronic device 102. As shown, the personal electronic device 102 can include at least one sensor 202, a microcontroller 204 which includes a microprocessor, memory and input / output (I / O) for connecting to other peripheral devices, an energy storage device 206, a transceiver 208 and a data storage device 210.

[0036] The at least one sensor 202 of the personal electronic equipment 102 can be configured to detect various physical phenomena, including but not limited to acceleration, angular velocity, magnetic heading, barometric pressure, temperature, humidity, etc. For example, the personal electronic equipment 102 may include sensors 202 such as a 6 DoF (degrees of freedom) IMU to obtain acceleration and angular rotation velocity, a 9 DoF IMU to obtain the magnitude of the Earth's magnetic field, etc.

[0037] The transceiver 208 can include communication hardware to enable the exchange of information between the personal electronic devices 102 and / or a hub device 104. In addition to its use in sending and receiving data, the transceiver 208 can provide information regarding, among other things, the relative placement between devices, the environment, the energy or power levels used, the direction of communication reception, and so on. For example, communication between the transceivers 208 of the personal electronic devices 102 can provide information regarding: the distance between the personal electronic devices 102, the quality of the received signal between a pair of personal electronic devices 102, the radio channel state estimate representing the environment in which the personal electronic devices 102 are used, and so on.

[0038] It should be noted that there can be several communication modes implemented by one or more transceivers 208 of the personal electronic devices 102. Furthermore, the personal electronic devices 102 in the system 100 can have different communication modes among different groups.For example, an ultra-wideband or nearlink radio can provide the distance between pairs of devices and a CIR representation of the immediate environment; a Bluetooth Low Energy (BLE) or Bluetooth Classic Basic Rate (BR) / Enhanced Data Rate (EDR) radio can provide a received signal strength indicator (RSSI) as a representation of the relative separation between two devices and the type of environment between them (line of sight or obstructed); Wi-Fi can provide the distance between a pair of devices as well as a channel frequency response (CFR) representing the environment between the devices, and so on. All these representations of the RF channel as perceived by the receiver, such as RSSI, CIR, and CFR, are collectively referred to as channel state information (CSI).CSI refers to the RF communication channel between the sending (body device) and receiving device (hub / phone / other endpoint for body device data). The communication interfaces may also have the ability to scan for connectable devices, and the list of detected devices may also be reported information.

[0039] With renewed reference to Fig. 1. The Hub 104 can be configured to receive and aggregate data from other personal electronic devices 102 for further processing. The Hub 104 can be a separate wearable device worn by a person, or it can be integrated into the personal electronic devices 102 worn by the person. For example, the Hub 104 can be integrated into the person's headphones, smartwatch, shoes, belt, etc., or even into the fabric worn by the person. Or, in other examples, the Hub 104 can be a central, external device with which the rest of the personal electronic devices 102, such as a telephone, laptop, tablet, desktop computer, cloud server, etc., can communicate.

[0040] Fig. Figure 2B illustrates an exemplary detail of the components of the Hub 104. Similar to the personal electronic device 102, the Hub 104 can include a microcontroller 204, which contains a microprocessor, memory and I / O for connecting to other peripherals, an energy storage device 206, a transceiver 208, and a data storage device 210. In some examples, the Hub 104 can also include at least one sensor 202.

[0041] With renewed reference to Fig. 1. The Hub 104 can receive data from the sensors 202 of the personal electronic devices 102 and aggregate and store the information in its data storage 210. The Hub 104 can be configured to receive data from all personal electronic devices 102; this requires that the Hub 104 has at least the radio interfaces used by the personal electronic devices 102. For example, if groups of personal electronic devices 102 in the system 100 are equipped with BLE and ultra-wideband (UWB), then the Hub 104 can include at least one BLE transceiver 208 and also one UWB transceiver 208.

[0042] Fig. Figure 3A illustrates an exemplary system architecture 300A in which the personal electronic devices 102 stream information from their sensors 202 to an aggregator personal electronic device 102. In this example, the aggregator personal electronic device 102 performs an aggregation of data from the sensors 202 of the personal electronic devices 102 and streams the aggregated information to an off-body hub 104 for processing. For example, each of the personal electronic devices 102 sends data from its sensors 202 to the aggregator personal electronic device 102G (here a device attached to a belt, but which could be located anywhere on the body) via a respective communication link 302.Additionally, the aggregator personal electronic device 102G transmits the aggregated data via a communication link 302 between the aggregator personal electronic device 102 and the hub 104 located outside the body.

[0043] Fig. Figure 3B illustrates an exemplary system architecture 300B in which the personal electronic devices 102 stream information from their sensors 202 to a portable hub 104 (here a belt device, but it could be located anywhere on the body). In this example, the portable hub 104 performs both the aggregation and processing of the data from the sensors 202. For example, each of the personal electronic devices 102 sends data from its sensors 202 to the portable hub 104 via a respective communication link 302.

[0044] Fig. Figure 3C illustrates an exemplary system architecture 300B in which the personal electronic devices 102 each stream information from their sensors 202 to an extracorporeal hub 104 for processing. In this example, the extracorporeal hub 104 performs both the aggregation and processing of the data from the sensors 202. For example, each of the personal electronic devices 102 sends data from its sensors 202 to the extracorporeal hub 104 via a respective communication link 302.

[0045] Fig. Figure 4 illustrates an exemplary data flow 400 for the operation of the sensor hardware system 100. As shown, sensor data 402 is collected from various sensors 202. This can be performed by a personal electronic device 102 acting as an aggregator and / or by the hub 104. The data aggregation 404 can include temporal synchronization of the sensor data 402 from the various sensors 202, data validation, and / or other operations to facilitate downstream processing of the sensor data 402. The result of the data aggregation 404 is aggregated sensor data 406.

[0046] The aggregated sensor data 406 can be provided to a data intelligence block 408. As discussed in detail here, the data intelligence block 408 can include various operations that are performed by the hardware of the hub 104 and / or the personal electronic devices 102 to process the aggregated sensor data 406. The output of the data intelligence block 408 can include one or more actionable results 410. The one or more actionable results 410 can specify operations to be performed by controlled devices 412 based on the analysis of the aggregated sensor data 406. The controlled devices 412 can also include the personal electronic devices 102 or other devices.

[0047] Fig. Figure 5A illustrates an example block diagram 500A of the data intelligence block 408. As shown, data aggregation 404 of the incoming raw sensor data 402 can be performed as an input to the data intelligence block 408. A machine learning component 502 of the data intelligence block 408 receives the aggregated sensor data 406 and / or the sensor data 402. The machine learning component 502 then extracts a first stage of usable information from the inputs.

[0048] The first stage of data analysis via the machine learning component 502 can provide metadata regarding the radio environment of the personal electronic equipment 102. This section of the machine learning component 502 can be implemented on the device side of the personal electronic equipment 102 in some examples to extract metadata from data-heavy reports (such as CIR data from a UWB transceiver 208, CSI for Nearlink, or CFR from a Wi-Fi transceiver 208, etc.). This environmental metadata can include, for example, information regarding the presence or absence of nearby signal interference, the relative separation between devices, and so on. The machine learning component 502 can then pass this metadata to the next stage of the data intelligence block 408.

[0049] The machine learning component 502 can also perform individual limb movement recognition. This can include determining information about individual limb movements 504 to indicate which limb the user is moving and / or the trajectory of such a movement. The information about individual limb movements 504 can be merged with other information to determine higher-order actions by the user.

[0050] The machine learning component 502 can also perform posture detection. If aggregated sensor data 406 and / or sensor data 402 from the sensors 202 of several personal electronic devices 102 are available, the combined information about the several personal electronic devices 102 can be used to predict the user's current posture. This posture can be provided as posture detection information 506.

[0051] This first level of information, including information on individual limb movements 504 and posture detection information 506, can be fused using a first fusion block 508A to obtain second-level information. The potentially derived information at the second level may include, but is not limited to, activity detection information 510, intent detection information 512, and user detection information 514.

[0052] The activity detection information 510 can indicate an immediate activity that the user is currently engaged in, such as running, walking, swimming, exercising, etc.

[0053] The intent detection information 512 can indicate an immediate intention of the user. For example, the system 100 may observe over time that the user always performs a key-removal gesture before entering a house. Coupled with activity detection information 510, which provides a number of steps taken or a certain number of stairs climbed, this information can be used to predict that the user will attempt to enter the house. The tagging of actions such as entering or leaving the house can be automated based on a specific set of personal electronic devices 102, which are observed by participating personal electronic devices 102 to appear upon entering the house or disappear upon leaving.

[0054] The user identification information 514 can provide an identification of the user based on a usage pattern and relative placement of the personal electronic devices 102 on the user. For example, relative placement metrics of the sensor 202 of the personal electronic devices 102 can be merged with the type of activity detection information 510 performed (e.g., how the user walks or runs, etc.) to identify the user. This follows from human kinematics, which states that each person exhibits a certain signature when walking, climbing, etc.

[0055] The second stage of derived information can then be used in a further round of fusion by a second fusion block 508B. The output of the second fusion block 508B can yield a higher-order prediction of the user's behavior over time. This higher-order prediction can be referred to here as the actionable result 410. For example, if the user is identified as X and it is determined that the intention is "about to enter a car," then the user's identification can be passed to a keyless vehicle access system as additional validation. This ensures that someone else cannot go up to the car and unlock it using the user's phone.

[0056] Fig. 5B- Fig. 5G is illustrated by exemplary block diagrams of various variations of the operation of the Data Intelligence Block 408. As noted here, the operations performed by the Data Intelligence Block 408 can be carried out by the hardware of the Personal Electronic Equipment 102 (referred to here as the Equipment) and / or on the Hub 104 (referred to here as the Hub).

[0057] As in the example 500B from Fig. As shown in Figure 5B, the first stage of data analysis is performed via the machine learning component 502 and the first fusion block 508A, to obtain information for the second stage, by the personal electronic devices 102, while the second fusion block 508B is performed by the hub 104. This is similar to the example 500C from Figure 5B. Fig. As shown in Figure 5C, the first stage of data analysis is performed via the machine learning component 502 on the personal electronic devices 102, while the first fusion block 508A and the second fusion block 508B are performed on the hub. As shown in Example 500D of Fig. As shown in Figure 5D, data aggregation 404 is performed on-site to the personal electronic devices 102, while the entirety of the data intelligence block 508 is performed on the hub 104.

[0058] Fig. Sections 5E-G illustrate additional examples that involve the processing of CSI. As in example 500E from Fig. As shown in Example 5E, the first stage of data analysis is performed by the machine learning component 502 and the first fusion block 508A, which is used to obtain information for the second stage. This fusion block is performed by the personal electronic devices 102, while the second fusion block 508B is performed by the hub 104. Additionally, in Example 500E, the CSI (Configuration, Information, and Data) from the personal electronic devices 102 is provided to the hub 104 for consideration in the second fusion block 508B.

[0059] As in the example 500F from Fig. As shown in Figure 5F, the first stage of data analysis is performed on-site at the personal electronic devices 102 via the machine learning component 502, while the first fusion block 508A and the second fusion block 508B are performed on the hub. In this example 500F, the CSI data from the personal electronic devices 102 are provided to the hub 104 for consideration in the first fusion block 508A.

[0060] As in the example of 500G from Fig. As shown in the 5G example, data aggregation 404 is performed on-site at the personal electronic devices 102, while the entirety of the data intelligence block 508 is performed at the hub 104. In this 500G example, the CSI from the personal electronic devices 102 is also provided to the hub 104 for consideration in the first fusion block 508A.

[0061] Fig. Figure 6 illustrates an exemplary detail of the components of the fusion block 508. The fusion block 508 can be responsible for applying processing steps to a set of input values ​​602 and providing a set of output values ​​604 derived from the provided input values ​​602. The fusion blocks 508 can be applied at any point in the system 100. The components of the fusion block 508 include a time window component 606, a machine learning component 608, a heuristic component 610, and a decision component 612.

[0062] The time-window component 606 can be configured to receive the input values ​​602 and apply a time window of aggregation to the set of input values ​​602 before further processing steps are performed. Using the time-window component 606 allows upstream information applied to the fusion block 508 as the input values ​​602 to be analyzed with temporal context perception, thus enabling higher-order inferences.

[0063] The machine learning component 608 can be configured to apply machine learning techniques to the input values ​​602. The machine learning component 608 can be enabled or disabled by the system 100 designer or as a function of upstream processing blocks, as shown by the illustrated activation line. The models executed by the machine learning component 608 to infer the input values ​​602 can include simple classification models, such as Trees, Random Forest, Support Vector Machine (SVM), etc., or deep learning models, such as Convolutional Neural Networks (CNNs), Transformers, Fluid Neural Networks (LNNs), etc.

[0064] Heuristic component 610 can be configured to analyze the set of input values ​​602 using heuristics derived from domain knowledge. For example, heuristic component 610 can apply a set of business rules in the form of an if-else-then chain to the input values ​​602 to categorize the provided information as output. Heuristic component 610 can be enabled or disabled by the system 100 designer or as a function of upstream processing blocks, as shown by the illustrated activation line.

[0065] Decision component 612 can be configured to decide on the final output values ​​604 of fusion block 508. For example, the machine learning component 608 and the heuristic component 610 of fusion block 508 can both be enabled and operate independently. When both are enabled, the outputs provided by each are fed to decision component 612 to decide on the outcome. Techniques used by decision component 612 to determine the final output can include, but are not limited to, majority voting, weighting, k-means clustering, and other statistical procedures.

[0066] It should be noted that in each of the System 100 components where machine learning is applied, continuous user interaction and system usage enable further tuning of the machine learning models. System 100 allows newer versions of the models to be pushed back to the Sensors 202 and / or Hubs 104 on demand.

[0067] It should also be noted that System 100 does not implicitly obligate the user to always share data from their personal electronic devices 102. The user can fully control the visibility of such personal electronic devices 102 within System 100. The user can choose which personal electronic devices 102 are allowed into System 100 and which data from the available sensor 202 modalities on the personal electronic devices 102 are to be used. System 100, in turn, can inform the user about the levels of service that can be activated or deactivated based on these choices.

[0068] One of the challenges in setting up a System 100, as described here, is being able to connect a separate group of personal electronic devices 102 in a network in order to exchange data elements between the personal electronic devices 102 and / or between the personal electronic devices 102 and the hub 104. Since there may be personal electronic devices 102 using different communication modes, an approach to discovering and connecting devices can operate at a higher level than the physical (PHY) or medium access control (MAC) layer in the Open Systems Interconnection (OSI) architecture.

[0069] Fig. Figure 7 illustrates an exemplary sensor transmission plan for a multitude of sensors 202 of the system 100. As shown, a first sensor 202, S1, transmits with a first period, a first frequency channel, a first power value, and / or a first antenna selection. Additionally, a second sensor 202, S2, transmits with a different period, a different frequency channel, a different power value, and / or a different antenna selection. Furthermore, a third sensor 202, S3, transmits with yet another period, another frequency channel, another power value, and / or another antenna selection.

[0070] System 100 can be configured to use adaptive rate control with respect to the device data reporting intervals. For example, if an IMU sensor 202 on the personal electronic device 102 detects a significant change from a stable state (a state in which the personal electronic device 102 has been for a significant period), the reporting interval can be shortened, thus providing more data for analysis. Likewise, the personal electronic device 102 can also be instructed by the hub 104 to increase the reporting interval to conserve power. Furthermore, the reporting interval itself could be a function of the activity being detected. For example, a high update rate might be unnecessary or redundant for slower movements, but a shorter reporting interval could provide useful information for faster movements.The rate variation can also depend on the location of the sensor 202. For example, a sensor 202 on the wrist may report in every reporting interval, while a sensor 202 on the chest may report in every third reporting interval.

[0071] Another optimization that can be used by the System 100 is adaptive power control, whereby instead of sending each message with the same energy / power, subsequent messages from the sensor 202 to the hub 104 can follow a predetermined variable transmission power pattern to improve the accuracy of RF-based activity detection.

[0072] In another example, similar to adaptive power control, instead of transmitting on the same frequency channel, subsequent messages from the sensor 202 to the hub 104 can follow a predetermined variable transmit frequency channel power pattern to improve the accuracy of RF-based activity detection. While BLE frequency hopping may be built into the protocol, setting a predetermined pattern can further increase accuracy. For other technologies, a frequency hopping pattern can be explicitly provided by the hub 104 to the personal electronic devices 102.

[0073] In yet another example, multiple antennas can be used on the sensor 202 and / or the hub 104, and each of these antennas can be used as a separate training chain and benefit from antenna diversity. The choice of antenna usage can also depend on the activity, with some activities being given preferential antenna selection as an option.

[0074] One option for selecting optimal RF parameters (e.g., power, channel, antenna) may include a pattern communication exchange between the personal electronic devices 102 and the hub 104 at the start of activity detection.

[0075] Fig. Figure 8 illustrates an example message format 800 between personal electronic facilities 102 and / or the hub 104. As shown, the message format 800 includes a header 802, data fields 804, and a cyclic redundancy check (CRC) 806. The header 802 can contain identification information relating to the messages being sent, the data fields 804 can contain the application-level content of the messages, and the CRC 806 can specify parity or other verification information to ensure that the message is not corrupted.

[0076] Messages can be formed in the 800 message format to define standard messages between the personal electronic devices 102 and the hub 104. These messages can be transported using various lower-level communication constructs (Wi-Fi, BLE, UWB, Nearlink, other low-power wireless network protocols, etc.) to form an application-level network of devices. The set of such messages can be referred to here as a profile of a Cooperative Device Sensor Information Exchange Network (CoDeSInExNet).

[0077] While Fig. Section 8 shows a general structure of messages exchanged in System 100; next, specific illustrative examples are provided along with descriptions of individual messages.

[0078] Fig. Figure 9 illustrates an example of a setup information message 900. The setup information message 900 can be exchanged between the personal electronic devices 102 and the hub 104 during a general network setup discovery via any or all available communication interfaces. The formats of the setup information message 900 may vary depending on the implementation; however, information about the sensors 202 and communication interfaces can be included in the data fields 804 of the setup information message 900.

[0079] The facility information message 900 provides information about the message type, facility identification information including vendor ID, product ID, facility hardware ID, etc. Depending on the protocol, the fields of the facility information message 900 can be embedded in the communication protocol facility discovery messages between the personal electronic facilities 102 and the hub 104.

[0080] Another section of the information transmitted by the device information message 900 is information about the sensors 202 available to the communicating personal electronic device 102. Additionally, the information may specify the communication modes available via the personal electronic device 102 (this may include Wi-Fi, BLE, UWB, etc.). If the personal electronic device 102 was previously configured for the system 100, that previous information will also be made available as part of this message.

[0081] Upon receiving the setup information message 900, the endpoint (e.g., the personal electronic devices 102 and / or the hub 104) can parse the information and provide this information to the user interface for validating existing settings or configuring with new settings (for example, if the user accidentally wore the headphones for the left ear in the right ear, this can be corrected; or if a new personal electronic device 102 is used by the user, it can be configured for the system 100).

[0082] Fig. Figure 10 illustrates an example setup capability query message 1000. The setup capability query message 1000 can be broadcast by a personal electronic facility 102 and / or by the hub 104. A facility receiving this message can respond to the same source facility that broadcast the setup capability query message 1000 with a facility information message 900 via the communication interfaces on which it received the message.

[0083] Fig. Figure 11 illustrates an example setup message 1100. The setup message 1100 can be sent via the communication interface(s) through which the hub 104 has established a connection with the terminal equipment (e.g., the personal electronic device 102). The user provides the sensors 202 for monitoring, the radio interfaces for use in information exchange, and the message frequency for the terminal equipment. The message should include the message frequency as the minimum data field 804.

[0084] Fig. Figure 12 illustrates an example setup sensor information message 1200. The setup sensor information message 1200 can be sent from the personal electronic device 102 to the device acting as the hub 104 at a set reporting interval. The sensor data 402 included in the setup sensor information message 1200 is the collected data between the last report and the current time for all active sensors 202 on the personal electronic device 102, as set by the setup setup message 1100. As shown, the sensor data 402 from a single sensor 202 is included in the example setup sensor information message 1200; however, it should be noted that multiple sections of sensor data 402 can be included in the setup sensor information message 1200, allowing for space.In some examples, the sensor data 402 may be too large for a single setup sensor information message 1200 and may span multiple such setup sensor information messages 1200.

[0085] The data from individual sensors 202 contain a timestamp for temporal alignment with the device acting as the hub 104. A common time base can be established by any time synchronization protocols active on the network. If a communication protocol message and a device sensor information message 1200 overlap in transmission time, the communication protocol message takes precedence, and the device sensor information message 1200 is sent at the next available opportunity. This data packet could contain data from multiple sensors 202 on the personal electronic device 102 in a single message or in separate messages. It should contain at least one data length in bytes and, accordingly, sensor data 402 for each sensor 202 of the personal electronic device 102.

[0086] Fig. Figures 13-16 illustrate an exemplary message protocol using the setup information messages 900, setup capability query messages 1000, setup setup messages 1100, and setup sensor information messages 1200.

[0087] Fig. Figure 13 illustrates an example of 1300 personal electronic devices 102-1 to 102-N sending device information messages 900 to the hub 104. These device information messages 900 specify the device-specific interface and capability information of the respective personal electronic devices 102-1 to 102-N. This can be done during a phase of the message protocol in which the personal electronic devices 102-1 to 102-N announce their intention to join a network. The device acting as the hub 104 can store the information received in the device information messages 900.

[0088] When the user activates System 100, the data for the personal electronic devices 102-1 to 102-N can be displayed along with any past configuration information. In some cases, the user may choose to reconfigure the personal electronic devices 102-1 to 102-N as desired, and a new configuration will be created. In other cases, the old configuration will be considered acceptable and reused.

[0089] Fig. Figure 14 illustrates an example 1400 of the Hub 104, which sends the setup capability query messages 1000 to the personal electronic devices 102-1 to 102-N. As shown, the device operating as the Hub 104 device is connected to the personal electronic devices 102-1 to 102-N. The user has requested that a personal electronic device 102 be present in the system 100 and has provided a configuration for it. The configuration can specify, for example, a cadence for receiving sensor data 402 and / or information about which sensor data 402 is desired from the Hub 104. This can be communicated via the connected link(s) to the personal electronic devices 102-1 to 102-N using the device setup messages 1100.

[0090] Fig. Figure 15 illustrates an example 1500 of the facility, which acts as a hub 104 that broadcasts a facility query message 1502. The personal electronic facilities 102-1 to 102-N that receive this message can respond with a facility sensor information message 1200, which is intended for the source of the facility query message broadcast 1502.

[0091] Fig. Figure 16 illustrates an example 1600 of an ongoing transmission of the activated facility sensor information messages 1200 from the personal electronic devices 102-1 to 102-N to the hub 104. Once the personal electronic devices 102-1 to 102-N are connected and configured to be an information source in the system 100, the personal electronic devices 102-1 to 102-N send data collected from the activated sensors 202 via the configured communication interfaces.

[0092] The frequency of such updates is shown in example 1600 as the reporting interval 1602. This reporting interval 1602 may have been previously determined during the setup phase mentioned above. If any of the transmission event times of the reporting interval 1602 overlaps or conflicts with a communication protocol event, then the communication protocol event takes precedence. The conflicting messages are indicated by the dashed arrows. Once the communication protocol event is complete, the setup sensor information message(s) 1200 are then sent. Each of the reports from the sensors 202 within a setup sensor information message 1200 has a timestamp that can be used to align the messages at the facility, which acts as the hub 104.

[0093] Thus, personal electronic devices 102 can transmit their information directly to a device outside the body, such as a mobile phone, a laptop, etc. (as in Fig. 3A) or to a central hub 104 on the body (as in Fig. 3B shown) or to an external device via an internal hub / aggregator / bridge (as shown in Fig. (3C shown) stream. Data from sensors 202 of the personal electronic devices 102 can be analyzed, along with other available inputs, using machine learning algorithms to determine several factors about the person's current state. If the system 100 observes such changes in the person's state over time, a determination about the person's intention can be made, along with establishing a signature that helps identify the person and the environment in which they may be located.

[0094] The disclosed System 100 can be applied in various use cases. For example, in the automotive sector, System 100 can be used for occupancy detection; occupant identification; gait detection; gesture detection; activity detection; as an additional input to existing systems, such as keyless entry; and intent detection. In the building sector, System 100 can be used for gait detection; occupancy detection; activity detection; intent detection; patient activity monitoring in hospitals; monitoring the well-being of elderly people in nursing homes, and so on. In the health and fitness sector, System 100 can be used for consumer activity monitoring; activity detection; intent detection; condition monitoring; and vital sign monitoring.

[0095] Computing devices, such as those discussed here, generally contain computer-executable instructions, which may be executable by one or more computing devices. Computer-executable instructions can be compiled or interpreted from computer programs created using a variety of programming languages ​​and / or technologies, including, without limitation and either alone or in combination, Java™, C, C++, C#, Visual Basic, JavaScript, Python, Perl, etc. Generally, a processor (e.g., a microprocessor) receives instructions, e.g., from memory, a computer-readable medium, etc., and executes these instructions, thereby carrying out one or more processes, including one or more of the processes described here. Such instructions and other data may be stored and transmitted using a variety of computer-readable media.

[0096] With regard to the processes, systems, procedures, heuristics, etc., described herein, it is understood that, although the steps of such processes, etc., have been described as occurring according to a specific, ordered sequence, such processes could be carried out with the described steps in a different order than the one described here. It is further understood that certain steps could be performed simultaneously, that other steps could be added, or that certain steps described herein could be omitted. In other words, the descriptions of processes provided here are for the purpose of illustrating certain embodiments and should in no way be interpreted as limiting the claims.

[0097] Accordingly, it is understood that the above description is intended to be illustrative and not limiting. Many other embodiments and applications than the examples provided would be apparent upon reading the above description. The scope should not be determined by reference to the above description, but instead by reference to the attached claims together with the full scope of equivalents to which such claims entitle. It is expected and intended that future developments will take place in the technologies discussed herein and that the disclosed systems and methods will be incorporated into such future embodiments. Overall, it is understood that the application may be modified and varied.

[0098] All terms used in the claims shall be given their broadest reasonable constructions and ordinary meanings as understood by those skilled in the art of the technologies described herein, unless expressly stated otherwise. In particular, the use of singular articles such as "a", "an", "the", "a", etc., should be read as referring to one or more of the elements indicated, unless a claim expressly limits this interpretation.

[0099] The summary of disclosure is provided to enable the reader to quickly determine the nature of the technical disclosure. It is presented with the understanding that it is not intended to interpret or limit the scope or meaning of the claims. Furthermore, it is clear from the preceding detailed description that various features in different embodiments have been grouped together for the purpose of simplifying the disclosure. This method of disclosure is not to be construed as reflecting an intention that the claimed embodiments require more features than are expressly stated in each claim. Rather, as the following claims reflect, the inventive step lies in fewer than all the features of any single disclosed embodiment.Therefore, the following claims are hereby included in the detailed description, each claim standing alone as a separately claimed subject matter.

[0100] Although exemplary embodiments are described above, it is not intended that these embodiments describe all possible implementations of the disclosure. Rather, the words used in the description are words of the disclosure and not of the limitation, and it is understood that various modifications can be made without deviating from the basic idea and scope of the disclosure. In addition, the features of different implementing embodiments can be combined to form further embodiments of the disclosure.

Claims

[1] Methods for performing data pooling and analysis, comprising: Received, through a hub, from sensor data streamed by a variety of personal electronic devices worn by a user; Performing an initial analysis of the sensor data using an initial data fusion to determine momentary aspects of the user's movement; Conducting a second analysis of the current aspects using a second data fusion to determine behavioral aspects of the user's movement over time; Determining an actionable result based on the second analysis; and Performing one or more operations based on the actionable outcome. [2] Method according to claim 1, wherein the sensor data includes data from an inertial measurement unit (IMU) of one or more IMU sensors of the plurality of personal electronic devices. [3] Method according to claim 2, wherein the IMU data includes acceleration and / or velocity information relating to a movement of the user. [4] Method according to claim 1, wherein the sensor data includes radio frequency (RF) channel information data from one or more RF transmitters of the plurality of personal electronic devices. [5] Method according to claim 4, wherein the RF channel information data specify the distances between pairs of the plurality of personal electronic devices and channel state information (CSI) that is representative of the environment between the plurality of personal electronic devices. [6] Method according to claim 1, wherein the instantaneous aspects include one or more of user characteristics, attitude, actions and / or activities. [7] Method according to claim 1, wherein the behavioral aspects include one or more of user intent determination and / or person identification. [8] Method according to claim 7, wherein the implementable result includes authorizing the user to access a facility or location based on user intent determination and person identification. [9] Method according to claim 1, wherein the plurality of personal electronic devices includes one or more of headphones, a biometric device, a smartwatch and / or a mobile phone. [10] Method according to claim 1, further comprising: Identifying, through the hub, the multitude of personal electronic devices based on advertising messages sent by the respective personal electronic devices; Receiving, at the hub, facility information messages from the multitude of personal electronic devices, the facility information messages specifying facility-specific interfaces and capabilities of the respective personal electronic devices; and Sending, through the hub, configurations to the multitude of personal electronic devices, the configurations specifying a cadence for receiving sensor data and / or information about which elements of the sensor data should be provided to the hub. [11] Method according to claim 1, further comprising: Broadcasting, through the hub, of a facility query message broadcast requesting that the multitude of personal electronic facilities send facility sensor information messages to the hub; Received, through the hub, the facility sensor information messages requested by the multitude of personal electronic devices; and Continue to receive periodic facility sensor information messages from the multitude of personal electronic devices. [12] Method according to claim 11, wherein the plurality of personal electronic devices delays the sending of the periodic device sensor information messages when there is a conflict with protocol messages being sent or received by the plurality of personal electronic devices. [13] System for performing data pooling and analysis, comprising: a multitude of personal electronic devices worn by a user, wherein the multitude of personal electronic devices are configured to generate sensor data relating to the user; and a hub device for wireless communication with a variety of personal electronic devices, where the multitude of personal electronic devices and the hub device are configured to: Performing an initial analysis of the sensor data using an initial data fusion to determine momentary aspects of the user's movement, Conducting a second analysis of the current aspects using a second data fusion to determine behavioral aspects of the user's movement over time, Determining an actionable result based on the second analysis, and Performing one or more operations based on the actionable outcome. [14] System according to claim 13, wherein the sensor data includes data from an inertial measurement unit (IMU) of one or more IMU sensors of the plurality of personal electronic devices. [15] System according to claim 14, wherein the IMU data includes acceleration and / or velocity information relating to a movement of the user. [16] System according to claim 13, wherein the sensor data includes radio frequency (RF) channel information data from one or more RF transmitters of the plurality of personal electronic devices. [17] System according to claim 16, wherein the RF channel information data specify the distances between pairs of the plurality of personal electronic devices and channel state information (CSI) that is representative of the environment between the plurality of personal electronic devices. [18] System according to claim 13, wherein the instantaneous aspects include one or more of user characteristics, attitude, actions and / or activities. [19] System according to claim 13, wherein the behavioral aspects include one or more of user intent determination and / or person identification. [20] System according to claim 19, wherein the implementable result includes authorizing the user to access a facility or location based on user intent determination and person identification. [21] System according to claim 13, wherein the plurality of personal electronic devices includes one or more of headphones, a biometric device, a smartwatch and / or a mobile phone. [22] System according to claim 13, wherein the hub is further configured to: Identifying, through the hub, the multitude of personal electronic devices based on advertising messages sent by the respective personal electronic devices; Receiving, at the hub, facility information messages from the multitude of personal electronic devices, the facility information messages specifying facility-specific interfaces and capabilities of the respective personal electronic devices; and Sending, through the hub, configurations to the multitude of personal electronic devices, the configurations specifying a cadence for receiving sensor data and / or information about which elements of the sensor data should be provided to the hub. [23] System according to claim 13, wherein the hub is further configured to: Broadcast, through the hub, a facility query message broadcast requesting that the multitude of personal electronic facilities send facility sensor information messages to the hub; Received, through the hub, the facility sensor information messages requested by the multitude of personal electronic devices; and Continue to receive periodic facility sensor information messages from the multitude of personal electronic devices. [24] System according to claim 23, wherein the plurality of personal electronic devices delays the sending of the periodic device sensor information messages when there is a conflict with protocol messages being sent or received by the plurality of personal electronic devices. [25] One or more non-volatile, computer-readable media comprising instructions for performing data pooling and analysis, which, when executed by a system incorporating a hub facility and a variety of personal electronic devices, cause the system to perform operations to: Capturing sensor data streamed from a variety of personal electronic devices worn by a user; Performing an initial analysis of the sensor data using an initial data fusion to determine momentary aspects of the user's movement; Conducting a second analysis of the current aspects using a second data fusion to determine behavioral aspects of the user's movement over time; Determining an actionable result based on the second analysis; and Performing one or more operations based on the actionable outcome.

Images