A system for monitoring data quality during a research study or from a care setting using personal devices

Abstract

A technique for monitoring data quality during a research study or from a care setting using body wearable monitors includes storing sensor data, received from the body wearable monitors worn by participants in the research study, into a database. Data quality metrics for the sensor data are derived by determining whether the sensor data falls within specified value ranges. The sensor data stored within the database is annotated with the data quality metrics. Data quality decisions for the sensor data are generated based on data quality rules specified for the research study and applied to the sensor data and the data quality metrics. The data quality decisions are monitored in real-time or on a periodic schedule and a data quality alert is issued when one or more of the data quality decisions falls outside of an acceptable limit specified for the research study.

Description

A SYSTEM FOR MONITORING DATA QUALITY DURING A RESEARCH STUDY OR FROM A CARE SETTING USING PERSONAL DEVICESCROSS REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to U.S. Provisional Application No. 63 / 693,877. filed on September 12. 2024, the contents of which are incorporated herein by reference.TECHNICAL FIELD

[0002] This disclosure relates generally to data quality monitoring for research studies, and in particular but not exclusively, relates to monitoring data quality during a clinical study that employs wearable devices.BACKGROUND INFORMATION

[0003] A clinical study is a type of research study on a population having a goal to better understand the population, the population’s response to certain stimuli or environments, or test how7well a new medical approach works in the population. Accurate and reliable data is of paramount concern to achieve a successful clinical study / trial. The uality of the data collected affects downstream clinical research activities. Errors originating in the data collection process can propagate to these downstream research activities with profoundly negative implications and consequences.

[0004] A framework capable of generating study data quality metrics, monitoring these data quality metrics, identifying anomalies in the study data, filtering bad-quality study data, and timely notifying relevant study team members is desirable. Such a framew ork could provide a way to discover issues when remedial opportunities exist, thereby salvaging an otherwise doomed effort. A framework that increases the overall data quality, confidence in that data, and clinical study success rate is desirable.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Non-limiting and non-exhaustive embodiments of the invention are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified. Not all13960-P401WOinstances of an element are necessarily labeled so as not to clutter the drawings where appropriate. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles being described.

[0006] FIG. 1 is a functional block diagram illustrating a system for monitoring data quality during a research study using body wearable monitors, in accordance with an embodiment of the disclosure.

[0007] FIG. 2 is a flow chart illustrating a method for monitoring data quality during a research study using body wearable monitors, in accordance with an embodiment of the disclosure.

[0008] FIG. 3 is a data structure for storing sensor data annotated with data quality metrics and data quality decision using a common format, in accordance with an embodiment of the disclosure.

[0009] FIG. 4 illustrates an example configurable dashboard for visualizing aggregations of data quality metrics and data quality decisions, in accordance with an embodiment of the disclosure.DETAILED DESCRIPTION

[0010] Embodiments of a system and method of operation for monitoring the quality of data obtained during a research study or from a care setting using personal devices, such as body wearable monitors or cell phones, are described herein. In the following description numerous specific details are set forth to provide a thorough understanding of the embodiments. One skilled in the relevant art will recognize, however, that the techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring certain aspects.

[0011] Reference throughout this specification to “one embodiment'’ or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.23960-P401WO

[0012] Embodiments described herein disclose a technique and system architecture for monitoring data quality during a research study (e.g., clinical study), particularly research studies that use biomarker sensors such as body wearable monitors. However, the techniques can also extend to monitoring data quality of data obtained in a care setting, such as a nursing home. Example body wearable monitors include devices integrated with various biomarker sensors, motion sensors, light sensors, or otherwise. The body wearable monitors may assume various wearable form factors including watches or wristbands, angle bracelets, neckbands, headbands, rings, patches, belts, vests, compression socks, etc. Data collection from body wearable monitors may extend to personal devices that include body wearable monitors, cell phones, or otherwise. Example sensors integrated with the body wearable monitors, or other personal devices, may include an inertial measurement unit (IMU), a photopl ethy smogram (PPG) sensor, electrodermal activity (EDA) sensor, electrocardiogram (ECG) sensor, photosensors, capacitive touch sensors, temperature sensors, pressure sensors, etc. Accordingly, the body wearable monitors may include a variety of sensor types that produce raw sensor data that is analyzed to monitor participants of the research study. The raw sensor data, along with other forms of sensor data (e.g., device logs, configuration data, etc.) may be analyzed to ensure the participants are complying with study procedures, ensure correct operation of the body wearable sensors, or otherwise.

[0013] The framework described herein enables the derivation of data quality' metrics and generation of data quality' decisions, which characterize and qualify the sensor data gathered during a research study. The data quality' metrics and data quality decisions enable real-time monitoring and timely remedial action, as opposed to an after the fact error discovery when remedial action is no longer possible. Embodiments of the framework store the data quality metrics and data quality decisions as annotations in the database alongside the sensor data itself. Byusing a common format to store the annotations and the sensor data, the sensor data can be readily filtered for a user specified level of quality, and data subsets meeting these user selectable quality levels may be exported. The data quality metrics and decisions may be aggregated and visualized in a configurable dashboard. The configurable dashboard may be monitored in real-time by a data quality manager (DQM), while the technical experts of the study remain blinded during the clinical study as required by certain regulatory bodies. The configurable dashboard along33960-P401WOwith automated data quality alerts triggered by the data quality decisions can issue remediation alerts and investigation warnings to the DQM. These and other features of a framework for monitoring data quality during a research study employing body wearable monitors are described below.

[0014] FIG. 1 is a functional block diagram illustrating a sy stem 100 for monitoring data quality during a research study using body wearable monitors 101, in accordance with an embodiment of the disclosure. System 100 includes a database 105, a sensor data interface 1 10, a quality metrics computation module 1 15, derived digital measure algorithms 117, data collection user interface(s) 120, data formatting module 125. quality checker module 130, quality filtering module 135, and a quality metric visualizer 140.

[0015] During the course of a research study, such as a clinical study, a population is observed to better understand the population, the population’s response to certain stimuli or environments, or test how well a new medical approach w orks in the population. During the research study, participants may be asked to wear a body monitor that measures their biomarkers, activities, and compliance with the terms of the research study. System 100 provides mechanisms for collecting sensor data 145 from body w earable monitors 101 along with contextual data 150 from other sources, such as clinical reports, participant surveys, third party reports, or otherwise. The data is stored in tables 106 within database 105 where it can be analyzed for data quality'. In one embodiment, database 105 is an enterprise, cloud-based, relational database (e.g., BigQuery by Google LLC) that stores data using a columnar format and nested or linked tables 106. Of course, other database types and / or format types may be used.

[0016] The sensor data and contextual data stored within database 105 is analyzed by quality metric computation module 115 to derive data quality metrics. Quality checker module 130 may further analy ze the data stored within database 105 along with the data quality metrics to make data quality decisions. Quality checker module 130 performs a sort of level-2 quality check on the data (sensor data 145 and / or contextual data 150) while quality' metrics computation module 115 performs a sort of level- 1 check on sensor data 145. The data quality- decisions may be made in real-time, periodically, or ad-hoc. The data quality decisions and data quality metrics may be visualized in a configurable dashboard 141 using quality metric visualizer 140. In one embodiment, quality metric visualizer 140 is implemented43960-P401WOusing an embedded analytics software solution such as Looker by Google LLC. The data quality metrics and decisions may also drive issuance of automated flags & alarms 131, which prompt investigation or remedial action by a DQM of the research study. Finally, the data quality metrics and data quality decisions may also be stored within tables 106 using the very same format used to store the underlying sensor data. Doing so enables a quality’ filter module 135 to filter the data based upon a user selectable data quality to output data subsets 136 having selectable levels of data quality7.

[0017] FIG. 2 is a flow chart illustrating a process 200 for monitoring data quality during a research study using body wearable monitors 101, in accordance with an embodiment of the disclosure. Process 200 is described with reference to FIGs. 1, 3, and 4. The order in which some or all of the process blocks appear in process 200 should not be deemed limiting. Rather, one of ordinary7skill in the art having the benefit of the present disclosure will understand that some of the process blocks may be executed in a variety of orders not illustrated, or even in parallel.

[0018] In a process block 205, sensor data 145 is collected from body wearable monitors 101 worn by’ participants in the research study. Sensor data 145 is collected by body wearable monitors 101 and automatically communicated over one or more networks (e.g.. WiFi, LTE, the Internet, etc.) to sensor data interface 110, which populates the sensor data into tables 106 within database 105. Sensor data 145 may include a variety of different data types. For example, the sensor data may include raw sensor data from the onboard sensors (e.g., PPG data, IMU data, EDA data, temperature data, etc.). Raw sensor data refers to the unprocessed signals coming from the onboard sensors and flows automatically into database 105 via sync events with sensor data interface 110. In one embodiment, the raw sensor data is time series data. Sensor data 145 may7also include device data about the hardware, firmware, and software of each body wearable monitor 101. Examples of device data include hardware, firmware, software versions, operation logs, crash logs, sync events, battery logs, or otherwise.

[0019] In addition to sensor data 145 from body wearable monitors 101, system 100 collects contextual data 150 from various users including practitioners, study participants, third parties, etc. via data collection user interfaces 120. In one embodiment, data collection user interfaces 120 are web-based interfaces for soliciting contextual data 150 from the various parties. Contextual data 150 may53960-P401WOinclude user profile data (e.g., age, sex, weight, height, study commencement date, study completion date, etc.), clinical data, user feedback, responses to questions or questionnaires, clinical evaluations, etc. Clinical data refers to data collected in the course of a clinic visit (in person or remote) by a participant and includes clinical outcome assessments, patient reported outcomes, clinician reported outcomes, observer report outcomes, and performance outcomes. Clinical data may be used as truth labels, comparators, or a means to assess the performance of derived measures across demographics. In some embodiments, clinical data may be stored in its own table 106 or even a separate database altogether.

[0020] In one embodiment, sensor data 145 (including raw' sensor data and device data) along with the contextual data 150 is stored within database 105 using a common format (process block 215). While sensor data 145 may be directly populated into tables 106 contextual data 150 may first need to be formatted by data formatting module 125 prior to saving into table(s) 106 of database 105.

[0021] Once sensor data 145 and contextual data 150 is formatted and saved into database 105, at least some of sensor data 145 and / or contextual data 150 may be transformed into derivative digital measures such as pulse rate, total sleep time, step counts, etc. via various derived digital measure algorithms 117 (process block 220). Each derived digital measure algorithm 117 requires a specific type and amount of input data, which could include sensor data 145. other derived values, contextual data 150, or various combinations thereof. Typically, the most granular derived data (e.g., 10-second window' step count estimations) is not exported into output data subsets 136 for sharing with partners, but rather, aggregated at an hourly or daily level for output. Some derived digital measures are computed automatically on all incoming data (e.g., step counts, pulse rate) and persisted into database 105. Other derived digital measures are computed on a case-by-case basis and persisted in per-project instances of tables 106.

[0022] In process blocks 225, 230, and 235 quality metrics computation module 115 performs a series of level- 1 checks on the data to generate data quality metrics, which are saved back into database 105 (process block 250) as annotations on the underlying data. In one embodiment, the annotations are saved using a common format within database 105 as used to save the underlying data (e.g.. time series sensor data 145).63960-P401WO

[0023] Returning to process block 225, one type of data quality check performed by quality metrics computation module 115 is a foundational data quality check on sensor data 145. A foundational data quality check is a minimum set of characteristics that a particular data type should have to be useable. A foundational data quality check is a sort of “common sense” check that is performed on a small window of data (e.g., less than 1 hour) for a single body wearable monitor 101. Examples of foundational data quality checks include: 1) raw sensor data exists when it is expected, 2) raw sensor data falls within expected value ranges (e g., PPG sensor data falls within specification ranges), 3) derived digital values from derived digital measure algorithms 117 fall within physiological limits (e.g., pulse rate between 30 and 200 bpm for healthy cohorts or another range for diseased cohorts, etc.), etc. As quality7metrics computation module 115 performs foundational data quality checks, it annotates the associated data in data base 105 with a metric. Quality' checker module 130 monitors the accumulation of the data quality metrics generated byquality metric computation module 115. If the number of foundational data errors accumulating over a specified period of time (e.g.. hours, days, week, etc.) exceeds a threshold set for each error type or groups of error types (decision block 240), then the level-2 check performed by quality- checker module 130 may result in a flag / alarm 131 being issued in a process block 245. Errors in foundational data quality may be perceived as a high priority for triage and resolution. Flags / alarms 131 issued in process block 245 are monitored by a DQM for immediate investigation and / or remediation. Additionally, the data quality decisions being performed by quality- checker module 130 may also be recorded into database 105 as annotations on the underlying or related data.

[0024] Returning to process block 230, another type of data quality check performed by quality metrics computation module 115 is an aggregate data quality- check. An aggregated data quality check is defined as the expected behaviors of a data type when aggregated across devices or on one device over a period of type (e.g., greater than 1 hour). Examples of aggregated data quality checks include: 1) the quantity of raw data samples in a given window falls within an expected range (e.g., based upon know-n sampling rates, etc.), 2) raw sensor data falls w-ithin expected means or standard deviations for a given study, 3) derived metrics, such as daily step count values, fall within the expected distribution across users in a given study, 4) identify windows where accelerometry data exceeds threshold values for73960-P401WOextended periods, 5) identify windows of PPG data saturated at maximum value for extended periods, etc. As qualify metrics computation module 115 performs aggregated data quality checks, it annotates the associated data in database 105 with a metric. Qualify checker module 130 monitors the accumulation of the data qualify metrics generated by qualify metric computation module 115. If the number of aggregated data errors accumulating over a specified period of time (e.g, hours, days, week, etc.) exceeds a threshold set for each error type or groups of error types (decision block 240), then the level-2 check performed by qualify checker module 130 may result in a flag / alarm 131 being issued in a process block 245. Errors in aggregated data quality may require immediate triage and resolution, especially for issues aggregating in a single body wearable monitor 101. Issues aggregating across many body wearable monitors 101 may be interpreted as warnings suggesting investigation by the DQM. In some embodiments, the underlying aggregated data qualify checks may also be performed directly by qualify checker module 130. As mentioned above, the data quality decisions being performed by quality checker module 130 may also be recorded into database 105 as annotations on the underlying or related data.

[0025] Returning to process block 235, yet another type of data qualify check performed by quality metrics computation module 115 is a cascading data quality check. A cascading data quality check encompasses assessing the relationships between different types of sensor data that are interdependent. Cascading data qualify involves assessing relationships between interdependent ty pes of the data generated by derived digital measures algorithms 117. Examples of cascading data quality checks include: 1) if IMU data exists, then an associated ambulatory classification label should also exist, 2) if step counts exist in a 10- second window, then the associated ambulatory classification should be “ambulatory,” 3) assessing the impact of qualify / confidence level for interbeat interval (IBI) computation on downstream pulse rate qualify / confidence, 4) assessing the impact of noise in raw IMU data on downstream activity classification or step counting algorithms, etc. As qualify metrics computation module 115 performs cascading data qualify checks, it annotates the associated data in database 105 with a metric. Qualify checker module 130 monitors the accumulation of the data quality metrics generated by quality metric computation module 115. If the number of cascading data errors exceeds thresholds set for the study (decision block 240), then83960-P401WOthe level-2 check performed by quality checker module 130 may result in a flag / alarm 131 being issued in a process block 245. Errors in cascading data quality may require immediate triage and resolution, especially for issues like examples 1 and 2 where data is missing during a window in which we expect it to exist based on the availability' of other interdependent data. As mentioned above, the data qualitydecisions performed by quality checker module 130 may also be recorded into database 105 as annotations on the underlying or related data.

[0026] As sensor data 145 and / or contextual data 150 accumulates, it is populated into database 105, analyzed by quality metrics computation module 115 and quality- checker module 130, and annotated with data quality metrics and data quality decisions. FIG. 3 is an example data structure 300 stored into database 105 for storing sensor data 145 annotated with data quality metrics (e.g., DQM1, DQM2, etc.) and data quality decision (e.g., DQD1, DQD2, etc.) using a common format, in accordance with an embodiment of the disclosure. By storing the data qualityannotations 305 using a common format as the sensor data 145, and even the contextual data 150 in some situations, the data can be filtered (process block 260) to generate data subsets 136 (process block 265) upon request (decision block 255). Exported data subsets 136 can have user defined levels of quality- by filtering using data quality annotations 305 including the data quality metrics and the data quality decisions that fall within the user specified level of quality.

[0027] In general, the highest priority- data quality issues to escalate by the DQM are those that identify a particular body wearable monitor 101 in the field is broken or needs replacement, unexpected missing data that is incoming from sensor data interface 110, or unexpected missing data that is outgoing from qualify filtering module 135 when generating data subsets 136 in process block 265. A remediation alert may be issued in response to an error in the aggregated data quality aggregated over time for a given one of body wearable monitors 101 while an investigation warning may be issued in response to an error in the aggregated data quality aggregated across a plurality of the body wearable monitors 101. In various embodiments, data quality decisions generated by qualify checker module 130 may include analyzing sensor data 145 from different sensor types disposed onboard one or more body- wearable monitors 101 and identifying combinations of the sensor data, occurring contemporaneously or in a specified sequence, that are indicative of a data93960-P401WOquality error. These data quality decisions may be based upon user generated rules that are generic across research studies or unique to a specific research study.

[0028] Similarly, the data quality metrics derived by uality metrics computation module 115 may be generic across research studies, or uniquely specified for a given research study. The data quality metrics may be derived from combinations and sequences of a variety of types of sensor data 145 including the raw sensor data, derived sensor data, and system data, as well as from contextual data 150. For example, low frequency data readings received from a given body wearable monitor 101 may be used to identify overlapping data in higher frequency readings. In one embodiment, data quality metrics for sensor data 145 are derived by analyzing battery readings associated with a given one of the body wearable monitors 101 to determine a battery reading frequency for the given one of the body wearable monitors 101. An overlapping data flag may be asserted when the battery reading frequency exceeds a threshold. The overlapping data flag may be recorded into database 105 as a data quality annotation 305 that annotates at least some of the sensor data associated with the given body wearable monitor 101 while the battery reading frequency exceeds the threshold. The overlapping data flag (annotation) may even be used to disambiguate valid sensor data from erroneous sensor data annotated with the overlapping data flag by assuming monotonically changing battery readings for the given body wearable monitor 101. For example, if the charge level is expected to be monotonically changing during a period of time when many sensor readings were acquired, spurious battery charge readings during that period may be interpreted as redundant or overlapping data and annotated as such.

[0029] FIG. 4 illustrates a configurable dashboard 400 for visualizing aggregations of data quality metrics and data quality decisions, in accordance with an embodiment of the disclosure. In particular, FIG. 4 illustrates configurable dashboard 400 setup to display various data quality metrics and data quality decisions related to overlapping data annotations. In the example, 80 body wearable monitors 101 are being tracked and 12.5% of those tracked devices have data quality annotation related to an overlapping data flag. While configurable dashboard 400 is merely demonstrative, the dashboard can be configured to display various aggregations of data quality metrics including percent of devices with annotations vs devices by firmware, active devices by firmware, percentage of time devices have annotations, longest annotation duration, annotation count by firmware, etc.103960-P401WOAccordingly, the illustrated embodiment of dashboard 400 visualized to the DQM that a significant number of body wearable monitor 101 are having overlapping data error flags, which should cause the DQM to investigate the source of the error and potentially filter out the data annotated with the particular error. FIG. 4 is merely intended to illustrate how the data quality7metrics and data quality decision may be aggregated and visualized for periodic and / or real-time monitoring by a DQM. Other graphs, tables, or visualization techniques may be implemented.

[0030] The processes explained above are described in terms of computer software and hardware. The techniques described may constitute machineexecutable instructions embodied within a tangible or non-transitory machine (e.g., computer) readable storage medium, that when executed by a machine will cause the machine to perform the operations described. Additionally, the processes may be embodied within hardware, such as an application specific integrated circuit (“ASIC’') or otherwise.

[0031] A machine-readable storage medium includes any mechanism that stores information in a non-transitory form accessible by a machine (e.g., a computer, network device, personal digital assistant, manufacturing tool, any device with a set of one or more processors, etc.). For example, a machine-readable storage medium includes recordable / non-recordable media (e.g., read only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.).

[0032] The above description of illustrated embodiments of the invention, including what is described in the Abstract, is not intended to be exhaustive or to limit the invention to the precise forms disclosed. While specific embodiments of, and examples for, the invention are described herein for illustrative purposes, various modifications are possible within the scope of the invention, as those skilled in the relevant art will recognize.

[0033] These modifications can be made to the invention in light of the above detailed description. The terms used in the following claims should not be construed to limit the invention to the specific embodiments disclosed in the specification. Rather, the scope of the invention is to be determined entirely by the following claims, which are to be construed in accordance with established doctrines of claim interpretation.113960-P401WO

Claims

CLAIMSWhat is claimed is:

1. A computer implemented method for monitoring data quality during a research study or from a care setting using body wearable monitors, the method comprising: stonng sensor data, received from the body wearable monitors worn by participants in the research study or the care setting, into a database; deriving data quality metrics for the sensor data by determining whether the sensor data falls within specified value ranges; annotating the sensor data stored within the database with the data quality metrics; storing contextual data associated with each of the body wearable monitors into the database, wherein the contextual data is acquired from one or more sources other than the body wearable monitors; generating data quality decisions for the sensor data based on data quality rules specified for the research study or the care setting and applied to the sensor data, the contextual data, and the data quality7metrics; monitoring the data quality decisions in real-time or on a periodic schedule; and issuing a data quality alert when one or more of the data quality decisions falls outside of an acceptable limit specified for the research study or the care setting.

2. The computer implemented method of claim 1, further comprising: annotating the sensor data in the database with the data quality decisions.

3. The computer implemented method of claim 2, wherein the sensor data comprises time series sensor data and wherein the data quality decisions and the data quality metrics are saved in the database using a common format with the time series sensor data.

4. The computer implemented method of claim 2, further comprising: receiving a request for a subset of the sensor data and the contextual data stored in the database having a user specified level of quality;123960-P401WOfiltering the sensor data and the contextual data based upon the data quality metrics and the data quality decisions that fall within the user specified level of qualify: and exporting the subset of the sensor data and the contextual data having the user specified level of qualify based upon the filtering.

5. The computer implemented method of claim 1. wherein deriving the data qualify' metrics comprises: deriving at least one aggregated data qualify metric that defines an aggregated data qualify of the sensor data either aggregated across a plurality of the body wearable monitors or aggregated over time for a given one of the body wearable monitors.

6. The computer implemented method of claim 5, wherein issuing the data qualify alert comprises: issuing a remediation alert in response to a first error in the aggregated data qualify' aggregated over time for the given one of the body wearable monitors; or issuing an investigation warning in response to a second error in the aggregated data quality aggregated across the plurality of the body wearable monitors.

7. The computer implemented method of claim 1, wherein the sensor data includes raw sensor data and derived sensor data, the method further comprising: generating derived sensor data based on the raw sensor data; determining cascading data qualify for the derived sensor data by assessing relationships betw een interdependent types of the sensor data; and issuing a cascading data qualify alert in response to an error in the cascading data quality.

8. The computer implemented method of claim 1, further comprising: presenting a configurable dashboard that visualizes aggregations of one or more types of the data qualify metrics and one or more types of the data quality decisions.133960-P401WO9. The computer implemented method of claim 1, wherein generating the data quality decisions comprises: analyzing the sensor data from different sensors disposed onboard one or more of the body wearable monitors; and identifying combinations of the sensor data, occurring contemporaneously or in a specified sequence, that are indicative of a data quality error.

10. The computer implemented method of claim 1, wherein deriving the data qualify metrics for the sensor data comprises: analyzing batten- readings associated with a given one of the body wearable monitors to determine a battery reading frequency for the given one of the body wearable monitors; asserting an overlapping data flag when the battery reading frequency exceeds a threshold.

11. The computer implemented method of claim 10, further comprising: annotating at least some of the sensor data associated with the given one of the body wearable monitors, other than the battery readings, with the overlapping data flag while the battery reading frequency exceeds the threshold.

12. The computer implemented method of claim 1 1 , further comprising: disambiguating valid sensor data from erroneous sensor data annotated with the overlapping data flag by assuming monotonically changing battery readings for the given one of the body wearable monitors.

13. At least one machine-readable storage medium having instructions stored thereon that, in response to execution, cause a computing system to perform operations for monitoring data qualify during a research study using body wearable monitors, the operations comprising: storing sensor data, received from the body wearable monitors worn by participants in the research study, into a database; deriving data qualify metrics for the sensor data by determining whether the sensor data falls within specified value ranges;143960-P401WOannotating the sensor data stored within the database with the data quality metrics; generating data quality decisions for the sensor data based on data quality rules specified for the research study and applied to the sensor data and the data quality metrics; monitoring the data quality decisions in real-time or on a periodic schedule; and issuing a data quality alert when one or more of the data quality decisions falls outside of an acceptable limit specified for the research study.

14. The at least one machine-readable storage medium of claim 13, wherein the operations further comprise: annotating the sensor data in the database with the data quality decisions.

15. The at least one machine-readable storage medium of claim 14, wherein the sensor data comprises time series sensor data and wherein the data quality decisions and the data quality metrics are saved in the database using a common format with the time series sensor data.

16. The at least one machine-readable storage medium of claim 14, wherein the operations further comprise: receiving a request for a subset of the sensor data and the contextual data stored in the database having a user specified level of quality; filtering the sensor data and the contextual data based upon the data quality metrics and the data quality decisions that fall within the user specified level of quality; and exporting the subset of the sensor data and the contextual data having the user specified level of quality based upon the filtering.

17. The at least one machine-readable storage medium of claim 13, wherein deriving the data quality metrics comprises: deriving at least one aggregated data quality metric that defines an aggregated data quality of the sensor data either aggregated across a plurality of the body153960-P401WOwearable monitors or aggregated over time for a given one of the body wearable monitors.

18. The at least one machine-readable storage medium of claim 17, wherein issuing the data quality alert comprises: issuing a remediation alert in response to a first error in the aggregated data quality aggregated over time for the given one of the body wearable monitors; or issuing an investigation warning in response to a second error in the aggregated data quality aggregated across the plurality of the body wearable monitors.

19. The at least one machine-readable storage medium of claim 13, wherein the sensor data includes raw sensor data and derived sensor data, the method further comprising: generating derived sensor data based on the raw sensor data; determining cascading data quality for the derived sensor data by assessing relationships between interdependent ty pes of the sensor data; and issuing a cascading data quality alert in response to an error in the cascading data quality.

20. The at least one machine-readable storage medium of claim 13, wherein deriving the data quality' metrics for the sensor data comprises: analyzing batten' readings associated with a given one of the body wearable monitors to determine a battery reading frequency for the given one of the body wearable monitors; asserting an overlapping data flag when the battery reading frequency exceeds a threshold.

21. The at least one machine-readable storage medium of claim 20, wherein the operations further comprise: annotating at least some of the sensor data associated with the given one of the body wearable monitors, other than the battery readings, with the overlapping data flag while the battery reading frequency exceeds the threshold.163960-P401WO22. The at least one machine-readable storage medium of claim 21, wherein the operations further comprise: disambiguating valid sensor data from erroneous sensor data annotated with the overlapping data flag by assuming monotonically changing battery readings for the given one of the body wearable monitors.173960-P401WO

Images