Techniques for analyzing non-compliant wearable device data

Abstract

A method is provided for determining a quality of at least some data acquired by a plurality of sensors, the method including: receiving data acquired by the plurality of sensors during an acquisition period from a plurality of sensors associated with a plurality of subjects, the data being indicative of signals related to the bodies of the plurality of subjects; dividing the acquisition period into a plurality of sub-periods; aggregating at least a portion of the data into a plurality of data subsets, each of the plurality of data subsets including data acquired during a different one of the plurality of sub-periods; determining, for each particular sub-period of the plurality of sub-periods, a quality of the subset of data acquired during the particular sub-period; and outputting an indication of the determined quality of at least one of the plurality of data subsets.

Description

[Background technology]

[0001] Sensors are used to collect data about the human body. Sensors may detect physiological or biodynamic signals, such as acceleration, ECG signals, temperature, or glucose levels. The signals may be used to monitor a patient's overall health, monitor fluctuations in certain parameters, and detect the onset of disease, among other applications. Summary of the Invention [Means for solving the problem]

[0002] According to an exemplary embodiment of the present disclosure, a method is provided that includes: receiving, using one or more processors, data from a plurality of actigraphy sensors associated with a plurality of subjects, the data being acquired by the plurality of actigraphy sensors during an acquisition period, the data being indicative of a signal related to physical motion of the plurality of subjects, the receiving including receiving first data from a first actigraphy sensor of the plurality of actigraphy sensors associated with a first subject of the plurality of subjects, the first data being acquired by the first actigraphy sensor, the first data being indicative of a signal related to physical motion of the first subject; dividing the acquisition period into a plurality of sub-periods; and filtering at least a portion of the data, including the first data, by a plurality of sub-periods. The method includes: aggregating the data into a plurality of data subsets, each of the plurality of data subsets including data acquired during a different sub-period of the plurality of sub-periods; determining, for each particular sub-period of the plurality of sub-periods, a quality of the subset of data acquired during the particular sub-period, wherein determining includes determining whether a first subject of the plurality of subjects was compliant in using a first actigraphy sensor of the plurality of actigraphy sensors during the particular sub-period; and outputting an indication of the determined quality of at least one of the plurality of data subsets.

[0003] According to another embodiment of the present disclosure, a system is provided, including a memory storing instructions and a processor configured to execute the instructions to implement a method, the method including receiving data from a plurality of actigraphy sensors associated with a plurality of subjects, the data acquired by the plurality of actigraphy sensors during an acquisition period, the data indicative of a signal related to physical movement of the plurality of subjects, the receiving including receiving first data from a first actigraphy sensor of the plurality of actigraphy sensors associated with a first subject of the plurality of subjects, the first data acquired by the first actigraphy sensor, the first data indicative of a signal related to physical movement of the first subject; dividing the acquisition period into a plurality of sub-periods; and converting at least a portion of the data, including the first data, into a plurality of data sub-periods. aggregating the data into a plurality of data subsets, each of the plurality of data subsets including data acquired during a different sub-period of the plurality of sub-periods; determining, for each particular sub-period of the plurality of sub-periods, a quality of the subset of data acquired during the particular sub-period, wherein determining includes determining whether a first subject of the plurality of subjects was compliant in using a first actigraphy sensor of the plurality of actigraphy sensors during the particular sub-period; and outputting an indication of the determined quality of at least one subset of data of the plurality of data subsets.

[0004] According to another embodiment of the present disclosure, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium includes instructions that, when executed by one or more processors on a computing device, are operable to cause the one or more processors to perform a method including receiving data from a plurality of actigraphy sensors associated with a plurality of subjects, the data being acquired by the plurality of actigraphy sensors during an acquisition period, the data being indicative of a signal related to physical motion of the plurality of subjects, the receiving including receiving first data from a first actigraphy sensor of the plurality of actigraphy sensors associated with a first subject of the plurality of subjects, the first data being acquired by the first actigraphy sensor, the first data being indicative of a signal related to physical motion of the first subject; dividing the acquisition period into a plurality of sub-periods; and converting at least a portion of the data, including the first data, into a plurality of data sub-periods. aggregating the data into a plurality of data subsets, each of the plurality of data subsets including data acquired during a different sub-period of the plurality of sub-periods; determining, for each particular sub-period of the plurality of sub-periods, a quality of the subset of data acquired during the particular sub-period, wherein determining includes determining whether a first subject of the plurality of subjects was compliant in using a first actigraphy sensor of the plurality of actigraphy sensors during the particular sub-period; and outputting an indication of the determined quality of at least one subset of data of the plurality of data subsets. [Brief explanation of the drawings]

[0005] Additional embodiments of the present disclosure, and its features and advantages, will become more apparent by reference to the description herein taken in conjunction with the accompanying drawings, in which elements are not necessarily drawn to scale. Moreover, in the drawings, like reference numerals designate corresponding parts throughout the different views. [Figure 1A]FIG. 1 illustrates an exemplary method for determining the quality of data acquired by a sensor, according to some embodiments. [Figure 1B] FIG. 1 illustrates an exemplary method for determining the quality of data acquired by a sensor, according to some embodiments. [Figure 2A] FIG. 1 is a block diagram illustrating an example system for determining the quality of data acquired by a sensor, according to some embodiments. [Figure 2B] FIG. 1 is a block diagram illustrating an example system for determining the quality of data acquired by a sensor, according to some embodiments. [Figure 3A] 1 is a flowchart illustrating an exemplary computerized method for determining the quality of data acquired by a sensor, according to some embodiments. [Figure 3B] 1 is an exemplary plot illustrating data acquired by a sensor during an acquisition period, according to some embodiments. [Figure 3C-1] 1 is an exemplary graphical user interface (GUI) illustrating the quality of data acquired by a sensor, according to some embodiments. [Figure 3C-2] 1 is an exemplary graphical user interface (GUI) illustrating the quality of data acquired by a sensor, according to some embodiments. [Figure 3C-3] 1 is an exemplary graphical user interface (GUI) illustrating the quality of data acquired by a sensor, according to some embodiments. [Figure 3C-4] 1 is an exemplary graphical user interface (GUI) illustrating the quality of data acquired by a sensor, according to some embodiments. [Figure 3C-5]1 is an exemplary graphical user interface (GUI) illustrating the quality of data acquired by a sensor, according to some embodiments. [Figure 3C-6] 1 is an exemplary graphical user interface (GUI) illustrating the quality of data acquired by a sensor, according to some embodiments. [Figure 3C-7] 1 is an exemplary graphical user interface (GUI) illustrating the quality of data acquired by a sensor, according to some embodiments. [Figure 3C-8] 1 is an exemplary graphical user interface (GUI) illustrating the quality of data acquired by a sensor, according to some embodiments. [Figure 4A] FIG. 1 is a block diagram illustrating subjects participating in one or more clinical studies, according to some embodiments. [Figure 4B] 1 is an exemplary report showing compliance in using sensors of subjects participating in a clinical study, according to some embodiments. [Figure 4C] 1 is an exemplary report showing compliance in using sensors for subjects associated with a particular clinical study site, according to some embodiments. [Figure 4D-1] 10 is an exemplary report showing an individual subject's compliance in using their sensor, according to some embodiments. [Figure 4D-2] 10 is an exemplary report showing an individual subject's compliance in using their sensor, according to some embodiments. [Figure 5] 1 is a flowchart illustrating an exemplary computerized method for determining a subject's pattern of non-compliance when using a sensor, according to some embodiments. [Figure 6A]10 is an exemplary report showing the extent to which a first subject was compliant in using a sensor during each hour of multiple days of an acquisition period, according to some embodiments. [Figure 6B] 1 is an exemplary table showing whether a first subject was compliant in using a sensor during each hour of multiple days of an acquisition period, according to some embodiments. [Figure 6C] 10 is an exemplary table showing one hour of days when a first subject was frequently non-compliant in using a sensor over multiple days, according to some embodiments. [Figure 6D] 1 is an exemplary table showing days on which a first subject was frequently non-compliant in using a sensor, according to some embodiments. [Figure 6E] 10 is an exemplary plot showing a trend in a first subject's compliance when using a sensor, according to some embodiments. [Figure 7A] 10 is an exemplary report showing the extent to which a second subject was compliant in using the sensor during each hour of multiple days of the acquisition period, according to some embodiments. [Figure 7B] 10 is an exemplary table showing whether a second subject was compliant in using the sensor during each hour of multiple days of the acquisition period, according to some embodiments. [Figure 7C] 10 is an exemplary table showing one hour of days when a second subject was frequently non-compliant in using a sensor over multiple days, according to some embodiments. [Figure 7D] 10 is an exemplary table showing days when a second subject was frequently non-compliant in using a sensor, according to some embodiments. [Figure 7E] 10 is an exemplary plot showing a trend in compliance of a second subject when using a sensor, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0006] For the purposes of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings and specific language will be used to describe the same, it being understood nevertheless that no limitation of the scope of the invention is intended.

[0007] Provided herein are techniques for determining quality of data acquired by one or more sensors. The data can be acquired during an acquisition period. For example, the sensors can be configured to detect signals associated with a subject's body during the acquisition period. In some embodiments, the techniques include (a) dividing the acquisition period into multiple subperiods, (b) aggregating the data acquired by the sensors into multiple subsets of data, and (c) determining quality of each subset of data. In some embodiments, determining the quality of the subset of data includes determining whether the subject used the sensor during a particular subperiod.

[0008] Relevant clinical trials enable sensor-based collection of physiological signals for the research and development of digital biomarkers (dBMs), which are objective, quantifiable physiological and behavioral characteristics that can explain and predict health-related outcomes that cannot be captured during in-clinic visits or through questionnaires.

[0009] The computer-based platform can be configured to capture and process data from sensors in real time for clinical trials. The inventors recognized that developing and discovering novel dBMs is a hypothesis-driven research process that involves processing unprecedented amounts of digital data generated from digital technologies. As one non-limiting example, at a sampling frequency of 50 Hz, over 4 million triaxial data points are collected from a single patient's accelerometer to understand that patient's daily activities. Accessing, processing, understanding, and visualizing such a large amount of data is resource-intensive and inefficient.

[0010] Furthermore, effective results depend on trusted and understood data collected from sensors and digital devices. However, relevant clinical trials are often conducted under free-living conditions, with subjects wearing sensor devices on a best-effort basis as they attempt to follow instructions communicated or provided to the subjects during study enrollment. Such conditions can result in problems such as missing or invalid data when subjects are not wearing the devices or are wearing them incorrectly. Missing and invalid data can introduce inaccuracies into dBMs derived from such data because they do not accurately reflect the subject's physical state. For example, if a subject does not wear sensors for a significant portion of the day, the resulting acceleration data may suggest that the subject is inactive during that portion of the day, which may not actually be an accurate reflection of the subject's activity.

[0011] Therefore, the present inventors have developed a technology that addresses the above-mentioned limitations of conventional biosensing technologies. The technology can automatically analyze the quality of acquired data and provide an indication of data quality (e.g., including a graphical display). In some embodiments, the technology includes determining the quality of data acquired by one or more sensors worn by a user. As used herein, the "quality" of data acquired by such sensors can refer to the suitability of such acquired data for drawing scientific and / or medical conclusions. Quality can be determined for data during an acquisition period, such as the time a user is tasked with wearing a sensor to collect data for one or more clinical trials. For example, determining the quality of the data can include determining whether the subject was using the sensor properly, whether the device was charged (or not), errors (e.g., connectivity, synchronization with a data source, etc.), whether the data contains nonsense values ​​(e.g., indicating a damaged device), whether the data contains values ​​outside the dynamic output range of the sensor, or any other suitable quality metric. The data can be analyzed as part of a computer-based platform that ingests and processes data from user-worn sensors. In some embodiments, the quality of the data may be used to determine whether the data should be used in downstream analysis, such as to monitor the health of a subject, to develop biomarkers, or for any other suitable application. The data can be analyzed in real time as part of the platform.

[0012] The quality of the data can be assessed in various ways in accordance with the techniques described herein. The data can be assessed based on the test from which the data is being collected. For example, a user may be instructed to wear a sensor while sleeping, exercising, etc. The technology can aggregate the data and assess quality over time (e.g., over 5 minutes, a day, a week, etc.) and / or space (e.g., different geographic sites or locations, such as by city, state, country, etc.).

[0013] In some embodiments, determining the quality of data acquired by sensors during an acquisition period includes (a) dividing the acquisition period into subperiods, (b) aggregating the data into subsets of data, each subset of data including data acquired during a different subperiod, and (c) determining the quality of each subset of data. In some embodiments, aggregating the data in this manner enables efficient and accurate evaluation of the data without analyzing each data point acquired by each sensor during the acquisition period. For example, in a large-scale clinical study, a single data subset may represent data acquired by thousands of sensors during a particular period of the clinical study, allowing for a single determination of the quality of that data. As described herein, despite the data representing a large amount of information, among other benefits, the quality determination can be used to understand the quality of the dataset at a more holistic level. Such overall quality information can be used to develop clinical trials, including more accurately determining biomarker values ​​and predicting health outcomes.

[0014] In some embodiments, the technology provides for viewing and analyzing reports of data quality analysis. Data can be viewed and filtered in different ways, such as by calendar day and / or adjusted study date. Quality can be provided based on a time period, such as based on minutes per day (e.g., minutes of good quality data or as a percentage of minutes per day). In some embodiments, reports can be generated or viewed at a patient level, visit level, and / or time period level (e.g., daily). Thus, data can be aggregated in various ways (e.g., based on study, site, and / or user) and reported at different levels (e.g., daily, visit, and / or user) to provide in-depth, custom data analysis, which can improve clinical trial development.

[0015] In particular, quality analysis can be used to improve the development and / or execution of clinical trials by informing when data should be acquired and / or used for the clinical trial. This can help avoid acquiring or using data during times when subjects are typically non-compliant in using sensors. Being able to analyze when data should be used or acquired can result in one or more improvements to the operation of a clinical trial. For example, the efficiency of a clinical trial can be increased because less time is required to filter large amounts of acquired data to distinguish between data representing biological signals and bad or unusable data, such as data representing sensors not being used by the subject or data generated by a broken or malfunctioning sensor. As another example, the accuracy of metrics determined using the acquired data can be improved because the acquired data represents biological signals (e.g., body movement) as opposed to signals representing non-use of a sensor by the subject. Quality analysis can additionally or alternatively be used to improve the accuracy of biomarker values ​​determined using the acquired data. In particular, quality analysis can be used to identify low-quality data and exclude low-quality data from the determination of biomarker values. By excluding low-quality data, such as data representing sensor malfunction or non-use by the subject, and including only data representing the subject's bodily signals, the accuracy of biomarker values ​​determined using such data is improved.

[0016] While users can attempt to manually review quality analysis reports to visually identify patterns in the data, the inventors recognized that this is often inefficient and impractical, and that such an approach may also risk missing important patterns. For example, consider a clinical trial with hundreds or thousands of participants, each of whom may use sensors for days, weeks, months, or even longer periods. Such a large number of participants may result in a large amount of data, especially given the long study duration. Attempting to visually identify reports generated for each individual participant would be time-intensive and inaccurate, given the sheer volume of data and the presence of subtle patterns within the data. Furthermore, a human reviewer would likely need to trigger a next step to address the pattern of non-compliance, introducing additional inefficiencies (and possible errors) into the overall process.

[0017] Accordingly, the inventors have developed techniques for automatically determining a subject's patterns of non-compliance (or compliance) when using a sensor and summarizing such patterns of non-compliance for use by a user or clinical trial sponsor. For example, the techniques may include automatically identifying specific periods (e.g., days, hours of the day, weeks, etc.) during which a subject is frequently non-compliant when using a sensor. Such techniques have several practical applications, as further described below.

[0018] For example, in some embodiments, the technology can be used to efficiently enhance subject compliance when using a sensor (e.g., during a clinical trial). For example, determined patterns of non-compliance can be used to automatically trigger a notification to the subject (e.g., upon detection and / or during times when the subject is found to be frequently non-compliant). The notification can be in the form of an automatic text message, phone call, email, push notification, and / or any other suitable type of notification. The notification can prompt the subject to use the sensor during times when the subject is found to be typically non-compliant. By automatically reviewing patterns of non-compliance and prompting subjects to use the sensor in real time during a clinical trial, the technology can be used to obtain more usable data throughout the trial, thereby improving the overall quality of the data collected.

[0019] Additionally or alternatively, techniques for determining patterns of noncompliance can be used to understand the functionality of a sensor and / or a device comprising the sensor (e.g., a wearable device housing the sensor). For example, regular periods of noncompliance when using the sensor can be correlated to the amount of time a subject spends charging the device. This information can be used to understand device characteristics, such as average battery life and the time required to recharge the battery. Additionally or alternatively, this information can be used to design a second device and / or a second clinical trial with improved characteristics. For example, a second device can be deployed to participants and used to determine patterns of noncompliance when using the second device. The patterns of noncompliance when using the second device can be compared to the patterns of noncompliance when using the first device to measure the extent to which the second device improves over the first device. For example, if the regular periods of noncompliance when using the second device are shorter than the regular periods of noncompliance when using the first device, this can indicate that the second device has a shorter recharge time.

[0020] While various embodiments have been described, it will be apparent to those skilled in the art that many more embodiments and implementations are possible. Accordingly, the embodiments described herein are examples and are not intended to be the only possible embodiments and implementations. Furthermore, the advantages described above are not necessarily the only advantages, and it is not necessarily expected that all of the described advantages will be achieved in each embodiment.

[0021] 1A and 1B illustrate an example method 100 for determining the quality of data 120 acquired by a sensor 115, according to some embodiments.

[0022] As shown, the exemplary method 100 includes receiving data 120 from a sensor 115 associated with a subject 110. As described herein, in some embodiments, the data 120 is received by one or more processors configured to perform one or more steps of the exemplary method 100. For example, the one or more processors may be configured to determine a quality of the received data 120 (e.g., step 130) and output an indication 140 of the determined quality.

[0023] In some embodiments, the sensor 115 is worn by or implanted in the body of the subject 110. For example, the sensor 115 may be included in a device that takes the form of an accessory worn by the subject (e.g., a watch, eyeglasses, jewelry, etc.), a body-worn device (e.g., a patch attached to the subject's skin), a device embedded in the subject's clothing, an ear-worn device, an implantable device, or any other suitable type of device, as aspects of the technology described herein are not limited to any particular type of device.

[0024] In some embodiments, the sensor 115 is configured to detect signals related to the body of the subject 110. By way of non-limiting example, the sensor 115 may include an actigraphy sensor, an electrocardiogram (ECG) sensor, a blood glucose sensor, a thermometer, an electromyogram (EMG) sensor, a tissue oximeter, a pulse oximeter, a respiration rate sensor, a heart rate sensor, a skin sweat sensor, a motion sensor, an accelerometer, a position sensor, or any other suitable sensor, as aspects of the technology described herein are not limited in this respect.

[0025] In some embodiments, the sensor 115 is configured to acquire the signal during an acquisition period. For example, the sensor 115 may be configured to acquire the signal continuously during the acquisition period, or to acquire the signal periodically (e.g., periodically or intermittently) during the acquisition period. In some embodiments, the acquisition period is of any suitable duration, as aspects of the technology described herein are not limited in this respect. For example, the acquisition period may be in units of seconds, minutes, hours, days, weeks, months, or years.

[0026] In some embodiments, data 120 represents signals detected by sensor 115. For example, data 120 may include raw signal data acquired by sensor 115. Additionally, or alternatively, data 120 may include data processed using any suitable signal processing technique, as aspects of the technology described herein are not limited to any particular signal processing technique. As a non-limiting example, as shown in FIG. 1A , data 120 may include data indicative of the acceleration of sensor 115.

[0027] In some embodiments, at step 130 of exemplary method 100, one or more processors determine the quality of the received data 120. For example, determining the quality of the received data may include determining whether the subject 110 was compliant in using the sensor 115 during the acquisition period, determining whether the sensor 115 malfunctioned during the acquisition period, or determining any other suitable metric indicative of the quality of the data, as aspects of the technology described herein are not limited in this respect. Exemplary techniques for determining the quality of such data are described herein, including with respect to at least FIGS. 3A-3C .

[0028] In some embodiments, after determining the quality of the data in step 130, the one or more processors output an indication 140 of the determined quality. For example, the output may indicate whether the subject 110 was compliant in using the sensor 115 during the acquisition period, whether the sensor 115 malfunctioned during the acquisition period, and / or any other suitable indication of the determined quality of the data, as aspects of the technology described herein are not limited in this respect. FIG. 1B illustrates an example output 140 of the example method 100. The example output 140 identifies specific times on specific days during which the subject was compliant in using the sensor. In the example shown in FIG. 1B, each column is associated with a separate test day of the clinical trial, and each row is associated with a different time of a 24-hour day. Each cell in the illustrated matrix is ​​given a color or shade that corresponds to the quality of the data collected from the particular subject 110 during that associated day and time. For example, a lighter color may indicate that good quality data was collected, and a darker color may indicate that poor quality data was collected.

[0029] In some embodiments, outputting the indication 140 of the determined quality includes generating a graphical user interface that includes the indication, generating a report that includes the indication, transmitting the indication to another device, storing the indication, or outputting the indication according to any other suitable technique, as the aspects of the technology described herein are not limited in this respect.

[0030] Additionally or alternatively, in some embodiments, the determined data quality is used in operation 150 to determine a pattern of the subject's non-compliance (or compliance) when using the sensor 115. In some embodiments, determining a pattern of the subject's non-compliance includes identifying one or more days during the acquisition period during which the subject was frequently non-compliant when using the sensor. For example, this may include determining that the subject was relatively less compliant when using the sensor on particular days, such as the first, middle, and / or last days of the acquisition period (e.g., during the beginning and / or end of the acquisition period, or during one or more days when the patient was unable to wear the sensor). Additionally or alternatively, in some embodiments, determining a pattern of the subject's non-compliance includes identifying one or more hours during the day during which the subject was relatively less compliant when using the sensor over the acquisition period. For example, this may include determining that the subject was generally less compliant when using the sensor early in the day, in the middle of the day, and / or late in the day. Exemplary techniques for determining a subject's pattern of non-compliance are described herein, including with respect to at least Figures 3A, 5, and 6A-7E.

[0031] In some embodiments, the determined subject's pattern of non-compliance can be used to provide the subject with specific instructions regarding how to use the sensor. For example, the determined subject's pattern of non-compliance may be used to prompt the subject to use the sensor during days and / or times when the subject was previously non-compliant. Additionally, or alternatively, the determined subject's pattern of non-compliance can be used by another user (e.g., a clinical trial administrator, a healthcare provider, etc.) to instruct the subject to use the sensor and / or to design or improve future clinical trials. As yet another example, the determined subject's pattern of non-compliance can be used to determine data that will not be used as part of a clinical trial. Modifying and / or adjusting sensor usage can directly improve the data collected by the sensor, which, in turn, will improve the accuracy with which the data is used to determine biomarkers and predict health outcomes.

[0032] 2A is a block diagram illustrating an example system 200 for determining the quality of data acquired by a sensor, according to some embodiments. As shown, system 200 includes a sensor 215, a computing device 230, and a network 220. However, it should be understood that a system for determining the quality of data acquired by a sensor may include one or more additional or alternative components, as aspects of the technology described herein are not limited in this respect.

[0033] In some embodiments, sensor 215 (e.g., sensor 115 shown in FIG. 1A) is associated with subject 210. In some embodiments, sensor 215 includes one or more sensors associated with a single subject. For example, one or more sensors may be used to detect different biodynamic and / or physiological signals associated with the subject's body. Additionally or alternatively, sensor 215 may include multiple sensors associated with multiple subjects. For example, sensors may be used to monitor multiple subjects during a clinical study.

[0034] In some embodiments, after acquiring a signal related to the subject's body, the sensor 215 transmits data indicative of the acquired signal to the computing device 230 over the network 220. In some embodiments, the sensor 215 transmits the data in real time. For example, the sensor 215 may transmit the data in response to detecting a signal. Additionally or alternatively, in some embodiments, the sensor 215 transmits the data within a threshold time (e.g., within seconds, minutes, hours, etc.) of detecting the signal. Additionally or alternatively, in some embodiments, the sensor 215 transmits the data in response to a request to transmit the data. For example, the sensor 215 may receive a request from the computing device 230 and / or via a user interface associated with the sensor 215.

[0035] In some embodiments, the data indicative of the acquired signal includes raw (e.g., unprocessed) signal data. Additionally, or alternatively, the data indicative of the detected signal includes signal data that has been processed using any suitable signal processing technique, as aspects of the technology described herein are not limited in this respect. For example, the signal may be processed to ensure that it is suitable for transmission over network 220.

[0036] Network 220 may be or may include a wide area network (e.g., the Internet), a local area network (e.g., a corporate Internet), and / or any other suitable type of network. Sensors 215 and / or computing devices 230 may connect to network 220 using one or more wired links, one or more wireless links, and / or any suitable combination thereof. Thus, network 220 may be, for example, a hardwired network (e.g., a local area network within a medical facility), a wireless network (e.g., connected via Wi-Fi and / or a cellular network), a cloud-based computing network, or any combination thereof.

[0037] In some embodiments, computing device 230 is used to determine the quality of data received from sensor 215. For example, determining the quality of the received data may include determining whether subject 210 was compliant in using sensor 215 during the acquisition period, determining whether sensor 215 malfunctioned during the acquisition period, or determining any other suitable metric indicative of the quality of the data, as aspects of the technology described herein are not limited in this respect. Exemplary techniques for determining the quality of such data are described herein, including with respect to at least FIGS. 3A-3C .

[0038] In some embodiments, computing device 230 includes one or more computing devices. If computing device 230 includes multiple computing devices, the devices may be physically co-located (e.g., in a single room) or distributed across multiple physical locations. In some embodiments, computing device 230 may be part of a cloud computing infrastructure. In some embodiments, one or more computing devices 230 may be co-located within a facility operated by an entity.

[0039] In some embodiments, computing device 230 is associated with user 235. For example, user 235 may include researchers and / or medical professionals. Such researchers and / or medical professionals may monitor data acquired by sensor 215 to determine whether subject 210 is compliant in using sensor 215, monitor the health of subject 210, and / or determine whether sensor 215 is functioning properly. Additionally or alternatively, in some embodiments, user 235 includes subject 210. For example, subject 210 may use sensor data to monitor his or her health.

[0040] In some embodiments, computing device 230 is further configured to receive input from user 235 and / or generate output. For example, as described herein, including with respect to at least FIG. 2B , computing device 230 may include a user interface configured to receive user input and / or display output to user 235. As described herein, in some embodiments, the user input indicates one or more criteria for determining the quality of data from sensor 215. Additionally or alternatively, in some embodiments, the user input is used to generate output according to settings of user 235.

[0041] 2B is a block diagram of the computing device 230 shown in FIG. 2A, according to some embodiments. In some embodiments, the computing device 230 includes software 280 configured to perform various functions with respect to data received from the sensor 215. In some embodiments, the software 280 includes multiple modules. The modules may include processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform the functions of the module. Such modules are sometimes referred to herein as "software modules." As described herein, the system of FIG. 2A may be used, for example, to provide a computer-based platform for collecting data for clinical trials.

[0042] As shown in FIG. 2B, in some embodiments, software 280 includes a user interface module 282, a quality determination module 284, a report generation module 286, and a pattern determination module 288.

[0043] In some embodiments, user interface module 282 is a graphical user interface (GUI), a text-based user interface, and / or any other suitable type of interface to which a user may provide input and / or receive output. For example, in some embodiments, user interface module 282 may be a web page or web application accessible via an internet browser. In some embodiments, user interface module 282 may be the GUI of a software application (app) running on a user's mobile device. In some embodiments, user interface module 282 may include several selectable elements with which a user can interact. For example, user interface module 282 may include a drop-down list, a check box, a text field, or any other suitable element, as aspects of the technology described herein are not limited in this respect.

[0044] In some embodiments, the quality determination module 284 processes the data acquired by the sensor 215 to determine the quality of the data. For example, the quality determination module 284 may determine whether the subject was compliant in using the sensor 215 during the acquisition period, whether the sensor 215 malfunctioned during the acquisition period, and / or any other suitable metric indicative of the quality of the data acquired by the sensor 215.

[0045] In some embodiments, processing the data includes (a) dividing the acquisition period into multiple sub-periods; (b) aggregating the data into multiple data subsets, each subset of data including data acquired during a different sub-period; and (d) determining a quality of the subset of data including data acquired during a particular sub-period. For example, the sub-periods may be determined based on user input provided via user interface module 282. Additionally or alternatively, the quality of the subset of data may be determined based on one or more criteria provided via user interface module 282 and / or stored in data store 272. Techniques for processing data to determine the quality of the data are described herein, including with respect to at least FIGS. 3A-3C .

[0046] In some embodiments, the quality determination module 284 obtains sensor data from the sensors 215 and / or from a data store 272 configured to store data from the sensors 215. For example, the software 280 may include one or more interface modules (not shown), such as a sensor interface module and / or a data store interface module. The sensor interface module may be configured to obtain (pull or be provided) data from the sensors 215. The data store interface module may be configured to obtain (pull or be provided) data from the data store 272. The data may be provided over the network 220.

[0047] In some embodiments, data store 272 includes one or more data stores configured to store data from sensors 215, data related to clinical studies, and / or data obtained via user interface module 282. Data store 272 may include any suitable data store, such as a flat file, a database, a multi-file, or any suitable type of data storage, as aspects of the technology described herein are not limited to any particular type of data store.

[0048] In some embodiments, data associated with a clinical study may include identifying information for subjects participating in the clinical study, information about different clinical study sites, and / or any other suitable information related to the conduct of the clinical study, as the aspects of the technology described herein are not limited in this respect.

[0049] In some embodiments, the data obtained via user interface module 282 includes criteria for determining the quality of the data acquired by sensor 215. For example, as described herein, the criteria may indicate how to divide the acquisition period (e.g., the duration of sub-periods of the acquisition period), the type and / or amount of data to be processed to determine the quality of the data, the type of quality metric to be determined, one or more thresholds to compare the data to, and / or any other suitable type of criteria, as aspects of the technology described herein are not limited in this respect.

[0050] In some embodiments, the pattern determination module 288 is configured to determine a pattern of non-compliance for the subject based on the quality of the data determined by the quality determination module 284. In some embodiments, the pattern determination module 288 obtains the quality data from the quality determination module 284 and / or the data store 272. In some embodiments, the pattern determination module 288 is configured to perform frequent itemset mining to identify one or more periods (e.g., one or more days, one or more hours during a day, etc.) during which the subject is non-compliant in using the sensor. Techniques for determining a pattern of non-compliance for the subject are described herein, including with respect to at least Figures 3A and 6A-7E.

[0051] In some embodiments, the report generation module 286 generates a report indicating the quality of the data determined using the quality determination module 284. For example, the report generation module 286 may generate a report indicating the subject's compliance in using the sensor 215. Additionally or alternatively, the report generation module 286 may generate a report indicating whether the sensor 215 malfunctioned during the acquisition period. Additionally or alternatively, the report generation module 286 may generate a report indicating a pattern of non-compliance by the subject, as determined by the pattern determination module 288. Additionally or alternatively, the report generation module 286 may generate a report that includes instructions to a user (e.g., a subject) on how to use the sensor. However, it should be understood that the report generation module 286 may generate any suitable type of report, as aspects of the present technology are not limited in this respect. Exemplary reports are described herein, including with respect to at least FIGS. 3C and 4B-4D.

[0052] In some embodiments, the report generated by the report generation module 286 is output using any suitable output technique, such as, for example, outputting the report via the user interface module 282, storing the report in the data store 272, and / or transmitting the report to another device.

[0053] 3A is a flowchart illustrating an example method 300 for determining the quality of data acquired by a sensor, according to some embodiments. Method 300 may be implemented on any suitable processor, such as, for example, computing device 230 shown in FIGS. 2A and 2B and / or any other suitable processor, as aspects of the technology described herein are not limited in this respect.

[0054] In step 302, the processor receives data acquired by multiple sensors associated with multiple subjects during an acquisition period from multiple sensors. For example, the data acquired by the multiple sensors may be indicative of biokinetic and / or physiological signals associated with the subjects' bodies. The sensors may include sensors 115 shown in FIG. 1A , sensors 215 shown in FIGS. 2A and 2B , actigraphy sensors, electrocardiogram (ECG) sensors, blood glucose sensors, thermometers, electromyogram (EMG) sensors, tissue oximeters, pulse oximeters, respiration rate sensors, heart rate sensors, skin sweat sensors, motion sensors, accelerometers, position sensors, or any other suitable sensors, as aspects of the technology described herein are not limited in this respect.

[0055] In some embodiments, the plurality of subjects includes two or more subjects, each using one or more respective sensors. For example, each subject may use one or more sensors to monitor their own individual health. Additionally or alternatively, each subject may use one or more sensors as part of a clinical study. For example, the plurality of subjects may be participating in different clinical trials and / or the same clinical trial. Subjects participating in the same clinical study may be associated with different clinical study sites and / or the same clinical study site. Exemplary clinical study structures are described herein, including with respect to at least FIG. 4A .

[0056] In some embodiments, receiving the data includes receiving the data in real time (e.g., as the sensor acquires the data), within a threshold time of the sensor acquiring the data (e.g., within seconds, minutes, hours, days, etc.), in response to a request by the processor, and / or in response to a user input requesting transmission of the data. In some embodiments, the processor receives the data over a network, such as network 220 shown in FIG. 2A.

[0057] In some embodiments, the acquisition period is the period during which data is acquired by the sensor. The acquisition period may be of any suitable duration, as aspects of the present technology are not limited in this respect. For example, FIG. 3B illustrates an initial time t o Starting at time t and ending at time t f 1 illustrates an exemplary acquisition period 360 ending at . The initial and final times may be specified using any suitable technique, as aspects of the technology are not limited in this respect. For example, if data is received in real time, the acquisition period may include the elapsed time between the initial time and the current time. Additionally or alternatively, the acquisition period may be specified by a user and / or by a computing device (e.g., a user may provide user input indicating the initial and final times of the acquisition period).

[0058] In step 304, the processor divides the acquisition period into multiple sub-periods. In some embodiments, a sub-period of an acquisition period refers to a period within the acquisition period that is shorter in duration than the duration of the acquisition period. For example, as shown in FIG. 3B , acquisition period 360 is divided into multiple sub-periods, including sub-period 370. The sub-periods may be of any suitable duration within the acquisition period, as aspects of the present technology are not limited in this respect. As a non-limiting example, a one-month acquisition period may be divided into sub-periods of one week duration, one hour duration, one minute duration, one second duration, or any other suitable duration.

[0059] In some embodiments, the acquisition period is divided into multiple sub-periods based on user input. For example, a user (e.g., user 235 shown in FIGS. 2A and 2B) can indicate the duration of a sub-period. As a non-limiting example, a user, such as a researcher, may be interested in determining the quality of the data from moment to moment and therefore may indicate that the acquisition period should be divided into sub-periods each one hour in duration. In this example, based on the user input, the processor may divide the acquisition period into sub-periods each one hour in duration.

[0060] In step 306, the processor aggregates at least some of the data into multiple subsets of data. For example, the subsets of data may include data acquired during a particular subperiod. For example, as shown in FIG. 3B, subset 380 of data includes data acquired during subperiod 370 of acquisition period 360. In some embodiments, the subset of data includes data acquired by a single sensor during the particular subperiod. Additionally or alternatively, in some embodiments, the subset of data includes data acquired by multiple sensors during the particular subperiod.

[0061] At step 308, for each particular sub-period of the plurality of sub-periods, the processor determines the quality of the subset of data acquired during that particular sub-period. For example, with reference to FIG. 3B , the processor may determine the quality of subset of data 380. While various examples of determining the quality of a subset of data are described herein, it should be understood that aspects of the techniques described herein are not limited to any particular technique for determining the quality of a subset of data, and therefore any suitable technique may be implemented to determine the quality of the data.

[0062] In some embodiments, determining the quality of the subset of data includes determining whether one or more subjects were compliant in using one or more sensors during a particular subperiod. For example, if the sensors include an accelerometer, determining whether a subject was compliant in using the sensors during a particular subperiod may include evaluating acceleration data acquired by the accelerometer during the particular subperiod. This may, in some embodiments, include evaluating the standard deviation and / or range of values ​​of the data included in the subset of data acquired during the particular subperiod. For example, if the standard deviation of the acceleration data acquired during the particular subperiod is less than a threshold value (e.g., a compliance threshold), this may indicate that the subject was not using the sensors during the particular subperiod. In some embodiments, the threshold value includes any suitable value, such as a value indicated by a user via user input, as aspects of the present technology are not limited in this respect. Examples of determining whether a subject was compliant in using the sensors during a particular subperiod of an acquisition period are described herein, including at least the "Example 4 - Exemplary Technique for Determining Data Quality" section.

[0063] In some embodiments, determining the quality of the subset of data includes determining whether one or more sensors malfunctioned during a particular subperiod. For example, in some embodiments, determining whether a sensor malfunctioned includes determining whether the subset of data acquired during a particular subperiod includes values ​​outside the sensor's output range. For example, if the data includes values ​​that exceed the sensor's maximum output value or if the data includes values ​​that are below the sensor's minimum output value, this may indicate that some of the data is invalid. In some embodiments, if the subset of data includes values ​​that are outside the output range, such values ​​may be filtered out and excluded from further analysis. Additionally or alternatively, the processor may output an indication that the subset of data includes invalid values.

[0064] In some embodiments, determining whether one or more sensors have malfunctioned includes determining the percentage of values ​​in the subset of data that are at or near (e.g., within a threshold) the maximum or minimum output value of the sensor. If a threshold percentage of values ​​in the subset of data are at or near the maximum or minimum output value of the sensor, this may indicate that “clipping” has occurred and that the data is invalid. For example, if at least 40%, at least 45%, at least 50%, at least 55%, at least 60%, at least 65%, at least 70%, at least 75%, at least 80%, at least 90%, or at least 100% of the values ​​in the subset of data are at or near the maximum (or minimum) output value of the sensor, the processor may determine that clipping has occurred and that the sensor has malfunctioned during a particular sub-period. Examples of determining whether a sensor has malfunctioned during a particular sub-period are described herein, including at least those in connection with the “Example 4—Exemplary Technique for Determining Data Quality” section.

[0065] Further, in some embodiments, determining the quality of a subset of data may include aggregating quality metrics derived from multiple periods within a particular subperiod. For example, if a particular subperiod is a day, determining the quality of a subset of data corresponding to that day may include first determining the quality of data for multiple periods within that day (e.g., using any of the methods described above, such as determining whether a sensor has malfunctioned, detecting “clipping,” etc.). Each period within a particular subperiod may span 1 minute, 5 minutes, 10 minutes, 1 hour, or any other suitable period. In some embodiments, each such period within a particular subperiod may be referred to as an “epoch.” Determining the quality of data for each epoch may include determining a binary indication as to whether the data collected within that epoch was of good quality or poor quality, or may include a value (e.g., a numerical scale from 0 to 10) indicating the value of the data collected within that epoch in a more granular manner. Once indicators of data quality for each epoch within a particular subperiod have been determined, these indicators may be aggregated to determine a single indication of data quality for the entire particular subperiod (e.g., the entire day). For example, this aggregation may include calculating an average or median mean of the indicators for each epoch within the particular subperiod. Alternatively or additionally, this aggregation may include an indication of the number of epochs within the particular subperiod that had “good” data, e.g., data meeting some predetermined threshold criteria. Alternatively or additionally, this aggregation may include an indication of whether the number of epochs with good data is above or below a certain predetermined threshold (e.g., whether the subject collected at least 6 hours of good data during the day, or whether at least 50% of the epochs had good data). While the examples presented above correspond to scenarios where a particular subperiod corresponds to a day and epochs correspond to 1 minute, 5 minutes, 10 minutes, or 60 minutes, it should be understood that subperiods and epochs may be set to different lengths.For example, a sub-period may correspond to 5 days, 1 week, 2 weeks, or 1 month, and an epoch may correspond to a period of 12 hours, 24 hours, or 2-3 days.

[0066] At step 310, the processor outputs an indication of the determined quality of at least one subset of data of the plurality of subsets of data. In some embodiments, the processor outputs the indication using any suitable output technique, as aspects of the technology described herein are not limited to any particular technique. For example, the processor may output the indication of quality via a user interface such as the user interface module 282 shown in FIG. 2B, the exemplary GUI shown in FIG. 3C, or through any other suitable user interface. Additionally or alternatively, in some embodiments, outputting the indication includes storing the indication. For example, the indication may be stored in a data store such as the data store 272 shown in FIG. 2B. Additionally or alternatively, outputting the indication includes transmitting the indication to another device, such as another computing device, a printer, or any other suitable device for viewing, processing, and / or storing the indication.

[0067] An indication of data quality may improve the accuracy of determining biomarker values ​​and predicting health outcomes. For example, the indication of quality may be used to exclude low-quality data from such biomarker calculations or health outcome predictions. For example, low-quality data may reflect data acquired during times when the subject was not using the sensor and / or when the sensor was malfunctioning. Data acquired during these times may not accurately reflect the subject's biosignals (e.g., body movements) and therefore may contribute to inaccurate biomarker estimation and health outcome predictions based on the biosignal data.

[0068] Additionally or alternatively, the indication of data quality may be used to instruct the subject on how to use the sensor and / or to address any issues of sensor malfunction. Addressing such issues will help improve the quality of future acquired data. This, in turn, will improve the accuracy of biomarker value determinations and health outcome predictions because such determinations and predictions will be based on a larger amount of high-quality data acquired during the acquisition period.

[0069] In step 312, the processor optionally determines a pattern of non-compliance for one or more subjects in using each of one or more sensors of the plurality of sensors based on the quality determined for each sub-period in step 308. For example, in some embodiments, the output of step 308 includes, for each sub-period, an indication of whether the subject was compliant in using the sensor during the particular sub-period. In step 312, the indication of subject compliance can be used to determine a pattern of non-compliance over the acquisition period.

[0070] In some embodiments, determining patterns of compliance includes (a) determining subsets of acquisition periods to evaluate for patterns of non-compliance, (b) calculating a support value for each determined subset based on the quality determined in operation 308, (c) comparing the support value for each subset of data to a threshold (e.g., a user-defined threshold), and (d) identifying a set of subsets whose support values ​​exceed the threshold. Exemplary techniques for determining patterns of non-compliance are described herein, including with respect to at least Figures 5-7E.

[0071] In some embodiments, the pattern of non-compliance indicates a period during which the subject was frequently non-compliant in using the sensor during the acquisition period. For example, the pattern of non-compliance may indicate a set of one or more days during the acquisition period during which the subject was frequently non-compliant in using the sensor. Additionally or alternatively, the pattern of non-compliance may indicate a set of one or more hours per day during the acquisition period during which the subject was frequently non-compliant in using the sensor. Additionally or alternatively, the pattern of non-compliance may indicate any other suitable period during which the subject was frequently non-compliant in using the sensor during the acquisition period, as aspects of the technology described herein are not limited in this respect.

[0072] In step 314, the processor optionally outputs an indication of the determined pattern of non-compliance. In some embodiments, this indication may be output together with or independently of the quality indication output in step 310. In some embodiments, the processor outputs the indication using any suitable output technique, as aspects of the technology described herein are not limited to any particular technique. For example, the processor may output the indication of the pattern via a user interface, such as user interface module 282 shown in FIG. 2B , or through any other suitable user interface. Additionally or alternatively, in some embodiments, outputting the indication includes storing the indication. For example, the indication may be stored in a data store, such as data store 272 shown in FIG. 2B . Additionally or alternatively, outputting the indication includes transmitting the indication to another device, such as another computing device, a printer, or any other suitable device for viewing, processing, and / or storing the indication.

[0073] In some embodiments, the indication of the determined pattern of non-compliance includes instructions for the subject. For example, the instructions may prompt the subject to use the sensor during one or more instances in which the pattern indicates that the subject was not compliant in past sensor use. Providing instructions to the subject may improve the quality of data obtained during future uses of the sensor, as the data no longer contains as much "missing" data during the period of non-compliance. The instructions may be provided directly to the subject (e.g., via a notification or report) or indirectly to the subject (e.g., through a healthcare provider or clinical trial administrator). For example, the instructions may be provided in the form of a text message, phone call, email, push notification, and / or any other suitable type of notification, as aspects of the technology described herein are not limited in this respect.

[0074] In some embodiments, instructions may be automatically provided to the subject upon detection of a pattern of non-compliance. Additionally or alternatively, instructions may be provided to the subject after a threshold period of time has elapsed since the pattern of non-compliance was first detected. For example, the present technology may include initially detecting a pattern of non-compliance and, if the subject still exhibits the pattern of non-compliance after the threshold period of time has elapsed, providing instructions to the subject. The threshold period of time may include any suitable threshold, as aspects of the technology described herein are not limited in this respect. For example, the threshold period of time may be at least 1 day, at least 5 days, at least 1 week, at least 2 weeks, at least 1 month, at least 2 months, 6 hours to 1 year, 1 day to 6 months, 5 days to 2 months, 1 week to 1 month, or any other suitable threshold period of time.

[0075] Additionally or alternatively, in some embodiments, the technology includes providing instructions in different formats depending on the subject's historical pattern of non-compliance. For example, in some embodiments, if a pattern of non-compliance is detected for a subject less than a threshold number of times, instructions may be provided to the subject via an automatic notification (e.g., an automatic text message, email, push notification, etc.). If a pattern of non-compliance is detected for a subject more than a threshold number of times, this may indicate that a higher level of intervention is needed, and instructions may be provided to the subject in a different format. For example, in addition to or as an alternative to sending an automatic notification to the subject, the automatic notification may be sent to a researcher and / or administrator, who may then arrange for a person (e.g., a healthcare provider, clinical trial administrator, etc.) to contact the subject directly to provide instructions for using the sensor and / or to solicit input regarding why the subject is not using the sensor.

[0076] Additionally or alternatively, in some embodiments, outputting an indication of the determined pattern of non-compliance in operation 314 may include outputting an indication of whether the subject has improved in sensor use relative to the subject's past use. For example, a first pattern of non-compliance may have been previously determined for the subject. The first pattern of compliance may be compared to the pattern of compliance determined in step 312 of process 300 (e.g., a second pattern of non-compliance) to determine the degree of improvement (or non-improvement or deterioration) in the subject's sensor use since the first pattern of non-compliance was determined. If the results of the comparison indicate that the subject has not improved in sensor use, an intervention may be triggered (e.g., instructions for using the sensor may be provided to the subject).

[0077] In some embodiments, method 300 may be implemented to determine a first pattern of non-compliance of one or more subjects when using a sensor included in a first device. In some embodiments, the first pattern of non-compliance may be used to determine a characteristic of the first device. For example, periodic periods of non-compliance may indicate the amount of time the subject spent charging the first device. In some embodiments, these periods of non-compliance may be used to estimate battery life and / or the amount of time required to charge the first device.

[0078] The determined characteristics of the first device may, in some embodiments, be used to design a second device. For example, the second device may be designed to improve on the characteristics of the first device. The second device may be distributed to one or more subjects (e.g., the same subject who used the first device or different subjects). Method 300 may be repeated to determine a second pattern of non-compliance of one or more subjects when using a sensor included in the second device. In some embodiments, the second pattern of non-compliance may be compared to the first pattern of non-compliance. The results of the comparison may be used to determine whether one or more characteristics of the second device are improving relative to the characteristics of the first device. For example, if the second pattern of non-compliance indicates that periodic periods of non-compliance are less frequent or of shorter duration, this may indicate that the second device has a longer battery life and / or a shorter charging time and / or that the second device is more convenient for a user to charge and use in a compliant manner.

[0079] 3C is an exemplary graphical user interface (GUI) showing the quality of data acquired by a sensor, according to some embodiments. As shown, the GUI displays a compliance heat map and a compliance bar plot. The compliance heat map includes a row for each of multiple subjects and a column for each day of the acquisition period. The compliance heat map shows, for each subject, the percentage of minutes of the day that the subject used the sensor for each day using different colors and / or shading. For example, cooler colors (e.g., blue and / or green) may indicate higher compliance, while warmer colors (e.g., red and / or orange) may indicate lower compliance. The compliance bar plot shows, for each subject, the average number of minutes each day that the sensor was used during the acquisition period. In some embodiments, a user of the GUI may interact with the GUI by hovering a cursor over different elements shown on the GUI.

[0080] Additionally or alternatively, in some embodiments, a user may use drop-down menus to manipulate indications of the quality of the determined data. For example, a user may use a "Compliance Type" drop-down menu to select different types of reports to be generated. The "Compliance Type" drop-down menu may include options for generating reports based on different sub-period durations. For example, selecting a "Weekly," "Bi-Weekly," or "Monthly" type of report may change the heat map so that each column corresponds to one week, two weeks, or one month, respectively.

[0081] Additionally or alternatively, the "Compliance Type" drop-down menu may include options for generating reports for different levels of a clinical study (e.g., by individual, by clinical study site, by study, etc.). For example, by selecting a "site-level" type report, the heat map can be modified so that each row corresponds to a site rather than an individual. Such a heat map shows aggregated compliance for all subjects within the indicated site. Such aggregation may include calculating a mean or median average of the quality indicators determined for each subject associated with the site. Alternatively or additionally, this aggregation may include an indication of the number of subjects associated with the site that had "good" data, e.g., data meeting certain predetermined threshold criteria. Alternatively or additionally, this aggregation may include an indication of whether the number of subjects with good data is above or below certain predetermined thresholds. In some embodiments, by selecting a "study-level" type report, the heat map can be modified so that each row corresponds to the entire clinical study rather than an individual subject or site. Such a heat map shows aggregated compliance for all subjects within the indicated study. Such aggregation may include calculating a mean or median average of the quality indicators determined for each subject associated with the study. Alternatively or additionally, the aggregation may include an indication of the number of subjects associated with the study that had "good" data, e.g., data that met certain predetermined threshold criteria. Alternatively or additionally, the aggregation may include an indication of whether the number of subjects with good data was above or below certain predetermined thresholds.

[0082] The exemplary GUI shown in FIG. 3C also includes a drop-down menu for selecting metrics to display. In some embodiments, this drop-down can be manipulated to allow a user to select a method for assessing compliance and / or data quality for one or more subjects. For example, the drop-down menu may allow a user to select a threshold to compare acceleration data to determine compliance. If the standard deviation of acceleration data acquired during a particular subperiod is less than the selected threshold, this may indicate that the subject was not using the sensor during the particular subperiod. Thus, the heat map may provide an indication as to whether the subject was compliant during the particular subperiod. Additionally, or alternatively, the drop-down menu may allow a user to select a clipping threshold. The techniques described herein may be used to (a) determine the percentage of values ​​in a subset of data that are at or near the maximum or minimum sensor output value, and (b) compare the determined percentage to a selected clipping threshold. If the percentage exceeds the clipping threshold, this may indicate that "clipping" is occurring and that the data is invalid. Thus, the heat map may provide an indication as to whether clipping was occurring.

[0083] Additionally or alternatively, in some embodiments, the processor may output the indication via one or more report documents. For example, Figures 4B-4D are example reports that may be in PDF format or any other suitable format, as aspects of the technology described herein are not limited in this respect.

[0084] Researchers and medical professionals frequently conduct large-scale clinical studies (e.g., clinical trials) to find new and better ways to detect, understand, and treat medical conditions, which involves the collection, organization, and analysis of large amounts of data. For example, multiple (e.g., tens, hundreds, thousands, etc.) subjects may participate in one of several clinical studies. Data may be obtained from each subject (e.g., using one or more sensors) continuously or periodically (e.g., periodically or intermittently) over an extended period of time (e.g., over days, weeks, months, or years).

[0085] For example, as shown in FIG. 4A, each clinical study (e.g., Clinical Studies 1-N) may have multiple sites. A site may refer to a location, such as a hospital, research center, or medical institution, participating in the clinical study. For example, as shown in FIG. 4A, each clinical study may have a respective number of participating clinical sites (e.g., Study 1 has M sites, Study N has P sites). Furthermore, one or more subjects may participate in each clinical study through available sites. For example, Q subjects may participate through Site 1 in Study 1, S subjects may participate through Site M in Study 1, R subjects may participate through Site 1 in Study N, and T subjects may participate through Site P in Study N. Data is collected from each subject participating in each clinical study.

[0086] In some embodiments, the techniques described herein are used to determine data quality at the clinical study level. For example, the techniques may be used to determine the compliance of subjects 1-Q and 1-S in using sensors 1-Q and 1-S during each subperiod (e.g., 1 minute, 1 hour, 1 day, 1 week, 1 month, etc.) of an acquisition period (e.g., 1 hour, 1 day, 1 week, 1 month, etc.) of clinical study 1. In such embodiments, the data acquired by each of sensors 1-Q and 1-S is processed according to the techniques described herein to determine the quality of the data.

[0087] Additionally or alternatively, in some embodiments, the techniques described herein are used to determine data quality at the site level of a clinical study. For example, the techniques may be used to determine compliance of subjects 1-Q at site 1 in using sensors 1-Q during each subperiod (e.g., 1 minute, 1 hour, 1 day, 1 week, etc.) of an acquisition period (e.g., 1 hour, 1 day, 1 week, 1 month, etc.) of clinical study 1. In such embodiments, data acquired by each of sensors 1-Q is processed according to the techniques described herein to determine data quality.

[0088] Additionally or alternatively, in some embodiments, the techniques described herein are used to determine data quality at the level of individuals participating in clinical studies. For example, the techniques may be used to determine subject Q's compliance in using sensor Q during each subperiod (e.g., 1 minute, 1 hour, 1 day, 1 week, 1 month, etc.) of an acquisition period (e.g., 1 hour, 1 day, 1 week, 1 month, etc.) of clinical study 1. In such embodiments, data acquired by sensor Q is processed according to the techniques described herein to determine data quality.

[0089] 4B is an exemplary report showing study-level compliance for subjects participating in a clinical study, according to some embodiments. Study-level reports may include metrics displaying overall enrollment and compliance at the site level. These may allow clinical trial teams to easily measure the progress of a particular study, i.e., the number of patients who have completed their time in the study and the number of patients who are still progressing. As shown in the report, data was acquired for 131 patients during a 66-day acquisition period. The acquisition period was divided into 1-hour subperiods.

[0090] The sensor data from each of the 131 patients was aggregated into subsets of data, each of which contained sensor data acquired by the sensor during a specific time period during the acquisition period. The data was processed to determine the quality of each subset of data during each specific time period. The results are shown in a compliance table. As shown, 75 patients used their sensors for more than (or equal to) 20 hours per day for more than (or equal to) 50% of the total number of days during the acquisition period.

[0091] Additionally, for each site, sensor data from patients associated with the particular site was aggregated into subsets of data, each of which included data acquired by the sensors during a particular time period during the acquisition period. For example, for site 148, sensor data from three patients was aggregated into multiple subsets of data. For each site, the data was processed to determine the quality of each subset of data during each particular time period. The results are shown in the site-based compliance table. As shown, on average, the three patients at site 148 used sensors for more than 20 hours per day during 90.91% of the days during the acquisition period.

[0092] FIG. 4C is an exemplary report showing compliance of subjects associated with a particular clinical study site, according to some embodiments. In some embodiments, generating a report based on site allows clinical teams to efficiently identify which sites may be experiencing issues with low compliance across their assigned patients. In some embodiments, the site report includes overall performance information along with details of patients who may be below a set compliance threshold. Patients with low compliance may be labeled with potential issues, such as low compliance overnight. Potential issues may be derived from the patient's hourly compliance. From this, the site can identify which of the patients are the biggest contributors to low compliance and attempt to resolve issues related to low compliance.

[0093] As shown in the report in Figure 4C, 40 patients were associated with clinical sites, 36 of whom had completed their clinical studies, and 4 of whom were still in the process of completing their clinical studies. The acquisition period was 66 days, divided into 1-hour subperiods.

[0094] The sensor data from each of the 36 patients who completed the clinical study was aggregated into subsets of data, each of which contained sensor data acquired by the sensor during a specific time period during the acquisition period. The data was processed to determine the quality of each subset of data during each specific time period. The results are shown in a compliance table for the completed patients. As shown, 23 of the completed patients used their sensor for more than (or equal to) 20 hours per day for more than (or equal to) 50% of the total days in the acquisition period.

[0095] Additionally, for each patient, the sensor data from the patient was aggregated into subsets of data, each of which included data acquired by the sensor during a particular time period during the acquisition period. For example, for patient 13220, the sensor data from the patient was aggregated into multiple subsets of data. For each patient, the data was processed to determine the quality of the subset of data acquired during each particular time period during the acquisition period. The results are shown in the ongoing patient table. As shown, on average, patient 13220 used the sensor for more than 20 hours per day during 43.75% of the days during the acquisition period.

[0096] 4D is an exemplary report showing an individual subject's compliance in using their sensor, according to some embodiments. In some embodiments, patient-level reports can provide insight into specific patterns of device wear. In these reports, in some embodiments, the number of visits, compliant days within each visit, and compliance rates per visit may be displayed. Additionally, in some embodiments, an hourly compliance heat map may be visible, allowing further understanding of when patients wear their devices over the duration of the study.

[0097] As shown in the example report in Figure 4D, the acquisition period was 66 days, divided into 1-hour subperiods.

[0098] The sensor data from the patient was aggregated into subsets of data, each of which contained data acquired by the sensor during a specific time period during the acquisition period. The data was processed to determine the quality of each subset of data acquired during each specific time period. The results of the analysis are shown in a compliance table and hourly compliance heat map.

[0099] The compliance table shows, for each of a plurality of date ranges, the number of days and percentage during the particular date range that the subject was compliant in using the sensor. A subject was considered compliant if they used the sensor for more than (or equal to) 20 hours per day. For example, as shown, during pre-treatment, the subject used the sensor for more than (or equal to) 20 hours per day for 10 days (or 66.67%) of the pre-treatment period.

[0100] The hourly compliance heatmap shows the specific times during each day of the acquisition period when the subject was or was not using the sensor, with darker shading indicating poorer compliance and lighter shading indicating better compliance.

[0101] 5 is a flowchart illustrating an exemplary computerized method 500 for determining a subject's pattern of non-compliance when using a sensor, according to some embodiments. Method 500 may be implemented on any suitable processor, such as, for example, computing device 230 shown in FIGS. 2A and 2B and / or any other suitable processor, as aspects of the technology described herein are not limited in this respect.

[0102] In step 502, method 500 includes determining a subset of the sensor's acquisition period for evaluating patterns of non-compliance. The subset may include any suitable subset, as aspects of the technology described herein are not limited in this respect. As a non-limiting example, the subset of the acquisition period may include a time of day, a combination of times of day, a day, or a combination of days within the acquisition period. For example, the subset of times may be determined in operation 502 to identify times of day (e.g., early morning, late evening, etc.) during which the subject is frequently non-compliant in using the sensor. As an additional or alternative example, the subset of days may be determined in operation 502 to identify particular days of the acquisition period (e.g., the first and / or last few days of the acquisition period) during which the subject is frequently non-compliant in using the sensor.

[0103] At step 504, a support value is determined for each of the subsets. In some embodiments, the support value is a ratio of (a) the number of instances of that subset in which the subject was not compliant in using the sensor to (b) the total number of instances of that subset. For example, if the subsets are determined to be times of day, then the support value for a particular subset (e.g., a particular time of day) may be a ratio of (a) the number of instances of a particular time of day in which the subject was not compliant in using the sensor during the acquisition period to (b) the total number of instances of a particular time during the acquisition period. In some embodiments, determining whether the subject was compliant in using the sensor during instances of a subset may be performed using any of the techniques described herein, including at least those with respect to FIG. 3A , or any other suitable technique, as the aspects of the techniques described herein are not limited in this respect.

[0104] In some embodiments, determining the support value for a particular subset includes subdividing the subset into multiple subdivisions and determining the support value using the multiple subdivisions. In such embodiments, the support value may be a ratio of (a) the number of subdivisions of the subset in which the subject was not compliant when using the sensor to (b) the total number of subdivisions of the subset. For example, if the subsets are determined to be days of the acquisition period, the days may be subdivided into hours, and the support value for a particular day may be (a) the number of hours of that day in which the subject was not compliant when using the sensor to (b) the total number of hours of that day. In some embodiments, determining whether the subject was compliant when using the sensor during a subdivision of the subset may be performed using any of the techniques described herein, including at least those with respect to FIG. 3A , or any other suitable technique, as aspects of the techniques described herein are not limited in this respect.

[0105] At operation 506, the support value is compared to a user-defined threshold. The threshold may include any suitable threshold, as aspects of the technology described herein are not limited in this respect. For example, the threshold may be at least 0.3, at least 0.4, at least 0.5, at least 0.6, at least 0.7, at least 0.8, at least 0.9, or at least any other suitable threshold. Additionally or alternatively, the threshold may be at most 0.9, at most 0.8, at most 0.7, at most 0.6, at most 0.5, at most 0.4, at most 0.3, or at most any other suitable threshold. Additionally or alternatively, the threshold may be between 0.3 and 0.9, 0.4 and 0.8, 0.5 and 0.7, or any other suitable value, as aspects of the technology described herein are not limited in this respect.

[0106] At operation 508, the set of one or more subsets whose support values ​​are greater than or equal to a user-defined threshold are output. In some embodiments, outputting the set of subsets may be performed using techniques described herein with respect to operation 314 of method 300 shown in FIG. 3A for outputting an indication of a pattern of non-compliance. Additionally, or alternatively, the set of one or more subsets may be output using any other suitable technique, as aspects of the techniques described herein are not limited in this respect.

[0107] While the foregoing description focuses on a subset of times within a day or days within an acquisition period, other methods of determining the subset to evaluate for patterns of noncompliance are possible. For example, the subset may include days of the week (e.g., Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, and Sunday). Additionally or alternatively, the subset may include weekdays, weekends, and / or holidays. Additionally or alternatively, the subset may include days on which the subject is physically located at the clinical site (e.g., for a visit) and / or days on which the subject is not physically located at the clinical site. Additionally or alternatively, the subset may include days on which the subject is experiencing symptoms and / or days on which the subject is not experiencing symptoms. It should be understood that the subset may include any other suitable subset, as aspects of the technology described herein are not limited in this respect.

[0108] 6A-6D illustrate an example of determining a pattern of a first subject's compliance in using an actigraphy sensor based on indications of quality determined for data acquired by the actigraphy sensor.

[0109] 6A is an exemplary report showing the extent to which a first subject was compliant in using a sensor during each hour of multiple days, according to some embodiments. The acquisition period has a duration of 67 days, divided into 67 days, each of which is further subdivided into one-hour subperiods. Each box indicates the extent to which the first subject was compliant in using the sensor during the corresponding hour and day of the acquisition period. The degree of compliance may be calculated based on the ratio of the number of minutes in the corresponding hour during which compliant data was received for the first subject to the total number of minutes in the corresponding hour. The higher this ratio, the higher the degree of compliance. Again, darker shading indicates lower compliance, and lighter shading indicates higher compliance.

[0110] FIG. 6B is an exemplary table showing whether a first subject was compliant in using a sensor during each hour of multiple days, according to some embodiments. The rows of the table correspond to days, and the columns correspond to hours on those days. A table entry of "true" indicates that the first subject was not compliant in using the sensor during a particular hour on a particular day. A table entry of "false" indicates that the first subject was compliant in using the sensor during a particular hour. In some embodiments, the table may be generated based on the quality indications shown in FIG. 6A. For example, for an hour having a quality equal to or greater than a threshold quality (e.g., 50% of maximum quality), the corresponding table entry may be filled with a value of "false." For an hour having a quality below the threshold, the corresponding table entry may be filled with a value of "true."

[0111] In some embodiments, the data in the table shown in FIG. 6B can be used to determine a pattern of non-compliance for a subject. For example, the data can be provided as input to a frequent itemset mining algorithm to determine the pattern of non-compliance. In some embodiments, the frequent itemset mining algorithm can determine support values ​​for different portions of the acquisition period, and the support values ​​can be used to determine the pattern of non-compliance.

[0112] For example, determining a pattern of noncompliance may include determining support values ​​for a subset of one or more hours of the day and using the support values ​​to determine whether the first subject was frequently noncompliant during those hours of the day over the duration of the acquisition period. FIG. 6C shows an exemplary table of support values ​​determined using the data shown in FIG. 6B. As shown in FIG. 6C, 8:00 PM of the day has a support value of 0.552239. Intuitively, this support value indicates that the first subject was noncompliant during 8:00 PM on 67 days of the acquisition period. This support value exceeds a threshold (e.g., 0.5) indicated by the dashed line, indicating that the first subject had a pattern of noncompliance when using the sensor during the 20th hour of the day. In contrast, as further shown in FIG. 6C, the first subject was typically compliant (e.g., support values ​​below the threshold) when using the sensor during the previous and subsequent hours (e.g., 7:00 PM and 9:00 PM).

[0113] Additionally or alternatively, determining a pattern of noncompliance may include determining support values ​​for a subset of one or more days of the acquisition period and using the support values ​​to determine whether the first subject was frequently noncompliant during those days of the acquisition period. FIG. 6D shows an example table of support values ​​determined using the data shown in FIG. 6B. As shown in FIG. 6D, day 1 of the acquisition period has a support value of 1.0 (indicating that the first subject was noncompliant in using the sensor during every hour of day 1), and day 67 has a support value of 0.6432 (indicating that the first subject was noncompliant at 0.6432 of the hours on day 67). Both of these values ​​are above a threshold (e.g., 0.5), as indicated by the dashed line, indicating that the first subject had a pattern of noncompliance in using the sensor during day 1 and the 67th (final) day of the acquisition period. In contrast, as further shown in FIG. 6D, the first subject was typically compliant in using the sensor (e.g., support values ​​below threshold) during the days following day 1.

[0114] It should be understood that a simple trend analysis may not be sufficient to capture patterns of non-compliance. For example, as shown in FIG. 6E, an exemplary trend analysis based on the data shown in FIG. 6A indicates that the first subject was generally compliant in using the sensor. Fitting a line to the compliance data does not provide any indication of outlier compliance data points, which indicate that the first subject was not compliant in using the sensor during a few days of the acquisition period.

[0115] 7A-7D show an example of determining a second subject's pattern of compliance in using an actigraphy sensor based on indications of quality determined for data acquired by the actigraphy sensor.

[0116] 7A is an exemplary report showing the extent to which a second subject was compliant in using a sensor during each hour of multiple days, according to some embodiments. The acquisition period is 41 days in duration and is divided into 41 days, which are further subdivided into 1-hour sub-periods. Each box indicates the extent to which the second subject was compliant in using the sensor during an hour on a particular day of the acquisition period.

[0117] FIG. 7B is an exemplary table showing whether a second subject was compliant in using the sensor during each hour of multiple days, according to some embodiments. The rows of the table correspond to days, and the columns correspond to hours. A table entry of "false" indicates that the second subject was compliant in using the sensor during a particular hour on a particular day. A table entry of "true" indicates that the second subject was not compliant in using the sensor during that hour. In some embodiments, the table may be generated based on the quality indications shown in FIG. 7A. For example, for an hour having a quality equal to or greater than a threshold quality (e.g., 50% of maximum quality), the corresponding table entry may be filled with a value of "false." For an hour having a quality below the threshold, the corresponding table entry may be filled with a value of "true."

[0118] In some embodiments, the data in the table shown in Figure 7B can be used to determine a pattern of non-compliance for a subject. For example, the data can be provided as input to a frequent itemset mining algorithm to determine the pattern of non-compliance. In some embodiments, the frequent itemset mining algorithm can determine support values ​​for different portions of the acquisition period, and the support values ​​can be used to determine the pattern of non-compliance.

[0119] For example, determining a pattern of noncompliance may include determining support values ​​for a subset of one or more hours of the day and using the support values ​​to determine whether the second subject was frequently noncompliant during those hours of the day over the duration of the acquisition period. Figure 7C shows an example table of support values ​​determined using the data shown in Figure 7B. As shown in Figure 7C, 21:00 of the day has a support value of 0.512195 (indicating that the second subject was noncompliant during 21:00 on 0.512195 days of the acquisition period). This support value is above a threshold (e.g., 0.5) indicated by the dashed line, indicating that the second subject had a pattern of noncompliance in using the sensor during the 21st hour of the day. In contrast, as further shown in Figure 7C, the second subject was typically compliant (e.g., support values ​​below threshold) in using the sensor during the preceding and subsequent times (e.g., 10:00 PM and 11:00 PM), even when evaluated along with compliance data obtained for 9:00 PM.

[0120] Additionally or alternatively, determining a pattern of non-compliance may include determining support values ​​for a subset of one or more days of the acquisition period and using the support values ​​to determine whether the second subject was frequently non-compliant during those days of the acquisition period. Figure 7D shows an exemplary table of support values ​​determined using the data shown in Figure 7B. As shown in FIG. 7D , day 1 of the acquisition period had a support value of 1.0 (indicating the second subject was noncompliant in using the sensor during every hour of day 1), the subset of days 1 and 2 had a support value of 0.875 (indicating the second subject was noncompliant for 0.875 of the entire time on days 1 and 2 combined), day 28 had a support value of 0.708333 (indicating the second subject was noncompliant for 0.70833 of the entire time on day 28), and the subset of days 28 and 1 had a support value of 0.708333 (indicating the second subject was noncompliant for 0.70833 of the entire time on days 28 and 1 combined). All of these values ​​were above a threshold (e.g., 0.5), indicating that the second subject had a pattern of non-compliance in using the sensor toward the beginning and end of the acquisition period.

[0121] It should be understood that a simple trend analysis may not be sufficient to capture patterns of non-compliance. For example, as shown in FIG. 7E, an exemplary trend analysis based on the data shown in FIG. 7A indicates that the second subject was generally compliant in using the sensor. Fitting a line to the compliance data does not provide any indication of several outlier compliance data points, which indicate that the second subject was not compliant in using the sensor during several days of the acquisition period.

[0122] Example 1 - Illustrative Data As described herein, in some embodiments, data is received and / or processed to determine the quality of the data. For example, the data may include data acquired by a sensor. Examples of types of data that may be received and / or processed are described herein. It should be understood that these examples are not intended to be limiting, and any other suitable type of data may be received and / or processed.

[0123] Sensor Data In some embodiments, the data is collected from a device that measures a particular physical environment (e.g., wrist movement or temperature) at a high sampling frequency, e.g., 50 Hz. This type of data is also called a high-frequency point-based time series, and is expressed as X={(T i ,X i ), where i=1,...,m}: a sequence of time-value pairs, where T i is the timestamp, and X i is the signal value at the i-th sample point.

[0124] In some embodiments, the device may collect data from multiple sensor signals at various pre-configured sampling frequencies. For example, the device may collect triaxial accelerometry data at 50 Hz, electrodermal activity (EDA) data at 4 Hz, and body temperature data at 1 Hz. In some embodiments, the sensor signals are collected nonstop 24 / 7 throughout the entire study, which may take place over several weeks to several months.

[0125] Scored data or digital biomarkers In some embodiments, in addition to the raw sensor signal, the device may process the sensor data and derive digital biomarkers (dBMs) therefrom. For example, heart rate and blood volume pulse may be derived from the raw photoplethysmography (PPG) sensor signal. The derived dBMs may be at a lower resolution than the sensor signal.

[0126] Electronically reported outcomes (ePROs) In some embodiments, electronic patient-reported outcomes are collected from study participants for comparison with digital biomarkers (dBMs).

[0127] Temporal Events In contrast to point-based time series, temporal events can be viewed as interval-based time series with a start and end timestamp for each event. An example is patient-reported migraine events with a start and end timestamp along with the reported pain level during the migraine on a scale of 1 to 10. This interval-based variable can be expressed as W={(T si ,T ei ,Y i ), where i=1,...,m}}: it can be expressed as a set of time intervals, T si and T ei represents the onset and offset timestamps, and Y i is the binary representation of the event in the ith interval.

[0128] Example 2 - Exemplary System for Receiving and Processing Data As described herein, one or more computing devices may be configured to receive data (e.g., from sensors) and process the data. For example, the computing devices may be configured to process the data using software to determine the quality of the data and / or a subset of the data. Exemplary systems for receiving and processing data are described herein. These examples are not intended to be limiting, and it should be understood that any other suitable technique may be used to receive and process data.

[0129] Data Source In some embodiments, two types of data may be received from study participants: high-resolution raw sensor signals from wearable sensors (e.g., wrist-worn devices, chest patches, and foot insole sensors) and ePROs submitted via a mobile application or web form. Additionally, metadata may be obtained from Clinical Research Organizations (CROs), such as mapping between sensor devices and participants, participant visit schedule information (a typical study or trial consists of a series of visits, with each visit spanning several days), and treatment group or cohort placement (e.g., placebo vs. treatment with a specific drug dose).

[0130] Data Transfer Tool In some embodiments, data transfer tools enable data to be received at scale at a digital data platform (DDP), interfacing with storage infrastructure without intermediate data staging storage or areas. In terms of data transfer frequency, secure file transfer protocols such as the Amazon Web Services Secure Shell File Transfer Protocol (AWS SFTP) service may enable data transfer in batches. Additionally, services such as AWS Kinesis services (i.e., Kinesis Data Streams and Firehose) can be leveraged to receive data streams (e.g., real-time data streams) directly from sensors. Additionally, various synchronization tools may enable the DDP to pull data from external sources at configurable frequencies, allowing for additional flexibility.

[0131] Storage Infrastructure In some embodiments, the storage infrastructure interfaces with data transfer tools through pre-configured listeners and triggers, allowing data to flow in (e.g., automatically) as it arrives. DDP can enable data to enter storage components tailored by three metrics: (1) I / O performance, (2) available interfaces (e.g., APIs) to access the data and interface with other storage / computing components, and (3) cost. Based on this principle, the infrastructure, in some embodiments, has four categories:

[0132] The first category may include data lakes targeting raw data such as sensor signals, consisting of a Parallel File System (PFS, e.g., Lustre) in an on-premise High Performance Computing (HPC) environment and storage buckets and tapes in the cloud (e.g., AWS S3 for hot data and Glacier for cold data).

[0133] Second, for structured data or data frames with read access, a table viewer (e.g., AWS Glue Tables) reads the underlying data lake with an additional layer for performance optimization and a SQL-based view / query interface for data access.

[0134] The third category can include application-specific high-performance data structures and data stores, for example, sensor signals and mission-critical data can be converted into Elasticsearch's internal data structures, i.e., Lucene indexes, to enable near real-time aggregation and on-the-fly data querying.

[0135] Fourth, for structured data that involves write operations or whose query performance cannot be guaranteed by table views, dedicated relational databases can be utilized.

[0136] Computing Infrastructure In some embodiments, the computing infrastructure acts as a heavy-duty computing engine. To support automation, in some embodiments, listeners and triggers are pre-configured to launch downstream computations when upstream storage supplies data.

[0137] In this example, the computing engine is divided into two logical tiers. The first tier may include a bottom-level computing framework tier that can support application-agnostic data handling and processing at scale. For example, crawlers and data catalogs, along with lambda event listeners, generate table viewers for structured data, Spark performs parallel generic data format conversions (e.g., from CSV to the more performant Apache Parquet), and custom scripts / frameworks support scalable data processing in HPC environments.

[0138] The second layer may include a business logic-specific pipeline layer on top. This layer may leverage the underlying computing framework layer to perform processing at scale. For example, an accelerometry sensor data analysis pipeline may be developed on top of open source algorithms (e.g., GGIR and UKBiobank) and deployed in the cloud (e.g., via Spark) with parallel array jobs. Additional algorithms may be deployed for further processing.

[0139] Front-End Portal In some embodiments, interactive and iterative visual queries are realized through a web-based portal that includes dashboards, enabling the concept of "human-in-the-loop" analysis. For example, each dashboard may have multiple filters and operators that iteratively refine the visual query. In addition to dashboards, the DDP may include practical information such as research portfolios, technical manuals, data processing pipelines, and pointers to internal GitHub repositories.

[0140] API and analytics environment In some embodiments, APIs and software development kits (SDKs) target different data access and query needs, including metadata queries, raw data queries, dynamic data aggregation, data ingestion, and business-specific utility APIs.

[0141] In some embodiments, a set of APIs and / or SDKs are used to facilitate data access. For example, the digital data platform can integrate APIs (e.g., AWS S3 and AWS Athena) to enable SQL data queries and direct file loads into the analysis environment. AWS S3 is a highly scalable, durable, and secure object storage cloud service. AWS Athena is a serverless query service backed by AWS S3.

[0142] In some embodiments, a search engine such as AWS-managed Elasticsearch enables near-real-time search and data aggregation. Elasticsearch is a data store, search, and analytics engine based on Lucene. AWS-managed services enable on-demand upscaling of Elasticsearch as data volumes grow.

[0143] Data Flow In some embodiments, sensor and ePRO data arrive at the storage layer through high-performance transfer tools and services. As the data arrives, in some embodiments, one or more processing pipelines are triggered, either in parallel (if there are no dependencies) or in a specific order (i.e., chained pipelines if sequence matters). One example is a Spark pipeline for raw data cleaning and quality checks, followed by concurrent feature extraction pipelines, each processing a specific sensor data type. For example, an accelerometry data processing pipeline may include one or more modules, including, but not limited to: (1) calibration to local gravity, (2) resampling, (3) gravity removal, (4) noise removal, (5) data segmentation, and (6) feature computation. In some embodiments, once features from data channels (e.g., accelerometer, electrodermal activity, and temperature) are prepared, a feature aggregation pipeline is launched to perform aggregation at different levels for each hour, day, week, or visit in the study.

[0144] Example 3 - Exemplary Visualization As described herein, in some embodiments, techniques for determining the quality of a signal acquired by a sensor include outputting an indication of the quality. Examples for visualizing the data and outputting an indication of the quality of the data are described herein. It should be understood that these examples are not intended to be limiting, and any other suitable technique for generating an output may be implemented, as the aspects of the present technology are not limited in this respect.

[0145] In some embodiments, an integrated tool such as AWS Kibana is used to leverage stored data (e.g., data stored in AWS Elasticsearch). AWS Kibana can create customized visualizations with near real-time interactivity. Multiple data types associated with clinical trials can be viewed in a variety of formats, including, for example, time series plots, histograms, heat maps, and data tables. In some embodiments, digital data is continuously ingested during associated clinical trials; therefore, visualizations can be rendered based on the data structure and purpose to help track the progress and quality of the digital data in the trial.

[0146] In some embodiments, the visualization is tailored to display accurate data based on a time filter or applied query. As a non-limiting example, a visualization can be generated for a biosensor signal. Such data can include, for example, raw accelerometer sensor data with a sampling frequency of 50 Hz (e.g., a data point every 20 milliseconds). Thus, in this example, the visualization can show data every 20 milliseconds and can zoom in and out based on a particular level of detail or pattern to be identified. In some embodiments, an existing AWS Elasticsearch indexed DataFrame can be selected as the data source using an integration tool such as AWS Kibana. Additionally, an aggregation method and plot type can be selected. Non-limiting examples of aggregations include average, maximum, minimum, percentile, standard deviation, sum, and variance. Depending on the zoom level, data aggregation can, in some embodiments, be dynamically updated to an interval (e.g., 1 second, 1 hour, 1 day) that matches the date and time range of the visualization.

[0147] In some embodiments, temporal events are plotted. Plotting events can be useful for understanding the ground truth of reported symptoms and events and comparing them with sensor data. For example, a step line plot allows data with timestamps and labels for each timestamp (i.e., the start and end timestamps of an event) to be converted into a time series plot showing different categories of events. Additionally or alternatively, for events with scale ratings, changes in reported rankings can be seen at indicated time points within the event.

[0148] In some embodiments, derived features such as, for example, step count, sleep time, or heart rate, can be viewed in a time series bar plot, similar to the time series plot for viewing accelerometer data. Additionally or alternatively, other derived information in associated clinical trials can be viewed, including, but not limited to, a matrix of data compliance percentages to track digital data quality throughout the associated clinical trial, and the geolocation of participants in distributed trials.

[0149] In some embodiments, the visualizations may be shown independently (e.g., separately), or multiple visualizations may be viewed simultaneously at a given time.

[0150] Example 4 - Exemplary Techniques for Determining Data Quality As described herein, including at least with respect to Figures 1A-4D, data acquired by a sensor may be processed to determine quality of the data. Examples of determining quality of sensor data are described herein. These examples are not intended to be limiting, and it should be understood that any other suitable technique for determining quality of data may be implemented, including those described herein with respect to Figures 1A-4D.

[0151] Validity Check In some embodiments, the sensor data is evaluated to filter out invalid data. For example, an expected number of valid data points can be determined based on a preconfigured sampling frequency. Invalid values ​​can be filtered using a range of valid values ​​to obtain valid data coverage, i.e., coverage of valid data points.

[0152] Additionally or alternatively, in some embodiments, since raw sensor signals are directly correlated with the derived dBM, a plausibility check can be performed independently on the two, and then their valid data coverage can be aligned to check for consistency. In some embodiments, device incident events can be overlaid to better understand the root cause of observed problems. Invalid data can be discarded.

[0153] Unused and Clipping Detection In some embodiments, data points may be initially collected at high resolution, e.g., a 50 Hz sampling frequency, but processing is performed on aggregated values ​​(e.g., short sub-periods (epochs) of 1 or 5 seconds or long sub-periods (epochs) of 15 minutes).

[0154] In some embodiments, accelerometer data is used to determine whether the subject was using the sensor during a particular subperiod. In some embodiments, accelerometer non-wearing time is estimated based on the standard deviation and value range of raw data from each accelerometer axis. As an example, classification may be performed for each 15-minute subperiod based on characteristics of a 60-minute acquisition period centered around the 15-minute subperiod. If the standard deviation of the 60-minute acquisition period is less than 13.0 mg (1 mg = 0.0098 m s), and the value range of the 60-minute window for at least two of the three accelerometer axes is less than 50 mg, the subperiod may be classified as non-wearing time.

[0155] In some embodiments, accelerometer data may also be screened for “clipping.” As an example, if more than 50% of the data points within a 15-minute time window (subperiod) are near the maximum dynamic range of the sensor (e.g., 7.5 g), the corresponding subperiod may be considered potentially corrupted due to the presence of mostly extreme values.

[0156] In some embodiments, the technique involves determining whether data corresponding to a particular subperiod is valid based on whether the subject used the sensor and / or whether clipping occurred. Table 1 shows an example of a validity table for different subperiods. The values ​​in the "Non-Use Score" column range from 0 to 3 and represent the sum of three independent scores from the x, y, and z axes, respectively. An axis receives a score of 1 when detected as non-use and a score of 0 otherwise. Furthermore, for the "Clipping Score" column, a value of 1 means corrupted data was detected, and 0 otherwise. Based on these two scores, a third column called "isValid" can be derived to indicate the validity of the data in the subperiod, where if the Non-Use Score is 1 or less and the Clipping Score is 0, the data is considered valid.

[0157] [Table 1]

[0158] In some embodiments, the data can be processed to determine data coverage at an hourly level. For example, this can include starting with a validity table, applying a filter on the "isValid" flag, and then grouping the results by subject, date, and hour to obtain minute-by-minute data coverage at the hourly level. In some embodiments, each long period lasts 15 minutes, so hourly data coverage can be derived by grouping and counting the records for each hour. Hourly data coverage can be the source for the finest granularity of data coverage reporting.

[0159] In some embodiments, data can be processed to determine coverage at an hourly level. For example, hourly data coverage can be aggregated over several days to provide a daily level of data coverage. Additionally, in some embodiments, intraday coverage can be derived.

[0160] In some embodiments, data can be processed to determine expanded data coverage using external mappings. For example, data coverage can be expanded with additional mappings, such as mapping between subjects and sites / visits, as reported by clinical operations sites. These additional fields can enable analysis-specific filtering and aggregation, for example, to find which participants have sufficient data and establish individual baselines. For example, such data can be used to identify subjects with at least three valid days (≥ 20 hours of data per day to qualify as a valid date) during a pre-treatment visit.

[0161] Example 5 - Exemplary Indication of Quality As described herein, in some embodiments, techniques for determining the quality of a signal acquired by a sensor include outputting an indication of the quality. Exemplary indications of the determined quality are described herein. It should be understood that these examples are not intended to be limiting, and any other suitable indication of quality may be implemented.

[0162] Quality map for every minute of the day Checking the signal at the minute level can help identify minutes where a device may have intermittent connectivity, or minor issues can be identified and further checked.

[0163] Time-based quality map of the exam In some embodiments, hourly level aggregation is used to construct a day-level plot. The hourly quality map shows the data coverage for each hour across all study days. This type of visualization allows for evaluation of patient compliance trends that may persist for specific times each day. For example, a patient may remove their wearable device to charge the battery for several hours each day, which may result in missing data. For example, FIG. 4D shows that a patient did not use their device for approximately an hour in the middle of the day for several days during the study period.

[0164] Study day-by-day population-level quality map In some embodiments, plotting data quality for all hours, days, and participants in a study provides insight into data quality patterns. Such study-level visualizations can be useful for gaining insight into overall data quality at the population level and compliance trends at the participant level throughout the study.

[0165] Compliance dates throughout the study In some embodiments, it is also useful to look at the number of compliant days throughout the study, with the definition of compliance depending on the study protocol. By plotting the number of patients who are compliant each day in a given study, device wear patterns can be recognized (e.g., as shown in Figure 3C).

[0166] Techniques operating according to the principles described herein may be implemented in any suitable manner. The processing and decision blocks of the flowcharts above represent steps and acts that may be included in algorithms that perform these various processes. The algorithms derived from these processes may be implemented as software integrated with and directing the operation of one or more single-purpose or multi-purpose processors, as functionally equivalent circuitry such as digital signal processing (DSP) circuitry or application-specific integrated circuits (ASICs), or in any other suitable manner. It should be understood that the flowcharts included herein do not depict any particular circuitry or the syntax or operation of any particular programming language or type of programming language. Rather, the flowcharts illustrate functional information that one skilled in the art may use to fabricate circuitry that performs the processing of, or implement computer software algorithms that perform the processing of, a particular device that implements the types of techniques described herein. It should also be understood that, unless otherwise indicated herein, the specific sequence of steps and / or acts set forth in each flowchart is merely illustrative of algorithms that may be implemented and varied in implementations and embodiments of the principles described herein.

[0167] Thus, in some embodiments, the techniques described herein may be embodied in computer-executable instructions implemented in software, including application software, system software, firmware, middleware, embedded code, or any other suitable type of computer code. Such computer-executable instructions may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and may be compiled as executable machine code or intermediate code that runs on a framework or virtual machine.

[0168] When the techniques described herein are embodied as computer-executable instructions, these computer-executable instructions may be implemented in any suitable manner, including as several functional facilities, each of which provides one or more operations for completing the execution of an algorithm operating according to these techniques. A "functional facility" is an instantiated structural component of a computer system that is integrated with one or more computers and, when executed by one or more computers, causes the one or more computers to perform a particular operational role. A functional facility may be part or all of a software element. For example, a functional facility may be implemented as a function of a process, as a separate process, or as other suitable processing units. When the techniques described herein are implemented as multiple functional facilities, each functional facility may be implemented in its own way and need not all be implemented in the same way. Additionally, these functional facilities may execute in parallel and / or serially as desired and may pass information between each other using shared memory on the computers on which they are executing, using a message-passing protocol, or in other suitable manners.

[0169] Generally, functional facilities include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically, the functionality of functional facilities may be combined or distributed as desired in the systems in which they operate. In some implementations, one or more functional facilities that implement the techniques herein may together form a complete software package. These functional facilities may, in alternative embodiments, be adapted to interact with other, unrelated functional facilities and / or processes to implement a software program application.

[0170] Several exemplary functional facilities for performing one or more tasks are described herein. However, it should be understood that the described functional facilities and divisions of tasks are merely exemplary of types of functional facilities that may implement the exemplary techniques described herein, and that embodiments are not limited to being implemented with any particular number, division, or type of functional facilities. In some implementations, all functionality may be implemented in a single functional facility. It should also be understood that in some implementations, some of the functional facilities described herein may be implemented together with others or separately (i.e., as a single unit or separate units), or some of these functional facilities may not be implemented.

[0171] Computer-executable instructions implementing the techniques described herein (when implemented as one or more functional facilities or in any other manner) may, in some embodiments, be encoded on one or more computer-readable media to provide functionality thereon. Computer-readable media include magnetic media such as hard disk drives, optical media such as compact discs (CDs) or digital versatile discs (DVDs), persistent or non-persistent solid-state memory (e.g., flash memory, magnetic RAM, etc.), or any other suitable storage media. Such computer-readable media may be implemented in any suitable manner. As used herein, "computer-readable media" (also referred to as "computer-readable storage media") refers to tangible storage media. Tangible storage media are non-transitory and have at least one physical structural component. In "computer-readable media," as used herein, the at least one physical structural component has at least one physical characteristic that can be altered in some way during the process of creating a medium having embedded information, recording information thereon, or any other process of encoding the medium with information. For example, the magnetization state of portions of the physical structure of the computer-readable medium may be altered during the recording process.

[0172] Furthermore, some of the technologies described above involve the act of storing information (e.g., data and / or instructions) in a particular manner for use by those technologies. In some implementations of these technologies, such as those in which the technologies are implemented as computer-executable instructions, the information may be encoded on a computer-readable storage medium. Where particular structures are described herein as advantageous formats for storing this information, those structures may be used to impart a physical organization to the information when encoded on the storage medium. These advantageous structures may then provide functionality to the storage medium by affecting the operation of one or more processors that interact with the information, e.g., by increasing the efficiency of computer operations performed by the processors.

[0173] While the techniques may be embodied as computer-executable instructions, in some, but not all, implementations, these instructions may be executed on one or more suitable computing devices operating in any suitable computer system, or one or more computing devices (or one or more processors of one or more computing devices) may be programmed to execute the computer-executable instructions. A computing device or processor may be programmed to execute the instructions when the instructions are stored in a manner accessible to the computing device or processor, such as a data store (e.g., an on-chip cache or instruction register, a computer-readable storage medium accessible via a bus, a computer-readable storage medium accessible via one or more networks and accessible by the device / processor, etc.). The functional facility containing these computer-executable instructions may be integrated with and direct the operation of a single general-purpose programmable digital computing device, a cooperative system of two or more general-purpose computing devices that share processing power and jointly implement the techniques described herein, a single computing device or cooperative system of computing devices (co-located or geographically distributed) dedicated to performing the techniques described herein, one or more field programmable gate arrays (FPGAs) for implementing the techniques described herein, or any other suitable system.

[0174] A computing device may include at least one processor, a network adapter, and a computer-readable storage medium. The computing device may be, for example, a desktop or laptop personal computer, a personal digital assistant (PDA), a smart mobile phone, a server, or any other suitable computing device. The network adapter may be any suitable hardware and / or software that enables the computing device to communicate wired and / or wirelessly with any other suitable computing device over any suitable computing network. The computing network may include wireless access points, switches, routers, gateways, and / or other network equipment, as well as any suitable wired and / or wireless communication medium for exchanging data between two or more computers, including the Internet. The computer-readable medium may be adapted to store data to be processed and / or instructions to be executed by the processor. The processor enables the processing of data and execution of instructions. The data and instructions may be stored on the computer-readable storage medium.

[0175] A computing device may additionally have one or more components and peripherals, including input devices and output devices. These devices may be used, among other things, to present a user interface. Examples of output devices that may be used to provide a user interface include a printer or display screen for visual presentation of output, and a speaker or other sound-generating device for audible presentation of output. Examples of input devices that may be used for a user interface include pointing devices such as a keyboard, mouse, touchpad, or digitizer tablet. As another example, a computing device may receive input information through voice recognition or in other audible forms.

[0176] Embodiments have been described in which the present technology is implemented in circuits and / or computer-executable instructions. It should be understood that some embodiments may be in the form of a method, of which at least one example is provided. The acts performed as part of this method may be ordered in any suitable manner. Thus, while exemplary embodiments show acts as sequential, embodiments may be constructed in which acts are performed in an order different from that illustrated, which may include performing some acts simultaneously.

[0177] Various aspects of the above-described embodiments may be used alone, in combination, or in various arrangements not specifically discussed in the foregoing embodiments, and therefore are not limited in their application to the details and arrangements of components set forth in the foregoing description or illustrated in the drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.

[0178] The use of ordinal terms such as "first," "second," "third," etc. to modify claim elements in the claims does not, by itself, imply any priority, precedence, or order of a claim element relative to another or the temporal order in which the actions of a method are performed, but is used merely as a label to distinguish a claim element having a particular name from another element having the same name (but due to the use of ordinal terms) to distinguish between claim elements.

[0179] Also, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," "having," "containing," "involving," and variations thereof is meant to encompass the items listed thereafter and equivalents thereof as well as additional items.

[0180] The term "exemplary" is used herein to mean serving as an example, instance, or illustration. Thus, any embodiment, implementation, process, feature, etc. described herein as exemplary is to be understood as an exemplary example and not as a preferred or advantageous example, unless otherwise specified.

[0181] By clarifying the use of the following and notifying the public: 、 , ...and <n> "At least one of the following" or "< / n> 、 、... <n> , or at least one of a combination thereof," or "< / n> 、 , ...and / or <n>The phrase "is defined by applicant in its broadest sense and supersedes any other implied definition above or below unless expressly asserted to the contrary by applicant, and means one or more elements selected from the group including A, B... and N. In other words, this phrase means any combination of one or more of the elements A, B,... or N, including any one element alone or that one element in combination with one or more of the other elements, which may also include additional, unlisted elements.

[0182] While various embodiments have been described, it will be apparent to those skilled in the art that many more embodiments and implementations are possible. Accordingly, the embodiments described herein are examples and are not intended to be the only possible embodiments and implementations. Furthermore, the advantages described above are not necessarily the only advantages, and it is not necessarily expected that all of the described advantages will be achieved in each embodiment.

[0183] Various aspects are described in this disclosure, including but not limited to the following aspects. 1. A method, using one or more processors, receiving data from a plurality of actigraphy sensors associated with a plurality of subjects, the data being acquired by the plurality of actigraphy sensors during an acquisition period, the data being indicative of a signal related to physical motion of the plurality of subjects, the receiving including receiving first data from a first actigraphy sensor of the plurality of actigraphy sensors associated with a first subject of the plurality of subjects, the first data being acquired by the first actigraphy sensor, the first data being indicative of a signal related to physical motion of the first subject; dividing the acquisition period into a plurality of sub-periods; and aggregating data from the plurality of data subsets into a plurality of data subsets, each of the plurality of data subsets including data acquired during a different sub-period of the plurality of sub-periods; determining, for each particular sub-period of the plurality of sub-periods, a quality of the subset of data acquired during the particular sub-period, wherein determining includes determining whether a first subject of the plurality of subjects was compliant in using a first actigraphy sensor of the plurality of actigraphy sensors during the particular sub-period; and outputting an indication of the determined quality of at least one of the plurality of data subsets. 2. The method of aspect 1, wherein determining whether the first subject was compliant in using the first actigraphy sensor includes determining whether the first subject was using the first actigraphy sensor. 3. The method of aspect 2, wherein the subset of data acquired during a particular sub-period indicates acceleration of a portion of the first subject's body during the particular sub-period, and determining whether the first subject was using the first actigraphy sensor during the particular sub-period includes comparing the acceleration of the portion of the first subject's body with an acceleration threshold, and determining whether the first subject was using the first actigraphy sensor during the particular sub-period based on a result of the comparison. 4. The method of any one of aspects 1-3, wherein determining the quality of the subset of data acquired during the particular sub-period further includes determining whether the first actigraphy sensor malfunctioned during the particular sub-period. 5. The method of aspect 4, wherein determining whether the first actigraphy sensor malfunctioned during a particular subperiod includes determining a percentage of the subset of data that is equal to a maximum output value or a minimum output value of the first actigraphy sensor, and wherein the indication of the determined quality of the at least one subset of data includes an indication of the determined percentage of the subset of data. 6. The method of aspect 4, wherein determining whether the first actigraphy sensor has malfunctioned includes determining whether the subset of data includes a value excluded from the output range of the first actigraphy sensor. 7. The method of any of aspects 1-6, wherein the plurality of subperiods are a plurality of first subperiods, each subperiod of the plurality of first subperiods being a first duration, the method further comprising: dividing the acquisition period into a plurality of second subperiods, each second subperiod being a second duration different from the first duration; aggregating at least a portion of the data into a plurality of second data subsets, each second data subset of the plurality of second data subsets including data acquired during a different second subperiod of the plurality of second subperiods; for each particular second subperiod of the plurality of second subperiods, determining a quality of the second data subset acquired during the particular second subperiod; and outputting an indication of the determined quality of at least one second data subset of the plurality of second data subsets. 8. The method of aspect 7, wherein the first duration is longer than the second duration. 9. The method of any of aspects 1-8, wherein each of the plurality of subjects is participating in a clinical study, the clinical study has one or more clinical study sites, each site associated with one or more subjects of the plurality of subjects, and the data acquired by the plurality of actigraphy sensors includes data related to the clinical study. 10. The method of aspect 9, further comprising determining study-level compliance for a plurality of subjects when using a plurality of actigraphy sensors during an acquisition period, and outputting a report indicating study-level compliance. 11. The method of aspect 9, wherein at least a portion of the data comprises data associated with a particular site of one or more clinical study sites, the particular site being associated with each of one or more subjects of the plurality of subjects, and the method further includes determining site-level compliance for each of the one or more subjects in using each of the one or more actigraphy sensors of the plurality of actigraphy sensors during an acquisition period, and outputting a report indicating the site-level compliance. 12. The method of any one of aspects 1-11, wherein outputting an indication of the determined quality of the at least one subset of data includes generating a graphical user interface and displaying the visual indication through the graphical user interface. 13. The method of any one of aspects 1 to 12, further comprising, before dividing the acquisition period into the plurality of sub-periods, receiving from the user an indication of a duration of each sub-period of the plurality of sub-periods, wherein dividing the acquisition period into the plurality of sub-periods comprises dividing the acquisition period according to the received indicated durations such that each sub-period of the plurality of sub-periods has a duration corresponding to the received indicated durations. 14. The method of any one of aspects 1-13, further comprising, prior to determining a quality of the subset of data acquired during the particular sub-period, receiving user input indicating one or more criteria for determining a quality of the subset of data acquired during the particular sub-period, wherein determining the quality of the subset of data acquired during the particular sub-period comprises determining a quality of the subset of data according to the received user input indicating the one or more criteria. 15. The method of any of aspects 1-14, wherein determining a quality of at least a portion of the data includes determining the quality in real time. 16. The method of any of aspects 1-15, further comprising: determining a pattern of non-compliance for one or more subjects among the plurality of subjects when using each of the one or more actigraphy sensors among the plurality of actigraphy sensors based on the quality determined for each sub-period of the plurality of sub-periods; and outputting an indication of the determined pattern of non-compliance. 17. The method of embodiment 16, further comprising generating an output that prompts the one or more subjects regarding how to use each of the one or more actigraphy sensors based on the determined pattern of non-compliance. 18. The method of any of aspects 1-17, wherein the acquisition period is multiple days in duration, and the method further includes identifying one or more days among the multiple days on which the first subject was non-compliant when using the first actigraphy sensor based on the quality determined for each sub-period of the multiple sub-periods, and outputting an indication of the one or more days on which the first subject was non-compliant when using the first actigraphy sensor. 19. The method of aspect 18, wherein identifying one or more days on which the first subject was not compliant when using the first actigraphy sensor includes: determining, for each of the multiple days, a percentage of the number of hours on which the first subject was not compliant when using the first actigraphy sensor relative to the total number of hours on that day; determining a subset of days on which the percentage is greater than or equal to a threshold; and identifying the determined subset of days as one or more days on which the first subject was not compliant when using the first actigraphy sensor. 20. The method of any of aspects 1-19, wherein dividing the acquisition period into a plurality of sub-periods includes dividing the acquisition period into a plurality of days, each day being further subdivided into a plurality of hours, and the method further includes identifying one or more hours among a plurality of times across the plurality of days during which the first subject was non-compliant when using the first actigraphy sensor based on the quality determined for each sub-period of the plurality of sub-periods, and outputting an indication of the one or more hours during which the first subject was non-compliant when using the first actigraphy sensor. 21. The method of aspect 20, wherein identifying one or more periods of time over a plurality of days during which the first subject was not compliant when using the first actigraphy sensor includes: for each period of the plurality of periods, determining a percentage of the number of days during the plurality of days during which the first subject was not compliant when using the first actigraphy sensor relative to the total number of days during the plurality of days; determining a subset of periods during which the percentage is greater than or equal to a threshold; and identifying the determined subset of periods as one or more periods during which the first subject was not compliant when using the first actigraphy sensor. 22. The method of any one of aspects 1-21, further comprising outputting instructions prompting at least one subject of the plurality of subjects regarding how to use at least one actigraphy sensor of the plurality of actigraphy sensors. 23. The method of any one of aspects 1-22, further comprising: for each particular subset of data among the plurality of subsets of data, determining whether the quality determined for the particular subset of data satisfies at least one criterion; and if the particular subset satisfies the at least one criterion, using the particular subset of data to predict a health outcome for at least one subject among the plurality of subjects; and if the particular subset does not satisfy the at least one criterion, refraining from using the particular subset of data to predict a health outcome for the at least one subject. 24. The method of any of aspects 1-23, wherein receiving data acquired by the plurality of actigraphy sensors includes receiving the data at least once per hour during the acquisition period. 25. The method of any one of aspects 1-24, wherein the acquisition period is at least 20 hours. 26. The method of embodiment 25, wherein the acquisition period is multiple days. 27. A system comprising: a memory that stores instructions; and a processor configured to execute the instructions to implement a method according to any one of aspects 1-26. 28. A non-transitory computer-readable medium containing instructions, the instructions, when executed by one or more processors on a computing device, being operable to cause the one or more processors to perform a method according to any of aspects 1-26.< / n>

Claims

1. 1. A method comprising: Using one or more processors receiving data from a plurality of actigraphy sensors associated with a plurality of subjects, the data acquired by the plurality of actigraphy sensors during an acquisition period, the data indicative of signals associated with physical movements of the plurality of subjects; receiving, from a first actigraphy sensor of the plurality of actigraphy sensors associated with a first subject of the plurality of subjects, first data acquired by the first actigraphy sensor, the first data indicative of a signal related to a physical movement of the first subject; Dividing the acquisition period into a plurality of sub-periods; aggregating at least some of the data, including the first data, into a plurality of data subsets, each of the plurality of data subsets including data acquired during a different one of the plurality of subperiods; determining, for each particular sub-period of the plurality of sub-periods, a quality of the subset of data acquired during the particular sub-period, said determining including: determining whether the first subject of the plurality of subjects was compliant in using the first actigraphy sensor of the plurality of actigraphy sensors during the particular sub-period; and outputting an indication of the determined quality of at least one subset of data of the plurality of subsets of data.

2. 10. The method of claim 1, wherein determining whether the first subject was compliant in using the first actigraphy sensor comprises determining whether the first subject was using the first actigraphy sensor.

3. The subset of data acquired during the particular sub-period indicates acceleration of a body portion of the first subject during the particular sub-period, and determining whether the first subject was using the first actigraphy sensor during the particular sub-period includes: comparing the acceleration of the portion of the body of the first subject to an acceleration threshold; and determining whether the first subject was using the first actigraphy sensor during the particular sub-period based on the results of the comparing.

4. 4. The method of claim 1, wherein determining the quality of the subset of data acquired during the particular sub-period further comprises determining whether the first actigraphy sensor malfunctioned during the particular sub-period.

5. determining whether the first actigraphy sensor malfunctioned during the particular subperiod includes determining a proportion of the subset of data equal to a maximum or minimum output value of the first actigraphy sensor; The method of claim 4 , wherein the indication of the determined quality of the at least one subset of data comprises an indication of the determined proportion of the subset of data.

6. 5. The method of claim 4, wherein determining whether the first actigraphy sensor has malfunctioned comprises determining whether the subset of data includes a value excluded from an output range of the first actigraphy sensor.

7. the plurality of subperiods are a plurality of first subperiods, each subperiod of the plurality of first subperiods being of a first duration; The method comprises: Dividing the acquisition period into a plurality of second sub-periods, each second sub-period being of a second duration different from the first duration; aggregating at least a portion of the data into a plurality of second data subsets, each second data subset of the plurality of second data subsets including data acquired during a different second sub-period of the plurality of second sub-periods; For each particular second sub-period of the plurality of second sub-periods, determining a quality of the subset of second data acquired during the particular second sub-period; 7. The method of claim 1, further comprising: outputting an indication of the determined quality of at least one second data subset of the plurality of second data subsets.

8. The method of claim 7 , wherein the first duration is greater than the second duration.

9. each of the plurality of subjects is participating in a clinical study, the clinical study having one or more clinical research sites, each of the one or more clinical research sites being associated with one or more subjects of the plurality of subjects; The method of any one of claims 1 to 8, wherein the data acquired by the plurality of actigraphy sensors includes data related to the clinical study.

10. determining a study-level compliance of the plurality of subjects in using the plurality of actigraphy sensors during the acquisition period; and 10. The method of claim 9, further comprising: outputting a report indicating said study-level compliance.

11. wherein the at least some of the data comprises data associated with a particular site of the one or more sites of the clinical study, the particular site being associated with each of one or more subjects of the plurality of subjects, and the method determining site-level compliance of each of the one or more subjects in using each of one or more actigraphy sensors of the plurality of actigraphy sensors during the acquisition period; 10. The method of claim 9, further comprising: outputting a report indicating the site-level compliance.

12. Outputting the indication of the determined quality of the at least one subset of data comprises: generating a graphical user interface; and displaying a visual indication through said graphical user interface.

13. receiving, from a user, an indication of a duration of each sub-period of the plurality of sub-periods prior to dividing the acquisition period into the plurality of sub-periods; 13. The method of claim 1, wherein dividing the acquisition period into the plurality of sub-periods comprises dividing the acquisition period according to the received indicated duration such that each sub-period of the plurality of sub-periods has a duration corresponding to the received indicated duration.

14. and, prior to determining the quality of the subset of data acquired during the particular sub-period, receiving user input indicating one or more criteria for determining the quality of the subset of data acquired during the particular sub-period; 14. The method of claim 1, wherein determining the quality of the subset of data acquired during the particular sub-period comprises determining the quality of the subset of data in accordance with the received user input indicating the one or more criteria.

15. The method of any preceding claim, wherein determining the quality of the at least part of the data comprises determining the quality in real time.

16. determining a pattern of non-compliance for one or more subjects among the plurality of subjects when using each of one or more actigraphy sensors among the plurality of actigraphy sensors based on the quality determined for each sub-period of the plurality of sub-periods; The method of any one of claims 1 to 15, further comprising: outputting an indication of the determined pattern of non-compliance.

17. 17. The method of claim 16, further comprising generating an output that prompts the one or more subjects regarding how to use each of the one or more actigraphy sensors based on the determined pattern of non-compliance.

18. The acquisition period is multiple days in duration, and the method comprises: identifying one or more days among the plurality of days on which the first subject was non-compliant in using the first actigraphy sensor based on the quality determined for each sub-period of the plurality of sub-periods; 18. The method of any one of claims 1-17, further comprising outputting an indication of the one or more days that the first subject was non-compliant in using the first actigraphy sensor.

19. Identifying one or more days on which the first subject was non-compliant in using the first actigraphy sensor includes: determining, for each of the plurality of days, a percentage of the number of hours that the first subject was non-compliant in using the first actigraphy sensor relative to a total number of hours for each of the days; determining a subset of days on which the percentage is equal to or greater than a threshold; and identifying the determined subset of days as the one or more days on which the first subject was non-compliant in using the first actigraphy sensor.

20. Dividing the acquisition period into a plurality of sub-periods includes dividing the acquisition period into a plurality of days, each day being further subdivided into a plurality of hours, and the method further comprises: identifying one or more hours among the plurality of times across the plurality of days in which the first subject was non-compliant in using the first actigraphy sensor based on the quality determined for each sub-period of the plurality of sub-periods; and 20. The method of any one of claims 1-19, further comprising outputting an indication of the one or more hours during which the first subject was non-compliant in using the first actigraphy sensor.

21. Identifying the one or more hours across the multiple days during which the first subject was non-compliant in using the first actigraphy sensor comprises: For each of the plurality of time periods, determining a percentage of the number of days during which the first subject was non-compliant in using the first actigraphy sensor relative to a total number of days during the plurality of time periods; determining a subset of times during which the percentage is greater than or equal to a threshold; and identifying a subset of the determined periods of time as the one or more periods during which the first subject was non-compliant in using the first actigraphy sensor.

22. 22. The method of any one of claims 1-21, further comprising outputting instructions prompting at least one subject of the plurality of subjects regarding how to use at least one actigraphy sensor of the plurality of actigraphy sensors.

23. determining, for each particular subset of data of the plurality of subsets of data, whether the quality determined for the particular subset of data satisfies at least one criterion; using the particular subset of data to predict a health outcome for at least one subject among the plurality of subjects if the particular subset meets the at least one criterion; 23. The method of any one of claims 1 to 22, further comprising refraining from using the particular subset of data to predict the health outcome of the at least one subject if the particular subset does not meet the at least one criterion.

24. 24. The method of claim 1, wherein receiving the data acquired by the plurality of actigraphy sensors comprises receiving the data at least once per hour during the acquisition period.

25. The method of any one of claims 1 to 24, wherein the acquisition period is at least 20 hours.

26. 26. The method of claim 25, wherein the acquisition period is multiple days.

27. A system comprising a memory for storing instructions and a processor configured to execute said instructions to perform the method of any one of claims 1 to 26.

28. 27. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors on a computing device, are operable to cause the one or more processors to perform the method of any one of claims 1 to 26.

Images