Data replication architecture using machine learning for clinical data from digital therapeutic application

Abstract

To provide a system and method for maintaining integrity of data during clinical trials of digital therapeutic applications.SOLUTION: A method is configured to: receive a data element generated based on interactions by a user with a digital therapeutic application during a clinical trial; send the data element to each data store of a plurality of data stores; access a first data store to retrieve a first instance of the data element; identify the first instance of the data element as invalid; and access a second data store to retrieve a second instance of the data element, responsive to the identification of the first instance of the data element in the first data store as invalid. The method may store the second instance of the data element onto a data repository for the clinical trial.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Non-Provisional Patent Application No. 18 / 740,338, filed June 11, 2024, entitled "DATA REPLICATION ARCHITECTURE USING MACHINE LEARNING FOR CLINICAL DATA FROM DIGITAL THERAPEUTIC APPLICATIONS," which is incorporated by reference in its entirety. [Background technology]

[0002] As servers hosting resources for applications on user devices become increasingly data-dependent, data integrity may become more important for such servers to accurately allocate and select resources to provide to user devices. Furthermore, maintaining data integrity is particularly important for some types of applications. For example, in the case of digital therapeutic applications, a server may aggregate data regarding user interactions and performance and then select appropriate digital therapeutic interventions for the user based on this data. These digital therapeutic interventions may be provided by the server in the form of messages to alleviate or treat various medical conditions of the user. Furthermore, data may also be aggregated during clinical trials to evaluate the efficacy and performance of digital therapeutic applications and, therefore, may be crucial to accurately conducting such evaluations.

[0003] Even a small reduction in the completeness of aggregated data can adversely affect the selection and implementation of appropriate digital therapeutic interventions. Reduced data validity can also adversely affect the ability to properly evaluate digital therapeutic applications during clinical trials. Both of these can significantly reduce the effectiveness of the therapy in addressing a user's condition. From a hardware perspective, reduced data completeness can waste computational and network resources (e.g., processing, memory, and network bandwidth) and can degrade performance and impair functionality of remote servers and user devices. Furthermore, from a human-computer interaction (HCI) perspective, reduced performance and impaired functionality can result in a reduced quality of HCI between the user and the digital therapeutic application.

[0004] Data may become unavailable for various reasons between the time of transmission from the user device and the subsequent retrieval from storage. For example, data integrity may be reduced due to an interruption in storage service or an error in the storage disk itself. This problem may be exacerbated when large amounts of user interaction data are accumulated offline on the user device and then all sent simultaneously immediately after establishing a connection with the remote server. Reduced data integrity may make it impossible for the remote server to select the appropriate functions or identify the appropriate resources to provide to the user device.

[0005] There can be many technical challenges associated with transferring and storing large amounts of data, especially when a user device transfers large amounts of data upon connection with a server. In this case, the large amount of data that can accumulate on the user device can reduce the likelihood of data integrity when the user device sends the data to a remote server upon connection establishment. In digital therapeutics, reduced data integrity can exacerbate issues with the appropriate selection and functionality of digital therapeutic content. This can occur, among other things, because it can be difficult to evaluate the performance or efficacy of digital therapeutic content delivered via a digital therapeutic application during clinical trials due to the paucity of interaction data, especially for relatively new application users who have insufficient application usage time.

[0006] Furthermore, failure to accurately and precisely receive user interaction data can result in inadequate allocation of resources (e.g., processing, memory, and network bandwidth) for delivering subsequent data and presenting and interacting with content via user interaction elements. Reduced data integrity can lead to biased or inaccurate clinical trial results, or further reduce the ability to properly evaluate the efficacy and performance of digital therapeutic applications, resulting in an inadequate digital therapy being delivered to users that has little or no effect on the user's condition. This can also lead to wasted computing resources by providing less effective digital therapeutics and reducing the quality of the HCI between the user and the application. Summary of the Invention

[0007] The present disclosure relates to systems and methods for maintaining the integrity and authenticity of data generated by user devices during clinical trials, particularly in the field of digital therapeutics. Given the critical role data plays in achieving clinical trial primary endpoints, ensuring data integrity and authenticity can reduce clinical trial costs, improve the accuracy of clinical trial results, and enable more effective use of developed treatments on patients. In particular, the novel data replication system of the present disclosure can interface with a set of data stores to replicate data related to digital therapeutic applications across all of the data stores, and then collate data from the data stores with a data repository. These data stores may serve to store the data replicas, and the data repository may serve as centralized storage for data from each of the data stores. Numerous advantages and benefits can be derived from this approach. By mitigating any risks to data integrity, this architecture can increase the overall quality of data used to evaluate the efficacy of digital therapeutics. Additionally, the quality of data collected from the data stores can be further improved by adding machine learning techniques. Ultimately, this can result in more accurate assessment of therapy outcomes and improve the allocation of digital therapeutic content, promoting the effectiveness of patient treatments while conserving computing resources.

[0008] Data integrity issues can be particularly acute when the source of data (e.g., a user device) frequently transitions between connected and disconnected states with a server on a network. The digital therapeutic application of the present disclosure can be uniquely built for frequent offline use, providing convenience to the user. When the user device is offline, the digital therapeutic application on the user device can accumulate or aggregate data generated from the user interacting with the therapeutic interventions provided through the application's user interface elements. The amount of data from the digital therapeutic application can be large, depending on the type of data, the complexity of the processing operations, and the speed of the user's interaction with the user interface elements. When the user device establishes a connection with the server, the digital therapeutic application may simultaneously send all user interaction data to the remote server. In response to receipt, the remote server may store the data regarding the user interaction on storage for subsequent retrieval and use, such as performing analysis or identifying content to provide to the user.

[0009] To address challenges such as these, a data replication system on the server can interface with a set of data stores to replicate data and a centralized data repository to further ensure data integrity. The data stores can serve to store replicas of data, and the data repository can serve as centralized storage aggregating data from each of the data stores. To this end, the server can provide digital therapeutic content to the user device for presentation via a user interface. The digital therapeutic content may instruct the user to perform activities to address the user's target condition. For example, the digital therapeutic content may be prompts to perform activities that may be in accordance with specific tasks from clinical trials, such as an emotional faces memory task (EFMT) or attention bias modification treatment (ABMT).

[0010] Additionally, the disclosed digital therapeutic application is uniquely and specifically configured to operate on a user device for frequent offline use. In particular, the digital therapeutic application may be designed and optimized to function without a constant connection to a server. First, the digital therapeutic application may have core functionality that provides digital therapeutic content to a user without relying on a constant connection to server hosting resources. This may enable a seamless user experience without relying on server connectivity. Second, the digital therapeutic application may store data locally on the user device and have the ability to write and store new data in storage when not connected to the server. When a connection to the server is re-established, the digital therapeutic application may synchronize with the server, receive updates to the digital therapeutic content from the server, and provide data accumulated while offline to the server. This may enable the digital therapeutic application to efficiently manage computing resources locally on the user device.

[0011] During a session, the user device can monitor interactions during the session and generate a set of data elements based on the interaction data. Each data element may include various information about the interaction and contextual factors, such as the type of task, the location where the user performed the task, a timestamp corresponding to the task's execution, and an indication of whether the user's response was correct for the task. The digital therapeutic application on the user device may obtain some of the data elements upon disconnection from the server hosting the digital therapeutic application. When the user device connects to the server, the digital therapeutic application can send the data elements to the server.

[0012] In response to receiving data from the user device, the data replication system communicates with the data stores to store instances of one or more data elements in each data store. The instances may correspond to copies or replicas of the interaction data generated on the digital therapeutic application. In response to receipt, each data store may store and maintain the instances of the data elements. Instances in one data store may differ from instances in subsequent data stores. For example, each data store may store instances of data elements according to a respective data store schema, such that an instance of a data element in one data store may differ from another instance of the data element in a different data store.

[0013] At a subsequent time, the data replication system may access the data store to obtain instances of the data elements. The data replication system may first identify a data store from which to obtain the machine learning model to utilize. For each instance of a given data element, the data replication system may identify whether the instance is plausible or implausible. An implausible instance may correspond to distortions (e.g., of format or structure), unexpected values ​​(e.g., outside the range of a given type of metric), reduced data integrity (e.g., of the value of the data element itself), data inconsistencies (e.g., a mismatch with another data element), or missing values ​​for the data element (e.g., a missing side effect, frequency of medication administration, or part of the diary). An implausible instance may result in deviations or invalidation of clinical data for further analysis in a digital therapeutic application. To identify whether data is plausible or implausible, the data replication system may use the trained machine learning model to determine a confidence value indicating the degree of plausibility based on the instance of the data element. For example, a data element whose value is missing or whose value deviates from the expected range may have a lower confidence value than a data element whose value is complete or whose value is within the expected range.

[0014] If the confidence value meets the threshold (e.g., greater than or equal to the threshold), the data replication system may identify the instance of the data element as valid and store the instance of the data element in the clinical trial data repository. Otherwise, if the confidence value does not meet the threshold (e.g., less than the threshold), the data replication system may identify the instance of the data element as invalid. In addition, the data replication system may access another data store to obtain a different instance of the corresponding data element. The data replication system may iterate to identify whether the data element from the other data store is valid or invalid. If the data element is identified as valid, the data replication system may store the instance of the data element in the data repository. On the other hand, if the data element is identified as invalid, the data replication system may access yet another data store and repeat the process.

[0015] As a result of replicating data elements and storing data elements identified as valid in a data repository, the data replication system can improve data integrity for clinical trials, even when the computing system receives data collected while the computing system is offline. Furthermore, the ability to replicate clinical trial results across multiple data stores provides several technical advantages and benefits. First, replicating interaction data can provide multiple copies of interactions within each data store to store instances of data related to a clinical trial when a user device's state changes from online to offline. Because each data store maintains a respective instance of the interaction data to maintain a record of the data related to a clinical trial, data elements with sufficient trust values ​​can be stored in the data repository, improving data integrity between each instance of the data.

[0016] The data in the data repository can be used to appropriately and more accurately evaluate the performance and efficacy of digital therapeutic applications, thereby compensating for the effort and time required to conduct clinical trials. The data in the data repository may also be used by the server to appropriately select and allocate more effective digital therapeutic content to be provided to user devices. This can reduce the consumption of computing resources (e.g., processor, memory, and network bandwidth) that would otherwise be wasted on delivering less effective digital therapeutics to users. From the user's perspective, the provision of higher performance and more effective digital therapeutics can lead to improvement or alleviation of the user's condition. Furthermore, special and unique configurations of digital therapeutic applications for use even when not connected to the server can significantly improve the usability and functionality of user devices, thereby improving the quality of HCI.

[0017] Aspects of the present disclosure relate to systems and methods for maintaining data integrity during clinical trials of a digital therapeutic application. One or more processors coupled to a memory may receive, in response to establishing communication with the digital therapeutic application, data elements generated based on a user's interaction with the digital therapeutic application during the clinical trial. The one or more processors may send the data elements to each data store of a plurality of data stores. The one or more processors may access a first data store of the plurality of data stores to obtain a first instance of the data element. The one or more processors may identify the first instance of the data element as lacking validity. In response to identifying the first instance of the data element in the first data store as lacking validity, the one or more processors may access a second data store of the plurality of data stores to obtain a second instance of the data element. The one or more processors may store the second instance of the data element on a clinical trial data repository.

[0018] In some embodiments, the one or more processors may identify a first instance of a second data element on a first data store as invalid. The one or more processors may determine that a second instance of the second data element on a second data store is invalid. The one or more processors may provide an indication that the second data element on the first data store and the second data store are invalid. The one or more processors may identify the second instance of the data element on the second data store as valid. The one or more processors may store the second instance of the data element in response to identifying the second instance of the data element as valid.

[0019] In some embodiments, the one or more processors may identify a first instance of a second data element on a first data store as valid. The one or more processors may store the first instance of the second data element from the first data store in a data repository without accessing the second data store to obtain the second instance of the second data element. The one or more processors may determine a confidence value indicating the validity of the first instance of the data element. The one or more processors may determine that the confidence value does not satisfy a threshold metric. In some embodiments, the one or more processors may apply a machine learning model to determine the confidence value. The one or more processors may determine, using the data elements stored on the data repository, a plurality of metrics that identify at least one of a degree of efficacy of the digital therapeutic application, a degree of performance of the digital therapeutic application, or a severity of a condition to be addressed in a user during a clinical trial.

[0020] In some embodiments, the one or more processors may provide a graphical user interface that shows information associated with data elements on at least one of the plurality of data stores or data repositories. In some embodiments, the one or more processors may identify a first data store of the plurality of data stores by applying a machine learning (ML) model. The one or more processors may obtain, from a user device running the digital therapeutic application, a plurality of data elements generated during a time when communication with the user device was not established. The one or more processors may provide a respective instance of the data element to each data store of the plurality of data stores, each instance including an encrypted copy of the data element. The digital therapeutic application is configured to generate the data elements during a clinical trial at least partially in parallel with a user receiving a medication to address a condition associated with the clinical trial. [Brief explanation of the drawings]

[0021] The above and other objects, aspects, features, and advantages of the present disclosure will become more apparent and be better understood from the following description taken in conjunction with the accompanying drawings. [Figure 1] FIG. 1 is a block diagram of a system for maintaining data integrity during clinical trials for digital therapeutic applications, according to an exemplary embodiment. [Figure 2] FIG. 1 is a block diagram of a process for generating data elements based on a user's interaction with a digital therapeutic application, according to an exemplary implementation. [Figure 3] FIG. 1 is a block diagram of a process for accessing multiple data stores to obtain an instance of a data element, validating the data element, and storing the data element in a data repository for a clinical trial, according to an exemplary embodiment. [Figure 4]FIG. 1 is a block diagram of a process for generating a plurality of metrics using data elements from a data repository and sending the output to a management computing device according to an exemplary implementation. [Figure 5] 1 illustrates an example of a dashboard provided by a data replication system, according to an illustrative embodiment. [Figure 6] FIG. 1 is a block diagram of a data replication architecture for clinical trials, according to an exemplary embodiment. [Figure 7] FIG. 1 is a block diagram of a method for maintaining data integrity during clinical trials for digital therapeutic applications, according to an exemplary embodiment. [Figure 8] FIG. 1 is a block diagram of a server system and a client computer system according to an exemplary embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0022] For purposes of reading the following description of the various embodiments, the following listed sections of the specification and their respective contents may be helpful:

[0023] Section A describes systems and methods for data replication of data related to clinical trials for digital therapeutic applications.

[0024] Section B describes network and computing environments that may be useful for implementing embodiments of the present disclosure. A. Systems and methods for data replication of data related to clinical trials for digital therapeutic applications

[0025] Referring now to FIG. 1 , a block diagram of a system 100 for maintaining data integrity during clinical trials for digital therapeutic applications is illustrated. Generally, system 100 may include at least one data management service 105, a management device 110, a set of user devices 120A-N (collectively referred to herein as user devices 120), a plurality of data stores 170A-N (collectively referred to herein as data stores 170), and a repository 165, communicatively coupled to each other via at least one network 115. At least one of the user devices 120 (e.g., the illustrated user device 120A) may include at least one application 125. Application 125 may include or provide at least one user interface 130 having one or more user interface (UI) elements 135A-N (collectively referred to herein as UI elements 135). The data management service 105 may include at least one session handler 140, at least one replication manager 145, at least one data validator 150, at least one metric calculator 155, at least one interface provider 160, and at least one machine learning (ML) model 180, among others. The data management service 105 may include or have access to at least one repository 165. The functionality of the application 125 on the user device 120 may be partially executed on the data management service 105, or vice versa. Each of the components of the system 100 may be embodied using the computing system described in Section B.

[0026] More specifically, data management service 105 (collectively referred to in this disclosure as a messaging service) may be any computing device comprising one or more processors coupled to memory and software and capable of executing the various processes and tasks described in this disclosure. Data management service 105 may be associated with an entity that manages the provision of resources and content for application 125 or the delivery of digital therapeutics to users of user devices 120. In some embodiments, data management service 105 may be associated with an entity that oversees at least one clinical trial associated with a digital therapeutic provided via application 125. Data management service 105 may communicate with one or more user devices 120 and repository 165 via network 115. Data management service 105 may be located within, located in, or associated with at least one computer system. The computer system may correspond to a data center, branch office, or location where one or more computers corresponding to data management service 105 are located.

[0027] Within the data management service 105, the session handler 140 may manage sessions and receive interactions from users. The replication manager 145 may identify data elements to distribute within each data store 170. The data validator 150 may validate the data elements in each data store 170 and store the validated data elements in the repository 165. The metric calculator 155 may calculate metrics associated with the data elements in the repository 165. The interface provider 160 may provide a dashboard of metrics associated with the data elements. In some embodiments, the data management service 105 may be part of a service (e.g., having the functionality of the illustrated session handler 140) that is managing the session. In some embodiments, the data management service 105 may be separate from the service. For example, the data management service 105 may include the functionality of the replication manager 145, the data validator 150, and the metric calculator 155 and may communicate with a separate service (e.g., having the functionality of the illustrated session handler 140) to exchange interaction data through the service.

[0028] The ML model 180 may include any machine learning model or artificial intelligence (AI) algorithm for selecting a data store 170 and determining whether an instance of a data element is plausible. In some implementations, the data management service 105 may include one ML model 180 for selecting a data store 170 and another ML model 180 for determining whether a data element is plausible. The architecture for the machine learning model of the predictive model 180 may include, for example, a deep learning neural network (e.g., a convolutional neural model architecture), a regression model (e.g., a linear or logistic regression model), a random forest, a (e.g., a linear or logistic) regression model, a support vector machine (SVM), a clustering algorithm (e.g., k-nearest neighbors), or a naive Bayes model. In general, the predictive model 180 may have at least one input and one output. The input and output may be related via a set of weights. The input may include various data about the data store 170 and the data elements stored on the data store 170. The output may include a selection indication and a confidence value. The set of weights may be based on a machine learning architecture. The ML model 180 may be trained using training data (e.g., according to supervised learning). In general, the ML model 180 may include a set of inputs and a set of outputs that are interrelated via a set of weights. The weights may be distributed according to an architecture for the machine learning model.

[0029] The management device 110 may be any computing device having one or more processors coupled to memory and software capable of executing the various processes and tasks described in this disclosure. The management device 110 may communicate with the data management service 105 and the data store 170 over a network 115. The management device 110 may be a laptop computer, tablet computer, or smartphone for receiving output from the data management service 105. The output may trigger one or more UI elements of a user interface of the management device 110 to provide a dashboard for managing data related to clinical trials.

[0030] User device 120 (sometimes referred to in this disclosure as an end-user computing device) may be any computing device including one or more processors coupled to memory and software capable of executing the various processes and tasks described in this disclosure. User device 120 may communicate with data management service 105 and data store 170 via network 115. User device 120 may be a smartphone, other mobile phone, tablet computer, wearable computing device (e.g., smartwatch, eyeglasses), or laptop computer. User device 120 may be used to access applications 125. In some implementations, applications 125 may be downloaded and installed on user device 120 (e.g., as a native app or a mobile app from a digital distribution platform). In some implementations, applications 125 may be web applications with resources accessible via network 115. The functionality of applications 125 may not depend on connectivity to network 115.

[0031] The application 125 executing on the user device 120 may be a digital therapeutic application. The application 125 may present or provide sessions (sometimes referred to in this disclosure as therapy sessions) according to clinical trials to address at least one condition of the user. The end user's condition may include, for example, chronic pain (such as those associated with or including arthritis, migraine, fibromyalgia, back pain, Lyme disease, endometriosis, repetitive strain injury, irritable bowel syndrome, inflammatory bowel syndrome, and cancer pain), skin conditions (e.g., atopic dermatitis, psoriasis, self-inflicted dermatopathy, and eczema), dementia (e.g., mild cognitive impairment (MCI), Alzheimer's disease, multiple sclerosis, and schizophrenia), psychiatric conditions (e.g., affective disorders, bipolar disorder, obsessive-compulsive disorder, borderline personality disorder, and attention-deficit / hyperactivity disorder), substance use disorders (e.g., opioid use disorder, alcohol use disorder, tobacco use disorder, or hallucinogen disorder), and other conditions (e.g., insomnia and tumors or cancer).

[0032] The end user may receive medication to address a condition at least partially concurrently with use of the application 125 (e.g., over any number of sessions). For example, if the medication is for pain, the end user may be taking acetaminophen, a nonsteroidal anti-inflammatory composition, an antidepressant, an anticonvulsant, or other composition. If the medication is for skin lesions, the end user may be taking a steroid, an antihistamine, or a topical antiseptic. If the medication is for dementia, the end user may be taking a cholinesterase inhibitor, memantine, or other medication. If the medication is for a psychiatric condition, the end user may be taking an antidepressant, a mood stabilizer, an antipsychotic, a tranquilizer, or a stimulant. If the medication is for substance use disorder, the end user may be taking naltrexone, disulfiram, acamprosate, or nicotine replacement therapy. The end user may also participate in other psychological therapies for these conditions. In some embodiments, digital therapeutic content may be provided to the end user within the digital therapeutic application toward achieving the end user's endpoint. The endpoint may be, for example, an end user's physical or behavioral goal, completion of a medication regimen, or an endpoint indicated by a physician or end user. At least one of the user devices 120 may have a digital therapeutic application and may provide sessions (sometimes referred to in this disclosure as therapy sessions) to address at least one condition of the end user. The application 125 may augment the efficacy of a medication the user is taking to address the condition.

[0033] The application 125 may include, present, or provide to a user of the user device 120 at least one user interface 130 including one or more user interface elements 135A-N (hereinafter collectively referred to as UI elements 135). The user interface 130 may be provided according to a configuration on the application 125. The UI elements 135 may correspond to visual elements of the user interface 130, such as command buttons, text boxes, check boxes, radio buttons, menu items, and sliders. In some implementations, the application 125 may be a digital therapeutic application and may provide sessions (sometimes referred to in this disclosure as therapy sessions) via the user interface 130 to address a condition. The user interface 130 may include a set of UI elements 135 for presenting digital therapeutic content.

[0034] Digital therapeutic content can be of any modality, such as text, images, audio, video, or multimedia content, or a combination thereof. Content items can be stored and maintained in the repository 165 using one or more files. For example, in the case of text, digital therapeutic content may be stored as text files (TXT), rich text files (RTF), extensible markup language (XML), hypertext markup language (HTML), etc. In the case of images, digital therapeutic content may be stored as Joint Photographic Experts Group (JPEG) format, Portable Network Graphics (PNG) format, Graphics Interchange Format (GIF), or Scalable Vector Graphics (SVG) format, etc. In the case of audio, digital therapeutic content may be stored as waveform audio files (WAV), moving image experts group formats (e.g., MP3 and MP4), Ogg-Volvis (OGG) format, etc. For video, the digital therapeutic content may be stored as a moving image expert group format (e.g., MP3 and MP4), QuickTime Movie (MOV), Windows Movie Video (WMV), etc. For multimedia content, the digital therapeutic content may be stored as Audio Video Interleave (AVI), a moving image expert group format (e.g., MP3 and MP4), QuickTime Movie (MOV), Windows Movie Video (WMV), etc.

[0035] Digital therapeutic content may include a set of stimuli (e.g., in the form of audio, visual, or text) that guide users through specific tasks according to clinical trials. Tasks may include, for example, implicit association tasks (IATs) (e.g., associating stimuli with concepts), attention bias modification training (ABMTs) (e.g., training users to shift attention away from specific stimuli), emotional faces memory tasks (EFMTs) (e.g., testing users' ability to recognize and remember specific facial emotions), digital support tools (DSTs) (e.g., providing messages based on the user's state), and adaptive goal setting (AGSs) (e.g., providing messages based on dynamic objectives for the user).

[0036] The application 125 may be provided to the user device 120 as part of a clinical trial. The clinical trial may be managed, overseen, or executed by an entity associated with the data management service 105. The clinical trial may include a study or survey to evaluate the performance or efficacy of a digital therapeutic provided to a user of the user device 120 via the application 125. The clinical trial may span a set period of time, ranging from seven days to six months. The clinical trial may have defined objectives, study design, cohorts, type of intervention, control, and one or more outcome measures. The objective may include evaluating the impact or effectiveness of the digital therapeutic on a user via the application 125. The design may include, for example, an exploratory trial, a basket trial, a pragmatic clinical trial, a comparative trial, a randomized control trial (RCT), or a single-arm study. The cohort (or study population) may identify which subset of the population is eligible to participate in the clinical trial. The intervention may identify what type of task the user should be instructed to perform via the application 125. The control may identify whether a subset of users should receive a placebo or a digital therapeutic intervention. The outcome measure may define one or more metrics associated with a condition, such as, for example, symptom severity, health outcomes, adherence rates, biomarkers, or behavioral data.

[0037] Repository 165 may be a centralized data storage for storing and maintaining data from a set of data stores 170. Data from a set of data stores 170 may be structured, semi-structured, or unstructured.

[0038] In some embodiments, repository 165 may be a data lake for collecting, storing, or managing data related to clinical trials. Repository 165 may collect structured clinical trial data (e.g., patient demographics, medical records, or study protocols), semi-structured data, and unstructured data (e.g., consultation notes, lab results, imaging scans, or patient-reported outcomes), etc. Repository 165 may prevent data integrity of patient information from being compromised during clinical trial collection. Repository 165 may include metadata for organizing, sorting, or categorizing the clinical trial data. For example, the metadata may organize the clinical trial data so that a user (e.g., a clinical trial clinician or administrator) can navigate within repository 165 to relevant clinical trial data. In some embodiments, repository 165 may be a structured database (e.g., a relational database management system (DBMS)) for storing clinical trial data in a structured format. The format may be structured based on a protocol for defining at least one of data types, data relationships, data constraints, etc. A user (e.g., a clinician) may use tables to access the data using structured query language (SQL). During the performance of various operations, data management service 105 and applications 125 may access repository 165 to retrieve identified data therefrom. Data management service 105 and applications 125 may write data from the performance of such operations onto repository 165.

[0039] Each data store 170 may store and maintain various resources and data related to the data management service 105 and the applications 125. Each data store 170 may be a structured, unstructured, or semi-structured repository. Each data store 170 may include a cloud storage service, a file system, a data lake, a key-value store, etc. In some embodiments, the data store 170 may include a database management system (DBMS) for arranging and organizing data maintained on the data store 170, such as instances of data elements. The DBMS may include a structured collection of data using tables with an organized schema. For example, the DBMS may be an SQL database or a NoSQL database. In some embodiments, the data store 170 may include an event queue (or message queue) for storing and maintaining data related to events (e.g., on the user device 120 or elsewhere). The data store 170 may communicate with the data management service 105 and one or more user devices 120 via the network 115. During the performance of various operations, data management service 105 and applications 125 may access, send and retrieve identified data from, data store 170. Data management service 105 and applications 125 may also write data to data store 170 from the performance of such operations.

[0040] In some embodiments, at least one data store 170 may be associated with a third party. In some embodiments, the third party may be at least one of a pharmaceutical company, a research laboratory, a health information exchange network, a health insurance company, a medical research organization, a medical imaging center, a medical teaching institution, or a hospital. For example, data management service 105 may access data store 170A of a hospital to store clinical trial data related to patients. In some embodiments, at least one data store 170 may be associated with an entity overseeing a clinical trial (e.g., the same entity as data management service 105).

[0041] Referring now to FIG. 2 , a process 200 for generating data elements and replicating data elements based on interactions between a user and a digital therapeutic application is illustrated. Process 200 may include or correspond to operations performed by system 100 to generate data elements from interactions with a session. In process 200, a session handler 140 running on data management service 105 may send, transmit, or provide instructions for at least one session 202 to be presented via application 125. The instructions may identify or include clinical trial content to be presented to a user 205 via a user interface 130 for application 125. The instructions may include specifications for which UI elements 135 to use and may also identify content related to the clinical trial to be displayed on the UI elements 135 of user interface 130. The instructions may be code, data packets, or controls for presenting a session to a user 205 via an application 125 running on user device 120.

[0042] In session 202, session handler 140 may create, write, or generate instructions. The generation of instructions may be based on digital therapeutic content associated with user 205. For example, session handler 140 may select content for application 125 based on digital therapeutic content associated with user 205. In some embodiments, session handler 140 may generate instructions for including digital therapeutic content in session 202 that includes a message for user 205. For example, the message may include exclusion criteria for a clinical trial. In some embodiments, session handler 140 may generate instructions for including digital therapeutic content in session 202 that prompts user 205 to perform a task. A clinical trial may include objectives, inclusion criteria, study design, exclusion criteria, intervention outcome measures, study duration, statistical analysis, expected benefits, potential risks, ethical considerations, etc. A session 202 may correspond to a set number of tasks to be performed by user 205 within a specified time period. In response to the generation, session handler 140 may send instructions for session 202 to user device 120.

[0043] An application 125 on user device 120 may obtain, identify, or receive instructions for session 202. The application 125 may execute, implement, or perform the instructions for session 202. In accordance with the instructions, application 125 may render, display, or present digital therapeutic content via UI elements 135 of user interface 130. For example, user 205 may participate in a clinical trial evaluating the effectiveness of performing a task to address mild cognitive impairment (MCI). Session 202 may include messages about how frequently to perform activities related to the task during the clinical trial. In some embodiments, application 125 may include (e.g., preloaded or installed) instructions for session 202. In some embodiments, application 125 may execute session 202 following receipt (e.g., without relying on connectivity with network 115).

[0044] In response to the presentation of content for session 202, application 125 on user device 120 may detect, record, or monitor at least one interaction 208 by user 205 with application 125. Each interaction 208 may correspond to one or more tasks, actions, interventions, assessments, criteria, etc. Application 125 may monitor interactions 208 using event listeners or handlers on UI elements 135 of user interface 130. In some implementations, application 125 may monitor interactions 208 using listeners associated with input / output devices on user device 120. For example, application 125 may use actionable objects on user interface 130 to detect button presses, fingerprints, keyboard interactions, etc.

[0045] Based on the interactions 208, the application 125 may write, create, or generate interaction data 204 identifying interactions 208 by the user 205 with the sessions 202 for the clinical trial. In some embodiments, the application 125 may generate interaction data 204 identifying interactions 208 with the clinical trial over a certain amount of time (e.g., 1 to 10 interactions 208 with the sessions 202 provided to the user 205). The user device 120 may be offline (e.g., disconnected from Wi-Fi, cellular data, Internet provider, etc.) for a certain amount of time (e.g., 1 minute, 1 hour, 1 day, 1 week, 1 month, etc.). The data management service 105 may be unable to establish a connection with the user device 120 when the user device 120 is offline. For example, the user device 120 may be offline for 1 to 3 days, yet the application 125 may record the interactions 208 of the user 205. In some implementations, the interaction data 204 may include a log of interactions 208 detected on a UI element 135 of the user interface 130 that presents digital therapeutic content and information derived from the set of interactions 208. The interaction data 204 may include timestamps and metadata to reference individual interactions with a session 202, the digital therapeutic content of the session 202, or the applications 125 of the user device 120, etc.

[0046] The timestamps may identify, for example, when the digital therapeutic content of a session 202 was presented to the user 205, when the session 202 was provided to the user 205, when the user device 120 was reported offline, the duration the user device 120 was offline, and when the user 205 interacted with the user interface 130 of the application 125. In some implementations, the interaction data 204 may include identification information to distinguish which session 202 the user 205 interacted with. For example, the user 205 may be involved in one or more clinical trials, and therefore the session handler 140 may provide one or more sessions 202 to the user device 120. As the user 205 interacts with each session 202, the interaction data 204 may include identification information corresponding to each session 202.

[0047] Interaction data 204 may include a set of data elements 206A-N (collectively referred to as elements 206). Each of the data elements 206 may include information related to an interaction 208. In some implementations, data elements 206 may be structured and include one or more fields and one or more corresponding values. Each field may include various measurements, such as the type of interaction, a timestamp associated with the interaction, whether the interaction was a correct or incorrect response to a task, user characteristic data, medical history, baseline characteristics, an indication of an adverse event (e.g., a side effect), an efficacy outcome, a safety assessment, adherence data, participant-reported outcomes, quality of life measures, or data related to concomitant medications. For example, a user 205 may report side effects, compliance data, and efficacy outcomes for digital therapeutic content within an actionable object of application 125. Application 125 may generate interaction data 204 that includes side effects, compliance data, efficacy outcomes, and the like as elements 206. In some embodiments, data elements 206 may be unstructured and may contain similar or identical information as described above with respect to structured data. For example, data elements 206 may include free text data entered by user 205 that identifies a patient-reported outcome.

[0048] The user device 120 may be offline or disconnected from the data management service 105 for a certain amount of time (eg, 1 hour to 4 days). During this time, the application 125 may generate, store, and maintain the interaction data 204 locally. For example, the application 125 may generate first interaction data 204, second interaction data 204, and third interaction data 204 when the user device 120 is offline. Each interaction data 204 may have a corresponding element 206 (e.g., the first interaction data 204 and the first element 206, the second interaction data 204 and the second element 206). After the device becomes online, the user device 120 may establish, link, or initiate communication with the data management service 105. In response to establishing a connection, the application 125 may send, provide, or transmit the interaction data 204 (e.g., the first interaction data 204, the second interaction data 204, and the third interaction data 204) to the data management service 105.

[0049] The replication manager 145 may receive, identify, or obtain the interaction data 204 from the user device 120. The replication manager 145 may obtain the interaction data 204 when the user device 120 establishes a connection with the data management service 105. In some implementations, the application 125 may transmit, send, or provide a signal to the data management service 105 to identify that the user device 120 has transitioned to an online state. The online state may indicate that the device is connected to a wireless or wired Internet connection. For example, before the application 125 transmits the interaction data 204 to the replication manager 145 of the data management service 105, the application 125 may transmit this signal to indicate that the user device 120 is online and ready to synchronize with the data management service 105. In response to receiving this signal, the data management service 105 may invoke the replication manager 145 by sending a signal preamble to the replication manager 145.

[0050] In response to receiving the interaction data 204, the replication manager 145 may parse or process the interaction data 204 to extract, obtain, or identify the elements 206 from the interaction data 204. In some embodiments, the replication manager 145 may use the interaction data 204 to generate, create, or determine the elements 206. In some embodiments, the signals from the applications 125 may include labeling, mapping, or associations that allow the replication manager 145 to identify the elements 206. For example, an actionable object involved in the interaction 208 may include a data field that indicates the elements 206 to the replication manager 145. In some embodiments, to extract the elements 206, the replication manager 145 may perform signal conditioning, analog-to-digital conversion, detection and synchronization, decoding algorithms, and the like.

[0051] In response to identifying each element 206, the replication manager 145 may replicate, copy, or recreate the element 206 as instances 210A-N (hereinafter collectively referred to as instances 210) for each data store 170. Each instance 210 may correspond to or include a copy of one or more data elements 206 of the interaction data 204. In response to creating the instance 210, the replication manager 145 may store the instance 210 of the element 206 on each data store 170. The data management service 105 may, for each instance 210, submit, transmit, or provide the instance 210 to each data store 170.

[0052] In some embodiments, the replication manager 145 may monitor, maintain, or track the number of data stores 170 connected to the data management service 105. The number of data stores 170 may increase or decrease depending on the entities involved in the clinical trial. For example, during a first clinical trial, the data management service 105 may connect to four to eight data stores 170 associated with a hospital, a research lab, a medical education institution, etc. Accordingly, the replication manager 145 may replicate elements 206 associated with the first clinical trial for each of the four to eight data stores 170. Subsequently, during a second clinical trial, the data management service 105 may connect to one to three data stores 170 associated with a hospital. Accordingly, the replication manager 145 may replicate elements 206 associated with the second clinical trial for each of the one to three data stores 170. The number of replicated elements 206 corresponds to the number of data stores 170 of the entities participating in the clinical trial. For example, if four data stores 170 (e.g., data store 170A, data store 170B, data store 170C, and data store 170D) are participating in a clinical trial, the replication manager 145 may generate at least four replicated elements 206.

[0053] In response to receiving the instance 210, each data store 170 may store and maintain the data element 206 of the instance 210. Each data store 170 may store, maintain, or maintain the instance 210 of the element 206. For example, a first data store 170A may maintain a first instance 210A of the element 206, while a second data store 170B may maintain a second instance 210B of the element 206. The instance 210 may store, maintain, or maintain the element 206 in a data structure according to the schema of the data store 170. The data structure may be at least one of a dataset, a table, an array, a list, a HashMap, or the like, specified by the schema of the data store 170 for storing each element 206 in association with a user 205 and a clinical trial. For example, the data structure of the instance 210 may contain each element 206 of the multiple elements 206 in a table within the data structure. Each instance 210 may represent a replica or copy of an element 206 across each data store 170. For example, each instance 210 (e.g., instance 210A, instance 210B, instance 210C) may store a replicated element 206. Replicating elements 206 across each data store 170 may allow the data management service 105 to maintain instances 210 within each data store 170. Each instance 210 may include the same replicated element 206. For example, data store 170A may include instance 210A including elements 206A-D, data store 170B may include instance 210B including elements 206A-D, data store 170C may include instance 210C including elements 206A-D, etc. The replication manager 145 can use the created data stores 170 to create or replicate elements 206 and increase the number of instances 210 of the element.

[0054] Referring now to FIG. 3 , a block diagram of a process 300 is shown for accessing multiple data stores 170 to obtain instances 210 of data elements 206, validating the data elements 206, and storing the data elements 206 in a repository 165 for clinical trial data 306. Process 300 may include or correspond to operations performed by system 100 to verify data elements 206 from interactions with a session. In process 300, data validator 150 may access one or more data stores 170 to obtain or acquire at least one element 206 for storage in repository 165. In some embodiments, data validator 150 may identify or select one or more data stores 170 to obtain or acquire according to a sequence or schedule. The sequence may specify a priority or order in which data validator 150 accesses data stores 170. The sequence may also identify a chronological order in which data stores 170 are accessed. In the illustrated example, data validator 150 may select or identify a first data store 170A for access.

[0055] In some implementations, the data validator 150 may use the ML model 180 to identify or select a data store 170 for accessing the element 206. The data validator 150 (or the data management service 105) may train the ML model 180 to select or identify a data store 170 from which the element 206 is accessed. The ML model 180 may be trained using a training dataset according to supervised learning, unsupervised learning, or weakly supervised learning. The training dataset may identify multiple examples. Each example may include a set of metrics associated with a given data store and a label indicating whether to select or exclude the given data store. The metrics may be various measures of the performance of the data store or any other measure that can predict data integrity in the data store. The metrics may include, for example, query response time, query throughput, query latency, resource utilization, remaining capacity, index utilization, transactions per section, commit rate, rollback rate, failure rate, or downtime. Data validator 150 may apply the set of metrics from each training data example to ML model 180 to generate a selection indication. The selection indication may indicate whether a given data store should be selected. Data validator 150 may compare the output selection indication with the label indications of the training data examples to generate a loss metric (e.g., mean squared loss, mean absolute error, or cross-entropy loss). Data validator 150 may use the loss metric to update the weights of ML model 180. This process may be repeated until a convergence condition is reached.

[0056] The data validator 150 may use the ML model 180 to identify or select a data store 170 for accessing the element 206. To make the selection, the data validator 150 may determine or identify network metrics for each of the data stores 170. In response to the identification, the data validator 150 may input or apply the network metrics of the given data store 170 to the ML model 180. In response to the application, the data validator 150 may process the input according to the weights of the ML model 180. The data validator 150 may create, output, or generate a selection indication from the processing. If the selection indication indicates that the data store 170 is not eligible for selection, the data validator 150 may repeat the process by applying network metrics of another data store 170 to the ML model 180. Conversely, if the selection indication indicates that the data store 170 is eligible for selection, the data validator 150 may select the data store 170 (e.g., the first data store 170A shown) as the source for the element 206.

[0057] The data validator 150 may transmit, send, or provide an access request 302A to the first data store 170A to access the first data store 170A. The access request 302A may be a signal or communication from the data management service 105 to access the first data store 170A, the instance 210, or the element 206. For example, the data management service 105 may transmit the access request 302A to the hospital data store 170 to access the instance 210A that includes the element 206A. In another example, the data validator 150 may transmit the access request 302A to the laboratory data store 170 to access the instance 210A that includes the element 206A. The access request 302A may include at least one of identification information, clinical trial information, and verification, etc., to distinguish between multiple data stores 170 (e.g., data store 170A, data store 170B). For example, access request 302A may correspond to data store 170A, and access request 302B may correspond to data store 170B.

[0058] A first data store 170A may receive, acquire, or obtain an access request 302A from data validator 150. Data store 170 may extract information about the access request 302A (e.g., identifying information, clinical trial information, etc.) and approve or disapprove the access request 302A based on a match between the information in the access request 302A and the data structures of instance 210. For example, data validator 150 may send an access request 302A, including a clinical trial and a user 205 associated with the clinical trial, to data store 170. Data store 170 may extract and match user 205 with elements 206 in instance 210A. After data store 170 completes the match, data store 170 can approve the access request 302A.

[0059] In response to approving the access request 302A, the first data store 170A may transmit, send, or provide the requested instance 210A that includes the element 206A. In some implementations, requested instance 210 may include one or more elements 206. For example, requested instance 210A may include elements 206A-C. In another example, requested instance 210A may include elements 206A-F. Data store 170A may route instance 210A according to the clinical trial in access request 302A. From there, data store 170A may route instance 210A according to the clinical trial.

[0060] When data store 170A transmits instance 210 including element 206A, data validator 150 may obtain, receive, or identify instance 210A including element 206A. In some embodiments, data validator 150 may verify receipt of instance 210 by sending a confirmation request. The confirmation request may include a signature, an indication, a flag, etc. to acknowledge secure receipt of instance 210A including element 206A. Data management service 105 may enable communication with clinical trial data 306 stored in data store 170 and may interact with multiple instances 210 (e.g., instance 210A, instance 210B) to determine which instances should be stored in repository 165. In response to receipt, data validator 150 may parse instance 210A to obtain, extract, or identify element 206A from instance 210A.

[0061] Additionally, the data validator 150 may identify, determine, or indicate whether the instance 210A of the element 206A is valid or invalid. A valid instance 210A of the element 206A may, for example, have valid values ​​for all fields of the element 206A. On the other hand, an invalid instance 210A of the element 206A may involve one or more factors. For example, the instance 210A of the element 206A may include one or more unexpected values ​​in the interaction data 204. If the user device 120 is offline for an extended period of time, the data validator 150 can store and maintain the interaction data 204 in the memory of the user device 120.

[0062] In some implementations, to determine whether instance 210A is valid or invalid, data validator 150 may generate, calculate, or determine a confidence value for instance 210A of element 206A. The confidence value may indicate, identify, or specify the validity of instance 210A of element 206A. Data validator 150 may determine the confidence value as a function of the value in element 206A. Generally, if element 206A has missing values, is incomplete, has a value outside the expected range for the field, etc., data validator 150 may determine a lower confidence value. Conversely, if element 206 has no missing values, is complete, or has a value within the expected range for the field, data validator 150 may determine a higher confidence value. In some implementations, data validator 150 may identify the field type of each field in element 206. Data validator 150 may compare the value corresponding to the field type to the expected range for that field. The data validator 150 may use the confidence values ​​of each field within an element 206 (eg, as a sum, average, or weighted sum) to determine an aggregate confidence value for the element 206 .

[0063] In some embodiments, data validator 150 may use ML model 180 to identify or determine whether instance 210A of element 206A is valid or invalid. In some embodiments, data validator 150 may use ML model 180 to calculate, generate, or determine a confidence value for instance 210A of element 206A. Data validator 150 (or data management service 105) may train ML model 180 to determine the confidence value. ML model 180 may be trained using a training data set according to supervised learning, unsupervised learning, or weakly supervised learning. The training data set may identify multiple examples. Each example may include a sample data element (e.g., a valid value, an invalid value, or a null value) and a label indicating whether the sample data element is valid or invalid. Data validator 150 may apply the sample data element from each example to ML model 180 to create or generate an output indicating whether the sample element is valid or invalid. Data validator 150 may compare the output representation with the label representations of examples in the training data to generate a loss metric (e.g., mean squared loss, mean absolute error, or cross-entropy loss). Data validator 150 may use the loss metric to update the weights of ML model 180. This process may be repeated until a convergence condition is reached.

[0064] Data validator 150 may use ML model 180 to identify or determine whether instance 210A of element 206A is valid or invalid. To determine, data validator 150 may input or apply at least a portion of instance 210A of element 206A to ML model 180. In response to the application, data validator 150 may process input instance 210A of element 206A according to the weights of ML model 180. From the processing, data validator 150 may create, output, or generate an indication of whether instance 210A of element 206A is valid or invalid. In some implementations, data validator 150 may create, output, or generate a confidence value for instance 210A from ML model 180. The confidence value may be used to determine whether instance 210A of element 206A is valid or invalid, as described later in this disclosure.

[0065] The data validator 150 may compare the confidence value to a threshold to determine whether the data validator 150 can store the element 206A in the repository 165. The threshold may indicate a maximum validity of the instance 210 of the element 206 for storing the element 206 in the repository 165. The threshold may be based on the element 206 in the repository 165. For example, the data validator 150 may generate, determine, or calculate the threshold using a confidence level of the element 206 in the repository 165 corresponding to a clinical trial. If the confidence value meets (e.g., is greater than or equal to) the threshold, the data validator 150 may identify the instance 210A that includes the element 206A as valid. In response to identifying the instance 210A as valid, the data validator 150 may store the element 206A in the repository 165. The data validator 150 may identify the next element 206 (e.g., data element 206B) and repeat the process 300 as described further herein.

[0066] Concurrent with storing element 206A in repository 165, data validator 150 may generate, create, or create an indication 304 indicating that instance 210A of element 206A is valid. In some embodiments, data validator 150 may send indication 304 to managing device 110 to identify the stored element 206A. For example, data validator 150 may send indication 304 of valid element 206A to managing device 110. Managing device 110 may approve the indication 304 of the stored element 206A. In some embodiments, data validator 150 may detect, identify, or monitor indication 304 within managing device 110. In response to managing device 110 approving indication 304, data validator 150 may provide element 206A to each data store 170 for inclusion of element 206A with the highest trust value. In some implementations, the data validator 150 may provide the elements 206A to each data store 170 for inclusion without detecting the indication 304, the elements 206A having the highest confidence values.

[0067] If the confidence value does not meet (e.g., is less than) the threshold, data validator 150 may determine or identify instance 210A that includes element 206A as invalid. In response to identifying element 206A as invalid, data validator 150 may access second data store 170B to retrieve, obtain, or identify a second instance 210B of the corresponding element 206A. To gain access, data validator 150 may provide, transmit, or send an access request 302B to second data store 170B. In response to approving access request 302B, second data store 170B may transmit, transmit, or provide a second instance 210B that includes element 206A. The second instance 210B may be different from the first instance 210A. In some cases, the second instance 210B of element 206A may include one or more factors more than the first instance 210A of element 206A, which may cause the second instance 210B of element 206A to have a lower trust value than the trust value associated with the first instance 210A of element 206A. In some cases, the second instance 210B of element 206A may include one or more factors less than the first instance 210A of element 206A, which may cause the second instance 210B of element 206A to have a higher trust value than the trust value associated with the first instance 210A of element 206A.

[0068] Data validator 150 may obtain, receive, or identify instance 210B, which includes element 206A, in a manner similar to that for instance 210A. In response to receiving instance 210B, which includes element 206A, data validator 150 may identify, determine, or indicate that instance 210B of element 206A is valid. To determine that instance 210B is valid, data validator 150 may generate, calculate, or determine a confidence value for instance 210B of element 206A. The confidence value may be determined in a manner similar to that described for instance 210A of element 206A. In response to the determination, data validator 150 may compare the confidence value of instance 210B of element 206A to a threshold. If the confidence value does not meet the threshold, data validator 150 may determine that element 206A of instance 210B also lacks validity. Otherwise, if the confidence value meets the threshold, data validator 150 may determine that element 206A of instance 210B is valid.

[0069] In some embodiments, if an instance 210A that includes element 206A lacks validity, data validator 150 may generate, create, or produce an indication 304. Indication 304 may identify element 206A as lacking validity across multiple instances 210 or multiple data stores 170. In some embodiments, data validator 150 may send indication 304 to managing device 110 to identify the invalid element 206A. For example, data validator 150 may send indication 304 of valid element 206A to managing device 110. Managing device 110 may present indication 304 to a clinical trial administrator. The administrator may use the information provided by indication 304 to diagnose or modify session 202, application 125 itself, or digital therapeutic content presented within the clinical trial.

[0070] In some implementations, data validator 150 may compare the trust value of instance 210B that includes element 206A with the trust value of instance 210A that includes element 206A based on a determination that none of the instances 210 of element 206 have a trust value above a threshold. For example, if none of the instances 210 on data store 170 that include element 206A have a trust value higher than a threshold, data validator 150 may select and store in repository 165 the instance 210 that includes element 206 with the highest trust value.

[0071] If instance 210B is valid, data validator 150 may generate, create, or construct display 304 and send this display to management device 110 as described above. Data validator 150 may store element 206B on repository 165. In some embodiments, data validator 150 may transmit, send, or provide element 206B to repository 165. Repository 165 may store, maintain, or store element 206 in clinical trial data 306. Repository 165 may include elements 206 with the highest confidence value or a confidence value above a threshold. For example, elements 206 in repository 165 may have confidence values ​​above a threshold. In this way, clinical trial data 306 may include accurate elements 206 for each clinical trial. Data management service 105 may utilize elements 206 in clinical trial data 306 to provide to a user of management device 110. For example, a user using the management device 110 may analyze elements 206 of clinical trial data 306 when evaluating the results of each clinical trial.

[0072] In some implementations, to avoid using an invalid instance 210 containing element 206A, data validator 150 may access a subsequent data store 170 (e.g., data store 170B) to attempt to retrieve a second instance 210B of the same element 206A. Data validator 150 may identify whether instance 210B of element 206A has a confidence value above a threshold and repeat the process until instance 210N of element 206A exceeds the threshold. In this manner, system 100 can improve the data integrity of clinical trials. Furthermore, system 100 can improve the accuracy of each clinical trial for each entity associated with data store 170.

[0073] 4, a block diagram of a process 400 for generating a plurality of metrics 404A-N using data elements 206 from a data repository 165 and transmitting an output 406 to the administration device 110 is shown. The process 400 may include or correspond to operations performed by the system 100 for generating data elements from interactions with a session. In the process 400, a metric calculator 155 executing on the data management service 105 may determine, calculate, or compute a plurality of metrics 404A-N (hereinafter collectively referred to as metrics 404) using clinical trial data 402 on the repository 165. The repository 165 may then store and maintain the clinical trial data 402, including elements 206 identified as relevant from the data store 170, as further described herein. The metric calculator 155 may access the repository 165 to obtain one or more of the elements 206. The metric calculator 155 may access the repository 165 to calculate the metrics 404 in response to a request or trigger related to a clinical trial. For example, the metric calculator 155 may access the repository 165 and use element 206 to determine the metrics 404 at set points during the clinical trial (e.g., at 1 week, 1 month, or end of trial).

[0074] The metric calculator 155 may use the elements 206 to generate one or more metrics 404 as a function of the elements 206. The metrics 404 may identify or indicate at least one of a degree of efficacy of the application 125 during the clinical trial, a degree of performance of the application 125, or a severity of a condition to be addressed in the user 205, etc. In some embodiments, the metric calculator 155 may use the elements 206 to calculate the metrics 404 based on a degree of performance of the application 125 while the user 205 is participating in the clinical trial. The degree of performance of the application 125 may indicate a level of functionality (e.g., speed, response time, or error rate) of the application 125 during the clinical trial.

[0075] In some implementations, the metric calculator 155 may use element 206 to calculate a metric 404 based on the efficacy of the application 125 to measure the level of deviation from the intended outcome of the clinical trial. High efficacy of the application 125 may indicate a low level of deviation from the intended outcome of the clinical trial associated with each clinical trial data 402 caused by the session 202. For example, element 206 may include information regarding the number or percentage of correct responses to a task to address mild cognitive impairment (MCI). The metric calculator 155 may determine the degree of efficacy based on the number or percentage of correct responses to the task.

[0076] In some embodiments, metric calculator 155 may use elements 206 to calculate metric 404 based on the severity of the condition being addressed in user 205. The severity may correspond to an objective indicator for measuring the impact of the condition on user 205. For example, for a migraine condition, element 206 may include an indication of the pain experienced by user 205 during a clinical trial. Metric calculator 155 may use elements 206 to calculate a pain catastrophizing score (PCS) value as one of metrics 404. In some embodiments, metric 404 may measure the uptake of a clinical trial medication concurrently with providing session 202 or generating data element 206. For example, data element 206 may indicate the consumption of a clinical trial medication, and metric calculator 155 may determine an indication of whether the medication has been consumed as one of metrics 404. In response to generating the metrics 404, the metric calculator 155 may store and maintain the metrics 404 on the data repository 165 (or another data store 170).

[0077] The interface provider 160 may generate, create, or generate at least one output 406 using one or more metrics 404. The output 406 may be generated based on digital therapeutic content associated with the user 205, the metrics 404, and information related to the metrics 404. For example, the interface provider 160 may select content for the output 406 based on the digital therapeutic content associated with the user 205. In some embodiments, the interface provider 160 may generate the output 406 for the session 202 to include the metrics 404 in a summary in a graphical user interface. For example, the summary in the graphical user interface may include the metrics 404 corresponding to the severity of the user's 205 condition. In some embodiments, the interface provider 160 may generate the output 406 to include an activity log using information related to the metrics 404. The interface provider 160 may transmit, send, or provide the output 406 to the managed device 110 as a graphical user interface. The information provided in the output 406 may relate to at least one of a plurality of data stores 170 or a data element 206 on a data repository 165, or the like.

[0078] The management device 110 may display, generate, or render the output 406 as shown in FIG. 5 . FIG. 5 illustrates an example of a dashboard 500 provided by the data replication system 100. The dashboard 500 may identify, display, or present information related to elements 206 on at least one of the plurality of data stores 170 or data repository 165. For example, the information related to the plurality of data stores 170 may indicate entities associated with the data stores 170, clinical trials the entities are observing, identities of the data stores 170, and participant activity. The information related to elements 206 on repository 165 may include identities of users 205, clinical trials in which the users 205 are participating, medications taken during the clinical trials, and instances 210 of elements 206. A user using the management device 110 may view and analyze the information in the dashboard 500 to advance the clinical trial.

[0079] In this manner, the data management service 105 can reduce the consumption of computing resources when receiving data from a user device 120 that transitions from an offline state to an online state. The user device 120 can store the interaction data 204 in on-device storage when the user device 120 is offline, allowing the user to access the digital therapeutic content and functionality of the application 125, greatly increasing the overall usability of the user device 120. When the user device 120 transitions to an online state, the user device 120 may transmit the interaction data 204 to the data management service 105. Using aspects described herein, the data management service 105 can reduce the consumption of computing resources by copying the interaction data 204 across all data stores 170 associated with the system 100.

[0080] FIG. 6 shows a block diagram of a data replication architecture 600 for clinical data. The data replication architecture 600 may include data from an application 602 (e.g., application 125), multiple transfer media 604A-N (collectively referred to as transfer media 604), and at least one of a data store 170. The data store 170 may validate (606) an instance 210 of an element 206 and store (608) the element 206 in a repository (e.g., repository 165). The data from the application 602 may include interaction data 204, digital therapeutic content for a session 202, and the element 206. The data management service 105 may send the data from the application 602 to each transfer media 604. Each transfer media 604 may be associated with a respective data storage provider for transferring the data to the data store 170.

[0081] The data store 170 may validate (606) the instance 210 that includes the element 206 by querying the element 206 within the instance 210 (e.g., using the data validator 150). In this manner, the validation 606 may identify the optimal instance 210 that includes the element 206 by comparing each instance 210 with the elements 206 as described above and selecting the instance 210 that includes the element 206 with the highest confidence value. Once the validation 606 is performed, the repository 608 may receive the optimal instance 210 that includes the element 206 and store the element 206. The element 206 may be applied to a calculator (e.g., the metric calculator 155) to calculate a metric (e.g., the metric 404) using at least one data store 170.

[0082] Referring now to FIG. 7 , a flow diagram of a method 700 for maintaining data integrity during a clinical trial for a digital therapeutic application is illustrated. Method 700 may be implemented or performed using any of the components described herein, such as the data management service 105 and the user device 120, or any combination thereof. In method 700, a computing system (e.g., the data management service 105 or the user device 120) may receive a data element generated based on a user's interaction with the digital therapeutic application during the clinical trial in response to establishing communication with the digital therapeutic application (705). The computing system may send the data element to each data store of a plurality of data stores (710). The computing system may access one of the plurality of data stores to obtain an instance of the data element (715). The computing system may identify whether a data element in the instance lacks validity (720). If the instance of the data element lacks validity, the computing system may identify another data store of the plurality of data stores (725) and repeat from (715). If the first instance of the data element is valid, the computing system may store the instance in a data repository (730). The computing system may provide a graphical user interface that shows information related to the data element on at least one of the plurality of data stores or the data repository (735). B. Network and Computing Environment

[0083] Various operations described herein may be implemented on a computer system. FIG. 8 illustrates a simplified block diagram of an exemplary server system 800, client computer system 814, and network 826 that can be used to implement some embodiments of the present disclosure. In various embodiments, server system 800 or a similar system may implement a service or server, or portions thereof, described herein. Client computer system 814 or a similar system may implement a client described herein. System 100 described herein may be similar to server system 800. Server system 800 may have a modular design incorporating multiple modules 802 (e.g., blades in a blade server implementation). While two modules 802 are illustrated, any number may be provided. Each module 802 may include a processing unit 804 and local storage 806.

[0084] Processing unit 804 may include a single processor, which may have one or more cores, or multiple processors. In some implementations, processing unit 804 may include a general-purpose primary processor and one or more special-purpose coprocessors, such as a graphics processor, digital signal processor, etc. In some implementations, some or all of processing unit 804 may be implemented using customized circuitry, such as an application-specific integrated circuit (ASIC) or field-programmable gate array (FPGA). In some implementations, such integrated circuits execute instructions stored on the circuitry itself. In other implementations, processing unit 804 may execute instructions stored in local storage 806. Any combination of any type of processor may be included in processing unit 804.

[0085] The local storage 806 may include volatile storage media (e.g., DRAM, SRAM, SDRAM, etc.) and / or non-volatile media (e.g., magnetic or optical disks, flash memory, etc.). The storage media incorporated in the local storage 806 may be fixed, removable, or updatable, as appropriate. The local storage 806 may be physically or logically divided into various subunits, such as system memory, read-only memory (ROM), and permanent storage devices. The system memory may be a read-and-write memory device or a volatile read-and-write memory, such as dynamic random access memory. The system memory may store some or all of the instructions and data needed by the processing unit 804 at runtime. The ROM may store static data and instructions needed by the processing unit 804. The permanent storage device may be a non-volatile read-and-write memory device that can store instructions and data even when the module 802 is not powered on. The term "storage medium" as used in this disclosure includes any medium capable of storing data therein indefinitely (even in the face of overwriting, electrical disturbances, power outages, etc.), and does not include carrier waves and transitory electronic signals propagated wirelessly or via wired communications.

[0086] In some embodiments, local storage 806 may store one or more software programs to be executed by processing unit 804, such as an operating system and / or programs that embody various server functions, such as functions of system 100 or any other system described in this disclosure, or any other server functions associated with system 100 or any other system described in this disclosure.

[0087] "Software" generally refers to sequences of instructions that, when executed by processing unit 804, cause server system 800 (or portions thereof) to perform various operations and thus define one or more specific machine implementations that implement and execute the operations of the software program. The instructions may be stored as firmware resident in read-only memory and / or as program code stored on a non-volatile storage medium that can be loaded into a volatile working memory for execution by processing unit 804. The software may be embodied as a single program or as a collection of separate programs or program modules that interact as necessary. Processing unit 804 may retrieve program instructions to execute and data to process to perform the various operations described above from local storage 806 (or non-local storage, as described below).

[0088] In some server systems 800, multiple modules 802 may be interconnected via a bus or other interconnect 808 to form a local area network that facilitates communication between the modules 802 and other components of the server system 800. The interconnect 808 may be embodied using a variety of technologies, including server racks, hubs, routers, etc.

[0089] A wide area network (WAN) interface 810 may provide data communication capability between a local area network (e.g., via interconnect 808) and a network 826, such as the Internet. Other technologies may be used to communicatively couple the server system to the network 826, including wired technologies (e.g., Ethernet, IEEE 802.3 standard) and / or wireless technologies (e.g., Wi-Fi, IEEE 802.11 standard).

[0090] In some implementations, local storage 806 is intended to provide working memory for processing unit 804, providing rapid access to programs and / or data to be processed while reducing traffic on interconnect 808. One or more mass storage subsystems 812 connectable to interconnect 808 may provide storage for larger amounts of data on the local area network. Mass storage subsystem 812 may be based on magnetic, optical, semiconductor, or other data storage media. Direct-attached storage, storage area networks, network-attached storage, etc. may be used. Any data store or other collection of data described in this disclosure as created, used, or maintained by a service or server may be stored in mass storage subsystem 812. In some implementations, additional data storage resources may be accessible (potentially with higher latency) via WAN interface 810.

[0091] Server system 800 may operate in response to requests received via WAN interface 810. For example, one of modules 802 may perform monitoring functions in response to received requests and assign individual tasks to other modules 802. Work allocation techniques may be used. As requests are processed, results may be transmitted back to the requester via WAN interface 810. Such operations may be largely automated. Furthermore, in some embodiments, WAN interface 810 may interconnect multiple server systems 800 to provide a scalable system capable of managing large volumes of activity. Other techniques for managing server systems and server farms (collections of cooperating server systems), including dynamic resource allocation and reallocation, may also be used.

[0092] Server system 800 may interact with various user-owned or user-operated devices over a wide area network such as the Internet. An example of a user-operated device is illustrated in Figure 6 as client computing system 814. Client computing system 814 may be embodied as a consumer device such as a smartphone, other mobile phone, tablet computer, wearable computing device (e.g., smart watch, eyeglasses), desktop computer, laptop computer, etc.

[0093] For example, client computing system 814 may communicate over WAN interface 810. Client computing system 814 may include computer components such as a processing unit 816, a storage device 818, a network interface 820, a user input device 822, and a user output device 824. Client computing system 814 may be a computing device embodied in various form factors, such as a desktop computer, a laptop computer, a tablet computer, a smartphone, other mobile computing devices, a wearable computing device, etc.

[0094] The processing unit 816 and storage device 818 may be similar to the processing unit 804 and local storage 806 described above. Suitable devices may be selected based on the demands placed on the client computing system 814. For example, the client computing system 814 may be embodied as a "thin" client with limited processing capabilities or as a high-performance computing device. The client computing system 814 may be provided with program code executable by the processing unit 816 to enable various interactions with the server system 800.

[0095] A network interface 820 may provide a connection to a network 826, such as a wide area network (e.g., the Internet), to which the WAN interface 810 of the server system 800 is also connected. In various embodiments, the network interface 820 may include a wired interface (e.g., Ethernet) and / or a wireless interface implementing various RF data communication standards, such as Wi-Fi, Bluetooth, or cellular data network standards (e.g., 3G, 4G, LTE, etc.).

[0096] User input device 822 may include any device (or devices) through which a user can provide signals to client computing system 814. Client computing system 814 may interpret these signals as representing particular user requests or information. In various embodiments, user input device 822 may include some or all of the following: a keyboard, a touchpad, a touchscreen, a mouse or other pointing device, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, a microphone, etc.

[0097] The user output device 824 may include any device through which the client computing system 814 can provide information to a user. For example, the user output device 824 may include a display-to-display image generated by or delivered to the client computing system 814. The display may include various image generation technologies, such as, for example, a liquid crystal display (LCD), a light emitting diode (LED) including an organic light emitting diode (OLED), a projection system, a cathode ray tube (CRT), etc., along with supporting electronics (e.g., digital-to-analog converters, analog-to-digital converters, signal processors, etc.). Some embodiments may include devices such as a touchscreen that function as both an input and output device. In some embodiments, other user output devices 824 may be provided in addition to or instead of a display. Examples include indicator lights, speakers, tactile "display" devices, printers, etc.

[0098] Some embodiments include electronic components, such as a microprocessor, storage, and memory, that store computer program instructions on a computer-readable storage medium. Many of the features described herein may be implemented as processes specified as a set of program instructions encoded on a computer-readable storage medium. When executed by one or more processing units, these program instructions cause the processing units to perform various operations indicated in the program instructions. Examples of program instructions or computer code include machine code, such as that produced by a compiler, and files containing higher-level code executed by a computer, electronic component, or microprocessor using an interpreter. Through suitable programming, processing units 804 and 816 may provide functionality for server system 800 and client computing system 814, including any or other functions described herein as being performed by a server or user.

[0099] It will be understood that the server system 800 and the user computing system 814 are illustrative and subject to variations and modifications. Computer systems used in conjunction with embodiments of the present disclosure may have other capabilities not specifically described herein. Furthermore, while the server system 800 and the user computing system 814 are described with reference to certain blocks, it should be understood that these blocks are defined for convenience of explanation and are not intended to imply a particular physical arrangement of components. For example, different blocks may, but need not, exist within the same facility, the same server rack, or even on the same motherboard. Furthermore, the blocks need not correspond to physically discrete components. Blocks may be configured to perform various operations, for example, by programming a processor or providing appropriate control circuitry, and various blocks may be reconfigurable or non-reconfigurable, depending on how the initial configuration was obtained. Embodiments of the present disclosure may be realized in a variety of apparatuses, including electronic devices embodied using various combinations of circuitry and software.

[0100] While the present disclosure has been described with reference to specific embodiments, those skilled in the art will appreciate that numerous modifications are possible. Embodiments of the present disclosure may be implemented using a variety of computer systems and communication technologies, including, but not limited to, the specific examples described herein. Embodiments of the present disclosure may be implemented using any combination of dedicated components and / or programmable processors and / or other programmable devices. The various processes described herein may be performed on the same processor or on any combination of different processors. While components are configured to perform certain operations, such configuration may be achieved, for example, by designing electronic circuitry to perform the operations, by programming programmable electronic circuitry (such as a microprocessor) to perform the operations, or any combination thereof. Furthermore, while the above-described embodiments may refer to specific hardware and software components, those skilled in the art will appreciate that different combinations of hardware and / or software components may also be used, and that certain operations described as being implemented in hardware may also be implemented in software, or vice versa.

[0101] A computer program incorporating various features of the present disclosure may be encoded on and stored on a variety of computer-readable storage media. Suitable media include magnetic disks or tapes, optical storage media such as compact discs (CDs) or digital versatile discs (DVDs), flash memory, and other non-transitory media. A computer-readable medium on which the program code is encoded may be packaged with a compatible electronic device, or the program code may be provided separately from the electronic device (e.g., via internet download or as a separately packaged computer-readable storage medium).

[0102] Therefore, although the disclosure has been described in terms of specific embodiments, it will be understood that the disclosure is intended to cover all modifications and equivalents within the scope of the following claims.

Claims

1. 1. A method of maintaining data integrity during a clinical trial of a digital therapeutic application, comprising: receiving, by one or more processors, data elements generated based on interactions with the digital therapeutic application by a user during a clinical trial in response to establishing communication with the digital therapeutic application; sending, by said one or more processors, said data elements to each data store of a plurality of data stores; accessing, by the one or more processors, a first data store of the plurality of data stores to obtain a first instance of the data element; identifying, by the one or more processors, the first instance of the data element as being invalid; accessing, by the one or more processors, a second data store of the plurality of data stores to obtain a second instance of the data element in response to the identification that the first instance of the data element in the first data store is invalid; storing, by the one or more processors, the second instance of the data element on the clinical trial data repository; and A method comprising:

2. identifying, by the one or more processors, a first instance of a second data element on the first data store as being invalid; determining, by the one or more processors, that a second instance of the second data element on the second data store is invalid; and providing, by the one or more processors, an indication that the second data element on the first data store and the second data store lacks validity; The method of claim 1 further comprising:

3. identifying, by the one or more processors, the second instance of the data element on the second data store as valid. Further comprising: storing the second instance of the data element further comprises storing the second instance of the data element in response to identifying the second instance of the data element as valid. The method of claim 1.

4. identifying, by the one or more processors, a first instance of a second data element on the first data store as valid; storing, by the one or more processors, the first instance of the second data element from the first data store in the data repository without accessing the second data store to obtain a second instance of the second data element; The method of claim 1 further comprising:

5. determining, by the one or more processors, a confidence value indicative of the validity of the first instance of the data element. Further comprising: Identifying the first instance of the data element as lacking validity further includes determining that the confidence value does not meet a threshold metric. The method of claim 1.

6. The method of claim 5 , wherein determining the confidence value further comprises applying a machine learning (ML) model to determine the confidence value.

7. 10. The method of claim 1, further comprising determining, by the one or more processors, a plurality of metrics using the data elements stored on the data repository that identify at least one of a degree of efficacy of the digital therapeutic application, a degree of performance of the digital therapeutic application, or a severity of a condition to be addressed in the user during the clinical trial.

8. 10. The method of claim 1, further comprising providing, by the one or more processors, a graphical user interface presenting information related to the data elements on at least one of the plurality of data stores or the data repository.

9. 10. The method of claim 1, further comprising identifying, by the one or more processors, the first data store of the plurality of data stores by applying a machine learning (ML) model.

10. 10. The method of claim 1, wherein receiving the data elements further comprises obtaining, from a user device running the digital therapeutic application, a plurality of data elements generated during a time when communication with the user device was not established.

11. 2. The method of claim 1 , wherein transmitting the data element further comprises providing a respective instance of the data element to each data store of the plurality of data stores, the respective instances including an encrypted copy of the data element.

12. 10. The method of claim 1, wherein the digital therapeutic application is configured to generate the data elements during the clinical trial at least partially in parallel with the user receiving a medication to address a condition associated with the clinical trial.

13. 1. A system for maintaining data integrity during a clinical trial of a digital therapeutic application, comprising: one or more processors coupled to a memory, receiving, in response to establishing communication with the digital therapeutic application, data elements generated based on a user's interaction with the digital therapeutic application during the clinical trial; sending said data element to each data store of a plurality of data stores; accessing a first data store of the plurality of data stores to obtain a first instance of the data element; Identifying the first instance of the data element as invalid; in response to the identification that the first instance of the data element in the first data store is invalid, accessing a second data store of the plurality of data stores to obtain a second instance of the data element; storing the second instance of the data element on a data repository for the clinical trial; One or more processors configured to A system comprising:

14. The one or more processors: identifying a first instance of a second data element on the first data store as being invalid; determining that a second instance of the second data element on the second data store is invalid; providing an indication that the second data element on the first data store and the second data store lacks validity; The system of claim 13 further configured to:

15. The one or more processors: Identifying the second instance of the data element on the second data store as valid. further configured as follows: the one or more processors are further configured, in storing the second instance of the data element, to store the second instance of the data element in response to identifying the second instance of the data element as valid. The system of claim 13.

16. The one or more processors: identifying a first instance of the second data element on the first data store as valid; Storing the first instance of the second data element from the first data store in the data repository without accessing the second data store to obtain a second instance of the second data element. The system of claim 13 further configured to:

17. the one or more processors are further configured to determine a confidence value indicative of the validity of the first instance of the data element; Identifying the first instance of the data element as lacking validity further includes determining that the confidence value does not meet a threshold metric. The system of claim 13.

18. 20. The system of claim 17, wherein determining the confidence value further comprises applying a machine learning (ML) model to the first instance to determine the confidence value.

19. 14. The system of claim 13, wherein the one or more processors are further configured to determine, using the data elements stored on the data repository, a plurality of metrics that identify at least one of a degree of efficacy of the digital therapeutic application, a degree of performance of the digital therapeutic application, or a severity of a condition to be addressed in the user during the clinical trial.

20. 14. The system of claim 13, wherein the one or more processors are further configured to provide a graphical user interface that presents information related to the data elements on at least one of the plurality of data stores or the data repository.

21. 14. The system of claim 13, wherein the one or more processors are configured to identify the first data store of the plurality of data stores by applying a machine learning (ML) model.

22. 14. The system of claim 13, wherein the one or more processors are configured to obtain, from a user device running the digital therapeutic application, a plurality of data elements generated during a time when communication with the user device was not established.

23. 14. The system of claim 13, wherein the one or more processors are further configured to provide a respective instance of the data element to each data store of the plurality of data stores, the respective instances including an encrypted copy of the data element.

24. 14. The system of claim 13, wherein the digital therapeutic application is configured to generate the data elements during the clinical trial at least partially in parallel with the user receiving a medication to address a condition associated with the clinical trial.

Images