Methods, data processing systems, computer-readable media, and computer program products implemented by a computer system

Abstract

A method, data processing system, computer-readable medium, and computer program product implemented by a computer system are disclosed. The method is used to improve data security by detecting personally identifiable information (PII) in clinical trial data before submitting it to a hardware storage device, comprising: rendering a graphical user interface for inputting clinical trial data and one or more controls; receiving the input clinical trial data through the graphical user interface; associating the clinical trial data with a pending state in memory before submitting it to the hardware storage device; accessing the clinical trial data input to at least one of one or more input sections from memory when it is determined that the clinical trial data does not contain a PII; releasing the clinical trial data from the pending state; and submitting the released clinical trial data to the hardware storage device.

Description

Technical Field

[0001] This specification typically relates to systems and methods for improving the data security of clinical trial data in electronic data capture systems by using a pending status indicator to delay the submission of clinical trial data to storage until it is determined that no sensitive information exists, and then releasing the clinical trial data and submitting it to storage. Summary of the Invention

[0002] An implementation of this disclosure includes a method implemented by a computer system for detecting personally identifiable information (PII) in clinical trial data before it is submitted to a hardware storage device to improve data security. The method includes: causing the computer system to render a graphical user interface having one or more input sections for inputting clinical trial data and one or more controls; in response to selection of at least one of the one or more controls, the computer system receiving clinical trial data input to at least one of the one or more input sections of the graphical user interface via the graphical user interface; and before submitting the clinical trial data to the hardware storage device. Previously, the clinical trial data input to at least one of the one or more input sections of the graphical user interface was associated with a pending state in memory; a machine learning model detected whether the clinical trial data included a PII; the clinical trial data included a PII was determined based at least in part on the output of the machine learning model; if it was determined that the clinical trial data did not include a PII, the clinical trial data input to at least one of the one or more input sections of the graphical user interface and associated with the pending state was accessed from memory; the clinical trial data was released from the pending state; and the released clinical trial data was submitted to the hardware storage device.

[0003] In some implementations, the action includes receiving an indication from the machine learning model that the clinical trial data does not include a PII. The action includes: receiving an indication from the machine learning model that the clinical trial data includes a PII; causing an overlay to be rendered in the graphical user interface, wherein the rendered overlay displays a notification that a PII was detected and also displays: a first prompt confirming that the clinical trial data does not include a PII, wherein the first prompt is juxtaposed with the notification in the overlay; and a second prompt updating the clinical trial data to exclude a PII, wherein the second prompt is juxtaposed with the notification in the overlay. The action includes receiving data confirming that the clinical trial data does not include a PII. The overlay is a first overlay, and the method further includes: receiving updates to the input clinical trial data via the graphical user interface by the computer system; associating the updates to the input clinical trial data with a pending status indicator in memory before submitting them to the hardware storage device; causing the machine learning model to detect whether the updates to the input clinical trial data include a PII; and rendering a second overlay in the graphical user interface if the machine learning model detects that the updates to the input clinical trial data include a PII, wherein the rendered second overlay displays another notification of the detection of a PII and also displays: information to confirm the detection of a PII. The update of the clinical trial data does not include a first notification, wherein the first notification is juxtaposed with the other notification in the second overlay; and a second notification is used to update the updated clinical trial data to exclude the PII, wherein the second notification is juxtaposed with the other notification in the second overlay; step ad is repeated until confirmation that the update of the clinical trial data does not include the PII is received, or until the machine learning model detects the absence of the PII; once confirmation that the update of the clinical trial data does not include the PII is received or the machine learning model detects the absence of the PII is received, the most recently submitted update of the input clinical trial data is submitted to the hardware storage device.

[0004] In some implementations, the machine learning model is a Large Language Model (LLM), and the method further includes: generating system prompts to guide the LLM in interpreting input corresponding to clinical trial data input to at least one of the one or more input sections of the graphical user interface and providing output of a specified type; and inputting the system prompts and the input corresponding to the clinical trial data input to at least one of the one or more input sections of the graphical user interface into the LLM. Submission includes: creating entries in a specified table in the hardware storage device, wherein data corresponding to the input clinical trial data is stored in the created entries; and creating entries in an audit trail table to track modifications to the input clinical trial data or data corresponding to the input clinical trial data. Submitting the clinical trial data to the hardware storage device includes: submitting data corresponding to the clinical trial data to the hardware storage device.

[0005] In other implementations, a software development system is provided for developing software to improve data security by detecting personally identifiable information (PII) in clinical trial data before submitting it to a hardware storage device. The software development system includes: one or more processors; and one or more machine-readable hardware storage devices storing instructions executable by the processor to: (a) receive code that, when executed, performs operations including: causing a computer system to render a graphical user interface having one or more input sections for inputting clinical trial data and one or more controls; and receiving, via the graphical user interface, the one or more input sections input to the graphical user interface in response to selection of at least one of the controls. The system includes: (a) clinical trial data in at least one of the input sections of the graphical user interface; (b) associating the clinical trial data in at least one of the input sections of the graphical user interface with a pending state in memory before submitting the clinical trial data to the hardware storage device; (c) detecting by a machine learning model whether the clinical trial data includes personally identifiable information (PII); (d) determining whether the clinical trial data includes PII based at least in part on the output of the machine learning model; (e) if it is determined that the clinical trial data does not include PII, accessing the clinical trial data in at least one of the input sections of the graphical user interface and associated with the pending state from memory; (f) releasing the clinical trial data from the pending state; and (b) submitting the released clinical trial data to the hardware storage device; and (c) storing the code.

[0006] Other embodiments of this aspect include corresponding computer systems (e.g., data processing systems), devices, and computer programs recorded on one or more computer storage devices, each computer system, device, and computer program configured to perform the actions or operations described herein. A system of one or more computers may be configured to perform specific actions by means of software, firmware, hardware, or combinations thereof installed on the system, which in operation cause the system to operate. A computer program may be configured to perform specific actions by means of including instructions that, when executed by a data processing device, cause the device to operate.

[0007] A system of one or more computers can be configured to perform specific operations or actions by means of software, firmware, hardware, or combinations thereof installed on the system, which in operation cause the system to perform actions. A system of one or more computers can be configured to perform specific operations or actions by means of instructions that, when executed by a data processing device, cause the device to perform actions.

[0008] The technology described herein provides an Electronic Data Capture (EDC) system that initially captures or stores data in a pending state (e.g., temporarily stored in memory), which serves as a security measure for data submission within the EDC system. When a user registers data, it is temporarily held in this pending state, allowing the system to perform real-time detection of sensitive information (such as, for example, PIIs) before the data is released into the database and audit trail. If no PII is detected, the data is released from the pending state and securely registered into the system. If a PII is identified, the system notifies the user, giving the user an opportunity to correct or verify the data before submission is permitted. This system effectively acts as a protection against unauthorized or sensitive information being logged in. Keeping data in a pending state (and submitting it to memory only after confirming the absence of sensitive information) improves memory and processing resource allocation by preventing the storage of data (containing sensitive information) that will subsequently require modification (e.g., in the audit trail). Such modifications to data and / or audit trails consume processing resources, which are saved if the data is cleaned up before being committed to storage (e.g., by removing sensitive information). In other words, the system described herein consumes fewer processing resources by simply committing data that does not contain sensitive information to storage, compared to the amount of resources required to commit data containing sensitive information to storage and then, for example, to repeatedly modify that data (or audit trail). The technique described herein also reduces storage resource consumption. This is because less storage is needed per instance (as this occurs when the data does not contain sensitive information) compared to the amount of storage required to store data containing sensitive information and then to store a version of that data where the sensitive information has been removed or modified.

[0009] Additionally, as described herein, the data processing system generates software to function as a pending status marker, prompter, PII prevention ML engine, PII detector, and data submitter. Because the techniques described herein eliminate the need for multiple accesses to stored (including PII) data to modify or delete audit trails targeting that data or modify the data itself, the software includes new code that allows the computer processor to process the data more efficiently. Instead, the data is first checked against the PII, and only when it is confirmed that the data does not contain the PII is the data submitted to storage—eliminating the need for subsequent accesses to the data to modify the data or its associated audit trails. Therefore, the technical problem (e.g., how to process data more efficiently) has been solved because the resulting technical effects significantly exceed the normal physical effects produced by ordinary software.

[0010] The EDC system, which keeps data in a pending state, addresses the following key technical challenges. First, it protects the database and audit trail from unverified PIIs, thereby reducing the risk of compliance violations. Second, it automates PII checks in real time before releasing data, minimizing the need for costly manual reviews. Third, it ensures that only compliant and secure data is stored, promoting compliance with regulatory standards such as the General Data Protection Regulation (GDPR) and the Health Insurance Portability and Accountability Act (HIPAA). By acting as a filter before data enters the system, the techniques described in this paper enhance data security, integrity, and compliance within the EDC system.

[0011] The techniques described in this paper offer several advantages for storing PIIs within an EDC system. First, the techniques provide increased efficiency. Unlike current systems that typically rely on manual review after PIIs have been registered, the techniques described in this paper automate the detection process in real time. This reduces the need for labor-intensive checks and significantly reduces the time required to ensure data integrity and compliance. Second, the techniques described in this paper improve the accuracy of PII detection. By introducing real-time PII detection in the pending state, these techniques minimize the risk of human error, which is common in manual reviews. The system ensures consistent identification of PIIs before data storage, resulting in greater accuracy in data disposal. Third, enhanced data security. The pending state workflow acts as a shield, preventing unverified or sensitive data from entering the database or audit trail. This proactive measure ensures that PIIs are not unintentionally stored, reducing the risk of compliance violations and improving overall data security. Fourth, increased regulatory compliance. Automated PII detection and prevention mechanisms help ensure that data entries comply with stringent privacy regulations such as GDPR and HIPAA. This reduces the risk of penalties and audits associated with the mishandling of sensitive information. Fifth, increased speed of data processing. The technology described in this paper accelerates data processing by automating PII detection and processing and releasing secure data into a database without human intervention. This speed is crucial in environments such as clinical trials, where data must be handled quickly and efficiently. Sixth, consistency across data sources is achieved.

[0012] These technologies ensure consistent handling of PII across different data entry points and formats, providing a consistent level of protection and reducing the risk of gaps in data security. Compared to existing systems that often suffer from inefficient, human-error-prone, and inconsistent PII handling, the technologies described in this paper offer a faster, more accurate, and secure method for managing PII in EDC systems.

[0013] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0014] Figure 1 A data processing environment for improving data security is shown.

[0015] Figure 2 An example data processing system for improving data security is shown.

[0016] Figure 3 and Figure 7 Each example demonstrates a process for improving data security.

[0017] Figure 4 Example input and detection of PII in the input are shown.

[0018] Figure 5 An example of training a machine learning system is described.

[0019] Figure 6A and Figure 6B Each is a sample graphical user interface.

[0020] Figure 8 An example computing system based on an implementation of this disclosure is described. Detailed Implementation

[0021] Managing PIIs presents significant challenges to EDC systems used in clinical trials. PIIs include data that can directly or indirectly identify individuals, such as name, date of birth, Social Security number, or medical records. Regulatory bodies such as the FDA, EMA, and GDPR require strict controls over the handling, storage, and transfer of PIIs to protect patient privacy and ensure compliance.

[0022] When PIIs are not adequately checked before being registered in the database, data security is compromised. This means that users could inadvertently register clinical trial data, including sensitive information such as patient names, addresses, or other PIIs, without real-time verification. Once this data is stored in the database, it becomes part of the audit trail.

[0023] Audit trails, designed to track every change made to data, capture and retain every action and entry, including any PIIs that may have been incorrectly registered. Once a PII is logged into an audit trail, it cannot be easily modified or removed due to stringent regulatory requirements. This leads to several major problems. First, it reduces data integrity and increases privacy risks. Sensitive information is permanently logged in the system, even if it is later identified as inappropriate or incorrect. This creates privacy risks and compliance challenges, especially in jurisdictions with strict data protection laws such as GDPR. Second, once a PII is logged into an audit trail, correcting the problem can be costly (in terms of the computational resources required to do so) and time-consuming. Modifying or removing audit trail entries typically requires complex ex post facto processes that must be fully archived and certified to meet regulatory requirements. This process is not only resource-intensive but also error-prone, increasing the risk of non-compliance. The techniques described in this paper improve data security by detecting PIIs before they enter the system and are captured in the audit trail. The techniques described in this paper also improve the accuracy of machine learning (ML) models trained to detect PII based on both the values ​​and structure of the data.

[0024] In particular, the technology described in this paper has recognized and addressed the following technical issues: unverified PII registration (where PIIs are not checked before being registered into the database, leading to the unintentional logging and storage of sensitive data), immutable audit trails (once a PII is registered, it becomes part of an audit trail that cannot be easily modified, resulting in compliance and privacy risks), human review bottlenecks (many systems rely on manual processing to detect PIIs after registration, which is time-consuming, error-prone, and expensive), inconsistent PII disposal (different data sources and forms may dispose of PIIs differently, resulting in gaps in data security and privacy protection), and compliance risks (complying with stringent data protection laws such as GDPR and HIPAA is challenging, especially when PIIs are unintentionally captured and stored).

[0025] refer to Figure 1The diagram illustrates a data processing environment 10. The data processing environment 10 includes a client device 12, a network 16, and a data processing system 20 (e.g., an EDC system). In this example, the data processing environment 10 includes an environment for detecting whether input data (such as clinical trial data) includes a PII. In this example, the data processing system 20 includes a user interface (UI) generator 18 for providing or generating UI data 14, which, when rendered on the client device 12, causes the client device 12 to display one or more user interfaces. The data processing system 20 also includes a pending status marker 21 for receiving input data and storing (temporarily storing) input data with a pending status indicator (also referred to herein as pending status) to allow a machine learning engine to detect whether the input data includes a PII and, if so, to correct the input data before submitting it to a hardware storage device. The data processing system 20 also includes a prompter 30 for generating prompts to be transmitted to a PII prevention ML engine 34. The PII prevention ML engine 34 includes an ML model for detecting PIIs and for outputting an indication 36 to the PII detector 38 indicating whether the input data includes a PII. The PII detector 38 then sends a notification 42 of a PII, or, if no PII indication exists, transmits a message 40 to the data submitter 44 for the actual submission of the input data (or a copy of the input data) to a hardware storage device, such as hardware storage device 46. In this example, the UI generator 18 receives the notification 42. As described herein, based on the received notification 42, the UI generator 18 generates one or more additional UI prompts the user to specify that the input does not include a PII or to correct the PII. The data processing environment 10 also includes hardware storage devices 24 and 46 for storing data, structured data, and tables as described herein.

[0026] In operation, client device 12 renders one or more user interfaces based on UI data 14. These user interfaces include interfaces for users to input data 20. 1…N (Including, for example, clinical trial data) is input (or otherwise specified) into the input section of the data processing system 20. In this example, the client device 12 inputs data 20. 1…N The data is transmitted to the data processing system 20 and input to the data processing system 20. 1…N According to the pending status marker 21, it is received. The pending status marker 21 is configured to receive the input data 20. 1…N Input data 20 is identified as needing to be checked against the PII before being submitted to memory (e.g., in hardware storage device 46). 1…N Thus, the pending status marker 21 transmits input data 20 to the hardware storage device 24. 1…N And instruction 22, in response to input data 20 1…NThe entry is generated in the pending status table 26.

[0027] In this example, hardware storage device 24 stores a pending state table 26 to store and / or indicate which data is being temporarily stored, for example, to make the result of PII detection of the data pending. The pending state table 26 includes columns 26a, 26b, 26c specifying a key (e.g., a unique identifier), the input data itself (or an entry representing the input data), and the status of that input data. In this example, the input data includes a key that uniquely identifies the input data or is associated with that key. The pending state table 26 includes rows 26d…N, for example, rows corresponding to input data received as part of a specific request or transmission sent to data processing system 20. In this example, row 26d is for input data 20. 1…N In this example, entry 1 (e.g., the entry in the cell defined by row 26d and column 26b) includes a pointer to storage input 20 in hardware storage device 24. 1…N A pointer to the memory location. In this example, key ID1 specifies input data 20. 1…N The values ​​of the keys included. If the input data is 20... 1…N If the value of the key is not included or the value of the key is otherwise specified, the pending status marker 21 transmits an instruction to the hardware storage device 24 to receive the input data 20 using the specified data processing system 20. 1…N The timestamp of the time is used to populate the cells defined by column 26a and row 26d. The value of key ID allows for subsequent times (e.g., input data 20). 1…N Identify and retrieve input data at the time it is submitted to memory (20). 1…N .

[0028] The pending status marker 21 will then input data 20 1…N The data is transmitted to the prompter 30. The prompter 30 generates a prompt 32, which includes the input data 20. 1…N and detection input data 20 1…N Whether to include a PII request. The prompt 30 transmits the prompt 32 to the PII prevention ML engine 34, which applies one or more (trained) machine learning models to the input data 20. 1…N To detect input data 20 1…N Whether a PII is included. As mentioned earlier, when the PII prevention ML engine 34 detects a PII, the PII detector 38 generates a notification 42. When the PII prevention ML engine 34 does not detect a PII, the PII detector 38 transmits message 40 to the data submitter 44. Message 40 specifies the input data 20. 1…N Excluding PII. Message 40 via input data 20 1…NAssociated keys to identify input data 20 1…N The key is included in the input data 20 received by the prompt 30. 1…N The input data may be specified in the prompt 32 and message 40.

[0029] Upon receiving message 40, the data submitter 44 transmits the input data 20 to the hardware storage device 24. 1…N The request includes input data 20. 1…N The associated key. In response, the hardware storage device 24 will input data 20. 1…N The data is transmitted to the data submitter 44, which then processes the input data 20. 1…N The data is transferred to hardware storage device 46 for storage.

[0030] Hardware storage device 46 includes a data point table 48 and an audit table 50. For example, through individual cells in the data point table 48 that specify or store individual data (which may be referred to as attributes), the data point table 48 includes cells for storing input data 20. 1…N The data point table 48 includes columns 48a…48M corresponding to various attributes of the input data. The data point table 48 also includes column 48M+1, which specifies a primary key (PK), which links to or otherwise points to a specific row in the audit table 50, enabling the data processing environment 10 to specify audit trails for various received input data.

[0031] Data point table 48 also includes rows 49a…49n, where each row corresponds to input data received from the client device. In this example, input data 20 1…N It is stored in row 49a of data point table 48. In this example, data submitter 44 includes data processing rules and / or a parser to identify input data 20. 1…N The data submitter 44 can store the values ​​of various attributes in the appropriate cells of row 49a. The cells in data point table 48 include specific entries defined by specific rows and columns in data point table 48. In another example, data submitter 44 can submit input data 20... 1…N The primary key associated with the primary key used to access a specific row in audit table 50 is stored in hardware storage device 46.

[0032] Audit table 50 includes columns 50a, 50b, 50c and rows 50d…50n. In this example, column 50a includes data specifying information about a particular entry (e.g., a timestamp). Column 50b specifies audit data, including, for example, data specifying one or more modifications to the input data. Column 50c specifies a foreign key (FK) against the primary key specified by column 48M+1 in data point table 48. By using the foreign key specified in column 50c, data processing environment 10 can identify the corresponding primary key in data point table 48, thereby enabling data processing environment 10 to identify which audit data corresponds to a row in data point table 48. Audit table 50 includes rows 50d…50n specifying audit data for various received input data. In this example, row 50d includes data for input data 20… 1…N Audit data. In this example, the audit data includes specifying the input data 20. 1…N The data is submitted to the data point table 48 at the time of submission, and also specifies any other updates or modifications that may have been made to the data since it was originally submitted. In some examples, the data submitter 44 transmits the input data 20 to the hardware storage device 46. 1…N Related audit data.

[0033] exist Figure 1 In the variant, the data processing system 20 is or includes the aforementioned functional software development system for generating or receiving codes for pending status marker 21, prompter 30, PII prevention ML engine 34, PII detector 38, and data submitter 44.

[0034] PII prevents ML engines 34 from including machine learning models. Various types of machine learning models exist, including generative artificial intelligence (AI) models, such as those with one or more large language models (LLMs). Example LLMs include models with one or more generative pre-trained transformers (GPTs), such as models implemented using one or more artificial neural networks.

[0035] Generally, machine learning can encompass a variety of different techniques used to train machines to perform specific tasks without being specifically programmed to do so. Machines can be trained using various machine learning techniques, including, for example, supervised learning, unsupervised learning, and reinforcement learning. In supervised learning, the machine is given an input of interest and a corresponding output. The machine adjusts its functions to produce the desired output when given the input. Supervised learning is often used to teach computers to solve problems of deterministic outcomes; for example, a training set can be used to train a trained machine learning model to detect PIIs. In contrast, in unsupervised learning, input is given without providing a corresponding desired output. Reinforcement learning describes algorithms where machines make decisions using trial and error. Feedback informs the machine when a good or bad choice is made. The machine then adjusts its algorithm accordingly. For example, a trained learning model can be embodied as a generalized linear model (GLM). Different types of GLMs may be suitable for various scenarios. Zero-inflated negative binomial GLMs can be used because they model discrete counts of events (such as transitions) that occur over a given time period.

[0036] In another example, the trained learning model can be represented as an artificial neural network. An artificial neural network (ANN), or connection system, is a computational system inspired by the biological neural networks that make up the animal brain. An ANN is based on a collection of connected units or nodes called artificial neurons. Similar to synapses in a biological brain, each connection can transmit a signal from one artificial neuron to another. The receiving artificial neuron can process the signal and then use it to notify additional artificial neurons connected to it.

[0037] In common ANN implementations, the signals at the connections between artificial neurons are real numbers, and the output of each artificial neuron is calculated by some nonlinear function of the sum of its inputs. The connections between artificial neurons are called "edges." Artificial neurons and edges can have weights that adjust as learning progresses (e.g., the individual inputs of an artificial neuron can be weighted individually). These weights increase or decrease the signal strength at the connection. Artificial neurons can have thresholds such that a signal is only sent if the aggregated signal exceeds that threshold. The transfer functions along the edges typically have an sigmoid shape, but they can also take the form of other nonlinear functions, piecewise linear functions, or step functions. Typically, artificial neurons are aggregated into layers. Different layers can perform different kinds of transformations on their inputs. The signal may travel from the first layer (input layer) to the last layer (output layer) after multiple traversals of layers.

[0038] Typically, supervised learning techniques are used to train neural networks based on training data. The neural network is configured to receive training data as input (e.g., input data records) and process the input to generate an output, for example, indicating whether a PII has been detected. In this example, the neural network includes multiple artificial neurons connected by edges and aggregated into multiple neural network layers, each comprising at least an input layer and an output layer. Each edge is configured to transmit a signal from one artificial neuron to another, and the output of each artificial neuron is computed based on multiple weights according to the input of the artificial neuron. In this example, new data is processed using the multiple artificial neurons in the trained neural network to detect a PII based on the values ​​of the multiple weights. The artificial neurons in the input layer are configured to receive new data as input, and the artificial neurons in the output layer are configured to generate a new output indicating whether a PII has been detected.

[0039] Figure 2 Various aspects of the data processing system 20 are illustrated. Typically, the data processing system 20 includes several operational modules that perform specific functions related to the operation of the data processing system 20. For example, as previously described, the data processing system 20 includes a pending status marker 21, a prompter 30, a PII prevention ML engine 34, a PII detector 38, and a data submitter 44. Furthermore, the data processing system 20 includes a database module 62, a communication module 64, and a processing module 66. The operational modules can be provided as one or more computer-executable software modules, hardware modules, or combinations thereof. For example, one or more operational modules can be implemented as software code blocks having instructions that cause one or more processors of the data processing system 20 to perform the operations described herein. Alternatively or concurrently, one or more operational modules can be implemented in electronic circuitry, such as programmable logic circuits, field-programmable arrays (FPGAs), or application-specific integrated circuits (ASICs).

[0040] Database module 62 maintains and uses PII prevention ML engine 34 detects PII-related information.

[0041] As an example, database module 62 may store training data 62a for training or prompting the PII prevention ML engine 34. In some implementations, training data 62a may include examples of PIIs from clinical trial documents (such as clinical trial documents previously generated by data processing system 20 and / or clinical trial documents manually generated by one or more human users). For example, training data 62a may include documents describing clinical trial protocols, informed consent forms, clinical research reports, and / or any other documents that facilitate the conduct of clinical trials.

[0042] As another example, database module 62 may store input data 62b that serves as input to the PII prevention ML engine 34. As an example, input data 62b may include commands or instructions provided by the user, including information about specific desired outputs of the data processing system 20 and information input into the graphical user interface. For example, input data 62b may include information about users participating in clinical trials (e.g., symptoms or medications they are taking), the subject of the clinical trial (e.g., the intervention being tested), the type of clinical trial to be conducted (e.g., the “phase” of the clinical trial), the intended audience of the document (e.g., a specific government agency), and / or any other information about the clinical trial. In addition, input data 62b may include information supporting clinical trials retrieved by data processing system 20. As an example, input data 62b may include data retrieved from one or more drug information databases (e.g., information about drug composition, drug interactions, drug dosage, indications for use, side effects, etc.).

[0043] As another example, input data 62b may include data retrieved from one or more medical or scientific journals. For instance, input data 62b may include one or more articles or other publications describing the use, safety, and / or efficacy of certain drugs or other medical interventions.

[0044] As another example, input data 62b may include data about one or more existing clinical trial protocols. For example, input data 62b may include data about the names and other information of users participating in the clinical trial, and a series of steps, processes, actions, or operations previously (e.g., by clinical trial researchers) performed to assess the safety and / or efficacy of a particular medical intervention.

[0045] As another example, input data 62b may include data regarding one or more rules, regulations, and / or guidelines for conducting clinical trials. For instance, input data 62b may include data regarding government rules, regulations, and / or guidelines (e.g., those specified by government agencies such as the FDA). Input data 62b may also include data regarding institutional rules, regulations, and / or guidelines (e.g., those specified by public or private hospitals).

[0046] Furthermore, database module 62 may store output data 62c generated by PII prevention ML engine 34. As an example, output data 62c may include one or more portions of content (e.g., text, images, charts, graphs, tables, etc.) specifying whether a PII was detected or generated by PII prevention ML engine 34 based on input data 62b. As another example, output data 62c may include one or more documents generated by PII prevention ML engine 34 based on input data 62b.

[0047] In addition, database module 62 can store processing rules 62d that specify how the data in database module 62 can be processed to use PII prevention ML engine 34 to detect PII (e.g., specifying the type of PII and the rules used to detect PII).

[0048] As an example, processing rule 62d may include one or more rules for implementing, instructing, or prompting the PII prevention ML engine 34 to produce output data 62c. For example, one or more rules may specify that training data 62a is provided to the PII prevention ML engine 34 for training or prompting (e.g., enabling the PII prevention ML engine 34 to detect PIIs, identify trends and / or correlations between the content of clinical trial input data and the topics therein, and generate new outputs based on these identified trends and / or correlations).

[0049] As another example, one or more rules may specify that input data 62b be provided to the PII prevention ML engine 34 (e.g., to generate output data 62c indicating whether a PII has been detected).

[0050] As another example, one or more rules may specify that the generated output data 62c is presented to the user and / or stored for future retrieval and / or processing (e.g., using database module 62).

[0051] As another example, one or more rules can specify one or more tools that prompt the PII prevention ML engine 34 to perform specific actions. For example, tools can specify certain actions or operations that the PII prevention ML engine 34 can perform to detect PIIs, retrieve data, and generate content based on the retrieved data.

[0052] The example data processing techniques are described in further detail below.

[0053] As described above, the data processing system 20 also includes a communication module 64. The communication module 64 allows data to be transmitted to and from the data processing system 20. For example, the communication module 64 can be communicatively connected to network 16. Figure 1This allows it to send data to client device 12 ( Figure 1 The system transmits data and receives data from the client device 12. Information received from the client device 12 can be processed (e.g., using processing module 66) and stored (e.g., using database module 62).

[0054] As described above, the data processing system 20 also includes a processing module 66. The processing module 66 processes data stored by the data processing system 20 or otherwise accessible. For example, the processing module 66 may be used to perform one or more operations described herein (e.g., operations associated with the PII prevention ML engine 34). In some implementations, software applications can be used to facilitate the tasks described herein. As an example, the application can be installed on the data processing system 20 and / or the client device 12. Furthermore, users can interact with the application to input data and / or commands to the data processing system 20 and to review the data generated by the data processing system 20.

[0055] refer to Figure 3The diagram illustrates a process 70 for placing clinical trial data in a pending state before submitting it to a hardware storage device to detect whether the clinical trial data includes a PII. In operation, client device 12 receives (72) the PII into an EDC system (e.g., data processing system 20). Data processing system 20 receives the registered information and places it in a pending state (74). While the data is in a pending state, data processing system 20 uses prompt 30 to transmit or send (76) the data to a PII prevention ML engine 34. Using the received data, PII prevention ML engine 34 detects (78) that the data does not contain a PII or detects (82) that the data does contain a PII. When the PII prevention ML engine 34 detects that the data does not contain a PII, the data submitter 44 releases the data from the pending state (80), for example, by sending instructions to the hardware storage device 24 to retrieve the data and (in some examples) other instructions to remove one or more records associated with the data from table 26 (because the data is about to be submitted to the hardware storage device 46 and will no longer be pending). The hardware storage device 46 submits the data to the hardware storage device 46 by creating (81) an entry in the data point table 48 for the received data. The hardware storage device 46 also creates an entry in the audit trail table 50 to store audit data associated with the received data now submitted to the hardware storage device 46. When the PII prevention ML engine 34 detects that the data contains a PII, the PII detector 38 transmits a notification to the UI generator 18, which in turn causes the client device 12 to display (84) a pop-up window to the user to inform the user of the presence of a PII and request confirmation that the data does not contain a PII. When the data is confirmed (88) to not include a PII, the client device 12 transmits data indicating the absence of a PII to the PII detector 38. The process continues as described above, and the data is released from the pending state. However, when the data does include a PII, the client device updates (86) the input data and transmits the updated input data to the pending state marker 21. The process described herein is repeated until no PII is detected or until it is confirmed that the input data does not include a PII.

[0056] refer to Figure 4 The input data 92 is shown. As described herein, the pending status marker 21 ( Figure 1 The input data 92 is associated with the pending status. Using the input data 92, the prompt 30 ( Figure 1 ) Generates prompt 94 with parts 94a and 94b. Part 94a instructs PII to prevent ML engine 34 ( Figure 1The input data 92 is evaluated to detect whether it contains a PII. Section 94b specifies the type of PII. Using input data 92 and hint 94, the PII prevention ML engine 34... Figure 1 Apply the ML model and output a response specifying whether a PII was detected (96).

[0057] In this example, the prompt 30 ( Figure 1 This generates detailed input instructions (e.g., prompt 94) that guide the machine learning model (e.g., LLM) in reading, interpreting, and generating the desired output. The machine learning model will be provided with values ​​for text fields and instructed to evaluate whether the text includes a PII. In this example, prompt 94 specifies the role (instructions on what role the LLM should take as part of the scheme), domain knowledge (the definition of the PII and what it covers (name, address, etc.)), text values ​​(the content of the text fields), and output format (0 / 1 (or true / false)).

[0058] Figure 5 An example of training a machine learning system is shown. For example, training sets 102a…102n are input with various types of input data (e.g., including text entries). Training sets 102a…102n may include input data previously received by various users during a specified time period. Training sets 102a…102n are mapped to labels 104a…104n via mapping structures 106a…106n. The mapping structure includes pointers or other types of structures that associate one data item with another. Labels 104a…104n specify whether a particular data item is a PII.

[0059] Training sets 102a…102n can include any suitable number of training datasets. Each training example in training sets 102a…102n can include contextual data for a name (e.g., SSN, first and last name) specifying the data type, combined with values ​​for that particular data type. The contextual data can also include information about the structure of the data, such as whether hyphens exist in the data and whether any values ​​are expected to be uppercase.

[0060] Using the machine learning techniques described above, a model can be trained by machine learning model trainer 108 to detect PIIs. In some implementations, machine learning model trainer 108 can compute a numerical representation (e.g., in vector form) of the training data used for machine training. In some implementations, machine learning techniques include feature extraction to build different neurons within a neural network. One or more features can be translated into one or more instances of a PII. For example, a particular feature can correspond to a particular type of PII, such that the strength of the present feature leads to the determination of a particular metric (e.g., an indication of a PII) for the corresponding event (e.g., the detection of a specified type of data in the input data).

[0061] In some implementations, a single machine learning model can be trained. In others, multiple models can be trained based on a training set 102a…102n that is mapped to labels 104a…104n via a mapping structure 106a…106n. Once trained, the trained machine learning model is able to receive and process requests. For example, given clinical trial input data, the trained machine learning model can detect PIIs. In some implementations, the trained machine learning model can filter inputs where the probability of experiencing an event (e.g., detecting a PII) is below a threshold (e.g., less than 50%) for that input. In some implementations, the threshold can be provided along with the training data; for example, the threshold probability can be higher for some events (e.g., a specified type of PII) and lower for others (e.g., other types of PII). Model Training - Historical Data Extraction and Processing

[0062] In some examples, training datasets 102a…102n are formed by mapping the mapping structure 106a…106n to labels 104a…104n. To fine-tune the PII prevention ML engine 34 (which includes a machine learning model), small, high-quality, and diverse training datasets are used. This strategy prevents overfitting while ensuring the model performs well across different scenarios. The training dataset includes positive cases (PII detected) and negative cases (PII not detected) to provide balanced training data. The positive dataset (e.g., the training dataset with positive cases) will be primarily derived from historical data. To generate this data, data processing system 20 extracts entries from an audit trail table of detected and edited PIIs. By linking these edited entries to internal work requests (e.g., requests to remove or edit PIIs), data processing system 20 retrieves raw values ​​submitted by the user. For example, an entry such as “John Doe, 123 Main St, john.doe@example.com” will be used as input, where the expected result is “PII detected”. These real-world examples form the core of the positive dataset, reflecting real-world scenarios for successfully managing PII.

[0063] To further enhance the positive dataset, the data processing system 20 generates synthetic examples of PIIs. These synthetic scenarios will cover situations that may not be well represented in historical data. By simulating diverse PII patterns, the model will be exposed to edge cases and anomalous formats, thereby improving its robustness and generalization.

[0064] Negative entries will include those from the audit trail that have not been edited, indicating that the content does not contain a PII. For example, an entry such as "The patient is stable and vital signs are within normal range" will be marked as non-PII. To enhance the dataset, the data processing system 20 generates synthetic non-PII examples (such as "The subject arrived on time") to ensure that the model is well trained to identify non-sensitive content and avoid false positives.

[0065] Using the prepared dataset (as described above), the machine learning model is fine-tuned through supervised learning, where the input consists of text entries (both PII and non-PII cases) and labels indicate the expected result: PII detected or PII not detected.

[0066] To optimize for inference latency, the data processing system 20 implements a binary classification method, which simplifies the output of the machine learning model to a direct 1 / 0 or true / false format, thereby minimizing computational complexity and accelerating prediction. Additionally, the data processing system applies model quantization techniques, reducing the precision of model weights and activations (e.g., from 32-bit floating-point to 8-bit integer). This significantly reduces the model's memory footprint and computational requirements, enabling faster inference without sacrificing accuracy. These optimizations together ensure the model operates efficiently in real-time environments.

[0067] refer to Figure 6A A graphical user interface 120 for inputting clinical trial data is shown. The graphical user interface 120 includes controls 126 and input sections 124, 128 for specifying and inputting data (including, for example, clinical trial input data). The graphical user interface 120 also includes a control 129, the selection of which causes the client device to communicate with the data processing system 20. Figure 1 The data corresponding to the data input to input units 124, 128 and specified by control 126 is transmitted. In some examples, the data corresponding to the data input to input units 124, 128 and specified by control 126 is the data corresponding to the data input to input units 124, 128 and specified by control 126.

[0068] refer to Figure 6B The graphical user interface 130 is shown with an overlay 132. In this example, the overlay 132 includes an overlay of a graphical user interface (e.g., graphical user interface 120 or another graphical user interface). For example, the overlay 132 is displayed once the PII prevention ML engine 34 detects a PII in the input data entered into the graphical user interface 120, for example. The overlay 132 displays a notification that a PII has been detected and prompts the user to select either control 132a or control 132b. When control 132a is selected, the PII detector 38 transmits information to the data submitter 44 indicating that no PII has been detected. When control 132b is selected, a graphical user interface for editing the original input data is provided to the user. In some examples, the provided graphical user interface is a version of graphical user interface 120, which displays a visual representation of the previously entered data, thereby facilitating editing of that data. In this example, controls 132a and 132b are each juxtaposed with notification 132c in the overlay 132.

[0069] refer to Figure 7 This illustrates a process 140 for detecting PIIs in clinical trial data before it is submitted to a hardware storage device to improve data security. In operation, a computer system (e.g., Figure 1The data processing system 20 causes (142) to render a graphical user interface having one or more input sections and one or more controls for inputting clinical trial data. In response to selection of at least one of the one or more controls, the computer system receives (144) clinical trial data input into at least one of the one or more input sections of the graphical user interface via the graphical user interface. Before submitting the clinical trial data to the hardware storage device, the computer system associates the clinical trial data input into at least one of the one or more input sections of the graphical user interface with a pending state in memory (146). For example, the computer system may temporarily store the input clinical trial data and associate the data with a tag or other indication having a pending state value. In another example, the computer system may identify one or more keys associated with the input clinical trial data and may store entries in a table specifying one or more keys and the location in memory where the clinical trial data is stored. Then, the data submitter 44 ( Figure 1 A key can be used to request input data associated with that key. The hardware storage device that temporarily stores the input data can then look up the location associated with the key in the table to access the input data.

[0070] Using the input data, a machine learning model detects (148) whether the clinical trial data includes a PII. Based at least in part on the output of the machine learning model, a computer system or a component thereof determines (150) whether the clinical trial data includes a PII. When it is determined that the clinical trial data does not include a PII, the computer system accesses (152) the clinical trial data that has been input into at least one of the input sections of the graphical user interface and is associated with a pending status from memory or hardware storage device. The computer system releases the clinical trial data from a pending status, for example, by sending an instruction to the hardware storage device to submit the clinical trial data, and by removing one or more entries in a table that represent the clinical trial data and are associated with a pending status, etc. (154). The computer system submits the released clinical trial data to the hardware storage device. Example computer system

[0071] Figure 8An example computing system according to an implementation of this disclosure is depicted. System 160 can be used for any of the operations described with respect to the various implementations discussed herein. System 160 may include one or more processors 162, memory 166, one or more storage devices 164, and one or more input / output (I / O) devices 172 controllable via one or more I / O interfaces 170. The various components 162, 166, 164, 170, or 172 may be interconnected via at least one system bus 168, which enables data transfer between various modules and components of system 160.

[0072] One or more processors 162 may be configured to process instructions for execution within system 160. One or more processors 162 may include one or more single-threaded processors, one or more multi-threaded processors, or both. One or more processors 162 may be configured to process instructions stored in memory 166 or on storage device 164. One or more processors 162 may include one or more hardware-based processors, each including one or more cores. One or more processors 162 may include one or more general-purpose processors, one or more special-purpose processors, or both.

[0073] Memory 166 may store information within system 160. In some implementations, memory 166 includes one or more computer-readable media. Memory 166 may include any number of volatile memory cells, any number of non-volatile memory cells, or both. Memory 166 may include read-only memory, random access memory, or both. In some examples, memory 166 may be used as active or physical memory by one or more software modules. One or more storage devices 164 may be configured to provide (e.g., persistent) mass storage to system 160. In some implementations, one or more storage devices 164 may include one or more computer-readable media. For example, one or more storage devices 164 may include a floppy disk device, a hard disk device, an optical disk device, or a magnetic tape device. One or more storage devices 164 may include read-only memory, random access memory, or both. One or more storage devices 164 may include one or more of an internal hard disk drive, an external hard disk drive, and a removable drive.

[0074] One or more of memory 166 and storage devices 164 may include one or more computer-readable storage media (CRSM), including, for example, non-transitory computer-readable media. The CRSM may include one or more of electronic storage media, magnetic storage media, optical storage media, magneto-optical storage media, quantum storage media, mechanical computer storage media, etc. The CRSM may provide storage for computer-readable instructions describing data structures, processes, applications, programs, other modules, or other data used for the operation of system 160. In some implementations, the CRSM may include a data store that provides storage for computer-readable instructions or other information in a non-transitory format. The CRSM may be incorporated into system 160 or may be external to system 160. The CRSM may include read-only memory, random access memory, or both. One or more CRSMs suitable for tangibly representing computer program instructions and data may include any type of non-volatile memory, including but not limited to: semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. In some examples, processor 162 and memory 166 may be supplemented by or incorporated into one or more application-specific integrated circuits (ASICs).

[0075] System 160 may include one or more I / O devices 172. The I / O device 172 may include one or more input devices, such as a keyboard, mouse, pen, game controller, touch input device, audio input device (e.g., microphone), gesture input device, haptic input device, image or video capture device (e.g., camera), or other devices. In some examples, the I / O device 172 may also include one or more output devices, such as a display, one or more LEDs, audio output devices (e.g., speakers), printers, and haptic output devices. The I / O device 172 may be physically incorporated into one or more computing devices of system 160, or may be external to one or more computing devices of system 160. System 160 may include one or more I / O interfaces 170 to enable components or modules of system 160 to control, interact with, or otherwise communicate with (one or more) I / O devices 172. The I / O interfaces 170 may enable information to be transmitted within or outside system 160 or between components of system 160 via serial communication, parallel communication, or other types of communication. For example, the I / O interfaces 170 may conform to a version of the RS-232 standard for serial ports or a version of the IEEE 1284 standard for parallel ports. As another example, the I / O interfaces 170 may be configured to provide connectivity via a Universal Serial Bus (USB) or Ethernet. In some examples, the I / O interfaces 170 may be configured to provide a serial connection compliant with a version of the IEEE 1394 standard. One or more I / O interfaces 170 may also include one or more network interfaces that enable communication between computing devices in system 160 or between system 160 and other network-connected computing systems. One or more network interfaces may include one or more network interface controllers (NICs) or other types of transceiver devices configured to send and receive communications on one or more networks using any network protocol. The computing devices of System 160 can communicate with each other or with other computing devices using one or more networks. Such networks can include public networks such as the Internet, private networks such as institutional or personal intranets, or any combination of private and public networks. Networks can include any type of wired or wireless network, including but not limited to local area networks (LANs), wide area networks (WANs), wireless local area networks (WWANs), wireless LANs (WLANs), and mobile communication networks (e.g., 3G, 4G, Edge, etc.). In some implementations, communication between computing devices can be encrypted or otherwise protected. For example, communication can employ one or more public or private encryption keys, passwords, digital certificates, or other credentials supported by security protocols such as Secure Sockets Layer (SSL) or Transport Layer Security (TLS) protocols. System 160 may include any number of computing devices of any type. One or more computing devices may include, but are not limited to: personal computers, smartphones, tablet computers, wearable computers, implantable computers, mobile gaming devices, e-book readers, automotive computers, desktop computers, laptop computers, notebook computers, game consoles, home entertainment devices, network computers, server computers, mainframe computers, distributed computing devices (e.g., cloud computing devices), microcomputers, system-on-a-chip (SoC), and system-in-package (SiP). Although the examples herein may describe one or more computing devices as one or more physical devices, implementations are not limited thereto. In some examples, a computing device may include one or more of a virtual computing environment, hypervisor, emulation, or virtual machine executing on one or more physical computing devices. In some examples, two or more computing devices may include clusters, clouds, farms, or other groups of multiple devices that coordinate operations to provide load balancing, failover support, parallel processing capabilities, shared storage resources, shared networking capabilities, or other aspects.

[0076] This specification uses the term "configured to" in conjunction with system and computer program components. For a system of one or more computers configured to perform a particular operation or action, this means that the system has software, firmware, hardware, or a combination thereof installed on it that, in operation, causes the system to perform that operation or action. For a computer program configured to perform a particular operation or action, this means that one or more programs include instructions that, when executed by a data processing device, cause the device to perform that operation or action.

[0077] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed in this specification and their equivalents, or in one or more of these combinations. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs (i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium) for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more of these combinations. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals) generated to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0078] The term data processing device (also referred to herein as a data processing system) means data processing hardware and encompasses all kinds of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The device may also be or further include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0079] Computer programs (which may also be referred to or described as programs, software, software applications, apps, modules, software modules, scripts, or code) can be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages); and computer programs can be deployed in any form (including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment). A program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file holding other programs or data (e.g., stored in one or more scripts within a markup language document), stored in a single file dedicated to the program in question, or stored in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code sections). Computer programs can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected via a data communication network.

[0080] In this specification, the term "database" is used broadly to refer to any collection of data: data that does not need to be structured in any particular way, or does not need to be structured at all, and can be stored on storage devices in one or more locations. Therefore, for example, an indexed database may include multiple data collections, which can be organized and accessed differently.

[0081] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in others, multiple engines may be installed and run on the same or multiple computers.

[0082] The processing and logic flow described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by manipulating input data and generating outputs. The processing and logic flow can also be performed by dedicated logic circuitry (e.g., FPGA or ASIC), or by a combination of dedicated logic circuitry and one or more programmable computers.

[0083] A computer suitable for executing computer programs can be based on a general-purpose microprocessor, a special-purpose microprocessor, or both, or any other type of central processing unit (CPU). Typically, the CPU receives instructions and data from read-only memory (ROM) or random access memory (RAM), or both. The basic components of a computer are the CPU for making or executing instructions and one or more memory devices for storing instructions and data. The CPU and memory may be supplemented by or incorporated into special-purpose logic circuitry. Typically, a computer will also include one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or operatively coupled to receive data from or transfer data to one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks), or both. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in another device (e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few).

[0084] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including by way of example: semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROMs and DVD-ROMs.

[0085] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) via which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Furthermore, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving response messages from the user in response.

[0086] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer with a graphical user interface, web browser, or app through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs) (e.g., the Internet).

[0087] A computing system may include clients and servers. Clients and servers are typically geographically separated and interact via a communication network. The client-server relationship is established by means of computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., HTML pages) to a user device, for example, for the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from that user. Data generated at the user device (e.g., the result of user interaction) can be received from that device at the server.

[0088] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in the specification within the context of separate embodiments may also be implemented in combination in a single embodiment. On the other hand, various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, one or more features from a claimed combination may, in some cases, be removed from that combination, and a claimed combination may refer to a sub-combination or a variation of a sub-combination.

[0089] Similarly, although operations are depicted in a specific order in the drawings and described in the claims, this should not be construed as requiring such operations to be performed in the specific order shown or in sequence, or requiring all illustrated operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0090] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions described in the claims may be performed in a different order and still achieve the desired result. As an example, the processes depicted in the figures do not necessarily require the specific order or sequence shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method implemented by a computer system for detecting personally identifiable information (PII) in clinical trial data before submitting the data to a hardware storage device to improve data security, the method comprising: This enables a computer system to render a graphical user interface with one or more input sections and one or more controls for inputting clinical trial data; In response to the selection of at least one of the one or more controls, the computer system receives clinical trial data input to at least one of the one or more input sections of the graphical user interface via the graphical user interface; Before submitting the clinical trial data to the hardware storage device, the clinical trial data in at least one of the one or more input sections of the graphical user interface is associated with a pending status in the memory; The machine learning model is used to detect whether the clinical trial data includes personally identifiable information (PII). Whether the clinical trial data includes PII is determined, at least in part, based on the output of the machine learning model; If it is determined that the clinical trial data does not include PII, the clinical trial data associated with the pending status is accessed from memory in at least one of the one or more input sections of the graphical user interface. Release the clinical trial data from the pending state; as well as The released clinical trial data is submitted to the hardware storage device.

2. The method according to claim 1, further comprising: The clinical trial data received from the machine learning model does not include indications of PII.

3. The method according to claim 1, further comprising: The clinical trial data, including indications of PII, are received from the machine learning model. This causes an overlay to be rendered in the graphical user interface, wherein the rendered overlay displays a notification of the detected PII, and also displays: A first notification used to confirm that the clinical trial data does not include a PII, wherein the first notification is juxtaposed with the notification in the overlay; and This is used to update the clinical trial data to a second prompt that excludes the PII, wherein the second prompt is juxtaposed with the notification in the overlay.

4. The method according to claim 3, further comprising: Data confirmed to be received that does not include PII in the clinical trial data.

5. The method according to claim 3, wherein, The superimposed material is a first superimposed material, and the method further includes: a) The computer system receives updates to the input clinical trial data through the graphical user interface; b) Before submitting the update to the input clinical trial data to the hardware storage device, the update to the clinical trial data is associated with a pending status indicator in the memory; c) The computer system enables the machine learning model to detect whether the update to the input clinical trial data includes a PII; d) If the machine learning model detects that an update to the input clinical trial data includes a PII, A second overlay is rendered in the graphical user interface, wherein the rendered second overlay displays another notification of PII detection and also displays: A first notification confirming that the update to the clinical trial data does not include a PII, wherein the first notification is juxtaposed with the other notification in the second overlay; and This is used to update the updated clinical trial data to a second notification that does not include the PII, wherein the second notification is juxtaposed with the other notification in the second overlay; e) Repeat step ad until confirmation is received that the update to the clinical trial data does not include the PII, or until the machine learning model detects the absence of the PII. When a confirmation is received that the update to the clinical trial data does not include a PII, or when the machine learning model detects that a PII is not present, the most recently submitted update to the input clinical trial data is submitted to the hardware storage device.

6. The method according to claim 1, wherein, The machine learning model is a Large Language Model (LLM), and the method further includes: Generate system prompts to guide the LLM in interpreting inputs corresponding to clinical trial data input to at least one of the one or more input sections of the graphical user interface and to provide output of a specified type; and The system prompts and the inputs corresponding to the clinical trial data input into at least one of the one or more input sections of the graphical user interface are entered into the LLM.

7. The method according to claim 1, wherein, Submissions include: Entries are created in a designated table in the hardware storage device, wherein data corresponding to the input clinical trial data is stored in the created entries; and Create entries in the audit trail to track modifications to the entered clinical trial data or data corresponding to the entered clinical trial data.

8. The method according to claim 1, wherein, Submitting the clinical trial data to the hardware storage device includes submitting data corresponding to the clinical trial data to the hardware storage device.

9. A data processing system for detecting personally identifiable information (PII) in clinical trial data before submitting the clinical trial data to a hardware storage device to improve data security, the data processing system comprising: One or more processing devices; as well as One or more machine-readable hardware storage devices storing instructions that can be executed by the one or more processing devices to perform operations, including: This enables the rendering to have a graphical user interface with one or more input sections and one or more controls for inputting clinical trial data; In response to the selection of at least one of the one or more controls, clinical trial data input to at least one of the one or more input sections of the graphical user interface is received through the graphical user interface; Before submitting the clinical trial data to the hardware storage device, the clinical trial data in at least one of the one or more input sections of the graphical user interface is associated with a pending status in the memory; The machine learning model is used to detect whether the clinical trial data includes personally identifiable information (PII). Whether the clinical trial data includes PII is determined, at least in part, based on the output of the machine learning model; If it is determined that the clinical trial data does not include PII, the clinical trial data associated with the pending status is accessed from memory in at least one of the one or more input sections of the graphical user interface. Release the clinical trial data from the pending state; and The released clinical trial data is submitted to the hardware storage device.

10. The data processing system according to claim 9, wherein, The operation also includes: The clinical trial data received from the machine learning model does not include indications of PII.

11. The data processing system according to claim 9, wherein, The operation also includes: The clinical trial data, including indications of PII, are received from the machine learning model. This causes an overlay to be rendered in the graphical user interface, wherein the rendered overlay displays a notification of the detected PII, and also displays: A first notification used to confirm that the clinical trial data does not include a PII, wherein the first notification is juxtaposed with the notification in the overlay; and This is used to update the clinical trial data to a second prompt that excludes the PII, wherein the second prompt is juxtaposed with the notification in the overlay.

12. The data processing system according to claim 11, wherein, The operation also includes: Data confirmed to be received that does not include PII in the clinical trial data.

13. The data processing system according to claim 11, wherein, The superimposed material is a first superimposed material, wherein the operation further includes: a) Receive updates to the input clinical trial data through the graphical user interface; b) Before submitting the update to the input clinical trial data to the hardware storage device, the update to the clinical trial data is associated with a pending status indicator in the memory; c) Enable the machine learning model to detect whether the update to the input clinical trial data includes PII; d) If the machine learning model detects that an update to the input clinical trial data includes a PII, A second overlay is rendered in the graphical user interface, wherein the rendered second overlay displays another notification of PII detection and also displays: A first notification confirming that the update to the clinical trial data does not include a PII, wherein the first notification is juxtaposed with the other notification in the second overlay; and This is used to update the updated clinical trial data to a second notification that does not include the PII, wherein the second notification is juxtaposed with the other notification in the second overlay; e) Repeat step ad until confirmation is received that the update to the clinical trial data does not include the PII, or until the machine learning model detects the absence of the PII. When a confirmation is received that the update to the clinical trial data does not include a PII, or when the machine learning model detects that a PII is not present, the most recently submitted update to the input clinical trial data is submitted to the hardware storage device.

14. The data processing system according to claim 9, wherein, The machine learning model is a Large Language Model (LLM), and the operation further includes: Generate system prompts to guide the LLM in interpreting inputs corresponding to clinical trial data input to at least one of the one or more input sections of the graphical user interface and to provide output of a specified type; and The system prompts and the inputs corresponding to the clinical trial data input into at least one of the one or more input sections of the graphical user interface are entered into the LLM.

15. The data processing system according to claim 9, wherein, Submissions include: Entries are created in a designated table in the hardware storage device, wherein data corresponding to the input clinical trial data is stored in the created entries; and Create entries in the audit trail to track modifications to the entered clinical trial data or data corresponding to the entered clinical trial data.

16. The data processing system according to claim 9, wherein, Submitting the clinical trial data to the hardware storage device includes submitting data corresponding to the clinical trial data to the hardware storage device.

17. A non-transitory computer-readable medium for detecting personally identifiable information (PII) in clinical trial data before submitting the clinical trial data to a hardware storage device to improve data security, the non-transitory computer-readable medium storing instructions executable by one or more processing devices to perform operations including: This enables the rendering to have a graphical user interface with one or more input sections and one or more controls for inputting clinical trial data; In response to the selection of at least one of the one or more controls, clinical trial data input to at least one of the one or more input sections of the graphical user interface is received through the graphical user interface; Before submitting the clinical trial data to the hardware storage device, the clinical trial data in at least one of the one or more input sections of the graphical user interface is associated with a pending status in the memory; The machine learning model is used to detect whether the clinical trial data includes personally identifiable information (PII). Whether the clinical trial data includes PII is determined, at least in part, based on the output of the machine learning model; If it is determined that the clinical trial data does not include PII, the clinical trial data associated with the pending status is accessed from memory in at least one of the one or more input sections of the graphical user interface. Release the clinical trial data from the pending state; as well as The released clinical trial data is submitted to the hardware storage device.

18. The non-transitory computer-readable medium according to claim 17, wherein, The operation also includes: The clinical trial data received from the machine learning model does not include indications of PII.

19. The non-transitory computer-readable medium according to claim 17, wherein, The operation also includes: The clinical trial data, including indications of PII, are received from the machine learning model. This causes an overlay to be rendered in the graphical user interface, wherein the rendered overlay displays a notification of the detected PII, and also displays: A first notification used to confirm that the clinical trial data does not include a PII, wherein the first notification is juxtaposed with the notification in the overlay; and This is used to update the clinical trial data to a second prompt that excludes the PII, wherein the second prompt is juxtaposed with the notification in the overlay.

20. The non-transitory computer-readable medium according to claim 19, wherein, The operation also includes: Data confirmed to be received that does not include PII in the clinical trial data.

21. A computer program product storing instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1-8.

Images