System and method for converting electronic data capture dataset to standard data tabulation model (SDTM) dataset

Abstract

To provide a method of converting an Electronic Data Capture (EDC) dataset to a Standard Data Tabulation Model (SDTM) dataset.SOLUTION: A method includes a step of processing metadata for an EDC dataset, the metadata comprising EDC field names, to produce vectors of a particular dimensionality. The vectors are processed in an embedding model to produce embedded vectors of a lesser dimensionality. The embedded vectors are processed, in a prediction model, to produce class predictions for the EDC field names, the classes corresponding to SDTM field names. The EDC field names are associated with the SDTM field names based on the class predictions for the EDC field names. The embedding model is obtained from a trained Siamese neural network.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] This disclosure relates generally to mapping Electronic Data Capture (EDC) fields of clinical trial data to the Study Data Tabulation Model (SDTM) and converting EDC datasets to SDTM datasets. [Background technology]

[0002] Clinical trials are becoming increasingly large with highly sophisticated protocols and multiple, diverse data sources, large volumes of data, decentralization, and increasingly complex adaptive designs. By one estimate, data points collected in Phase III trials have tripled over the past decade, reaching an average of 3.6 million, and the complexity of trials is driving the expansion of data volumes generated from an increasingly wide range of sources.

[0003] Electronic Data Capture (EDC) systems are widely used in clinical trials for data collection and management. EDC systems include software applications that allow direct data entry at the point of data collection, replacing traditional paper-based data collection methods. Before the start of a clinical trial, the clinical trial protocol is used to design an electronic case report form (eCRF) in the EDC system. The eCRF contains a structured format for entering the trial data. This design includes all fields required to collect data specified in the clinical trial protocol, such as patient demographics, medical history, treatment assignment, laboratory results, adverse events, and outcomes.

[0004] An important advancement in clinical data management has been standardization by mapping clinical data to the Standard Data Tabulation Model (SDTM), which defines a standard structure for human clinical trial and non-clinical trial data tabulations submitted as part of product applications to regulatory authorities such as the U.S. Food and Drug Administration (FDA), streamlining the handling and comparison of clinical data across different clinical trials. Summary of the Invention [Problem to be solved by the invention]

[0005] Mapping EDC fields to SDTM can be a challenging technical problem for several reasons. EDC systems are primarily designed for data collection, with fields organized around the needs of clinical data entry and site management. EDC fields, such as those related to the collection of adverse events, demographics, and other clinical endpoints, may be specified in a study- or company-specific manner and therefore cannot be easily mapped to the SDTM standard. SDTM, on the other hand, is designed for data submission to regulatory agencies such as the FDA and follows a standard structure and terminology set by the Clinical Data Interchange Standards Consortium (CDISC). Differences in structure and terminology make a direct mapping between EDC and SDTM difficult. Furthermore, there is a large variability in how different EDC systems are designed and used. The same type of data may be captured differently in different systems, and the same therapeutic drug may be captured in different studies. This variability complicates the mapping process.

[0006] Therefore, the traditional process of mapping data from electronic data capture (EDC) systems to SDTM generally requires the involvement of programmers who write scripts to read annotated case report forms (CRFs) and convert the data into SDTM format. Furthermore, EDC systems used in clinical trials are often designed to facilitate study conduct, making mapping to SDTM a secondary priority. Thus, with such traditional approaches, the process of mapping data from EDC systems to SDTM is very laborious and often not reproducible.

[0007] To address these issues, attempts have been made to standardize EDC so that once a mapping exists for a study, all future studies will essentially follow the same pattern. However, EDC standardization has drawbacks, including loss of flexibility, difficulty in establishing common standards, and most importantly, inapplicability to mapping legacy studies that have never been submitted in SDTM format.

[0008] One of the barriers to building a practical machine learning (ML) solution for mapping EDC fields to SDTM is the relative lack of training data: even the largest pharmaceutical companies have only a few thousand clinical trials available to train their algorithms. [Means for solving the problem]

[0009] In disclosed embodiments, a Siamese network can be used to generate electronic data capture (EDC) metadata-based embeddings, and machine learning (ML) classifiers can be used on these embeddings to predict associated Standard Data Tabulation Model (SDTM) fields. The Siamese neural network is trained to have lower dimensional embeddings where similar data points are closer based on a distance metric, such as Euclidean distance, than dissimilar data points. This allows learning of sub-networks that can compare the low dimensional embedding of each field to all other fields. In disclosed embodiments, this allows expanding N data points into an O(N2) set from which a classifier neural network can be trained. After obtaining this embedding, the low dimensional representation can be used to generate a final predictive model to classify the fields of the EDC form into SDTM domains.

[0010] The disclosed embodiments enable large-scale mapping of legacy clinical trials to the SDTM format, significantly increasing the value of legacy data by enabling cross-analysis of clinical trials without the overhead of manually aligning datasets prior to analysis.

[0011] In the disclosed embodiment, a Siamese neural network is used to calculate the similarity between form fields that are typically specified in an EDC system. These similarity metrics can be fed into further machine learning stages for final classification. The output can be used to scale map EDC data to SDTM and / or drive tools that provide mapping recommendations.

[0012] In one aspect, the disclosed embodiments provide a method, system, and computer-readable medium for converting an electronic data capture (EDC) dataset into a standard data compilation model (SDTM) dataset. The method includes processing metadata of the EDC dataset, the metadata including EDC field names, to generate a vector of dimension n1, where n1 is an integer. The method further includes processing the vector of dimension n1 in an embedding model to generate an embedding vector of dimension n2, where n2 is an integer and less than n1. The method further includes processing the embedding vector in a predictive model to generate class predictions for the EDC field names, where the classes correspond to the SDTM field names. The method further includes mapping the EDC field names to the SDTM field names, respectively, based at least in part on the class predictions for the EDC field names. The embedding model is obtained from a trained Siamese neural network including a first embedding sub-network and a second embedding sub-network.

[0013] Implementations can include one or more of the following features, either alone or in combination.

[0014] The method further includes training a Siamese neural network based on a subset of EDC field names pre-mapped to a subset of SDTM field names. The subset of EDC field names may be manually curated. The method may further include generating a pair of vectors of dimension n1 to be input to the first and second embedding sub-networks, respectively. The method may further include processing the vectors of dimension n1 with a long short-term memory neural network by the first and second embedding sub-networks to generate embedding vectors.

[0015] The method may further include determining a distance between each pair of embedding vectors generated by the first and second embedding sub-networks, respectively, in an outer layer of the Siamese neural network. The method may further include applying at least one of batch normalization and activation functions to the determined distances of the embedding vectors output by the first and second embedding sub-networks, and backpropagating the results obtained by the application to the first and second embedding sub-networks. Various architectures, including different neural network layers, activation functions, and distance metrics, can be used for the Siamese network and its sub-networks. The method may further include training a classification model using a subset of EDC field names pre-mapped to a subset of SDTM field names (the subset of EDC field names is processed by the embedding model before being input to the classification model), and using the trained classification model as a predictive model. The method further includes converting the EDC dataset to an SDTM dataset based at least in part on the mapping. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a block diagram of a system for converting an electronic data capture (EDC) dataset into a standard data aggregation model (SDTM) dataset in accordance with disclosed embodiments. [Diagram 2] FIG. 1 is a block diagram of an EDC to SDTM field mapping system according to disclosed embodiments. [Diagram 3] FIG. 1 is a block diagram of a subsystem for generating an embedded model using a Siamese neural network embedded model. [Figure 4] FIG. 1 is a block diagram of a subsystem that trains an ML classification model using an embedding model generated by a Siamese neural network. [Diagram 5] FIG. 1 is a block diagram of a Siamese neural network embedding model and a pair generation unit. [Figure 6] This table compares the overall accuracy and macro F1 score of the disclosed method using three methods for evaluating predicted labels with two prior approaches. [Figure 7] A plot of accuracy against the number of training samples per class. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. However, the components described in the following embodiment are merely examples, and are not intended to limit the technical scope of the present invention.

[0018] [First embodiment] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of embodiments of the present invention. However, those skilled in the art will appreciate that embodiments of the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the present invention.

[0019] In a disclosed embodiment, a Siamese neural network is used to calculate the similarity between electronic data capture (EDC) fields that map to Standard Data Tabulation Model (SDTM) entries (referred to herein as "SDTM fields"). The resulting similarity metric produces a low-dimensional embedding in which EDC fields that map to the same SDTM field are closer in distance (e.g., Euclidean distance) than fields that are farther apart. This low-dimensional embedding is fed into a secondary classification algorithm to convert the embedding into an actual prediction.

[0020] 1 is a block diagram of a system 100 for converting an electronic data capture (EDC) dataset 110 to a standard data compilation model (SDTM) dataset 120. The EDC dataset 110 is generated by output from an EDC system 130, which may receive EDC data from many sources, such as clinical trial sites. To enable the conversion of the data, an EDC-SDTM field mapping system 140 retrieves metadata from the EDC dataset 110, including EDC field names and other metadata such as field descriptions, units in which the fields are recorded, and which form and field IDs are associated with the fields, and maps the EDC fields to standardized fields in SDTM.

[0021] The results of the field mapping are output to an EDC-SDTM data conversion system 150. The conversion system 150 uses the field mapping to convert the EDC dataset 110 into a standard format for the SDTM dataset 120. Once the data is converted, the SDTM dataset 120 is output to an SDTM data distribution system 160, which controls the transmission of the SDTM dataset 120. For example, the SDTM dataset 120 is received by the SDTM data distribution system 160, which transmits the data to various SDTM-based systems, e.g., via a cloud-based network 170. In an embodiment, the SDTM dataset 120 may be transmitted to a regulatory agency, such as the U.S. Food and Drug Administration (FDA).

[0022] Converting EDC datasets of clinical trial data into datasets with the standardized SDTM format is a technical challenge because complex methodologies and tools must be developed, implemented, and maintained to ensure that fields in the EDC datasets are accurately mapped to the standardized SDTM fields. While the underlying motivation to perform EDC field mapping and convert EDC datasets to SDTM datasets may stem from legal or regulatory requirements, the actual process of achieving the mapping and conversion involves addressing a number of technical challenges, as discussed in more detail below.

[0023] FIG. 2 is a block diagram of a neural network-based system 200 for performing EDC-to-SDTM field mapping. In the disclosed embodiment, the system 200 corresponds to the EDC-to-SDTM field mapping system 140 described above (see FIG. 1). The system 200 trains and uses an architecture of neural networks and machine learning models to process an unclassified EDC field data set 205 obtained from the EDC data set 110 (see FIG. 1). The EDC field data set 205 is processed by: (i) an embedding model 210, and (ii) a secondary prediction model 220. The result of processing the EDC field data set 205 by these two models is a set of final class (i.e., classification) predictions of the EDC field mapping. In this way, the EDC fields are classified with a certain determined probability in relation to the SDTM fields. The final class predictions 225 can be output to a data conversion and distribution system 230, such as the EDC-to-SDTM data conversion system 150 and the SDTM data distribution system 160 described above (see FIG. 1). In the disclosed embodiments, confidence scores may be determined and the closest matches may be presented as suggestions via a user interface.

[0024] In the disclosed embodiment, the embedded model 210 and the secondary predictive model 220, which may be referred to as the "principal runtime models," are generated by training corresponding "proxy models" and using the resulting trained model as the respective "principal runtime models." Specifically, the embedded model 210 is generated by training a Siamese neural network embedded model 245, as described below, and exporting the resulting trained model for use as the embedded model 210. Similarly, the secondary predictive model 220 is generated by training a machine learning (ML) classification model 250 and using the resulting trained model as the secondary predictive model 220. In the disclosed embodiment, the training of the proxy models may be performed during runtime, in which case the trained proxy models may be used to periodically update the corresponding principal runtime models. Alternatively, some or all of the training may be performed prior to runtime, with periodic updates occurring at runtime.

[0025] Training of the proxy models, i.e., Siamese neural network embedding model 245 and ML classification model 250, can be performed using a database of pre-mapped, e.g., manually curated, fields 240, including EDC fields that have been manually mapped to corresponding SDTM fields. The embedding model 210 receives the curated EDC fields and outputs a set of n-dimensional training embeddings 255 (e.g., 32-dimensional embeddings). The curated EDC fields and the SDTM fields to which the curated EDC fields were manually mapped, i.e., the final class labels, are output as training data to the Siamese neural network embedding model 245 via a second pass 252.

[0026] A pair generator 260 is provided along the second path 252 to generate all possible pair combinations between the curated EDC fields and the final class labels that are input to the Siamese neural network embedding model 245. The final class labels are also output via a third path 253 to the ML classification model 250 to be used as ground truth.

[0027] As described above, the system 200 trains and uses an architecture of neural networks and machine learning models to process the unclassified EDC field data set 205 retrieved from the EDC data set 110 (see FIG. 1 ). In a disclosed embodiment, the retrieved data may include metadata related to the electronic case report form (eCRF) used in the clinical trial, such as the form name (FormOID), pre-text, such as a description of the field in question, post-text, such as additional description, such as the format the input is expected to take, such as the expected date format, and FieldOID, which is an internal identifier for the particular field in question. In a disclosed embodiment, these four information fields may be associated with SDTM domains. However, while the disclosed embodiment focuses on these four fields, the methodology is not specific to these fields and may be implemented using other fields, such as the distribution of recorded data.

[0028] For our evaluation, the dataset was limited to the following domains: Adverse Events (AE), Treatments (DS), Concomitant Medications (CM), Exposures (EC), Medical History (MH), Vital Signs (VS), and Demographics (DM). Fields related to these domains were manually mapped to SDTM by human curators for evaluation. While this is not a comprehensive list of domains that are part of the SDTM dataset, these domains reasonably represent the constructs that require mapping.

[0029] Typically, domains in SDTM format are represented in a mix of unpivoted (i.e. narrow) and pivoted (i.e. wide) forms. To simplify the evaluation, all domains were represented as unpivoted tables. Note that, based on the SDTM standard, there is a simple way to convert from pivoted to unpivoted representations. Furthermore, the evaluation did not include composite fields (fields that are the result of a mathematical operation between one or more fields, such as BMI). The training data used for the evaluation had approximately 9000 entries corresponding to the 57 SDTM fields.

[0030] 3 is a block diagram of a subsystem (part of system 200 described above) that generates an embedding model 210 using a Siamese neural network embedding model 245. The use of a Siamese neural network provides a data embedding where data in the same class are close to data points in another class with respect to a given distance metric.

[0031] Training of the Siamese neural network 245 uses a database of pre-mapped fields 240, which are EDC fields mapped to corresponding SDTM fields. The database 240 provides EDC metadata to the pair generator 260, including manually curated EDC fields and final class (i.e., classification) labels, which are the corresponding SDTM fields.

[0032] In a disclosed embodiment, the pair generator 260 can generate all possible pair combinations between the curated EDC fields and output the resulting pairs to two independent inputs of the Siamese neural network 245. In a disclosed embodiment, the pairs may be generated randomly. As an example, for each SDTM field, 100 pairs are generated in which the two EDC fields map to the same SDTM entry (e.g., EDC fields labeled AESTD and AECODED are both mapped to AEDECOD), and 100 pairs are generated in which the two fields are associated with different SDTM elements (e.g., EDC fields labeled AGE and WEIGHT). The output variable of the Siamese neural network 245 is 0 if they belong to the same SDTM field and 1 if they do not belong to the same SDTM field.

[0033] This mapping operation effectively expands the amount of training data by an order of magnitude. For example, if we pair 1000 data points with another 1000 data points in every possible combination, the resulting training set will have 1 million data points. This is advantageous, since a typical language model may require billions of data points. Moreover, as we show later in the results discussion, using such an effectively expanded dataset leads to a significant improvement in accuracy compared to traditional approaches.

[0034] The mapping operation sets a minimum distance between two entries of the same class, and an arbitrary distance if they are not the same class. A holdout set is generated, stratified by SDTM fields, to ensure that the training and test sets contain the same proportion of SDTM fields. For example, 10% of the entries for each SDTM field can be excluded from the model training process and evaluated. This ensures that each class in the dataset is represented in both the training and test sets.

[0035] The Siamese neural network 245 is trained to map text into a low-dimensional space such that the geometric (e.g., Euclidean) distance between points in that space corresponds to the semantic similarity between related text (e.g., EDC metadata). Thus, similar inputs will have lower dimensional embeddings that are closer based on a distance metric such as Euclidean distance than dissimilar data points. This allows for the learning of sub-networks that can compare the lower dimensional embedding of each field to every other field. The lower dimensional embeddings produced by training the Siamese neural network 245 are exported for use as the embedding model 210, which in turn is used in training the ML classification model 250 (see FIG. 4).

[0036] FIG. 4 is a block diagram of a subsystem (e.g., part of system 200 described above) that uses an embedding model 210 generated by a Siamese neural network 245 to train an ML classification model 250. The training uses a database 240 of manually curated fields, which are EDC fields that have been manually mapped to corresponding SDTM fields. The database 240 provides EDC metadata, including the manually curated EDC fields, to the embedding model 210. The database 240 also provides final class (i.e., classification) labels, which are the corresponding SDTM fields, to the ML classification model 250 to use as ground truth.

[0037] In the disclosed embodiment, the ML classification model 250 may be a k-nearest neighbor (kNN) classifier. Alternatively, techniques such as XGBoost (optimized distributed gradient boosting library), random forests, or additional neural networks may be used to perform the classification. The use of a kNN classifier facilitates the evaluation of the embedding quality by providing information about the number of nearest neighbors that belong to the same class. Specifically, it can be evaluated whether any of the k nearest neighbors has the correct class, whether the correct class is predicted by the nearest neighbors, or whether the correct class is predicted by a majority vote of the k nearest neighbors.

[0038] The embedding model 210 is generated by a Siamese neural network 245, as described above, and is adapted to convert text into embeddings of a particular dimension, e.g., dense vectors. In a disclosed embodiment, the curated EDC fields from the database 240 are converted into a set of training embeddings in the form of n-dimensional (e.g., 32-dimensional) embeddings 255. The training embeddings 255 are, in turn, used to train the ML classification model 250.

[0039] 5 is a block diagram of the Siamese Neural Network embedding model 245 and the pair generator 260. The Siamese Neural Network architecture is designed to learn the similarity or dissimilarity between two different inputs. In the context of the disclosed embodiment, EDC fields that are mapped to the same SDTM domain should be more similar than EDC fields that are mapped to different SDTM fields.

[0040] The Siamese neural network 245 trains a first embedding sub-network 510, which is a neural network that generates an embedding for a first input from the pair generator 260. The second embedding sub-network 520 is a "clone" of the first embedding network 510, i.e., the sub-network shares the same architecture and weights, and generates an embedding for a second input from the pair generator 260. The embeddings for the first and second inputs are then compared in an outer layer 530, which has a distance module 535 that calculates the distance (e.g., Euclidean distance) between the first and second embeddings.

[0041] Each of the embedding sub-networks (510, 520) includes a text embedding layer 540 to convert words into vectors. The text embedding layer 540 converts linear vectors of dictionary-encoded text into matrices with one row for each element of the original vector. Words found in the same context are encoded into N-dimensional vectors such that similar words are mapped to similar vectors.

[0042] As described above, the data retrieved from the manually curated fields database 240 may include metadata associated with the electronic case report form (eCRF), such as the form name (FormOID), pre-text, such as a description of the field in question, post-text, additional description, such as the format the input is expected to take, such as an expected date format, and a FieldOID, which is an internal identifier for the particular field in question.

[0043] In the disclosed embodiment, to pre-process the input into a format usable by the neural network, the text is first converted to raw text by stripping out any HTML tags that may be present in the text. This can be done, for example, using Python's html2text library. After conversion, the four fields can be concatenated. The maximum length of this string is N words. This long string can be dictionary encoded, for example, via Gensim. In another embodiment, a more sophisticated neural network may be used. In this case, each type of data is encoded separately before being combined by the aggregation layer.

[0044] Encoding the dictionary can be a relatively simple process of assigning a 1 to the first word observed in the text, a 2 to the second word observed in the series, etc. The result of this encoding is a numerical vector that can be processed by a neural network. These embeddings are typically high-dimensional (e.g., a 1000-dimensional vector corresponding to the length of the input), so they are further condensed into lower-dimensional embeddings by an embedding sub-network, as described below.

[0045] In the disclosed embodiment, the text embedding layer 540 is trained along with the embedding sub-network 510 rather than using a pre-trained word embedding model. This is because the amount of vocabulary specific to EDC to SDTM conversion, such as FormOID and FieldOID, does not fit embeddings trained on a more general vocabulary, along with the technical terms and abbreviations given in the text description.

[0046] The encoded text is output by the text embedding layer 540 to a long-short-memory (LSTM) neural network 545, a type of recurrent neural network that is particularly good at processing sequences of data. The LSTM 545 can take a sequence of word vectors as input and output a single vector that represents the entire sequence. As explained above, pairs of fields are used to train the Siamese neural network embedding model 245. If a pair of fields is similar, the LSTM 545, and the corresponding LSTMs in the clone sub-network 520, are trained to make their outputs as similar as possible. On the other hand, if the pair of fields is dissimilar, the LSTMs are trained to make their outputs as different as possible. This can be achieved by using certain types of loss functions, such as contrastive loss, as explained in more detail below.

[0047] The output of the LSTM 545 is processed by a flattening module 547 to constrain the output to a desired distribution. For example, a tanh activation function can be applied to the output of the LSTM 545 to provide an embedding that is centered at 0 and has an approximately equal distribution of positive and negative values.

[0048] The resulting output of the LSTM 545 and the flattening module 547 is a set of n-dimensional field embeddings 550, where n is an integer, e.g., 8, 16, 32, 64, etc. As explained above, the field embeddings 550 of the first and second embedding sub-networks (510, 520) are compared in the outer layer 530 by a distance module 535 that calculates the distance, e.g., Euclidean distance, between the embeddings of the first and second embedding sub-networks (510, 520). A batch normalization module 555 may process the output of the distance module 535 to achieve a set of normalized values, specifically standardizing the scores across each batch of input pairs, ensuring that the batch has a mean of 0 and a standard deviation of 1. This can make the network more robust to different scales of raw similarity scores, improving the speed, performance, and stability of the twin neural network 245.

[0049] The final activation module 560 of the outer layer 530 applies a final activation function to the outputs of the embedding sub-networks (510, 520). The final activation layer is used to convert the embeddings generated by the sub-networks (510, 520) into a final value that reflects the difference between the different fields. For example, if the goal is to determine the similarity between two inputs provided by the pair generator 260, the desired output may be a single number that represents the degree of similarity. In this case, a sigmoid activation function may be used in the final layer to push the output into the range [0,1].

[0050] Thus, the Siamese neural network 245 effectively takes two copies of the sub-networks (510, 520) and combines them in an outer layer 530 that determines the distance between the embeddings that result when pairs of data are input together. Specifically, each sub-network (510, 520) processes the inputs and generates an output vector. The outer layer 530 (or "final activation layer") receives these two output vectors and calculates a final output. This final output can be, for example, a similarity score (e.g., based on Euclidean distance) between the two input sequences. A loss function (e.g., contrast loss or triplet loss) is used to calculate a scalar loss value. The loss function is adapted to encourage similar input pairs to have similar output vectors and dissimilar input pairs to have dissimilar output vectors.

[0051] In the backpropagation process, the gradient of the loss function is calculated with respect to each parameter of the model (i.e., weights and biases). This is effectively done by calculating from the loss function, through the final activation layer, and back to the sub-networks (510, 520). The gradients calculated during backpropagation are used to adjust the model parameters in a way that reduces the loss. This is done using an optimization algorithm such as stochastic gradient descent (SGD).

[0052] In a Siamese neural network architecture, the sub-networks (510, 520) share weights, meaning they have the same parameters. When gradients are calculated during backpropagation, they are accumulated across both sub-networks (510, 520) and the weights are updated based on the combined gradients. This process is repeated over many epochs (i.e., iterations over the training data), gradually adjusting the weights and biases to minimize the loss and learn a useful representation of the input. Over time, the sub-networks (510, 520) learn to output similar vectors for similar inputs and different vectors for different inputs.

[0053] One advantage of training the subnetwork such that the distance between similar data points, e.g., Euclidean distance, is minimized is that it tends to produce compact clusters that are amenable to other classifiers. Thus, once the Siamese neural network 245 is trained, the embedding subnetwork can be used as a pre-processor for training and evaluating machine learning techniques, or as a feed to a secondary neural network.

[0054] Figure 6 shows a table comparing the overall accuracy and Macro F1 of the disclosed method with two prior approaches using three methods of evaluating the predicted labels. Namely, 10% of the entries for each SDTM field were excluded from the model learning process for evaluation. This ensured that each class in the dataset was represented in both the training and test sets. The basic accuracy, i.e., the number of predicted values ​​that matched the SDTM fields, was evaluated, as well as the Macro F1 score. The results show that the disclosed approach can accurately predict most of the appropriate SDTM mappings. In most cases, nearest neighbors are good predictors of a given class. In the evaluation, the nearest field in the embedding space was able to predict the appropriate SDTM field with a probability of ~90%.

[0055] As a comparison, we present two prior machine learning (ML) approaches that we implemented for the task of mapping EDC fields to SDTM fields. The first approach used a traditional ML tool, XGBoost (optimized distributed gradient boosting library), on the metadata of the same fields, which were encoded via term frequency-inverse document frequency (TF-IDF). The second method attempted to directly predict the field mappings using a deep learning implementation that uses a pre-trained text embedding layer (RoBERTa) along with two bidirectional LSTMs implemented in Keras. As is evident from the results presented in the table, the approach disclosed herein (using a Siamese neural network and an ML classifier) ​​is significantly more accurate than previous attempts using traditional ML techniques and direct prediction with neural networks.

[0056] Analysis of the clusterings produced by the embeddings showed that the Siamese neural network can generate embeddings that are compact enough to allow machine learning (ML) techniques (e.g., ML classification models) to accurately separate the classes. We found that EDC fields that map to the same SDTM field are closely spaced in the embedding space, allowing them to be classified with high accuracy. Note that the results presented here are for the holdout set only. The self-prediction accuracy of the algorithm was above 99%. However, the accuracy on the holdout set was lower due to issues such as vocabulary discontinuity between the test and training sets.

[0057] Figure 7 plots accuracy against the number of training samples per class. On average, accuracy is high, but there is a relatively large variation in accuracy across classes. While the majority of classes have an accuracy of over 90%, the minimum accuracy is only 25%. However, there is a clear trend of accuracy increasing as the amount of training data increases. We observe that for classes with fewer than 100 samples in the training data, there is insufficient information to capture the variation in how fields are represented. For classes with more than 100 training samples, more than 90% of the fields were correctly predicted.

[0058] Compared to the conventional implementations evaluated, we observed a significant improvement in evaluation accuracy, regardless of the degree of tuning of the neural network architecture. On average, we observed a 10-15% improvement in overall classification accuracy when compared to conventional techniques, as defined by correctly predicted labels. Furthermore, prediction results were more consistent across different classes, as evidenced by significantly higher macro F1 scores.

[0059] Evaluation of the disclosed embodiments showed that for the set of domains in which the data was trained and tested, the metadata associated with EDC fields contained sufficient information to accurately determine the SDTM fields with which they match. Using a Siamese neural network, we were able to generate embeddings in which EDC fields mapped to the same SDTM field were embedded closely together in the same space. Most of the inaccuracies in the evaluation could be explained by the relative paucity of training data for certain domains, and the relative paucity of overlapping vocabulary, which tended to prevent the embedding layer from generating useful embeddings for some of the training data. These are all data-related issues that could potentially be addressed by expanding the dataset.

[0060] During the training process of the Siamese neural network, the ability to predict whether two entries map to the same SDTM domain reached an accuracy of over 99%. In cases where sufficient training data existed, the overall accuracy of the holdout set was over 95%. Even with issues of vocabulary inconsistency and under-representation in certain domains, the model was relatively accurate in the holdout set. Another area of ​​inaccuracy in the evaluation concerns fields that may overlap across multiple fields, such as dates. For example, many forms have EDC fields that reference dates, and while most of their metadata is identical, the SDTM mapping is determined only by the FormOID. In such cases, the approach described here can be used, but instead of looking at each EDC field individually, the metadata is aggregated across all fields, and the SDTM domain is targeted instead of the SDTM field.

[0061] Aspects of the present invention may be embodied in the form of a system, a computer program product, or a method. Likewise, aspects of the present invention may be embodied as hardware, software, or a combination of both. Aspects of the present invention may also be embodied as a computer program product stored on one or more computer readable mediums in the form of computer readable program code embodied thereon.

[0062] The computer readable medium may be a computer readable storage medium, which may be, for example, an electronic, optical, magnetic, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof.

[0063] The computer program code in the embodiments of the present invention can be written in any suitable programming language and / or scripting language. The program code may be executed on a single computer or on multiple computers. The computer may include a processing unit in communication with a computer usable medium, the computer usable medium including a set of instructions, and the processing unit is designed to execute the set of instructions and / or a trained machine learning algorithm.

[0064] The above discussion is intended to illustrate the principles and various embodiments of the present invention. Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.

Claims

1. 1. A method for converting an Electronic Data Capture (EDC) dataset into a Standard Data Collection Model (SDTM) dataset, comprising: processing metadata including EDC field names of the EDC dataset to generate a vector of dimension n1, where n1 is an integer; processing a vector of dimension n1 in an embedding model to generate an embedding vector of dimension n2, where n2 is an integer and less than n1; processing the embedding vector through a predictive model to generate class predictions for the EDC field names that correspond to SDTM field names; mapping each of the EDC field names to the SDTM field names based at least in part on a class prediction of the EDC field names; Including, The method, wherein the embedding model is obtained from a trained Siamese neural network, the embedding model including a first embedding sub-network and a second embedding sub-network.

2. 2. The method of claim 1, further comprising training the Siamese neural network based on a subset of the EDC field names pre-mapped to the subset of the SDTM field names.

3. The method of claim 2 , wherein the subset of EDC field names is manually curated.

4. The method of claim 2 , further comprising the step of generating a pair of vectors of dimension n1 to be input to the first and second embedded sub-networks, respectively.

5. 5. The method of claim 4, further comprising processing a vector of dimension n1 with a long short-term memory neural network to generate the embedding vector by the first embedding sub-network and the second embedding sub-network.

6. 5. The method of claim 4, further comprising determining a distance between each pair of embedding vectors generated by the first embedding sub-network and the second embedding sub-network, respectively, in an outer layer of the Siamese neural network.

7. training a classification model using a subset of metadata pre-mapped to the subset of SDTM field names; using the trained classification model as a predictive model; Further comprising: The method of claim 2 , wherein the subset of metadata is processed by an embedding model before being input to a classification model.

8. The method of claim 1 , further comprising converting the EDC data set to an SDTM data set based at least in part on the mapping.

9. 1. A conversion system for converting an Electronic Data Capture (EDC) dataset into a Standard Data Collection Model (SDTM) dataset, comprising: a computer having one or more processors in communication with a memory storing instructions executable by said one or more processors; The instruction: processing metadata including EDC field names from the EDC dataset to generate a vector of dimension n1, where n1 is an integer; processing a vector of dimension n1 in an embedding model to generate an embedding vector of dimension n2, where n2 is an integer and less than n1; processing the embedding vector through a predictive model to generate class predictions for the EDC field names that correspond to SDTM field names; mapping each of the EDC field names to the SDTM field names based at least in part on a class prediction of the EDC field names; is an instruction to execute The embedding model is a transformation system obtained from a trained Siamese neural network, the transformation system including a first embedding sub-network and a second embedding sub-network.

10. 10. The conversion system of claim 9, further comprising training the Siamese neural network based on a subset of the EDC field names pre-mapped to a subset of the SDTM field names.

11. The conversion system of claim 10 , wherein the subset of EDC field names is manually curated.

12. 11. The system of claim 10, further comprising instructions for generating a pair of vectors of dimension n1 to be input to the first and second embedded sub-networks, respectively.

13. The instruction:

13. The conversion system of claim 12, further comprising the step of processing a vector of dimension n1 with a long short-term memory neural network by the first embedding sub-network and the second embedding sub-network to generate the embedding vector.

14. The instruction:

13. The conversion system of claim 12, further comprising the step of determining a distance between each pair of embedding vectors generated by the first and second embedding sub-networks, respectively, in an outer layer of the Siamese neural network.

15. The instruction: training a classification model using a subset of the metadata that has been pre-mapped to a subset of the SDTM field names; using the trained classification model as a predictive model; Further execute The conversion system of claim 10 , wherein the subset of metadata is processed by an embedding model before being input to a classification model.

16. The instruction:

10. The conversion system of claim 9, further performing the step of converting the EDC data set to an SDTM data set based at least in part on the mapping.

17. 1. A non-volatile computer readable medium having stored thereon instructions that, when executed by one or more processors of a computer, cause the one or more processors to perform a method for converting an electronic data capture (EDC) dataset to a standard data compilation model (SDTM) dataset, the method comprising: The method comprises: processing metadata including EDC field names from the EDC dataset to generate a vector of dimension n1, where n1 is an integer; processing a vector of dimension n1 in an embedding model to generate an embedding vector of dimension n2, where n2 is an integer and less than n1; processing the embedding vector through a predictive model to generate class predictions for the EDC field names that correspond to SDTM field names; mapping each of the EDC field names to the SDTM field names based at least in part on a class prediction of the EDC field names; Including, The computer-readable medium, wherein the embedding model is obtained from a trained Siamese neural network, the Siamese neural network including a first embedding sub-network and a second embedding sub-network.

18. The method comprises:

20. The computer-readable medium of claim 17, further comprising training the Siamese neural network based on a subset of the EDC field names pre-mapped to a subset of the SDTM field names.

19. 20. The computer-readable medium of claim 18, wherein the subset of EDC field names is manually curated.

20. 20. The computer-readable medium of claim 18, wherein the method further comprises generating a pair of vectors of dimension n1 to be input to the first and second embedded sub-networks, respectively.

21. The method comprises:

21. The computer-readable medium of claim 20, further comprising processing, by the first embedding sub-network and the second embedding sub-network, a vector of dimension n1 with a long short-term memory neural network to generate the embedding vector.

22. The method comprises:

21. The computer-readable medium of claim 20, further comprising determining, in an outer layer of the Siamese neural network, a distance between each pair of embedding vectors generated by the first and second embedding sub-networks, respectively.

23. training a classification model using a subset of metadata pre-mapped to the subset of SDTM field names; using the trained classification model as a predictive model; Further comprising:

20. The computer-readable medium of claim 18, wherein the subset of metadata is processed by an embedding model before being input to a classification model.

24. 20. The computer-readable medium of claim 17, further comprising converting the EDC data set to an SDTM data set based at least in part on the mapping.

Images