System and method for converting EDC data sets to SDTM data sets

Abstract

The invention relates to a system and method for converting an EDC data set into an SDTM data set. The method includes processing metadata of an EDC dataset to generate a vector of a particular dimension, the metadata including an EDC field name. The vectors are processed in an embedding model to generate embedded vectors of lower dimensions. The embedded vectors are processed in a prediction model to generate category predictions for the EDC field names, the categories corresponding to the SDTM field names. And mapping the EDC field name and the SDTM field name based on category prediction of the EDC field name. The embedded model is obtained from a trained twin neural network.

Description

Technical Field

[0001] The present disclosure generally relates to mapping electronic data capture (EDC) fields of clinical trial data to a study data tabulation model (SDTM) and converting EDC datasets to SDTM datasets. Background Art

[0002] Clinical trials are proliferating, with highly complex protocols and multiple data sources, resulting in large, decentralized, and increasingly complex adaptive designs. One estimate suggests that the number of data points collected in Phase III trials has tripled over the past decade to an average of 3.6 million, and the increase in trial complexity is driving the growth in data volumes and from a wider range of sources.

[0003] Electronic data capture (EDC) systems are widely used in clinical trials to collect and manage data. They include software applications that allow data to be entered directly at the point of collection, replacing traditional paper-based data collection methods. Before the start of a trial, the clinical trial protocol is used to design an electronic case report form (eCRF) within the EDC system. The eCRF contains a structured format for entering trial data. The design includes all the required fields to capture the data specified in the trial protocol, such as patient demographics, past medical history, treatment assignments, laboratory results, adverse events, and outcomes.

[0004] An important advancement in clinical data management is the standardization of trial data by mapping them to the Standard Data Tabulation Model (SDTM), which defines a standard structure for tabulating data from human clinical trials and nonclinical studies that are submitted to regulatory agencies, such as the U.S. Food and Drug Administration (FDA), as part of product applications. This improves the efficiency of handling and comparing clinical data between different trials.

[0005] Mapping EDC fields to SDTM can be a challenging technical problem for a number of reasons. EDC systems are primarily designed for data collection, with fields organized around the needs of clinical data entry and site management. EDC fields (e.g., fields referencing adverse events, demographics, and other clinical endpoints) may be specified in a study- or company-specific manner and therefore not easily mapped to the SDTM standard. SDTM, on the other hand, is designed for submission of data to regulatory agencies such as the FDA and follows a standardized structure and terminology set by the Clinical Data Interchange Standards Consortium (CDISC). Differences in structure and terminology can make direct mapping between EDC and SDTM difficult. In addition, there are significant differences in how different EDC systems are designed and used. The same type of data may be captured differently in different systems, even in different studies for the same therapeutic agent. This variation complicates the mapping process.

[0006] Therefore, the traditional process of mapping data from electronic data capture (EDC) systems to SDTM usually requires the involvement of programmers who are responsible for reading annotated case report forms (CRFs) and writing scripts to convert the data into SDTM format. In addition, EDC systems used in clinical trials are usually designed to facilitate trial conduct, and mapping to SDTM is secondary. Therefore, in these traditional methods, the process of mapping EDC system data to SDTM is very labor-intensive and in many cases non-reproducible.

[0007] Attempts have been made to address these issues by standardizing EDC construction so that once a mapping of a trial exists, all future trials will essentially follow the same pattern. However, standardization of EDCs has several disadvantages, such as loss of flexibility, difficulty in establishing general standards, and most importantly, this approach is not suitable for mapping legacy trials that were not previously submitted in SDTM format.

[0008] One of the barriers to building a viable machine learning (ML) solution to map EDC fields to SDTM is the relative paucity of training data, especially when compared to the number of fields that need to be mapped, which can be in the thousands. Even the largest pharmaceutical companies may only have a few thousand trials available to train their algorithms. Summary of the invention

[0009] In a disclosed embodiment, a Siamese network can be used to generate embeddings based on electronic data capture (EDC) metadata, and a machine learning (ML) classifier can use these embeddings to predict related Standard Data Tabulation Model (SDTM) fields. The Siamese network is trained in such a way that similar data points will have lower dimensional embeddings that are closer based on some distance metric (e.g., Euclidean distance) than dissimilar data points. This allows training of subnetworks where the lower dimensional embeddings for each field can be compared to all other fields. In the disclosed embodiment, this allows scaling to O(N) for N data points. 2 ) pairs in order to train a classifier neural network therefrom. After obtaining this embedding, the lower dimensional representation can be used to generate a final predictive model to classify EDC form fields into their SDTM domains.

[0010] In the disclosed embodiments, legacy trials can be mapped to the SDTM format at scale, greatly increasing the value of legacy data by allowing cross-trial analysis without the overhead of manually reconciling pre-analytical datasets.

[0011] In the disclosed embodiment, a twin neural network is used to calculate similarities between form fields commonly specified in EDC systems. These similarity measures can be input into further machine learning stages for final classification. This output can be used to scale EDC data to SDTM and / or drive tools that provide mapping suggestions.

[0012] In one aspect, the disclosed embodiments provide a method, system, and computer-readable medium for converting an electronic data capture (EDC) data set into a standard data tabulation model (SDTM) data set. The method includes processing metadata of the EDC data set (the metadata includes EDC field names) to generate a dimension n 1 A vector of n 1 is an integer. The method further includes processing the dimension n in the embedding model 1 The vector of dimension n is generated 2 The embedding vector of 2 is an integer less than n 1 The method further includes processing the embedding vectors in a prediction model to generate category predictions for the EDC field names, the categories corresponding to the SDTM field names. The method further includes mapping the EDC field names to the SDTM field names, respectively, based at least in part on the category predictions for the EDC field names. The embedding model is obtained from a trained Siamese neural network, the Siamese neural network comprising a first embedding subnetwork and a second embedding subnetwork.

[0013] Embodiments may include one or more of the following features, alone or in combination.

[0014] The method may further include training a twin neural network based on a subset of EDC field names pre-mapped to a subset of SDTM field names. The subset of EDC field names may be manually curated. The method may further include generating a n-dimensional 1 The method may further include processing the vector of dimension n in the long short-term memory neural network through the first embedding subnetwork and the second embedding subnetwork. 1 to generate the embedding vector.

[0015] The method may further include determining, in an outer layer of the twin neural network, the distance between each pair of embedding vectors generated by the first embedding subnetwork and the second embedding subnetwork, respectively. The method may further include applying at least one of batch normalization and an activation function to the determined distance of the embedding vectors output by the first embedding subnetwork and the second embedding subnetwork, and backpropagating the result obtained by the application to the first embedding subnetwork and the second embedding subnetwork. The twin neural network and its subnetworks may use various architectures, including different neural network layers, activation functions, and distance metrics. The method may further include training a classification model using a subset of EDC field names pre-mapped to a subset of SDTM field names, wherein the subset of EDC field names is processed by the embedding model before being input to the classification model; and using the trained classification model as a prediction model. The method may further include converting the EDC dataset into an SDTM dataset based at least in part on the mapping. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a block diagram of a system for converting an electronic data capture (EDC) data set to a standard data tabulation model (SDTM) data set according to a disclosed embodiment.

[0017] Figure 2 is a block diagram of an EDC to SDTM field mapping system according to the disclosed embodiment.

[0018] Figure 3 is a block diagram of the subsystem for generating an embedding model using a Siamese neural network embedding model.

[0019] Figure 4 is a block diagram of the subsystem for training an ML classification model using the embedding model generated by a Siamese neural network.

[0020] Figure 5 It is a block diagram of the twin neural network embedding model and pair generator.

[0021] Figure 6 is a table of overall accuracy and macro F1 scores of the disclosed method, which evaluates the predicted labels using three methods and compares with two previous methods.

[0022] Figure 7 is a plot of accuracy versus the number of training samples for each class.

[0023] Where appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. Additionally, some of the blocks depicted in the figures may be combined into a single functionality. DETAILED DESCRIPTION

[0024] In the following detailed description, many specific details are set forth in order to fully understand the embodiments of the present invention. However, it will be appreciated by those skilled in the art that embodiments of the present invention may be practiced without these specific details. In other cases, well-known methods, procedures, components, and circuits are not described in detail to avoid obscuring the present invention.

[0025] In the disclosed embodiment, a Siamese neural network is used to compute similarities between electronic data capture (EDC) fields mapped to Standard Data Tabulation Model (SDTM) entries (referred to herein as "SDTM fields"). The similarity measure thus generated results in the generation of a low-dimensional embedding in which EDC fields mapped to the same SDTM field are closer in distance (e.g., Euclidean distance) than fields that are farther away. This low-level embedding is then input into a secondary classification algorithm to convert the embedding into an actual prediction.

[0026] Figure 1 is a block diagram of a system 100 for converting an electronic data capture (EDC) data set 110 into a standard data tabulation model (SDTM) data set 120. The EDC data set 110 is generated by the output of an EDC system 130, which can receive EDC data from multiple sources (e.g., clinical trial sites). To achieve data conversion, an EDC to SDTM field mapping system 140 retrieves metadata from the EDC data set 110 to map the EDC fields to the standardized fields of the SDTM, the metadata including the EDC field names and other metadata (e.g., field descriptions, units of field records, and forms and field IDs associated with the fields).

[0027] The results of the field mapping are output to an EDC to SDTM data conversion system 150, which uses the field mapping to convert the EDC dataset 110 into a standardized form of an SDTM dataset 120. Once the data is converted, the SDTM dataset 120 may be output to an SDTM data distribution system 160, which controls the transmission of the SDTM dataset 120. For example, the SDTM dataset 120 may be received by the SDTM data distribution system 160, which in turn may transmit the data to various SDTM-based systems via, for example, a cloud-based network 170. In an embodiment, the SDTM dataset 120 may be transmitted to a regulatory agency, such as the U.S. Food and Drug Administration (FDA).

[0028] Converting EDC datasets of clinical trial data into datasets with standardized SDTM format is a technical problem because it requires the development, implementation, and maintenance of complex methods and tools to ensure that the fields of the EDC datasets are accurately mapped to the standardized SDTM fields. While the underlying motivation for performing EDC field mapping and converting EDC datasets into SDTM datasets may stem from legal and regulatory requirements, the actual process of achieving the mapping and conversion involves addressing many technical challenges, as discussed in further detail below.

[0029] Figure 2 is a block diagram of a neural network-based system 200 for performing EDC to SDTM field mapping. In the disclosed embodiment, the system 200 corresponds to the EDC to SDTM field mapping system 140 described above (see Figure 1 ). The system 200 trains and uses the architecture of neural networks and machine learning models to process the unclassified EDC field dataset 205 retrieved from the EDC dataset 110 (see Figure 1 ). The EDC field dataset 205 is processed by the following models: (i) an embedding model 210; and (ii) a secondary prediction model 220. The result of the processing of the EDC field dataset 205 by these two models is a set of final category (i.e., classification) predictions for the EDC field mapping. Thus, the EDC field is classified with a specific determined probability associated with the SDTM field. The final category predictions 225 can be output to a data conversion and distribution system 230, such as the EDC to SDTM data conversion system 150 and the SDTM data distribution system 160 discussed above (see Figure 1 ). In the disclosed embodiments, a confidence score can be determined and the closest matches can be presented as suggestions through a user interface.

[0030] In the disclosed embodiment, the embedding model 210 and the secondary prediction model 220 (which may be referred to as the "primary runtime model") are generated by training the corresponding "proxy model" and using the resulting trained model as the corresponding primary runtime model. Specifically, the embedding model 210 is generated by training the twin neural network embedding model 245 described below and deriving the resulting trained model for use as the embedding model 210. Similarly, the secondary prediction model 220 is generated by training the machine learning (ML) classification model 250 and using the resulting trained model as the secondary prediction model 220. In the disclosed embodiment, the training of the proxy model can be performed during runtime, in which case the corresponding primary runtime model can be periodically updated using the trained proxy model. Optionally, some or all of the training can be completed before runtime and may be periodically updated during runtime.

[0031] Training of the proxy models (i.e., the twin neural network embedding model 245 and the ML classification model 250) can be performed using a pre-mapped database (e.g., manually managed fields 240) that includes EDC fields that have been manually mapped to corresponding SDTM fields. The managed EDC fields are output as training data to the ML classification model 250 via a first path 251, which includes an embedding model 210 that receives the managed EDC fields and outputs a set of n-dimensional (e.g., 32-dimensional embedding) training embeddings 255. The managed EDC fields and the SDTM fields that are manually mapped to the managed EDC fields (i.e., the final class labels) are output as training data to the twin neural network embedding model 245 via a second path 252.

[0032] A pair generator 260 is provided along the second path 252, which generates all possible pair combinations between the managed EDC fields and the final class labels for input to the Siamese neural network embedding model 245. The final class labels are also output to the ML classification model 250 via the third path 253 for use as ground truth.

[0033] As described above, the system 200 trains and uses a neural network and machine learning model architecture to process the unclassified EDC field dataset 205 retrieved from the EDC dataset 110 (see Figure 1 ). In the disclosed embodiments, the retrieved data may include metadata associated with an electronic case report form (eCRF) used in a clinical trial, such as: the form name (FormOID); pre-text, such as a description of the field in question; post-text, such as additional descriptions, such as the format in which the input is expected, such as the expected date format; and FieldOID, which is an internal identifier for the specific field in question. In the disclosed embodiments, these four information fields may be associated with an SDTM domain. However, while the disclosed embodiments focus on these four fields, the method is not specific to these fields and may be implemented using other fields, such as the distribution of record data.

[0034] In the evaluation performed by the inventors, the dataset was limited to the following domains: Adverse Events (AE), Medical Conditions (DS), Concomitant Medication (CM), Exposed Medication (EC), Past Medical History (MH), Vital Signs (VS), and Demographics (DM) domains. The fields associated with these domains were manually mapped to SDTM by a human curator for evaluation. While this does not represent a complete list of domains that are part of the SDTM dataset, these domains are a reasonable representation of the structure that needs to be mapped.

[0035] Typically, domains in SDTM format are represented as a mix of unpivoted (i.e., narrow) and pivoted (i.e., wide) formats. To simplify the evaluation, each domain is represented as an unpivoted table. It is worth noting that there is a simple way to convert a pivoted representation to an unpivoted representation based on the SDTM standard. In addition, the evaluation does not include composite fields (i.e., fields resulting from mathematical operations between one or more fields, such as BMI). In the training data used in the evaluation, there are approximately 9000 separate entries corresponding to 57 different SDTM fields.

[0036] Figure 3 is a block diagram of a subsystem, such as a portion of system 200 described above, that generates embedding model 210 using a siamese neural network embedding model 245. Using a siamese neural network (which may be referred to as a "siamese neural network") provides data embeddings in which data within the same class are closer to data points in another class than they are to data points in another class with respect to a given distance metric.

[0037] The training of the twin neural network 245 uses a database of pre-mapped fields 240, which are EDC fields that have been mapped to corresponding SDTM fields. The database 240 provides EDC metadata to the pair generator 260, including manually curated EDC fields and final category (i.e., classification) labels, which are corresponding SDTM fields.

[0038] In the disclosed embodiment, the pair generator 260 can generate all possible pair combinations between the managed EDC fields, and can output the resulting pairs to two independent inputs of the twin neural network 245. In the disclosed embodiment, these pairs can be randomly generated. As an example, for each SDTM field, 100 pairs are generated in which two EDCEDC fields are mapped to the same SDTM entry, for example, EDC fields labeled AESTD and AECODED are both mapped to AEDECOD, and 100 pairs are generated in which two fields, such as EDC fields labeled AGE and WEIGHT, are associated with different SDTM elements. If the output variable of the twin neural network 245 belongs to the same SDTM field, the output variable of the twin neural network 245 is 0, otherwise it is 1.

[0039] In effect, this mapping operation expands the amount of training data by several orders of magnitude. For example, if 1,000 data points are paired with another set of 1,000 data points in every possible combination, the resulting training set will have a million data points. This is advantageous because typical language models may require billions of data points. Furthermore, as described in the discussion of the results below, accuracy is significantly improved compared to traditional methods using a dataset that is effectively expanded in this way.

[0040] The mapping operation sets the minimum distance between two entries from the same category, or an arbitrary distance if they do not belong to the same category. A holdout set can be generated that is stratified by SDTM field, so the same proportion of SDTM fields will be present in the training and test sets. For example, 10% of the entries for each SDTM field can be held out from the model training process for evaluation. This ensures that every category in the dataset is represented in both the training and test sets.

[0041] The Siamese neural network 245 is trained to map text to a low-dimensional space such that the geometric (e.g., Euclidean) distance between points in this low-dimensional space corresponds to the semantic similarity between related text (e.g., EDC metadata). Thus, similar inputs will have lower dimensional embeddings that are closer based on some distance metric (e.g., Euclidean distance) than dissimilar data points. This allows for training of subnetworks where the lower dimensional embedding for each field can be compared to all other fields. The low dimensional embeddings generated by training the Siamese neural network 245 are exported for use as the embedding model 210, which in turn is used to train the ML classification model 250 (see Figure 4 ).

[0042] Figure 4 It is a block diagram of a subsystem, such as a portion of the above-described system 200, which uses a twin neural network 245 to generate an embedding model 210 to train an ML classification model 250. The training uses a database of manually managed fields 240, which are EDC fields that have been manually mapped to corresponding SDTM fields. The database 240 provides EDC metadata to the embedding model 210, including the manually managed EDC fields. The database 240 also provides the final category (i.e., classification) labels to the ML classification model 250 for use as ground truth, which are the corresponding SDTM fields.

[0043] In the disclosed embodiment, the ML classification model 250 can be a k nearest neighbor (kNN) classifier. Optionally, the classification can be performed using techniques such as XGBoost (an optimized distributed gradient boosting library), random forests, or additional neural networks. Using a kNN classifier facilitates the evaluation of embedding quality by providing information about the number of nearest neighbors that belong to the same category. Specifically, the following can be evaluated: Is the correct category found within any of the k nearest neighbors; Is it the correct category predicted by the nearest neighbors; And is the correct category predicted by the majority vote of the k nearest neighbors?

[0044] As described above, the embedding model 210 is generated by the twin neural network 245 and is suitable for converting text into an embedding of a specific dimension, such as a dense vector. In the disclosed embodiment, the managed EDC fields from the database 240 are converted into a set of training embeddings in the form of n-dimensional (e.g., 32-dimensional) embeddings 255. The training embeddings 255 are then used to train the ML classification model 250.

[0045] Figure 5 245 and pair generator 260. The twin neural network architecture is designed to learn the similarity or dissimilarity between two different inputs. In the context of the disclosed embodiments, EDC fields mapped to the same SDTM domain should be more similar than EDC fields mapped to different SDTM fields.

[0046] The Siamese neural network 245 trains a first embedding subnetwork 510, which is a neural network that generates an embedding for a first input from the generator 260. The second embedding subnetwork 520 is a "clone" of the first embedding network 510 (i.e., the subnetworks share the same architecture and weights) and generates an embedding for a second input from the generator 260. The embeddings of the first and second inputs are then compared in an outer layer 530 having a distance module 535 that computes a distance, such as a Euclidean distance, between the first and second embeddings.

[0047] The embedding subnetworks (510, 520) each include a text embedding layer 540 to convert words into vectors. The text embedding layer 540 converts the linear vector of the dictionary-encoded text into a matrix with one row for each element in the original vector. Words found in the same context are then encoded into N-dimensional vectors so that similar words will be mapped to similar vectors.

[0048] As described above, the data retrieved from the manually managed fields database 240 may include metadata associated with the electronic case report form (eCRF), such as: the form name (FormOID); leading text, such as a description of the field in question; trailing text, such as additional descriptions, such as the format in which the input is expected, such as an expected date format; and FieldOID, which is an internal identifier for the specific field in question.

[0049] In the disclosed embodiment, in order to preprocess the input into a format usable by the neural network, the text is first converted to raw text by removing any HTML tags that may be present in the text. This can be done, for example, using the Python html2text library. After the conversion, the four fields can be concatenated together. The maximum length of this string is N words. This long string can be dictionary encoded, for example, via Gensim. In an alternative embodiment, a more complex neural network can be used, where each type of data is encoded separately before being concatenated by the aggregation layer.

[0050] Dictionary encoding can be done by a relatively simple process, where the first word observed in a text is encoded as 1, the second word observed in the sequence is encoded as 2, and so on. The result of the encoding is a numeric vector that can be processed by a neural network. These embeddings are typically high-dimensional (e.g., a 1000-dimensional vector corresponding to the length of the input), so they will be further compressed into a lower-dimensional embedding by the embedding subnetwork, as described below.

[0051] In the disclosed embodiment, the text embedding layer 540 is trained together with the embedding subnetwork 510, rather than using a pre-trained word embedding model. This is because the vocabulary specific to the EDC to SDTM conversion (e.g., FormOID and FieldOID) and the terms and abbreviations given in the text descriptions make it difficult to adapt embeddings trained on a more general vocabulary.

[0052] The text embedding layer 540 outputs the encoded text to a long short-term memory (LSTM) neural network 545, which is a type of recurrent neural network that is particularly good at processing data sequences. The LSTM 545 can take a sequence of word vectors as input and output a single vector representing the entire sequence. As described above, the twin neural network embedding model 245 is trained using field pairs. If the field pairs are similar, the LSTM 545 and the corresponding LSTM of the clone subnetwork 520 are trained so that their outputs are as similar as possible. On the other hand, if the field pairs are not similar, the LSTM is trained so that its outputs are as different as possible. This can be achieved by using a specific type of loss function (e.g., contrastive loss), as described in further detail below.

[0053] The output of LSTM 545 is processed by flattening module 547 to constrain the output to a desired distribution. For example, a tanh activation function can be applied to the output of LSTM 545 to provide an embedding that is centered at 0 and has approximately equal distributions of positive and negative values.

[0054] The final output of the LSTM 545 and the flattening module 547 is a set of n-dimensional field embeddings 550, where n is an integer, such as 8, 16, 32, 64, etc. As described above, in the outer layer 530, the distance module 535 compares the field embeddings 550 of the first and second embedding subnetworks (510, 520), and the distance module 535 calculates the distance between the embeddings of the first and second embedding subnetworks (510, 520), such as the Euclidean distance. The batch normalization module 555 can process the output of the distance module 535 to obtain a set of standardized values, specifically, normalizing the scores of each batch of input pairs to ensure that the mean of the batch is 0 and the standard deviation is 1. This can make the network more robust to raw similarity scores of different scales, and can improve the speed, performance, and stability of the twin neural network 245.

[0055] The final activation module 560 in the outer layer 530 applies the final activation function to the output of the embedding subnetwork (510, 520). The final activation layer is used to convert the embedding generated by the subnetwork (510, 520) into a final value that reflects the difference between different fields. For example, if the goal is to determine the similarity between two inputs provided to the generator 260, the desired output may be a single number representing the similarity. In this case, a sigmoid activation function can be used in the last layer to compress the output into the range [0, 1], which can be interpreted as the probability that the two sequences are similar.

[0056] Thus, the twin neural network 245 actually takes two copies of the subnetwork (510, 520) and connects them together with an outer layer 530, which determines the distance between the embeddings that appear when multiple pairs of data are input together. Specifically, each subnetwork (510, 520) processes the input and produces an output vector. The outer layer 530 (or "final activation layer") takes these two output vectors and calculates the final output, for example, which can be a similarity score between the two input sequences (e.g., based on Euclidean distance). A loss function (e.g., contrastive loss or triple loss) is used to calculate a scalar loss value. The loss function is adapted to encourage similar input pairs to have similar output vectors, while different input pairs have different output vectors.

[0057] During back-propagation, the gradient of the loss function is calculated with respect to each parameter in the model (i.e., weights and biases). In practice, this is done by calculating backwards from the loss function, through the final activation layer, and then through the sub-network (510, 520). The gradients calculated during back-propagation are used to adjust the model parameters to reduce the loss. This can be done using an optimization algorithm, such as stochastic gradient descent (SGD).

[0058] In a twin neural network architecture, the subnetworks (510, 520) share weights, meaning they have the same parameters. When gradients are calculated during backpropagation, the gradients are accumulated in both subnetworks (510, 520) and the weights are updated based on the combined gradients. This process is repeated over many epochs (i.e., iterated over the training data), gradually adjusting the weights and biases to minimize the loss and learn useful representations of the inputs. Over time, the subnetworks (510, 520) learn to output similar vectors for similar inputs and different vectors for different inputs.

[0059] One of the advantages of training a subnetwork to minimize the distance (e.g., Euclidean distance) between similar data points is that it tends to generate compact clusters suitable for other classifiers. Therefore, once the Siamese neural network 245 is trained, the embedding subnetwork can be used as a preprocessor to train and evaluate machine learning techniques or as input to a secondary neural network.

[0060] Figure 6 is a table of the overall accuracy and macro F1 of the disclosed method, which uses three methods to evaluate the predicted labels and compares with two previous methods. The final classification results of the disclosed method are evaluated using 10% stratified retention, that is, 10% of the entries of each SDTM field are retained from the model training process for evaluation. This ensures that each category in the data set will be represented in both the training set and the test set. The basic accuracy (that is, the number of predictions that match the SDTM field in the retention set) and the macro F1 score are evaluated. The results show that the disclosed method can accurately predict most of the correct SDTM mappings. In most cases, the nearest neighbor is a good predictor for a given category. In this evaluation, the closest field in the embedding space is able to predict the correct SDTM field about 90% of the time.

[0061] As a comparison, the inventors implemented two previous machine learning (ML) approaches to map EDC fields to SDTM fields. The first approach used a traditional ML tool, XGBoost (Optimized Distributed Gradient Boosting Library), on the same field metadata, where the metadata was encoded via Term Frequency-Inverse Document Frequency (TF-IDF). The second approach attempted to directly predict the field mapping using a deep learning implementation that used a pre-trained text embedding layer (RoBERTa) and two bidirectional LSTMs implemented in Keras. From the results shown in the table, it is clear that the disclosed approach - using a twin neural network and an ML classifier - is much more accurate than previous attempts to make predictions using either traditional ML techniques or using neural networks directly.

[0062] Analysis of the clusters produced in the embeddings shows that the Siamese neural network is able to produce embeddings that are compact enough to allow machine learning (ML) techniques (e.g., ML classification models) to accurately separate the categories. We found that EDC fields that map to the same SDTM field are located very close in the embedding space, providing high classification accuracy. It is important to note that the results presented in this article are only for the holdout set. The algorithm has a self-prediction accuracy of over 99%. However, issues such as lexical incoherence between the test and training sets lead to lower accuracy on the holdout set.

[0063] Figure 7 is a plot of accuracy versus the number of training samples for each class. While the average accuracy is high, the spread of accuracy for each class is relatively large. While most classes have an accuracy of over 90%, the lowest accuracy is only 25%. However, there is a clear upward trend in accuracy as the amount of training data increases. It is observed that in classes with less than 100 samples in the training data, there is insufficient information to capture the variation in how the fields are represented. For classes with more than 100 training samples, over 90% of the field predictions are correct.

[0064] Compared to the traditional implementation of the evaluation, the evaluation accuracy is significantly improved without any particular degree of neural network architecture tuning. The overall classification accuracy is improved by an average of 10 to 15% compared to the traditional technique, as defined by the correctly predicted labels. In addition, the predictions are more consistent across classes, as can be seen from the significantly higher macro F1 scores.

[0065] In an evaluation of the disclosed embodiments, it was shown that for a set of domains for both training and test data, there is enough information in the metadata associated with an EDC field to accurately determine its matching SDTM field. Using a twin neural network, embeddings can be generated where EDC fields that map to the same SDTM field are tightly embedded in the same space. Most of the inaccuracies in the evaluation can be explained by the relative paucity of training data for specific domains and the relative lack of lexical overlap, which often prevents the embedding layer from generating informative embeddings for certain training data. These appear to be data-related issues that can be addressed by expanding the dataset.

[0066] During the training of the twin neural network, the ability to predict whether two entries map to the same SDTM domain reached over 99% accuracy. For cases with sufficient training data, the overall accuracy on the holdout set was greater than 95%, indicating that the model is sufficient for practical purposes. Even with issues such as vocabulary incoherence and insufficient samples for some domains, the model was relatively accurate on the holdout set, meaning that expanding the training data will produce higher accuracy. Another aspect where there is inaccuracy in the evaluation is related to fields that may be repeated in multiple fields, such as dates. For example, many forms will have an EDC field referencing a date, where most of the metadata is the same, but its SDTM mapping is determined solely by the FormOID. For these cases, the approach described in this article can be used, but instead of looking at each EDC field individually, all fields aggregate metadata and the target will be SDTM domains rather than SDTM fields.

[0067] Aspects of the present invention may be implemented in the form of a system, a computer program product, or a method. Similarly, aspects of the present invention may be implemented in hardware, software, or a combination of the two. Aspects of the present invention may be embodied as a computer program product stored on one or more computer-readable media in the form of a computer-readable program code embodied thereon.

[0068] The computer readable medium may be a computer readable storage medium. The computer readable storage medium may be, for example, an electronic, optical, magnetic, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof.

[0069] The computer program code in the embodiments of the present invention may be written in any suitable programming and / or scripting language. The program code may be run on a single computer or on multiple computers. The computer may include a processing unit in communication with a computer usable medium, wherein the computer usable medium contains a set of instructions, and the processing unit is designed to execute the set of instructions and / or the trained machine learning algorithm.

[0070] The above discussion is intended to illustrate the principles and various embodiments of the present invention. Once the above disclosure is fully understood, those skilled in the art will appreciate that there are many variations and modifications. The following claims should be interpreted to cover all such variations and modifications.

Claims

1. A method for converting an electronic data collection data set, i.e., an EDC data set, into a standard data tabulation model data set, i.e., an SDTM data set, the method comprising: Processing metadata of the EDC dataset to generate a vector of dimension n1, where n1 is an integer, the metadata including EDC field names; Processing a vector of dimension n1 in the embedding model to generate an embedding vector of dimension n2, where n2 is an integer less than n1; processing the embedding vector in a prediction model to generate a category prediction for the EDC field name, the category corresponding to the SDTM field name; and mapping the EDC field names to the SDTM field names, respectively, based at least in part on the category predictions of the EDC field names, The embedding model is obtained from a trained twin neural network, which includes a first embedding subnetwork and a second embedding subnetwork.

2. The method of claim 1 , further comprising training the twin neural network based on a subset of the EDC field names, the subset of the EDC field names being pre-mapped to a subset of the SDTM field names.

3. The method according to claim 2, wherein: A subset of the EDC field names is manually curated.

4. The method according to claim 2 further comprises generating a vector pair of dimension n1 to be input into the first embedding sub-network and the second embedding sub-network respectively.

5. The method of claim 4, further comprising processing the vector of dimension n1 in a long short-term memory neural network through the first embedding sub-network and the second embedding sub-network to generate the embedding vector.

6. The method of claim 4, further comprising determining, in an outer layer of the Siamese neural network, a distance between each pair of embedding vectors generated respectively by the first embedding subnetwork and the second embedding subnetwork.

7. The method according to claim 2, further comprising: training a classification model using a subset of metadata pre-mapped to a subset of the SDTM field names, wherein the subset of metadata is processed by the embedding model before being input to the classification model; as well as The trained classification model is used as the prediction model.

8. The method of claim 1, further comprising converting the EDC data set to the SDTM data set based at least in part on the mapping.

9. A system for converting an electronic data collection data set (EDC data set) into a standard data tabulation model data set (SDTM data set), the system comprising: A computer having one or more processors in communication with a memory storing instructions executable by the one or more processors to: Processing metadata of the EDC dataset to generate a vector of dimension n1, where n1 is an integer, the metadata including EDC field names; Processing a vector of dimension n1 in the embedding model to generate an embedding vector of dimension n2, where n2 is an integer less than n1; processing the embedding vector in a prediction model to generate a category prediction for the EDC field name, the category corresponding to the SDTM field name; and mapping the EDC field names to the SDTM field names, respectively, based at least in part on the category predictions of the EDC field names, The embedding model is obtained from a trained twin neural network, which includes a first embedding subnetwork and a second embedding subnetwork.

10. The system of claim 9, further comprising training the Siamese neural network based on a subset of the EDC field names, the subset of the EDC field names being pre-mapped to a subset of the SDTM field names.

11. The system according to claim 10, wherein: A subset of the EDC field names is manually curated.

12. The system of claim 10, further comprising generating a vector pair of dimension n1 to be input to the first embedding sub-network and the second embedding sub-network, respectively.

13. The system of claim 12, further comprising processing the vector of dimension n1 in a long short-term memory neural network through the first embedding sub-network and the second embedding sub-network to generate the embedding vector.

14. The system of claim 12, further comprising determining, in an outer layer of the Siamese neural network, a distance between each pair of embedding vectors respectively generated by the first embedding subnetwork and the second embedding subnetwork.

15. The system according to claim 10, further comprising: training a classification model using a subset of metadata pre-mapped to the subset of SDTM field names, wherein the subset of metadata is processed by the embedding model before being input to the classification model; and The trained classification model is used as the prediction model.

16. The system of claim 9, further comprising converting the EDC dataset to the SDTM dataset based at least in part on the mapping.

17. A non-transitory computer readable medium storing instructions that, when executed by one or more processors of a computer, cause the one or more processors to perform a method for converting an electronic data capture dataset (EDC dataset) into a standard data tabulation model dataset (SDTM dataset), the method comprising: Processing metadata of the EDC dataset to generate a vector of dimension n1, where n1 is an integer, the metadata including EDC field names; Processing a vector of dimension n1 in the embedding model to generate an embedding vector of dimension n2, where n2 is an integer less than n1; processing the embedding vector in a prediction model to generate a category prediction for the EDC field name, the category corresponding to the SDTM field name; and mapping the EDC field names to the SDTM field names, respectively, based at least in part on the category predictions of the EDC field names, The embedding model is obtained from a trained twin neural network, which includes a first embedding subnetwork and a second embedding subnetwork.

18. The computer-readable medium of claim 17, further comprising training the Siamese neural network based on a subset of the EDC field names, the subset of the EDC field names being pre-mapped to a subset of the SDTM field names.

19. The computer-readable medium of claim 18, wherein: A subset of the EDC field names is manually curated.

20. The computer-readable medium of claim 18, further comprising generating a vector pair of dimension n1 to be input to the first embedding sub-network and the second embedding sub-network, respectively.

21. The computer-readable medium of claim 20, further comprising processing the vector of dimension n1 in a long short-term memory neural network through the first embedding subnetwork and the second embedding subnetwork to generate the embedding vector.

22. The computer-readable medium of claim 20, further comprising determining, in an outer layer of the Siamese neural network, a distance between each pair of embedding vectors generated by the first embedding subnetwork and the second embedding subnetwork, respectively.

23. The computer readable medium of claim 18, wherein the method further comprises: training a classification model using a subset of metadata pre-mapped to a subset of the SDTM field names, wherein the subset of metadata is processed by the embedding model before being input to the classification model; as well as The trained classification model is used as the prediction model.

24. The computer-readable medium of claim 17, further comprising converting the EDC data set to the SDTM data set based at least in part on the mapping.

Images