System and method for converting electronic data capture datasets into Standard Data Tabulation Model (SDTM) datasets

Abstract

To provide a method of converting an Electronic Data Capture (EDC) dataset to a Standard Data Tabulation Model (SDTM) dataset.SOLUTION: A method includes a step of processing metadata for an EDC dataset, the metadata comprising EDC field names, to produce vectors of a particular dimensionality. The vectors are processed in an embedding model to produce embedded vectors of a lesser dimensionality. The embedded vectors are processed, in a prediction model, to produce class predictions for the EDC field names, the classes corresponding to SDTM field names. The EDC field names are associated with the SDTM field names based on the class predictions for the EDC field names. The embedding model is obtained from a trained Siamese neural network.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This disclosure generally relates to mapping Electronic Data Capture (EDC) fields of clinical trial data to the Study Data Tabulation Model (SDTM) and converting EDC datasets to SDTM datasets. [Background technology]

[0002] Clinical trials are becoming increasingly complex, with highly sophisticated protocols, multiple and diverse data sources, large volumes of data, decentralization, and increasingly complex adaptive designs. By one estimate, the number of data points collected in Phase III trials has tripled over the past decade, reaching an average of 3.6 million, and the increasing complexity of trials is driving the expansion of data volumes generated from an ever-wider range of sources.

[0003] Electronic Data Capture (EDC) systems are widely used in clinical trials for data collection and management. EDC systems include software applications that allow direct data entry at the point of collection, replacing traditional paper-based data collection methods. Before the start of a clinical trial, the clinical trial protocol is used to design an electronic case report form (eCRF) within the EDC system. The eCRF contains a structured format for entering trial data. This design includes all fields necessary to collect data specified in the clinical trial protocol, such as patient demographics, medical history, treatment assignment, laboratory results, adverse events, and outcomes.

[0004] A key advancement in clinical data management is standardization through mapping clinical data to the Standard Data Tabulation Model (SDTM), which defines a standard structure for human clinical trial data tabulations and non-clinical trial data tabulations submitted as part of product applications to regulatory authorities such as the U.S. Food and Drug Administration (FDA). This has streamlined the handling and comparison of clinical data across different clinical trials. Summary of the Invention [Problem to be solved by the invention]

[0005] Mapping EDC fields to SDTM can be a challenging technical problem for several reasons. EDC systems are primarily designed for data collection, with fields organized to meet the needs of clinical data entry and site management. EDC fields, such as those related to the collection of adverse events, demographics, and other clinical endpoints, may be specified in a study- or company-specific manner and therefore cannot be easily mapped to the SDTM standard. SDTM, on the other hand, is designed for data submission to regulatory agencies such as the FDA and follows a standard structure and terminology established by the Clinical Data Interchange Standards Consortium (CDISC). Differences in structure and terminology make direct mapping between EDC and SDTM difficult. Furthermore, there is significant variation in the design and use of different EDC systems. The same type of data may be captured differently across systems, and the same therapeutic drug may be captured across different studies. This variation complicates the mapping process.

[0006] Therefore, the traditional process of mapping data from electronic data capture (EDC) systems to SDTM generally requires programmers to write scripts that read annotated case report forms (CRFs) and convert the data into SDTM format. Furthermore, EDC systems used in clinical trials are often designed to facilitate study execution, making mapping to SDTM a secondary priority. Therefore, with this traditional approach, the process of mapping data from EDC systems to SDTM is highly laborious and often irreproducible.

[0007] To address these issues, attempts have been made to standardize EDC. Once a study mapping exists, all future studies will essentially follow the same pattern. However, EDC standardization has drawbacks, including a loss of flexibility, difficulty in establishing a common standard, and, most importantly, an inapplicability to mapping legacy studies that have never been submitted in SDTM format.

[0008] One of the barriers to building practical machine learning (ML) solutions for mapping EDC fields to SDTM is the relative lack of training data: even the largest pharmaceutical companies only have a few thousand clinical trials available to train their algorithms. [Means for solving the problem]

[0009] In disclosed embodiments, a Siamese neural network can be used to generate embeddings for electronic data capture (EDC) metadata, and a machine learning (ML) classifier can be used on these embeddings to predict associated Standard Data Tabulation Model (SDTM) fields. The Siamese neural network is trained so that similar data points have lower-dimensional embeddings that are closer based on a distance metric, such as Euclidean distance, than dissimilar data points. This allows for learning of a subnetwork that can compare the low-dimensional embedding of each field to all other fields. In disclosed embodiments, this allows for expanding N data points into an O(N2) set from which a classifier neural network can be trained. After obtaining this embedding, the low-dimensional representation can be used to generate a final predictive model to classify EDC form fields into SDTM domains.

[0010] The disclosed embodiments enable large-scale mapping of legacy clinical trials to the SDTM format, significantly increasing the value of legacy data by enabling cross-clinical analysis without the overhead of manually aligning datasets prior to analysis.

[0011] In the disclosed embodiment, a Siamese neural network is used to calculate the similarity between form fields typically specified in EDC systems. These similarity metrics can be fed into further machine learning stages for final classification. This output can be used to scale-map EDC data to SDTM and / or drive tools that provide mapping recommendations.

[0012] In one aspect, the disclosed embodiments provide a method, system, and computer-readable medium for converting an electronic data capture (EDC) dataset into a standard data compilation model (SDTM) dataset. The method includes processing metadata of the EDC dataset, the metadata including EDC field names, to generate a vector of n1 dimension, where n1 is an integer. The method further includes processing the n1 vector in an embedding model to generate an embedding vector of n2 dimension, where n2 is an integer less than n1. The method further includes processing the embedding vector in a predictive model to generate class predictions for the EDC field names, where the classes correspond to the SDTM field names. The method further includes mapping the EDC field names to the SDTM field names, respectively, based at least in part on the class predictions for the EDC field names. The embedding model is obtained from a trained Siamese neural network including a first embedding subnetwork and a second embedding subnetwork.

[0013] Embodiments can include one or more of the following features, either alone or in combination.

[0014] The method may further include training a Siamese neural network based on a subset of EDC field names pre-mapped to a subset of SDTM field names. The subset of EDC field names may be manually curated. The method may further include generating a pair of n1-dimensional vectors to be input to a first embedding subnetwork and a second embedding subnetwork, respectively. The method may further include processing the n1-dimensional vectors with a long short-term memory neural network by the first embedding subnetwork and the second embedding subnetwork to generate embedding vectors.

[0015] The method may further include determining, in an outer layer of the Siamese neural network, a distance between each pair of embedding vectors generated by the first and second embedding subnetworks, respectively. The method may further include applying at least one of batch normalization and an activation function to the determined distances between the embedding vectors output by the first and second embedding subnetworks, and backpropagating the results to the first and second embedding subnetworks. Various architectures, including different neural network layers, activation functions, and distance metrics, can be used for the Siamese network and its subnetworks. The method may further include training a classification model using a subset of EDC field names pre-mapped to a subset of SDTM field names (the subset of EDC field names is processed by the embedding model before being input to the classification model), and using the trained classification model as a predictive model. The method further includes converting the EDC dataset to an SDTM dataset based at least in part on the mapping. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a block diagram of a system for converting an electronic data capture (EDC) dataset into a standard data aggregation model (SDTM) dataset, according to disclosed embodiments. [Figure 2] FIG. 1 is a block diagram of an EDC to SDTM field mapping system according to disclosed embodiments. [Figure 3] FIG. 1 is a block diagram of a subsystem that generates an embedded model using a Siamese neural network embedded model. [Figure 4] FIG. 1 is a block diagram of a subsystem that trains an ML classification model using an embedding model generated by a Siamese neural network. [Figure 5] FIG. 1 is a block diagram of a Siamese neural network embedding model and a pair generation unit. [Figure 6] This table compares the overall accuracy and macro F1 score of the disclosed method using three methods for evaluating predicted labels with two prior approaches. [Figure 7] A plot of accuracy against the number of training samples per class. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the components described in the following embodiments are merely examples and are not intended to limit the technical scope of the present invention.

[0018] [First embodiment] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of embodiments of the present invention. However, those skilled in the art will understand that embodiments of the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the present invention.

[0019] In a disclosed embodiment, a Siamese neural network is used to calculate the similarity between electronic data capture (EDC) fields that map to Standard Data Tabulation Model (SDTM) entries (referred to herein as "SDTM fields"). The resulting similarity metric produces a low-dimensional embedding in which EDC fields that map to the same SDTM field are closer in distance (e.g., Euclidean distance) than fields that are further apart. This low-dimensional embedding is fed into a secondary classification algorithm to convert the embedding into an actual prediction.

[0020] FIG. 1 is a block diagram of a system 100 that converts an electronic data capture (EDC) dataset 110 to a standard data compilation model (SDTM) dataset 120. The EDC dataset 110 is generated by output from an EDC system 130, which can receive EDC data from many sources, such as clinical trial sites. To enable the conversion of the data, an EDC-SDTM field mapping system 140 retrieves metadata from the EDC dataset 110, including EDC field names and other metadata such as field descriptions, the units in which the fields are recorded, and which form and field IDs are associated with the fields, and maps the EDC fields to standardized fields in SDTM.

[0021] The results of the field mapping are output to EDC-SDTM data conversion system 150. Conversion system 150 uses the field mapping to convert EDC dataset 110 into the standard format of SDTM dataset 120. Once the data is converted, SDTM dataset 120 is output to SDTM data distribution system 160, which controls the transmission of SDTM dataset 120. For example, SDTM dataset 120 is received by SDTM data distribution system 160, which then transmits the data to various SDTM-based systems, e.g., via cloud-based network 170. In an embodiment, SDTM dataset 120 may be transmitted to a regulatory authority, such as the U.S. Food and Drug Administration (FDA).

[0022] Converting EDC datasets from clinical trial data to datasets with the standardized SDTM format is a technical challenge because complex methodologies and tools must be developed, implemented, and maintained to ensure that fields in the EDC datasets are accurately mapped to the standardized SDTM fields. While the underlying motivation for performing EDC field mapping and converting EDC datasets to SDTM datasets may stem from legal or regulatory requirements, the actual process of achieving the mapping and conversion involves addressing numerous technical challenges, as discussed in more detail below.

[0023] FIG. 2 is a block diagram of a neural network-based system 200 for performing EDC-to-SDTM field mapping. In the disclosed embodiment, system 200 corresponds to EDC-to-SDTM field mapping system 140 described above (see FIG. 1). System 200 trains and uses a neural network and machine learning model architecture to process an unclassified EDC field dataset 205 obtained from EDC dataset 110 (see FIG. 1). EDC field dataset 205 is processed by: (i) an embedding model 210 and (ii) a secondary prediction model 220. The result of processing EDC field dataset 205 through these two models is a set of final class (i.e., classification) predictions for EDC field mapping. In this manner, EDC fields are classified with specific, determined probabilities in relation to SDTM fields. Final class predictions 225 can be output to a data conversion and distribution system 230, such as EDC-to-SDTM data conversion system 150 and SDTM data distribution system 160 described above (see FIG. 1). In the disclosed embodiment, confidence scores may be determined and the closest matches may be presented as suggestions via a user interface.

[0024] In the disclosed embodiment, the embedded model 210 and the secondary predictive model 220, which may be referred to as "principal runtime models," are generated by training corresponding "proxy models" and using the resulting trained models as their respective "principal runtime models." Specifically, the embedded model 210 is generated by training a Siamese neural network embedded model 245, as described below, and exporting the resulting trained model for use as the embedded model 210. Similarly, the secondary predictive model 220 is generated by training a machine learning (ML) classification model 250 and using the resulting trained model as the secondary predictive model 220. In the disclosed embodiment, the training of the proxy models may occur during runtime, in which case the trained proxy models may be used to periodically update the corresponding primary runtime models. Alternatively, some or all of the training may occur prior to runtime, with periodic updates occurring at runtime.

[0025] Training of the proxy models, i.e., Siamese neural network embedding model 245 and ML classification model 250, can be performed using a database of pre-mapped, e.g., manually curated, fields 240, including EDC fields manually mapped to corresponding SDTM fields. The embedding model 210 receives the curated EDC fields and outputs a set of n-dimensional training embeddings 255 (e.g., 32-dimensional embeddings). The curated EDC fields and the SDTM fields to which the curated EDC fields were manually mapped, i.e., the final class labels, are output as training data to the Siamese neural network embedding model 245 via a second pass 252.

[0026] A pair generator 260 is provided along the second path 252 to generate all possible pair combinations between the curated EDC fields and the final class labels that are input to the Siamese neural network embedding model 245. The final class labels are also output to the ML classification model 250 via a third path 253 to be used as ground truth.

[0027] As described above, the system 200 trains and uses a neural network and machine learning model architecture to process the uncategorized EDC field dataset 205 retrieved from the EDC dataset 110 (see FIG. 1 ). In a disclosed embodiment, the retrieved data may include metadata associated with electronic case report forms (eCRFs) used in clinical trials, such as the form name (FormOID), pre-text, e.g., a description of the field in question, post-text, e.g., additional descriptions such as the expected format the input is expected to take, e.g., expected date format, and FieldOID, an internal identifier for the specific field in question. In a disclosed embodiment, these four information fields can be associated with SDTM domains. However, while the disclosed embodiment focuses on these four fields, the methodology is not specific to these fields and can be implemented using other fields, such as the distribution of recorded data.

[0028] For our evaluation, the dataset was limited to the following domains: Adverse Events (AE), Treatments (DS), Concomitant Medications (CM), Exposures (EC), Medical History (MH), Vital Signs (VS), and Demographics (DM). Fields related to these domains were manually mapped to SDTM by human curators for evaluation. While this is not a comprehensive list of domains that are part of the SDTM dataset, these domains reasonably represent the constructs that require mapping.

[0029] Typically, domains in the SDTM format are represented in a mixture of unpivoted (i.e., narrow) and pivoted (i.e., wide) formats. To simplify the evaluation, all domains were represented as unpivoted tables. Note that, based on the SDTM standard, there is an easy way to convert from pivoted to unpivoted representations. Furthermore, the evaluation did not include composite fields (fields that are the result of a mathematical operation between one or more fields, such as BMI). The training data used for the evaluation had approximately 9,000 entries corresponding to 57 SDTM fields.

[0030] 3 is a block diagram of a subsystem (part of system 200 described above) that generates an embedding model 210 using a Siamese neural network embedding model 245. The use of a Siamese neural network provides a data embedding where data points in the same class are close to data points in another class in terms of a given distance metric.

[0031] Training of the Siamese neural network 245 uses a database of pre-mapped fields 240, which are EDC fields mapped to corresponding SDTM fields. The database 240 provides the pair generator 260 with EDC metadata, including manually curated EDC fields and final class (i.e., classification) labels, which are corresponding SDTM fields.

[0032] In a disclosed embodiment, the pair generator 260 can generate all possible pair combinations between the curated EDC fields and output the resulting pairs to two independent inputs of the Siamese neural network 245. In a disclosed embodiment, the pairs may be randomly generated. As an example, for each SDTM field, 100 pairs are generated in which the two EDC fields map to the same SDTM entry (e.g., EDC fields labeled AESTD and AECODED both map to AEDECOD), and 100 pairs are generated in which the two fields are associated with different SDTM elements (e.g., EDC fields labeled AGE and WEIGHT). The output variable of the Siamese neural network 245 is 0 if the fields belong to the same SDTM field and 1 if they do not.

[0033] This mapping operation effectively expands the amount of training data by an order of magnitude. For example, if we pair 1,000 data points with another 1,000 data points in every possible combination, the resulting training set will have 1 million data points. This is advantageous, as typical language models can require billions of data points. Furthermore, as we demonstrate later in the Results Discussion, using this effectively expanded dataset significantly improves accuracy compared to traditional approaches.

[0034] The mapping operation sets a minimum distance between two entries of the same class, or an arbitrary distance if they are not of the same class. A holdout set stratified by SDTM field is generated to ensure that the training and test sets contain the same proportion of SDTM fields. For example, 10% of the entries for each SDTM field can be excluded from the model training process for evaluation. This ensures that each class in the dataset is represented in both the training and test sets.

[0035] The Siamese neural network 245 is trained to map text into a low-dimensional space such that the geometric (e.g., Euclidean) distance between points in that space corresponds to the semantic similarity between related text (e.g., EDC metadata). Thus, similar inputs will have lower-dimensional embeddings that are closer together than dissimilar data points, based on a distance metric such as Euclidean distance. This enables the training of a sub-network that can compare the lower-dimensional embedding of each field with all other fields. The lower-dimensional embeddings produced by training the Siamese neural network 245 are exported for use as the embedding model 210, which in turn is used to train the ML classification model 250 (see Figure 4).

[0036] FIG. 4 is a block diagram of a subsystem (e.g., part of system 200 described above) that uses an embedding model 210 generated by a Siamese neural network 245 to train an ML classification model 250. This training uses a database 240 of manually curated fields, which are EDC fields that have been manually mapped to corresponding SDTM fields. Database 240 provides EDC metadata, including the manually curated EDC fields, to embedding model 210. Database 240 also provides final class (i.e., classification) labels, which are corresponding SDTM fields, to ML classification model 250 to use as ground truth.

[0037] In the disclosed embodiment, the ML classification model 250 may be a k-nearest neighbor (kNN) classifier. Alternatively, techniques such as XGBoost (an optimized distributed gradient boosting library), random forests, or additional neural networks may be used to perform the classification. The use of a kNN classifier facilitates evaluation of the embedding quality by providing information about the number of nearest neighbors that belong to the same class. Specifically, it is possible to evaluate whether any of the k nearest neighbors have the correct class, whether the correct class is predicted by the nearest neighbors, or whether the correct class is predicted by a majority vote of the k nearest neighbors.

[0038] The embedding model 210 is generated by a Siamese neural network 245, as described above, and is adapted to convert text into embeddings of a particular dimension, e.g., a dense vector. In the disclosed embodiment, the curated EDC fields from the database 240 are converted into a set of training embeddings in the form of n-dimensional (e.g., 32-dimensional) embeddings 255. The training embeddings 255 are, in turn, used to train the ML classification model 250.

[0039] 5 is a block diagram of the Siamese neural network embedding model 245 and pair generator 260. The Siamese neural network architecture is designed to learn the similarity or dissimilarity between two different inputs. In the context of the disclosed embodiments, EDC fields that map to the same SDTM domain should be more similar than EDC fields that map to different SDTM fields.

[0040] The Siamese neural network 245 trains a first embedding sub-network 510, which is a neural network that generates an embedding for a first input from the pair generator 260. The second embedding sub-network 520 is a "clone" of the first embedding network 510, i.e., the sub-network shares the same architecture and weights, and generates an embedding for a second input from the pair generator 260. The embeddings for the first and second inputs are then compared in an outer layer 530, which has a distance module 535 that calculates the distance (e.g., Euclidean distance) between the first and second embeddings.

[0041] Each embedding subnetwork (510, 520) includes a text embedding layer 540 for converting words into vectors. The text embedding layer 540 converts the linear vectors of dictionary-encoded text into a matrix with one row for each element of the original vector. Words found in the same context are encoded into N-dimensional vectors such that similar words map to similar vectors.

[0042] As described above, the data retrieved from the manually curated fields database 240 may include metadata associated with the electronic case report form (eCRF), such as the form name (FormOID), pre-text, e.g., a description of the field in question, post-text, e.g., additional description such as the format the input is expected to take, e.g., expected date format, and a FieldOID, which is an internal identifier for the particular field in question.

[0043] In the disclosed embodiment, to preprocess the input into a format usable by the neural network, the text is first converted to raw text by stripping out any HTML tags that may be present in the text. This can be done, for example, using Python's html2text library. After conversion, the four fields can be concatenated. This string has a maximum length of N words. This long string can be dictionary encoded, for example, via Gensim. In another embodiment, a more sophisticated neural network may be used, where each type of data is encoded separately before being combined by the aggregation layer.

[0044] Encoding the dictionary can be a relatively simple process: the first word observed in the text is assigned a value of 1, the second word observed in the series is assigned a value of 2, etc. The result of this encoding is a numeric vector that can be processed by a neural network. These embeddings are typically high-dimensional (e.g., a 1000-dimensional vector corresponding to the length of the input), so they are further condensed into lower-dimensional embeddings by an embedding sub-network, as described below.

[0045] In the disclosed embodiment, the text embedding layer 540 is trained alongside the embedding subnetwork 510 rather than using a pre-trained word embedding model. This is because the amount of vocabulary specific to EDC to SDTM conversion, such as FormOID and FieldOID, along with the terminology and abbreviations present in the text description, does not fit embeddings trained on a more general vocabulary.

[0046] The encoded text is output by the text embedding layer 540 to a long-short-memory (LSTM) neural network 545, a type of recurrent neural network that is particularly good at processing sequences of data. The LSTM 545 can take a sequence of word vectors as input and output a single vector that represents the entire sequence. As explained above, pairs of fields are used to train the Siamese neural network embedding model 245. If a pair of fields is similar, the LSTM 545 and the corresponding LSTM in the clone subnetwork 520 are trained to make their outputs as similar as possible. On the other hand, if the pair of fields is dissimilar, the LSTM is trained to make their outputs as different as possible. This can be achieved by using certain types of loss functions, such as contrastive loss, as described in more detail below.

[0047] The output of the LSTM 545 is processed by a flattening module 547 to constrain the output to a desired distribution. For example, a tanh activation function can be applied to the output of the LSTM 545 to provide an embedding that is centered at 0 and has an approximately equal distribution of positive and negative values.

[0048] The resulting output of the LSTM 545 and flattening module 547 is a set of n-dimensional field embeddings 550, where n is an integer, e.g., 8, 16, 32, 64, etc. As described above, the field embeddings 550 of the first and second embedding subnetworks (510, 520) are compared in the outer layer 530 by a distance module 535, which calculates the distance, e.g., Euclidean distance, between the embeddings of the first and second embedding subnetworks (510, 520). A batch normalization module 555 may process the output of the distance module 535 to achieve a normalized set of values, specifically, standardizing the scores across each batch of input pairs to ensure that the batch has a mean of 0 and a standard deviation of 1. This can make the network more robust to different scales of raw similarity scores, improving the speed, performance, and stability of the twin neural network 245.

[0049] The final activation module 560 of the outer layer 530 applies a final activation function to the outputs of the embedding subnetworks (510, 520). The final activation layer is used to convert the embeddings generated by the subnetworks (510, 520) into a final value that reflects the differences between the different fields. For example, if the goal is to determine the similarity between two inputs provided by the pair generator 260, the desired output may be a single number representing the degree of similarity. In this case, a sigmoid activation function can be used in the final layer to force the output into the [0, 1] range.

[0050] Thus, the Siamese neural network 245 essentially takes two copies of the sub-networks (510, 520) and combines them in an outer layer 530, which determines the distance between the embeddings that result when data pairs are input together. Specifically, each sub-network (510, 520) processes its input and generates an output vector. The outer layer 530 (or "final activation layer") receives these two output vectors and calculates a final output. This final output can be, for example, a similarity score (e.g., based on Euclidean distance) between the two input sequences. A loss function (e.g., contrastive loss or triplet loss) is used to calculate a scalar loss value. The loss function is adapted to encourage similar input pairs to have similar output vectors and dissimilar input pairs to have dissimilar output vectors.

[0051] In the backpropagation process, the gradient of the loss function is calculated with respect to each parameter of the model (i.e., weights and biases). This is effectively done by calculating from the loss function, through the final activation layer, and back to the sub-networks (510, 520). The gradients calculated during backpropagation are used to adjust the model parameters in a way that reduces the loss. This is done using an optimization algorithm such as stochastic gradient descent (SGD).

[0052] In a Siamese neural network architecture, the subnetworks (510, 520) share weights, meaning they have the same parameters. When gradients are calculated during backpropagation, they are accumulated across both subnetworks (510, 520), and the weights are updated based on the combined gradients. This process is repeated over many epochs (i.e., iterations over the training data), gradually adjusting the weights and biases to minimize loss and learn useful representations of the inputs. Over time, the subnetworks (510, 520) learn to output similar vectors for similar inputs and different vectors for different inputs.

[0053] One advantage of training a subnetwork to minimize the distance between similar data points, e.g., Euclidean distance, is that it tends to produce compact clusters that are amenable to other classifiers. Thus, once a Siamese neural network 245 is trained, the embedding subnetwork can be used as a preprocessor for training and evaluating machine learning techniques, or as a feed to a secondary neural network.

[0054] Figure 6 compares the overall accuracy and Macro F1 scores of the disclosed method with two prior approaches using three methods for evaluating predicted labels. Specifically, evaluation was performed by excluding 10% of the entries for each SDTM field from the model training process. This ensured that each class in the dataset was represented in both the training and test sets. Basic accuracy, i.e., the number of predicted values ​​that matched the SDTM fields, was evaluated, as well as the Macro F1 score. The results showed that the disclosed approach was able to accurately predict most of the appropriate SDTM mappings. In most cases, nearest neighbors are good predictors of a given class. The evaluation showed that the nearest field in the embedding space could predict the appropriate SDTM field with approximately 90% accuracy.

[0055] For comparison, we present two prior machine learning (ML) approaches we implemented for the task of mapping EDC fields to SDTM fields. The first approach uses a traditional ML tool, XGBoost (an optimized distributed gradient boosting library), on the same field metadata, encoded via term frequency-inverse document frequency (TF-IDF). The second method attempts direct prediction of field mapping using a deep learning implementation using a pre-trained text embedding layer (RoBERTa) along with two bidirectional LSTMs implemented in Keras. As is evident from the results presented in the table, the approach disclosed herein (using a Siamese neural network and an ML classifier) ​​is significantly more accurate than previous attempts using traditional ML techniques or direct neural network prediction.

[0056] Analysis of the clustering generated by the embeddings showed that Siamese neural networks can generate embeddings that are sufficiently compact so that machine learning (ML) techniques (e.g., ML classification models) can accurately separate classes. EDC fields that map to the same SDTM field are closely spaced in the embedding space, enabling high-accuracy classification. Note that the results presented here are for the holdout set only. The algorithm achieved self-prediction accuracy of over 99%. However, due to issues such as vocabulary discontinuity between the test and training sets, accuracy on the holdout set was poor.

[0057] Figure 7 plots accuracy against the number of training samples per class. While accuracy is high on average, there is a relatively large variation in accuracy across classes. While the majority of classes exhibit accuracies above 90%, the lowest accuracy was only 25%. However, there is a clear trend of accuracy increasing as the amount of training data increases. We observed that for classes with fewer than 100 samples in the training data, there was insufficient information to capture the variability in how fields were represented. For classes with more than 100 training samples, over 90% of the fields were correctly predicted.

[0058] Compared to the previous implementations evaluated, we observed significant improvements in evaluation accuracy, regardless of the degree of tuning of the neural network architecture. On average, we observed a 10-15% improvement in overall classification accuracy, as defined by correctly predicted labels, compared to previous techniques. Furthermore, prediction results were more consistent across different classes, as evidenced by significantly higher macro F1 scores.

[0059] Evaluation of the disclosed embodiments showed that for the set of domains in which data was trained and tested, the metadata associated with EDC fields contained sufficient information to accurately determine the SDTM fields to which they match. Using a Siamese neural network, we were able to generate embeddings in which EDC fields mapped to the same SDTM field were closely embedded in the same space. Much of the inaccuracy in the evaluation can be explained by the relative lack of training data for certain domains and the relative lack of overlapping vocabulary, which tended to prevent the embedding layer from generating useful embeddings for some of the training data. These are all data-related issues that could potentially be resolved by expanding the dataset.

[0060] During training, the Siamese neural network reached an accuracy of over 99% in predicting whether two entries map to the same SDTM domain. In cases with sufficient training data, the overall accuracy for the holdout set was over 95%. Even with issues of vocabulary inconsistency and under-representation in certain domains, the model was relatively accurate in the holdout set. Another area of ​​inaccuracy in the evaluation involved fields that can overlap across multiple fields, such as dates. For example, many forms have EDC fields that reference dates, and although much of their metadata is identical, the SDTM mapping is determined only by the FormOID. In such cases, the approach described here can be used, but rather than examining each EDC field individually, the metadata is aggregated across all fields and the SDTM domain is targeted instead of the SDTM field.

[0061] Aspects of the present invention may be embodied in the form of a system, a computer program product, or a method. Likewise, aspects of the present invention may be embodied in hardware, software, or a combination of both. Aspects of the present invention may also be embodied as a computer program product stored on one or more computer-readable medium(s) with computer-readable program code embodied thereon.

[0062] The computer-readable medium may be a computer-readable storage medium, which may be, for example, an electronic, optical, magnetic, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof.

[0063] The computer program code in embodiments of the present invention can be written in any suitable programming and / or scripting language. The program code can be executed on a single computer or on multiple computers. The computer can include a processing unit in communication with a computer-usable medium, the computer-usable medium including an instruction set, and the processing unit designed to execute the instruction set and / or a trained machine learning algorithm.

[0064] The foregoing discussion is meant to be illustrative of the principles and various embodiments of the present invention. Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.

Claims

1. 1. A computer-implemented conversion method for converting an Electronic Data Capture (EDC) dataset to a Standard Data Collection Model (SDTM) dataset, comprising: processing metadata including EDC field names of the EDC dataset to generate a vector of dimension n1, where n1 is an integer; processing a vector of dimension n1 in an embedding model to generate an embedding vector of dimension n2, where n2 is an integer less than n1; processing the embedding vectors through a predictive model to generate class predictions for the EDC field names corresponding to SDTM field names; a mapping step of associating each of the EDC field names with the SDTM field names based at least in part on a class prediction of the EDC field names; Including, 10. A computer-implemented transformation method, wherein the embedding model is obtained from a trained Siamese neural network, the Siamese neural network including a first embedding sub-network and a second embedding sub-network.

2. 2. The method of claim 1, further comprising training the Siamese neural network based on a subset of the EDC field names pre-mapped to the subset of SDTM field names.

3. The method of claim 2 , wherein the subset of EDC field names is manually curated.

4. The method of claim 2 , further comprising generating a pair of vectors of dimension n1 to be input to the first and second embedded sub-networks, respectively.

5. 5. The method of claim 4, further comprising processing a vector of dimension n1 with a long short-term memory neural network to generate the embedding vector by the first embedding sub-network and the second embedding sub-network.

6. 5. The method of claim 4, further comprising determining, in an outer layer of the Siamese neural network, a distance between each pair of embedding vectors generated by the first embedding sub-network and the second embedding sub-network, respectively.

7. training a classification model using a subset of metadata pre-mapped to the subset of SDTM field names; using the trained classification model as a predictive model; further comprising The method of claim 2 , wherein the subset of metadata is processed by an embedding model before being input to a classification model.

8. The method of claim 1 , further comprising converting the EDC data set to an SDTM data set based at least in part on the mapping step.

9. 1. A conversion system for converting an Electronic Data Capture (EDC) dataset into a Standard Data Collection Model (SDTM) dataset, the system comprising: a computer having one or more processors in communication with a memory storing instructions and capable of executing said instructions; The instruction: processing metadata including EDC field names of the EDC dataset to generate a vector of dimension n1, where n1 is an integer; processing a vector of dimension n1 in an embedding model to generate an embedding vector of dimension n2, where n2 is an integer less than n1; processing the embedding vectors through a predictive model to generate class predictions for the EDC field names corresponding to SDTM field names; a mapping step of associating each of the EDC field names with the SDTM field names based at least in part on a class prediction of the EDC field names; is an instruction to execute The embedding model is a transformation system obtained from a trained Siamese neural network, including a first embedding sub-network and a second embedding sub-network.

10. 10. The conversion system of claim 9, further comprising training the Siamese neural network based on a subset of the EDC field names pre-mapped to the subset of SDTM field names.

11. The conversion system of claim 10 , wherein the subset of EDC field names is manually curated.

12. 11. The conversion system of claim 10, wherein the instructions further comprise generating a pair of vectors of dimension n1 to be input to the first and second embedded sub-networks, respectively.

13. The instruction:

13. The conversion system of claim 12, further comprising the step of processing a vector of dimension n1 with a long short-term memory neural network by the first embedding sub-network and the second embedding sub-network to generate the embedding vector.

14. The instruction:

13. The conversion system of claim 12, further comprising: determining, in an outer layer of the Siamese neural network, a distance between each pair of embedding vectors generated by the first and second embedding sub-networks, respectively.

15. The instruction: training a classification model using a subset of metadata pre-mapped to a subset of SDTM field names; using the trained classification model as a predictive model; Further execute The conversion system of claim 10 , wherein the subset of metadata is processed by an embedding model before being input to a classification model.

16. The instruction:

10. The conversion system of claim 9, further performing the step of converting the EDC data set to an SDTM data set based at least in part on the mapping step.

17. 1. A non-volatile computer-readable medium having stored thereon instructions that, when executed by one or more processors of a computer, cause the one or more processors to perform a method for converting an electronic data capture (EDC) dataset to a standard data compilation model (SDTM) dataset, the method comprising: The method comprises: processing metadata including EDC field names of the EDC dataset to generate a vector of dimension n1, where n1 is an integer; processing a vector of dimension n1 in an embedding model to generate an embedding vector of dimension n2, where n2 is an integer less than n1; processing the embedding vectors through a predictive model to generate class predictions for the EDC field names corresponding to SDTM field names; a mapping step of associating each of the EDC field names with the SDTM field names based at least in part on a class prediction of the EDC field names; Including, The computer-readable medium, wherein the embedding model is obtained from a trained Siamese neural network, the Siamese neural network including a first embedding sub-network and a second embedding sub-network.

18. The method comprises:

20. The computer-readable medium of claim 17, further comprising training the Siamese neural network based on a subset of the EDC field names pre-mapped to the subset of SDTM field names.

19. 20. The computer-readable medium of claim 18, wherein the subset of EDC field names is manually curated.

20. 20. The computer-readable medium of claim 18, wherein the method further comprises generating a pair of vectors of dimension n1 to be input to the first and second embedded sub-networks, respectively.

21. The method comprises:

21. The computer-readable medium of claim 20, further comprising processing, by the first embedding sub-network and the second embedding sub-network, a vector of dimension n1 with a long short-term memory neural network to generate the embedding vector.

22. The method comprises:

21. The computer-readable medium of claim 20, further comprising determining, in an outer layer of the Siamese neural network, a distance between each pair of embedding vectors generated by the first and second embedding sub-networks, respectively.

23. training a classification model using a subset of metadata pre-mapped to the subset of SDTM field names; using the trained classification model as a predictive model; further comprising 20. The computer-readable medium of claim 18, wherein the subset of metadata is processed by an embedding model before being input to a classification model.

24. 20. The computer-readable medium of claim 17, further comprising converting the EDC data set to an SDTM data set based at least in part on the mapping step.

Images