System and method for improving the accuracy of first-order prediction models based on residual models

Abstract

This invention provides methods for improving the accuracy of first-order prediction models, etc., to enhance the accuracy of prediction models. [Solution] The method includes: determining a first training prediction residual group based on first labeled training data and a first test prediction residual group based on first labeled test data using a first-order prediction model; generating a second dataset for a residual model that includes second labeled training data labeled based on the first training prediction residual group and second labeled test data labeled based on the first test prediction residual group; training the residual model using the second labeled training data; determining a training prediction group based on the second labeled training data and a test prediction group based on the second labeled test data using the residual model; and adjusting the first-order prediction model based on the training prediction group and test prediction group of the residual model.

Description

Technical Field

[0001] The present disclosure relates to a technique for improving the accuracy of a primary prediction model based on a residual model.

Background Art

[0002] Maintaining data privacy among collaborators while performing joint modeling is a major challenge. In particular, even when two parties wish to cooperate in building a prediction model, due to privacy concerns and other constraints, the data may not be integrated as a single modeling dataset.

[0003] Federated learning aims to address data governance and privacy issues by enabling collaborative learning of algorithms without exchanging the data itself. In federated learning, a central aggregator coordinates machine learning tasks by multiple collaborators and protects the data privacy of each participant. This approach is particularly important in the life sciences field where strict confidentiality of health data is required. In typical federated learning, each user locally learns a model with their own data, uploads and aggregates the model to a shared server. The shared server updates the model based on the data of all collaborators, and each participant can access the improved model learned with a broader dataset.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, traditional federated learning has several drawbacks. These methods require consistency of datasets, verification of feature definitions, and checking of feature distributions among collaborators. Consistency of shared modeling algorithms (e.g., neural network architectures) is also required. Furthermore, a third party is needed to manage a central server to maintain data privacy, creating a risk of privacy breaches due to adversarial attacks. Identifying factors that improve model performance is also difficult, and algorithms suitable for federated learning tend to be limited to neural networks because they generally rely on updating weight parameters. [Means for solving the problem]

[0005] Given the shortcomings of the conventional methods described above, it is essential to provide a technical solution to the technical challenge of ensuring the accuracy of predictive models by improving model performance through interactive adjustments from residuals and secondary actors, while maintaining data privacy and security.

[0006] The disclosed embodiments provide a method for improving the accuracy of a predictive model. The method includes defining a predictive task, creating a dataset including training and test data, generating initial predictions and calculating residuals, providing the residuals to a secondary subject, generating additional predictions based on the residuals, and adjusting the initial predictions using the additional predictions to improve accuracy.

[0007] The disclosed embodiments provide a method for improving the performance of a predictive model. The method includes defining a modeling cohort and a predictive task; creating a modeling dataset including training data and test data, with the training samples labeled; making initial predictions and calculating predictive residuals; providing the predictive residuals to a collaborator; creating an additional dataset and fitting it to the initial dataset; training the model on the predictive residuals to generate additional predictions; and using the additional predictions to fine-tune the initial model and improve model accuracy.

[0008] The disclosed embodiments provide a method for improving predictive modeling accuracy. The method includes: defining a modeling cohort and predictive tasks by a primary modeling team; creating a modeling dataset, determining training and test data, and labeling training samples by the primary modeling team; generating initial predictions and calculating predictive residuals using the dataset by the primary modeling team; providing the labels for the training and test data and the corresponding predictive residuals to a collaborating team; creating a modeling dataset, designing features, and aligning them with the training / test split of the primary modeling team by the collaborating team; training a model on the predictive residuals using the trained model, generating additional predictions, and providing the additional predictions to the primary modeling team; and tuning the initial model using the additional predictions to create a tuned model with improved accuracy.

[0009] The disclosed embodiments provide a method for improving predictive modeling accuracy. The method includes: defining a modeling cohort and a predictive task by a primary modeling team; creating a modeling dataset, determining training and test data, and labeling training samples by the primary modeling team; making initial predictions using the dataset and calculating prediction errors referred to as residuals by the primary modeling team; delivering the labels for the training and test data and the corresponding predictive residuals to a collaborating team by the primary modeling team; creating a modeling dataset from cohorts consistent with the defined modeling cohort and predictive task, designing features, and aligning the training / test split with that of the primary modeling team by the collaborating modeling team; training a model on the predictive residuals received from the primary modeling team, making predictions on the test data using the trained residual model, and returning the resulting predictions to the primary modeling team; and adjusting the initial model using the predictions received from the collaborating modeling team to obtain an adjusted model with improved accuracy compared to the initial model, and evaluating the degree of improvement obtained from the residual model predictions.

[0010] In one embodiment, the disclosed embodiment provides a method, system, and computer-readable medium for improving the accuracy of a primary predictive model based on a residual model. The method includes: using the primary predictive model to determine a first training predictive residual group based on first labeled training data and a first test predictive residual group based on first labeled test data; generating a second dataset for the residual model, the second dataset including second labeled training data labeled based on the first training predictive residual group and second labeled test data labeled based on the first test predictive residual group; training the residual model using the second labeled training data; using the residual model to determine a training predictive group based on the second labeled training data and a test predictive group based on the second labeled test data; and adjusting the primary predictive model based on the training and test predictive groups of the residual model to generate more accurate predictions.

[0011] This embodiment, either alone or in combination, includes one or more of the following features:

[0012] The method may include generating a first dataset for a primary predictive model, which may include first labeled training data and first labeled test data. The primary predictive model may also be trained using the first labeled training data. The first dataset for the primary predictive model may be generated based on a first patient-level history data set, and the second dataset for the residual model may be generated based on a second patient-level history data set. In this case, the first patient-level history data set is different from the second patient-level history data set.

[0013] Generating a second dataset for the residual model may involve receiving values ​​and labels from the first training-predicted residual group and the first test-predicted residual group, but not receiving data from the first patient-level historical data group. Generating a first dataset for the primary predictive model may involve defining the modeling cohort and predictive tasks. Predictive tasks may include predicting clinical trial participation. Predictive tasks may include predicting patient disability. Determining the first training-predicted residual group and the first test-predicted residual group may involve performing predictions based on the first labeled training data and the first labeled test data, and comparing the predictions to the labels in the first labeled training data and the first labeled test data to calculate the respective residuals.

[0014] Tuning the primary prediction model may include subtracting the training and test prediction groups of the residual model from the predictions of the primary prediction model, where the predictions of the primary prediction model are based on the first labeled training data and the first labeled test data. The tuning step of the primary prediction model may also include reading the hyperparameters of the primary prediction model so that the subsequent training prediction residuals and subsequent test prediction residuals are closer to the first training labels and the first test labels. [Brief explanation of the drawing]

[0015] [Figure 1] This is a schematic diagram of a system for improving the accuracy of a predictive model according to an embodiment of this disclosure. [Figure 2] Figure 1 is a schematic diagram of the primary model in the system shown. [Figure 3] Figure 1 is a schematic diagram of the residual model in the system shown. [Figure 4] This is a flowchart of a method for improving the accuracy of a predictive model according to an embodiment of this disclosure. [Figure 5]This is a scatter plot of the selection features used in the full model and their SHAP (Shapley Additive Explanations) values ​​in an example of a patient cohort using a publicly available dataset. [Figure 6] This is a scatter plot of the selection features used in the full model and their SHAP (Shapley Additive Explanations) values ​​in an example of a patient cohort using a publicly available dataset. [Figure 7] This is a scatter plot of the selection features used in the full model and their SHAP (Shapley Additive Explanations) values ​​in an example of a patient cohort using a publicly available dataset. [Figure 8] This is a scatter plot of the selection features used in the full model and their SHAP (Shapley Additive Explanations) values ​​in an example of a patient cohort using a publicly available dataset. [Figure 9] This is a scatter plot of the selection features used in the full model and their SHAP (Shapley Additive Explanations) values ​​in an example of a patient cohort using a publicly available dataset. [Figure 10] This is a scatter plot of the selection features used in the full model and their SHAP (Shapley Additive Explanations) values ​​in an example of a patient cohort using a publicly available dataset. [Figure 11] This is a scatter plot of the selection features used in the full model and their SHAP (Shapley Additive Explanations) values ​​in an example of a patient cohort using a publicly available dataset. [Figure 12] This is a scatter plot of the selection features used in the full model and their SHAP (Shapley Additive Explanations) values ​​in an example of a patient cohort using a publicly available dataset. [Figure 13] This is a scatter plot of the selected features and their SHAP values ​​for the first-order model in the embodiment. [Figure 14] Scatter plot of the selected features of the primary model in the embodiment and their SHAP values. [Figure 15] Scatter plot of the selected features of the primary model in the embodiment and their SHAP values. [Figure 16] Scatter plot of the selected features of the residual model in the embodiment and their SHAP values. [Figure 17] Scatter plot of the selected features of the residual model in the embodiment and their SHAP values. [Figure 18] Scatter plot of the selected features of the residual model in the embodiment and their SHAP values. [Figure 19] Scatter plot of the selected features of the residual model in the embodiment and their SHAP values. [Figure 20] Scatter plot of the selected features of the residual model in the embodiment and their SHAP values. [Figure 21] Table showing the ranking of feature importance based on the SHAP values of the full model, primary model, and residual model in the embodiment. [Figure 22] Table showing the test set errors in the primary model, adjusted model generated based on the disclosed method, and full model in the embodiment. **Embodiments for Carrying Out the Invention**

[0016] Hereinafter, embodiments of the present invention will be exemplarily described in detail with reference to the drawings. However, the components described in the following embodiments are merely examples, and are not intended to limit the technical scope of the present invention thereto. Also, detailed descriptions of well-known methods, procedures, components, circuits, etc. are omitted in order not to unnecessarily obscure the present invention.

[0017] [First Embodiment] As mentioned above, conventional collaborative modeling methods require the establishment of a central server by a trusted third party, the exchange of large amounts of data, and data anonymization. These require considerable time for setup and implementation. In contrast, the method disclosed here does not require a central server, and the data points exchanged are limited. For example, in federated learning, model weights are exchanged via a shared server, which poses a risk of attack, but the method disclosed here does not rely on such exchange. Furthermore, conventional federated learning relies on a large network of devices and neural network algorithms, increasing computational cost and complexity, but the method disclosed here reduces the computational load.

[0018] The disclosed approach offers a high degree of independence for the modeling tasks, as the modeling task is divided between the primary model and the residual model. Therefore, each collaborator is free to choose the modeling approach that best suits their dataset, thereby improving the overall performance of the model. For example, instead of jointly updating model weights as in conventional federated learning techniques, feature importance can be generated independently for both the primary and residual models. The disclosed approach also provides transparency regarding model performance improvements, allowing both collaborators to obtain information representing prediction errors and, as a result, independently quantify the improvement or degree of improvement due to the residual model.

[0019] One advantage of the disclosed approach is that both collaborating teams can independently maintain their own data without the need for data sharing. Each team may have its own unique modeling characteristics, addressing concerns regarding the sensitivity of proprietary data and contractual constraints on data granularity. Ideally, both teams would want to integrate their data to build a unified model. However, due to various constraints, this is not possible. Therefore, the advantage of the disclosed approach lies in its ability to achieve effective collaborative modeling without sharing sensitive data. It should be noted that while the underlying motivations for ensuring data privacy and security may stem from ethical, legal, and regulatory concerns, the process of actually achieving and maintaining data privacy and security is a technical endeavor, as detailed herein.

[0020] In some alternative solutions to the approach described herein, the provision of features to the primary modeler (or "client") is anonymized to mask data points originating from a single source, which degrades the predictive signal. The data exchange process is relatively complex and tends to create a barrier to client adoption. In some cases, extensive feature definitions are provided, along with a file containing a dataset join key and, for example, around 110 features. A detailed walkthrough may be provided to assist the client in integrating these features into their own modeling dataset. In contrast, the disclosed approach involves providing the join key provided by the client and the errors (e.g., residuals) of the client's model. The disclosed approach preserves the privacy of the client's dataset because it does not require any actual values ​​(i.e., data points) from the client. Furthermore, alternative methods depend on the quality of the client's predictive model and the modeling choices of the data science team. For example, even if a very effective set of features is provided, its effectiveness may be diminished by inappropriate modeling choices on the client's side. With the method disclosed herein, feature utilization is decoupled from the client's modeling choices, ensuring more robust predictive performance. Furthermore, both the residual model and the first-order model can verify how much the residual model improved the predictions of the first-order model, enabling a fair evaluation.

[0021] While the embodiments of this disclosure relate to client modeling support, they are applicable to any situation where it is desired to collaboratively build a predictive model in a situation where data sharing is not possible.

[0022] Figure 1 shows a system 100 that improves the accuracy of a predictive model. In the disclosed embodiment, system 100 implements a predictive modeling method that uses both a primary predictive model, e.g., primary model 110, and a residual predictive model, e.g., residual model 140, to improve predictive accuracy. In the embodiment, this may include a collaborative predictive modeling approach that includes both a primary modeling team and a collaborating modeling team.

[0023] The primary model 110 employs statistical methods, machine learning methods, and / or other computational techniques to generate predictions based on input data. In embodiments, the primary model 110 may use a gradient boosting machine (GBM) because it can handle complex patterns and interactions and provide high predictive accuracy. A GBM can provide a boosting technique in which the model is built sequentially to correct errors in the preceding model. Various other types of machine learning tools, such as neural networks, can also be used to implement the primary model 110.

[0024] The modeling cohort definition 105 is input to the primary model 110 and the residual model 140. The modeling cohort definition 105 is a set of data points or instances that define the scope and characteristics of the data to be modeled. The modeling cohort definition 105 serves as a dataset for subsequent modeling processes. As will be further detailed below, system 100 may implement a method for defining the modeling cohort and prediction task and creating a modeling dataset that includes training and test data and labels the training samples.

[0025] In a typical scenario, a modeling team (or “primary modeling team”) has a specific objective, such as predicting clinical trial participation at the site level. This team defines the cohort they want to use, for example, a breast cancer trial. Its purpose might be, for example, to identify the best recruiting sites for a future breast cancer trial. The primary modeling team may create a modeling dataset, including determining the training and test data and labeling the training samples accordingly. The primary modeling team then performs the model learning process on, for example, the primary model 110, using both the training and validation (or test) data. In some cases, there may be a dedicated team focused on designing and testing algorithms to predict desired outcomes. This may include feature engineering, data processing, building predictive models, and evaluating their performance.

[0026] In the illustrated example, the primary model 110 produces two outputs: the primary model prediction 120 and the primary model residual 130 (i.e., the primary model prediction residual). The primary model prediction 120 represents the initial prediction or estimate generated by the primary model 110 for each data point in the modeling cohort. The primary model residual 130 is the difference between the actual observation and the primary model prediction 120. These residuals represent the prediction error or deviation of the primary model 110.

[0027] In the illustrated example, the primary model residuals 130 are input to a residual model 140, which is a predictive model (or algorithm) specifically trained to model the residuals generated by the primary model. The residual model 140 generates residual model predictions 150 based on the input residuals. These predictions are used to substantially correct or adjust the predictions made by the primary model 110. In an embodiment, the primary modeling team initiates a data transfer to send the labels of the training and test data, and the predictive residuals corresponding to those samples, to a system (or model) implemented by the collaborating modeling team. This data transfer allows the collaborating team to further refine the prediction process. Such data may be transferred in a specific format, such as CSV or JSON, or stored in cloud storage for the collaborating team to retrieve.

[0028] In the illustrated example, the residual model prediction 150 is subtracted from the primary model prediction 120 in the adjustment process 160. This subtraction results in an adjusted primary model prediction 170 that is enhanced for each data point in the modeling cohort and represents a more accurate prediction or estimate. These adjusted predictions substantially incorporate insights from both the primary model 110 and the residual model 140. Thus, system 100 effectively leverages the complementary strengths of the primary model 110 and the residual model 140 to achieve more accurate and reliable prediction performance. By addressing the residual error of the primary model 110 through the dedicated residual model 140, system 100 considers the relationship between primary model labels and residual model dataset features to provide more accurate predictions.

[0029] In some embodiments of System 100, the primary modeling team may define the prediction task and primary dataset, and arrange for the collaborating modeling team to implement the residual model. In this case, each team maintains the privacy and security of its own data. The primary model is fitted to the primary dataset, and the prediction errors (i.e., residuals) are provided to the collaborating team. The collaborating team fits a model to these residuals using its own dataset, features, and other parameters, and sends its predictions back to the primary team as residual predictions. The primary modeling team applies the predictions of the residual model as adjustments to its own original predictions, thereby achieving an overall improvement in model accuracy while maintaining the data privacy of both parties.

[0030] Figure 2 is a diagram showing the primary model 110 of system 100 in Figure 1, illustrating the detailed function of the primary model 110 within the predictive modeling system. The primary model 110 is responsible for generating initial predictions and calculating residuals based on the modeling cohort definition 105 (see Figure 1). As previously mentioned, the modeling cohort definition 105 defines the scope and characteristics of the data to be modeled, thereby forming the basis for subsequent data processing and modeling. For simplification and ease of explanation, the primary model 110 is demarcated by dashed lines enclosing its various components. However, it should be understood that these components are not necessarily part of the primary model 110 itself, but may rather be completely independent components and may be implemented entirely separately.

[0031] The primary dataset 210 is generated based on the modeling cohort definition 105. The primary dataset 210 contains raw data points or instances that fall within the defined cohort and serves as the initial data set to be processed and analyzed. The primary dataset 210 is shown within the dashed line defining the primary model 110, but as mentioned above, it is not necessarily a component of it.

[0032] The primary dataset 210 is input to the data preparation unit 220. In embodiments, the data preparation unit 220 may be a module or component responsible for preprocessing and feature engineering. The data preparation unit 220 is shown within the dashed line defining the primary model 110, but as mentioned above, it is not necessarily a component of it. Preprocessing may include, for example, data cleaning, normalization, transformation, and missing value handling to ensure data quality and integrity. Feature engineering may include, for example, the selection and / or generation of features (or variables) to better represent underlying patterns in the data.

[0033] The data preparation unit 220 outputs a structured and refined modeling dataset 230 for effective modeling. The modeling dataset 230 may include, for example, multiple observations, i.e., data instances or samples; multiple features, i.e., independent variables or attributes describing each observation; and multiple dependent variables, i.e., the target variable or outcome that the model attempts to predict. The modeling dataset 230 is shown within the dashed line defining the primary model 110, but as mentioned above, it is not necessarily a component of it.

[0034] The modeling dataset 230 is input to the modeling unit 240. The modeling unit 240 is responsible for facilitating exploratory data analysis (EDA) and the construction of the primary model 110. EDA includes statistical and graphical analysis to understand the data distribution, identify patterns, and detect anomalies. The construction (or modeling) of the primary model 110 includes selecting and training a predictive algorithm using the modeling dataset 230. This algorithm may be a statistical model, a machine learning model, and / or other computational technique.

[0035] The modeling unit 240 outputs two main results: the first-order model prediction 120 and the first-order model residual 130 (see also Figure 1). As previously mentioned, the first-order model prediction 120 represents the initial prediction generated by the first-order model 110 for each observation in the modeling dataset 230. These represent the prediction or estimation of the dependent variable. The first-order model residual 130 (i.e., the first-order model prediction residual) is the difference between the actual observation of the dependent variable and the first-order model prediction 120. These residuals represent the prediction error or deviation of the first-order model.

[0036] Figure 3 shows the residual model 140 of system 100 in Figure 1, illustrating the detailed function of the residual model 140 within the predictive modeling system. The residual model 140 is responsible for refining and correcting the initial predictions made by the primary model 110, based on the modeling cohort definition 105 and the primary model residuals 130 (see also Figure 1). As previously mentioned, the modeling cohort definition 105 defines the scope and characteristics of the data to be modeled, thereby forming the basis for subsequent data processing and modeling. For simplification and ease of explanation, the residual model 140 is demarcated by dashed lines enclosing its various components. However, it should be understood that these components are not necessarily part of the residual model 140 itself, but may rather be completely independent components and may be implemented entirely separately.

[0037] The secondary dataset 310 is generated based on the modeling cohort definition 105. The secondary dataset 310 contains raw data points or instances that fall within the defined cohort and serves as the initial data set to be processed and analyzed. The secondary dataset 310 is shown within the dashed line defining the residual model 140, but as mentioned above, it is not necessarily a component of it.

[0038] In one embodiment, the collaborating modeling team may generate a modeling dataset from a cohort definition that is consistent with the modeling cohort definition 105 generated by the primary modeling team. The modeling cohort definition 105 is shown with the same reference number in the primary model 110 and the residual model 140, but does not have to be the result of data transfer and may be generated independently by the collaborating team based on specifications known to both teams. The collaborating modeling team may create the cohort definition and design the feature set independently. The collaborating team typically matches the training / test split of the secondary dataset 310 to the split used for the primary dataset 210. This ensures consistency and compatibility between the datasets used by both teams.

[0039] The secondary dataset 310 is input to the data preparation unit 320. In embodiments, the data preparation unit 320 may be a module or component responsible for preprocessing and feature engineering. The data preparation unit 320 is shown within the dashed line defining the residual model 140, but as mentioned above, it is not necessarily a component of it. Preprocessing may include, for example, data cleaning, normalization, transformation, and missing value handling to ensure data quality and integrity. Feature engineering may include, for example, the selection and / or generation of features (or variables) to better represent underlying patterns in the data.

[0040] The data preparation unit 320 outputs a structured and refined modeling dataset 330 for effective modeling. The modeling dataset 330 may include, for example, data instances or samples, features (or independent variables or attributes) describing each observation, and dependent variables that are the target or result that the model attempts to predict. In embodiments, the modeling dataset 330 may include, for example, data instances or samples, and historical metrics or variables that describe each observation and are relevant to the modeling of residuals. The modeling dataset 330 is shown within the dashed lines defining the residual model 140, but as mentioned above, it is not necessarily a component of it.

[0041] The modeling dataset 330 is input to the modeling unit 340. The modeling unit 340 is responsible for facilitating exploratory data analysis (EDA) and the construction of the residual model 140. EDA includes statistical and graphical analysis to understand the data distribution, identify patterns, and detect anomalies. The construction of the residual model 140 (or "modeling") includes selecting and training a predictive algorithm using the modeling dataset 330. The predictive algorithm is specifically designed to model the residuals from the primary model (primary model residuals 130), thereby effectively capturing patterns or information that the primary model 110 may have overlooked. The predictive algorithm may use statistical models, machine learning models, and / or other computational techniques.

[0042] The modeling unit 340 outputs residual model predictions 150 (see also Figure 1). These predictions represent adjustments or corrections that should be applied to the initial predictions of the primary model 110 (primary model predictions 120). These are derived, for example, from the analysis of historical metrics and primary model residuals 130 by the residual model 140. As mentioned above, the residual model predictions 150 are subtracted from the primary model predictions 120 in the adjustment process 160 (see Figure 1). This subtraction yields adjusted primary model predictions 170, which represent enhanced and more accurate predictions or estimates for each data point in the modeling cohort. These adjusted predictions essentially incorporate insights from both the primary model 110 and the residual model 140.

[0043] In one embodiment, the collaborating modeling team initiates a data transfer to send residual model predictions 150 to a system implemented by the primary modeling team, which then integrates them with the primary model predictions 120. As previously mentioned, this involves subtracting the residual model predictions from the primary model predictions to generate an adjusted primary model prediction. Thus, the adjusted model is more accurate than the primary model alone because it benefits from the refinement provided by the residual model. The primary modeling team can also evaluate the degree of model improvement achieved by incorporating the residual model predictions.

[0044] Figure 4 is a flowchart of method 600 for improving the accuracy of a predictive model according to the disclosed embodiment. Method 600 includes the step (610) of generating a first dataset for a first-order model, the first dataset including first labeled training data and first labeled test data. Method 600 further includes the step (620) of training a first-order model using the first labeled training data. Method 600 further includes the step (630) of using the first-order model to determine a first training predictive residual group based on the first labeled training data and a first test predictive residual group based on the first labeled test data. Method 600 further includes the step (640) of generating a second dataset for a residual model, the second dataset including second labeled training data labeled based on the first training predictive residual group and second labeled test data labeled based on the first test predictive residual group. Method 600 further includes the step (650) of training a residual model using the second labeled training data. Method 600 further includes the step (660) of determining a training prediction group based on second-labeled training data and a test prediction group based on second-labeled test data using a residual model. Method 600 further includes the step (670) of adjusting the first-order model based on the training and test prediction groups of the residual model to make more accurate predictions.

[0045] In one embodiment, the adjustment step (670) of the primary model may include subtracting the training and test prediction groups of the residual model from the predictions of the primary predictive model based on the first labeled training data and the first labeled test data.

[0046] The adjustment step (670) of the primary model may include adjustments (or modifications) to the primary model itself. For example, the adjustments may include readjusting the hyperparameters of the primary predictor model so that the subsequent training predictor residuals and subsequent test predictor residuals are closer to the first training labels and first test labels.

[0047] In some embodiments, the primary modeling team may evaluate the overall performance of the tuned model (i.e., the primary model whose predictions have been adjusted based on the predictions received from the residual model) and provide feedback on the performance of the tuned model to the residual modeling team (or collaborating team). In some cases, based on such feedback, the collaborating team may provide suggestions on areas where the primary model can be improved or where additional features may be beneficial. For example, it may be proposed that the primary modeling team redesign features, which may include adding new features and / or transforming existing features to better capture patterns in the tuned predictions. This may further include model selection and tuning, such as trying different models and / or readjusting hyperparameters, to improve performance. This may further include data augmentation, including incorporating additional data sources and / or more samples to enhance the training process.

[0048] Improving the accuracy of predictive models presents a technical challenge, requiring the development, implementation, and maintenance of complex methodologies and tools. Predictive modeling is fundamental in various technological fields, including healthcare, finance, and marketing. The accuracy of such models is crucial for making reliable predictions that inform decision-making processes. However, achieving high accuracy is difficult because any individual predictive model has inherent limitations due to the data available to it. Traditional predictive models are often inadequate because they cannot incorporate additional relevant insights that may be available to other predictive models. This limitation leads to suboptimal model performance and inaccurate predictions, which can result in significant inefficiencies and errors in practical applications.

[0049] The disclosed approach provides a technical solution to this technical challenge by improving the accuracy of the predictive model through the use of residuals and mutual adjustments. The disclosed approach is designed to systematically improve model performance by incorporating additional data from a second independent predictive model.

[0050] Specifically, this technical solution involves generating predictions using a specific first-order prediction model and calculating residuals that represent the prediction error. These residuals and associated metadata are then used in an independent residual prediction model, or a second-order prediction model, which may utilize data and features unavailable to the first-order model. The collaborators train the residual model using the residuals, thereby leveraging its unique features to generate additional predictions. This introduces new insights that were not initially available to the first-order model. The predictions generated by the residual model are used to refine the predictions of the first-order model, resulting in a refined first-order model with improved accuracy. This reciprocal refinement effectively reduces the prediction error of the first-order model.

[0051] The application of the disclosed approach yields numerous technical benefits and advantages. As mentioned above, by leveraging residuals and additional insights from residual models, the disclosed approach significantly improves the accuracy of predictive models. This technical improvement is crucial in applications where precision is paramount, such as predicting the conduct of clinical trials and metrics related to their effectiveness or success.

[0052] The disclosed approach enables interaction between independently implemented predictive models without requiring the sharing of raw data, thereby contributing to the maintenance of data privacy and security. This aspect is particularly important in fields such as healthcare, where the confidentiality of patient data is critical. While the underlying motivations for ensuring data privacy and security may stem from ethical, legal, and regulatory concerns, the process of actually achieving and maintaining data privacy and security is itself a technical endeavor.

[0053] The disclosed approach offers scalability and flexibility, for example, by being applicable to a variety of predictive tasks across different domains. The disclosed approach allows multiple entities to independently model using their own datasets and features, thereby improving the scalability of predictive modeling.

[0054] The disclosed approaches offer model independence by not being constrained to a specific type of predictive model and allowing the use of various machine learning algorithms. This flexibility ensures that these approaches are adaptable to different technical requirements and advancements in machine learning techniques.

[0055] By using residuals to identify and correct prediction errors, the disclosed approach provides a systematic improvement in the accuracy and performance of the predictive model. These technical aspects ensure that adjustments to the predictive model are data-driven and target-oriented, thereby leading to a more reliable and robust model.

[0056] Therefore, in light of the points mentioned above, the disclosed approach provides a technical solution to the technical challenge of suboptimal predictive model accuracy by introducing methodologies and tools that leverage residuals and mutual adjustments from secondary actors. These approaches improve model performance, maintain data privacy, and are adaptable to various predictive tasks and machine learning frameworks. By addressing the limitations of conventional predictive models, the disclosed approach represents a significant advance in the field of predictive analytics. The systematic and collaborative approach makes the models not only more accurate but also more robust and reliable, thereby providing a technical solution to pressing technical challenges.

[0057] Furthermore, this technical solution includes, for example, a predictive model, which is a statistical model, a machine learning model, and / or other computational technique, and which has certain properties to produce an adjusted forecast that is enhanced for each data point in the modeling cohort and represents a more accurate prediction or estimate, and substantially incorporates insights from both primary and residual models.

[0058] As explained here, this technical solution focuses on specific improvements in the accuracy of predictive models. Rather than simply using computers, the specific implementation of multiple predictive models improves existing technical processes by providing more accurate predictive models without requiring significant data exchange. Therefore, this technical solution is not merely aimed at abstract results or effects that rely on general processing and machine tools, but provides a specific approach to improving the relevant technologies.

[0059] Figures 5–22 illustrate an example of applying a disclosure approach to a publicly known dataset for a patient cohort. The dataset consists of a cohort of 9,105 critically ill patients from 1989 to 1994 (Vanderbilt University Department of Biostatistics, Professor Frank Harrell 2022). Each row (or record) represents a hospitalized patient record meeting the criteria for one of nine disease categories. The prediction task was an ordinal regression predicting patient functional impairment on a five-point scale. The dataset included 35 features related to patient demographics and health.

[0060] To demonstrate the approach described here, three models were trained using different feature sets. The 35 features available in the dataset were split between a primary model (17 features) and a residual model (18 features). This feature split was intended to illustrate that the primary and residual models are independent of each other, meaning they do not need to use the same feature set, and are not typically used. A third "full model" was trained using all 35 features to represent an ideal scenario where all available information is freely shared to make the best prediction. In this example, Xgboost regression was used to train the primary and residual models on 80% of the dataset, i.e., the training data, with minimal hyperparameter tuning, and each model was evaluated on 20% of the holdout test data, i.e., the test data.

[0061] Figure 22 is a table showing the test data errors for three sets of outputs (or predictions): the output of an unadjusted primary model (e.g., the client's model), the output of a “adjusted model” corresponding to the output of a primary model adjusted with the output of a residual model according to the approach described herein, and the output of the full model. These three sets of outputs were evaluated by the mean absolute error (MAE) and mean squared error (MSE) on the excluded test data. As expected, the full model trained on all features performed best. Notably, the adjusted model, i.e., the primary model output adjusted according to the disclosed embodiment, outperformed the unadjusted primary model.

[0062] Figures 5–20 show scatter plots of the selected features and their Shapley Additive Explanations (SHAP) for the three models in the example described above: the full model, the first-order model, and the residual model. SHAP is a game theory-based method for explaining the output of machine learning models. SHAP can be used to calculate a value for each feature that represents its contribution to the model's output. The SHAP value indicates how each feature influences each final prediction, the importance of each feature to other features, and the model's dependence on feature interactions. Various other evaluation and / or explanatory techniques can also be used, such as Local Interpretable Model-agnostic Explanations (LIME).

[0063] Figures 5–12 are scatter plots of the selection features used in the full model and their SHAP values. Similarly, Figures 13–15 are plots of the selection features in the first-order model, and Figures 16–20 are plots of the selection features in the residual model. Each point in each plot represents one patient in the cohort.

[0064] The y-axis of each plot represents the SHAP value for a specific feature / patient, which indicates the magnitude (or influence) of that feature on the model prediction. Positive values ​​on the y-axis correspond to relatively high SHAP values ​​and therefore relatively high predictions, while negative values ​​on the y-axis correspond to relatively low SHAP values ​​and therefore relatively low predictions. Points near the top and bottom of the plot indicate a large influence on the model prediction.

[0065] The x-axis of each plot represents the standardized value for a specific feature / patient. Points on the left side of the plot, i.e., the negative range of the x-axis, indicate that the patient has a relatively low value for that feature, while points on the right side of the plot, i.e., the positive range of the x-axis, indicate that the patient has a relatively high value for that feature.

[0066] These plots provide a qualitative sense of the consistency of the influence of specific features between the full model, the first-order model, and the residual model. For example, as shown in Figure 8, the feature avtisst has a high density of points in the full model in the negative direction of SHAP values, i.e., negative values ​​on the y-axis, and low values ​​of patient features, i.e., negative values ​​on the x-axis. It also has a high density of points in the full model in the high range of SHAP values, i.e., positive values ​​on the y-axis, and positive values ​​of patient features, i.e., positive values ​​on the x-axis. These characteristics are consistent with the influence of that feature in the first-order model shown in Figure 15, thereby indicating that the predictive power of that feature, such as SHAP values ​​(represented on the y-axis) and patient feature values ​​(represented on the x-axis), is stable across different models.

[0067] In contrast, the direction of feature influence on model output is generally inconsistent between full models and residual models. This is because residual predictions are expected to be adjustments to primary predictions, i.e., corrections for overestimation or underestimation by the primary model, and residual models are designed to provide additional data and / or features that were not available in the full model (or primary model). For example, feature adls substantially changes direction, with relatively high values, such as SHAP values ​​(positive values ​​on the y-axis), having a positive feature influence in the full model for patients with relatively low feature values ​​(negative values ​​on the x-axis) (see Figure 6), but having a negative feature influence (negative values ​​on the y-axis) in the residual model (see Figure 16).

[0068] Figure 21 is a table of feature importance rankings based on SHAP values ​​for the full model, first-order model, and residual model. These rankings compare the usefulness of features in the full model, first-order model, and residual model. Specifically, this table helps to understand whether features that are important in the full model remain important in the first-order model, residual model, and even the adjusted model. For example, the feature avtisst is very important in the full model and is ranked second. It still ranks highly in the first-order model and is positioned as a most important feature, demonstrating its consistent importance.

[0069] As expected, the feature importance ranking is consistent between the full model and the first-order model. For example, the avtisst feature is ranked second in importance in the full model and first in importance in the first-order model. It should be noted that the full model features prg6m and sps were assigned to the residual model and therefore do not appear in the feature importance ranking of the first-order model.

[0070] As expected, the order of feature importance is consistent between the full model and the residual model. For example, feature prg6m is ranked 1st in importance in both the full model and the residual model. Feature sps is ranked 3rd in importance in the full model and 4th in importance in the residual model.

[0071] Aspects of the present invention may be embodied in the form of a system, a computer program product, or a method. Similarly, aspects of the present invention may be embodied in hardware, software, or a combination of both. Aspects of the present invention may be embodied in the form of computer-readable program code embodied therein as a computer program product stored on one or more computer-readable media.

[0072] The computer-readable medium may also be a computer-readable storage medium. The computer-readable storage medium may be, for example, an electronic, optical, magnetic, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof.

[0073] The computer program code in embodiments of the present invention may be written in any preferred programming language. The program code may be executed on a single computer or on multiple computers. The computer may include a processing unit that communicates with a computer-enabled medium, the computer-enabled medium containing a set of instructions, and the processing unit is designed to execute that set of instructions. The above description is intended to illustrate the principles and various embodiments of the present invention. A number of variations and modifications will become apparent to those skilled in the art upon full understanding of the above disclosure. The following claims are intended to be construed to encompass all such variations and modifications.

[0074] The disclosed approach enables interaction between independently implemented predictive models without requiring the sharing of raw data, thereby contributing to the maintenance of data privacy and security. This aspect is particularly important in fields such as healthcare, where the confidentiality of patient data is critical. While the underlying motivations for ensuring data privacy and security may stem from ethical, legal, and regulatory concerns, the process of actually achieving and maintaining data privacy and security is itself a technical endeavor.

Claims

1. The first step is to determine the first training predicted residual group based on the first labeled training data and the first test predicted residual group based on the first labeled test data using a first-order prediction model. A second dataset generation step for generating a second dataset for a residual model, which includes a second labeled training data labeled based on the first training predicted residual group and a second labeled test data labeled based on the first test predicted residual group. The steps include training the residual model using the second labeled training data, A decision step in which the training prediction group based on the second labeled training data and the test prediction group based on the second labeled test data are determined using the residual model, To make predictions with higher accuracy, the first-order prediction model is adjusted based on the training prediction group and the test prediction group of the residual model; A method for improving the accuracy of a first-order predictive model based on a residual model, including the following:

2. A first dataset generation step for the first prediction model, which generates a first dataset including the first labeled training data and the first labeled test data, The steps include training the first-order predictive model using the first labeled training data, The method for improving the accuracy of a primary prediction model according to claim 1, further comprising:

3. The first dataset generation step is performed based on the first patient-level history data set. The method for improving the accuracy of a primary prediction model according to claim 2, wherein the second dataset generation step is performed based on a second patient-level history data set that is different from the first patient-level history data set.

4. The method for improving the accuracy of a primary prediction model according to claim 3, wherein in the second dataset generation step, the values ​​and labels of the first training prediction residual group and the values ​​and labels of the first test prediction residual group are received, and data from the first patient level history data group is not received.

5. The method for improving the accuracy of a primary prediction model according to claim 1, wherein the first dataset generation step defines a modeling cohort and a prediction task.

6. The method for improving the accuracy of a primary prediction model according to claim 5, wherein the prediction task includes predicting participation in a clinical trial.

7. The method for improving the accuracy of a primary prediction model according to claim 5, wherein the prediction task includes predicting the functional impairment of a patient.

8. The aforementioned decision step is, The steps include performing a prediction based on the first labeled training data and the first labeled test data, The steps include comparing the prediction results with the labels of the first labeled training data and the first labeled test data, and calculating the respective residuals, A method for improving the accuracy of a primary prediction model according to claim 1, including the method described in claim 1.

9. The adjustment step described above is A method for improving the accuracy of a first-order prediction model according to claim 1, comprising the step of subtracting the training prediction group and the test prediction group of the residual model from the prediction of the first-order prediction model based on the first-labeled training data and the first-labeled test data.

10. The method for improving the accuracy of a first-order prediction model according to claim 9, wherein the adjustment step includes a step of readjusting the hyperparameters of the first-order prediction model so that the subsequent training prediction residuals and subsequent test prediction residuals are closer to the first training label and the first test label.

11. A system comprising one or more processors that communicate with memory for storing executable instructions, and which generates a composite dataset from road vehicle position data, The first step is to determine the first training predicted residual group based on the first labeled training data and the first test predicted residual group based on the first labeled test data using a first-order prediction model. A second dataset generation step for generating a second dataset for a residual model, which includes a second labeled training data labeled based on the first training predicted residual group and a second labeled test data labeled based on the first test predicted residual group. The steps include training the residual model using the second labeled training data, A decision step in which the training prediction group based on the second labeled training data and the test prediction group based on the second labeled test data are determined using the residual model, To make predictions with higher accuracy, the first-order prediction model is adjusted based on the training prediction group and the test prediction group of the residual model; A synthetic dataset generation system that causes one or more processors to perform the following actions.

12. A first dataset generation step for the first prediction model, which generates a first dataset including the first labeled training data and the first labeled test data, The steps include training the first-order predictive model using the first labeled training data, The synthetic dataset generation system according to claim 11, further comprising causing one or more processors to perform the following operation.

13. The first dataset generation step is performed based on the first patient-level history data set. The composite dataset generation system according to claim 12, wherein the second dataset generation step is performed based on a second patient-level history data set different from the first patient-level history data set.

14. The adjustment step described above is A synthetic dataset generation system according to claim 11, comprising the step of subtracting the training prediction group and the test prediction group of the residual model from the prediction of the primary prediction model based on the first labeled training data and the first labeled test data.

15. The synthetic dataset generation system according to claim 11, wherein the adjustment step includes a step of readjusting the hyperparameters of the first prediction model so that the subsequent training prediction residuals and subsequent test prediction residuals are closer to the first training labels and the first test labels.

16. A non-volatile computer-readable medium for storing instructions that can be executed by one or more processors, The first step is to determine the first training predicted residual group based on the first labeled training data and the first test predicted residual group based on the first labeled test data using a first-order prediction model. The second dataset generation step generates a second dataset for a residual model, which includes a second labeled training data labeled based on the first training predicted residual group and a second labeled test data labeled based on the first test predicted residual group. The steps include training the residual model using the second labeled training data, A decision step in which the training prediction group based on the second labeled training data and the test prediction group based on the second labeled test data are determined using the residual model, To make predictions with higher accuracy, the first-order prediction model is adjusted based on the training prediction group and the test prediction group of the residual model; A non-volatile computer-readable medium that stores instructions for causing one or more processors to execute the above.

17. A first dataset generation step for the first prediction model, which generates a first dataset including the first labeled training data and the first labeled test data, The steps include training the first-order predictive model using the first labeled training data, The non-volatile computer-readable medium according to claim 16, further storing instructions for one or more processors to execute.

18. The first dataset generation step is performed based on the first patient-level history data set. The non-volatile computer-readable medium according to claim 17, wherein the second dataset generation step is performed based on a second patient-level history data set different from the first patient-level history data set.

19. The adjustment step described above is The non-volatile computer-readable medium according to claim 16, comprising the step of subtracting the training prediction group and the test prediction group of the residual model from the prediction of the primary prediction model based on the first labeled training data and the first labeled test data.

20. The non-volatile computer-readable medium according to claim 16, wherein the adjustment step includes readjusting the hyperparameters of the first prediction model so that the subsequent training prediction residuals and subsequent test prediction residuals are closer to the first training label and the first test label.

Images