Scoring System Federation

Abstract

A system and method for federated scoring by multiple nodes, wherein each node includes confidential data and a first set of scoring model coefficients is generated based on the confidential data, the first set of scoring model coefficients is broadcast to the other nodes, and at least one node generates a federated scoring model based on the received contribution intermediate statistics and its respective first set of scoring model coefficients.

Description

[Technical Field]

[0001] The present disclosure generally relates to methods and systems for federation of scoring systems or scoring models. [Background technology]

[0002] This background discussion is provided to generally present the context of the present disclosure. The contents of this background section are not admitted explicitly or implicitly as prior art to the present disclosure.

[0003] Inter-institutional partnerships in research, including healthcare research, are becoming increasingly prevalent. As healthcare processes become more computerized, large amounts of healthcare data are being generated by individuals' interactions with the healthcare system. As healthcare service providers continue to automate or computerize more portions of their services, the amount of data generated reflecting healthcare interactions or outcomes continues to increase. The increasing amount of data provides a great opportunity to develop and implement computational constructs, such as machine learning (ML) models in the healthcare system. As the amount of data related to healthcare interactions increases, opportunities arise to develop and train more robust and accurate ML models. As more data becomes available to ML models, ML models can potentially be trained to more accurately model a variety of real-world situations.

[0004] However, data held by individual institutions is subject to constraints, including privacy constraints that limit the sharing or transmission of data to other entities or institutions. The inability to share data creates a significant obstacle in the development of ML models by individual institutions or data holders, and it is desirable to provide frameworks, systems, and methods that address the problem or at least provide an alternative to existing solutions. Summary of the Invention [Means for solving the problem]

[0005] In one embodiment, the present disclosure provides an associative scoring system, said system comprising: A plurality of nodes; a communication network enabling communication between a plurality of said nodes; Including, each node includes at least one processor and memory, the memory of each node including sensitive data and program code; the sensitive data includes a plurality of records; The program code generating a first set of scoring model coefficients based on confidential data accessible to each of said nodes; broadcasting said first set of scoring model coefficients to other said nodes; and the plurality of nodes is executable by the processor of each of the plurality of nodes, At least one of the nodes receiving contributing intermediate statistics from other said nodes; generating a federated scoring model based on the received contribution intermediate statistics and the first set of respective scoring model coefficients; It is configured as follows.

[0006] In another embodiment, the present disclosure provides: providing a plurality of nodes, each node including at least one processor and a memory, the memory of each node including sensitive data and program code, the sensitive data including a plurality of records; providing a communications network enabling communication between a plurality of said nodes; generating a first set of scoring model coefficients based on confidential data accessible to each of said nodes; executing, by the processor of each of a plurality of said nodes, said program code to broadcast said first set of scoring model coefficients to other said nodes; receiving contributing intermediate statistics from other said nodes; executing, by at least one of the nodes, the program code to generate a federated scoring model based on the received contribution intermediate statistics and a first set of respective scoring model coefficients; The present invention provides an associative scoring method, including:

[0007] Also disclosed are one or more non-transitory computer-readable storage media that store instructions that, when executed by one or more processors, cause the one or more processors to perform the methods described above. [Brief explanation of the drawings]

[0008] Some embodiments of systems and methods for privacy-preserving federated execution of scoring models will now be described, by way of non-limiting example only, with reference to the accompanying drawings, in which:

[0009] [Figure 1] 1 shows a schematic block diagram of a federated scoring system. [Figure 2] 1 shows data partitioning for an experiment using an associative scoring system. [Figure 3] We show the performance of different scoring models at different nodes in the experiments. [Figure 4] 1 shows the combined performance of various scoring models, including the integrated scoring model and the federated scoring model. [Figure 5(1)] 1 shows parsimony plots generated during the performance evaluation of the scoring model in the experiments. [Figure 5(2)] This is a continuation of Figure 5(1). [Figure 5(3)] This is a continuation of Figures 5(1) and 5(2). [Figure 6] FIG. 1 is a block diagram of a federated scoring system. DETAILED DESCRIPTION OF THE INVENTION

[0010] Federated learning, also known as distributed learning or distributed algorithms, can address issues associated with limitations on data sharing by collectively training algorithms without exchanging data. In the context of healthcare data, embodiments of the federated scoring system and method disclosed herein can protect patient privacy by distributing model training to data owners and aggregating results across various data owners / sites. Federated learning can break down data silos and more quickly develop much-needed scoring systems / models for analyzing healthcare data and making decisions based on healthcare data. Furthermore, the federated system disclosed herein enables the federation of interpretable models, which are favorable for clinical research.

[0011] Figures 1 and 6 illustrate a system architecture for associative scoring. First, as shown in Figure 6, a system 600 is used for associative scoring. The system 600 includes multiple nodes 630 that communicate with each other via a communication network 620. As shown with nodes 630(1) and 630(N) (indicating that there are N nodes in the system or network 600), each node includes at least one processor 632 and a memory 634. The memory 634 includes sensitive data 638 and program code 636. The sensitive data 638 can include multiple records, such as patient data records.

[0012] The program code 636 is executable by each processor 632 of each node 630 and generates a first set of scoring model coefficients based on the confidential data accessible to the respective node. Generally, two scenarios will exist: a node 630 only has access to the confidential data 638 stored in its own memory 634; or a node from a group of affiliated nodes (e.g., a group of clinics operated by a particular company) has access to the confidential data stored at all affiliated nodes. Regarding the coefficients, an exemplary model used includes logistic regression, where logit(Pr(Y=1|X))=β0+β1X1+β2X2+..., where p is the probability of a response of Y=1 (Y may only be either 0 or 1), and X1, X2... are predictors such as gender, age, etc. Because it is possible to have more than one predictor (X), the visualization may be multidimensional rather than 2D. logit(Pr(Y=1|X)) is the y-axis, but there may be more than one x-axis. Therefore, the coefficient is beta (β).

[0013] The instructions stored in memory 634 also cause the processor 632 to broadcast the first set of scoring model coefficients to the other nodes 630 .

[0014] Each node 630 is also configured to generate a node-specific scoring model based on the first set of scoring model coefficients.

[0015] At least one of the nodes 630 is configured to receive contributory intermediate model statistics (also referred to as contributing intermediate statistics, etc.) from the other nodes 630, as described below. The node then generates an association scoring model based on these intermediate statistics and its respective data. The intermediate statistics are, for example, components (terms) of equation (5) for any particular node.

[0016] Each of the plurality of nodes 630 is configured to transmit its node-specific scoring model and scoring model performance data to the central server 610. Because at least one of the nodes 630 generates a federated scoring model, the central server 610 is also configured to receive the federated model from the node 630 that generates such a federated scoring model. To ensure that a consistent model is implemented across some or all of the nodes 630, the central server 630 is configured to transmit the federated model to at least a subset of the plurality of nodes 630. If two or more federated scoring models are received at the central server 610, the central server 610 may select a particular federated scoring model to be implemented across all of the nodes 630. Model selection may be performed via a parsimony plot, as shown in FIG. 5. In some embodiments, this step involves user selection; in other embodiments, the selection is automatic based on a general criterion that adding new variables to the model does not result in an increase in model performance (increase in AUC value) above a predetermined threshold. The predetermined threshold may be set by the user. Variables may also be added based on domain knowledge, which can be accessed using natural language processing on publications in the relevant domain to identify the most frequently mentioned variables, or based on the user's domain knowledge (i.e., user-added variables). These additional variables may not necessarily align perfectly with the parsimony plot. For example, if the model performance in the parsimony plot is already 0.85 using the first five variables (high enough to stop adding new variables), then additional variables may be included based on domain knowledge or domain publications.

[0017] Some embodiments federate a scoring system across various nodes of a computer system. The scoring system may be a classification model that includes the definition of a set of calculations on input data, which are executed to make predictions based on the input data. The scoring model can be implemented using various types of models, such as linear classification models, logistic regression models, and clinical decision support models. The set of calculations includes addition, subtraction, multiplication, and other calculations. The model is used to assess the risk of a number of serious medical conditions to provide efficient and interpretable predictions.

[0018] Table 1 shows an example of a scoring system.

[0019] [Table 1]

[0020] A doctor can easily determine whether a patient is classified as positive for obstructive sleep apnea by adding up points for the patient's age, presence of diabetes, body mass index, and gender. If the score exceeds a threshold, the patient is referred to a clinic for diagnostic testing.

[0021] Traditional scoring systems have primarily been developed for single-source data. Therefore, training or sample data sets are often small or unrepresentative. For example, data obtained from wealthy communities are likely to have lower rates of adverse outcomes than data from poor communities. While it is possible to develop scoring systems for integrated data, the integration process is time-consuming and difficult to achieve for privacy reasons. The framework shown in Figure 6 addresses these challenges by building a federated scoring system.

[0022] The disclosed system and method (also referred to as FedScore) provides an approach for building a federated scoring system that can run across multiple computer system nodes hosted in various locations. Embodiments improve robustness and remove bias from medical research, especially in situations with relatively small sample sizes. Figures 1 and 6 show the overall architecture of a system according to embodiments.

[0023] FedScore Framework When implementing clinical models and other models (e.g., financial records) where data privacy should be maintained, users typically consider the degree of parsimony as an important characteristic of the model. A model is considered parsimonious if it is sparse (i.e., uses the minimum number of variables possible) and has high predictive accuracy. FIG. 6 shows a block diagram of a FedScore system / framework 600. The FedScore system of some embodiments may include a central server 610 and multiple nodes 630. Alternatively, specific nodes of the multiple nodes of the FedScore system may perform the functions of the central server and the node or system / framework 600, thereby eliminating the need for a designated central server. As described above, each node 630 includes at least one processor 632 and memory 634. The memory 634 includes program code 636 and sensitive data 638. The various nodes communicate with each other via a communication network 620. The sensitive data is accessible only to the respective nodes.

[0024] To achieve good parsimony and interpretability while incorporating privacy requirements, the FedScore framework consists of five modules: (1) Federated Variable Ranking Module, (2) Federated Variable Transformation Module, (3) Federated Score Derivation Module / Scoring Module, (4) Model Selection Module, and (5) Model Evaluation Module. Some or all of these modules may be located within a particular node, distributed across nodes, or located on a central server, depending on the architecture of a particular embodiment. In the embodiment shown in FIG. 6, these modules are stored in memory 614.

[0025] (1) Union variable ranking module To build a global model across several sites or nodes, some embodiments may pre-identify a set of unified variables as candidate variables for ranking to be performed independently across various nodes 630. For example, assume that both sites A and B use 0%, 25%, 50%, 75%, and 100% cuts on their variables. Due to heterogeneity in the data, the cutoffs for the age variable may be different for A: (,24), (24,49), (49,62), (62,) years, and B: (,24), (24,52), (52,67), (67,) years. This suggests that site B has a relatively older population. In this case, federation cannot be performed because these categorical variables have different meanings and are not unified. Instead, when federating, the two sites can specify cuts that are sensible for both sites, e.g., (,24), (24,50), (50,60), (60,) years, and then perform federation. This integration can be used to This may be achieved by collecting data from all contributing sites (e.g., all patient ages) and separating them based on statistical parameters such as percentiles as described above. In some embodiments, random forests may be utilized to perform variable ranking. In the FedScore framework, variable ranking is first performed at each local site / node 630. Each node 630 then submits its variable ranking to the central server 610, which generates a global variable ranking. The global variable ranking may be generated by ordering variables by their average rank at each site.

[0026] In some embodiments, a random forest may include a collection of randomized classification and regression trees. One important measure of a given variable in a random forest is the average increase in the tree's error when observations of that particular variable are randomly permuted in out-of-bag samples. More specifically, the importance of a variable may be quantified based on the mean squared error of regression and the misclassification rate of classification. In classification tasks, the Gini coefficient for each node θ of a decision tree Θ is defined as:

number

number

[0027] In some embodiments, the variable rankings at each node may be determined based on model parsimony statistics, such as model parsimony plots, which show the relevance of each variable or combination of variables to the performance of the scoring model. In some embodiments, variable rankings may be evaluated using area-under-the-curve statistics for various models. The associative scoring model of some embodiments may take into account variables in the global variable rank that exceed a predetermined threshold, such as an importance threshold. For example, assume that a random forest is used for importance measurement. After scaling the importance values ​​to 0 and 1, the following variable importance values ​​may result: var1: 0.8, var2: 0.5, var3: 0.3, var4: 0.2, var5: 0.15, var6: 0.08... The threshold may be empirically set to 0.1, resulting in only the first five variables being selected. By doing so, the associative scoring model reduces the number of variables incorporated into the model, resulting in a more interpretable model while maintaining its accuracy in a clinical setting; i.e., the model becomes sparse without significantly compromising accuracy.

[0028] (2) Associated variable transformation module The FedScore framework also converts continuous variables into categorical / discrete variables after global variable rankings are determined. For example, a person's age may be banded as follows: 0-20, 20-30, 30-45, 45-60, and 60+. The maximum number of categories for such conversions may be predetermined (e.g., 5), and if the maximum value for a particular variable is exceeded, categories may be combined to meet the maximum requirement. A global cutoff / discrete bucket for each continuous variable is calculated by averaging the k values ​​obtained at each site. The k values ​​are used to cut continuous variables into several categorical variables. For example, assuming there is only one 50% k value for the age variable and the 50% cutoff for age is 50, the age variable is converted from a continuous variable into two categorical variables: age(≦50) (true or false) and age(≧50) (true or false). After defining the discrete buckets / cutoffs for each variable, the defined cutoffs are transmitted to multiple nodes, allowing the nodes to process the continuous variable data in a uniform, standardized, and discrete manner. In some embodiments, the quantiles for the continuous variables were set to 0%, k1%, k2%, k3%, k4%, and 100%, and the values ​​of k1, k2, k3, and k4 were set to 5, 20, 80, and 95. The federated variable transformation by providing standardized discrete buckets for the continuous variables improves the accuracy of the federation because various nodes observe a common set of discrete buckets when providing input to their respective scoring models.

[0029] (3) Association score derivation module The score derivation process can be flexibly tailored for different clinical modeling purposes by incorporating appropriate ML models according to clinical needs and circumstances. For example, a logistic regression model may only support binary outcomes. By switching the logistic regression in Module 3 to other appropriate models, the framework can be extended to support survival outcomes, ordinal outcomes, etc. (i.e., non-binary outcomes). A step in the score derivation process includes generating a first set of scoring model coefficients based on the confidential data. The scoring model coefficients include various parameters of the scoring model. For example, in an embodiment where the scoring model is a linear model, the coefficients are linear parameters and intercepts, etc. Each node has access to different confidential data and therefore generates its own first set of scoring model coefficients. Generally, the data does not overlap. Each node uses the first set of scoring model coefficients to define a node-specific scoring model. The node-specific scoring model serves as a candidate model for comparison with the federated scoring model.

[0030] The first set of scoring model coefficients may be broadcast to the other nodes, and each node may generate a set of contributing / intermediate scoring model coefficients based on the received first set of scoring model coefficients and confidential data accessible to each respective node. The contributing scoring model coefficients may then be sent / broadcast to the other nodes. One or more nodes may generate a federated scoring model based on its first set of scoring mode coefficients and the contributing scoring mode coefficients received from the other nodes. An example of federated scoring model generation is described with reference to the ODAL2 algorithm.

[0031] As another example, logistic regression is a common choice for modeling binary outcomes. Concatenated logistic regression may be performed by a single iteration of logistic regression (a one-shot approach) or by some embodiments requiring multiple iterations of logistic regression.

[0032] FedScore is a privacy-preserving framework for providing a unified, robust scoring system across multiple sites without the need to share sensitive data such as confidential medical data or other personal information. FedScore was tested using a clinical scoring model for 30-day mortality prediction using emergency department (ED) data from Singapore General Hospital (SGH) and simulations of 10 nodes / sites that did not exchange sensitive patient data during the experiment. The robustness and generalizability of FedScore were established by achieving a high average area under the curve (AUC) on each site's test data with minimal variance compared to the baseline score.

[0033] The experiment was performed using ODAL2, a one-shot privacy-preserving distributed algorithm for performing federated logistic regression, as disclosed in R. Duan et al., "Learning from electronic health records across multiple sites: A communication-efficient and privacy-preserving distributed algorithm," J. Am. Med. Inform. Assoc., Vol. 27, No. 3, pp. 376-385, December 2019, doi:10.1093 / jamia / ocz199. The embodiment utilized information from local sites / nodes with linear (ODAL1) (first set of scoring model coefficients) and quadratic (ODAL2) gradients (contributing scoring model coefficients) of the likelihood function from remote sites to construct an approximation of the global likelihood function that forms part of the federated scoring model. Data from remote sites / nodes was not accessible in performing the logistic regression calculation. The coefficients generated at each node during the logistic regression were sent to a central server.

[0034] The coefficients in the global logistic regression model are generated by optimizing a likelihood function, which is then rounded to obtain a score based on each variable. A central server defines a scoring table, and an overall score is calculated by adding up all points. Both the upper limit of the total score and the normalization of the score classification may be adjusted to suit the needs of the intended clinical application.

[0035] The coefficients of the global logistic regression model may be obtained by optimizing a global likelihood function: x1, x2, ... x p-1 Let ∇ denotes the p-1 predictors and y denotes the binary outcome, then the logistic regression model may be expressed as: logit(Pr(y=1│x))=x T β (3) In the formula, x=(1,x1,x2,…x p-1 ) T and β is the vector of intercepts and coefficients, logit(t)=logt / (1-t).

number

number

number

[0036] However, if the data cannot be shared, the calculation of the joint likelihood function is not possible. As envisaged by the embodiment, the approximation of the likelihood function is performed as described herein. As an example, the ODAL2 algorithm proposes to apply the concept of Taylor expansion and perform the approximation using the linear and quadratic gradients of the LLR (Log-Likelihood Ratio).

number

number

number

number

number

number

number

number

number

[0037] When running the ODAL2 algorithm, first, the initial value

number

number

number

[0038] Global Beta Estimator

number

[0039] In some embodiments, the coefficients of the association logistic regression model (association scoring model) are rounded to obtain an association score for each variable. A scoring table is created and an overall score is calculated by adding all the points. Both the upper limit of the total score and the normalization of the score classification may be adjusted to suit the needs of the intended clinical application.

[0040] (4) Model Selection Module The scoring model trained at each node serves as a candidate that can be used by any or all of the other nodes in the future to obtain the most accurate results. Model evaluation and selection could be performed using a parsimony plot generated by the average AUC (area under the curve) of all sites / nodes. The parsimony plot shows the performance of the scoring model as a function of the number of variables incorporated into the model. Let i denote the site / node index, i∈{1, 2, …k}. A common model selection criterion is Ψ m =Σw i φ i (p1,p2,p3,...p m ) where w i is the weight of site i, and φ i is the performance of the scoring model on the i-th validation set, and m is a predefined number of total variables to include, which may be uniform across all sites. In some embodiments, the weights indicate equal weights for all sites.

number

number

[0041] We can also add different constraints to the maximization task. For example, the total number of variables m may not exceed an integer N. The variables {p1, p2, ... p m The set of variables {x1, x2, ... x} may also be constrained to meet certain standards required by the system. For example, the system of some embodiments may use the variables {x1, x2, ... x} in the federated scoring model. q}, and q≦m, as long as the increase in the number of variables from d to m has only an acceptable effect on the change in Ψ, i.e., |Ψ m -Ψ d Ψ may be maximized using a number d of variables less than m, as long as |≦ε (the size of ε may be determined intuitively by the user).

[0042] The final selection of variables may be confirmed based on a federated model selected from among multiple scoring models at each node. The new scoring model may be re-fitted to any new data using step (2). The performance of the selected federated scoring model is validated on each test dataset at each site participating in the federated learning process. The selected federated scoring model may be sent to each node, enabling the node to process new clinical data using the federated scoring model.

[0043] In particular, to maintain parsimony without losing accuracy, interrelated variables may be identified, e.g., variables that have substantially the same impact on patient outcomes and vary substantially proportionally with each other (e.g., 5% of each other); for example, weight and height may vary substantially proportionally with each other (e.g., within 5%) for a particular gender. Of the interrelated variables, the system may select optimal variables for the federated model (e.g., variables among the interrelated variables that most closely correlate with an outcome such as 30-day mortality) based on each node capturing the optimal variable in its confidential data or capturing a different variable that is interrelated with the optimal variable. Each node that does not capture the optimal variable may then substitute the relevant interrelated variable in its local model.

[0044] The resulting associative scoring model is interpretable. Unless the context dictates otherwise, interpretability means that the correlation between the sensitive data and the outcome is clear and explainable from the model; for example, as shown in Table 1, for people over 60, age correlates well with sleep apnea. This is in contrast to machine learning models that use hidden layers to identify features in the data, whereby the correlation between the data and the outcome may not be readily apparent.

[0045] (5) Model evaluation module The performance of the federated scoring model and / or node-specific scoring models is validated using a test dataset for each site participating in the FedScore framework. Model evaluation may be performed by a designated node or a central server, depending on the configuration of the FedScore framework. Ψ, as defined in step (4), m According to, the overall average performance of the associative scoring model is

number

number

[0046] Clinical trial design A retrospective analysis was conducted using emergency medical record data from Singapore General Hospital (SGH) extracted from the SingHealth Electronic Health Intelligence System. An initial study cohort of 86,527 admissions was identified by selecting ED admissions from the SGH data between 2016 and 2017. After excluding patients under 18 years of age and patients with missing values, a total of 80,613 admissions remained. These were then randomly divided into 10 simulation sites at 4%, 5%, 7%, 9%, 10%, 11%, 12%, 13%, 14%, and 15% of the total, respectively. Figure 2 shows the data division for experiments using the system for federated execution of ML models.

[0047] The outcome of the study was 30-day mortality, defined as death occurring within 30 days after ED admission. Candidate predictors included a total of 29 variables, including demographic information, PACS triage category, shift time, day of the week (Friday, Monday, weekend, midweek), vital signs, comorbidities, and previous healthcare use.

[0048] result The analysis was performed across three groups: (1) 10 local scores trained independently at each site; (2) one federated score trained using all sites without data sharing; and (3) one federated score generated using the combined data, which is an ideal case but typically not feasible in practical applications. Models were selected based on the corresponding parsimony plots using predetermined criteria: the maximum number of variables in a model should not exceed 10, and adding additional variables should be discontinued if there is no significant improvement in AUC. The variables selected for each model are included in the footnotes of Table 2. Figure 5 shows a series of parsimony plots obtained for various scoring models. Plots (a)–(j) relate to local models generated at sites 1–10. Plot (k) relates to the federated scoring model generated at any one of the sites / nodes. Plot (l) relates to the scoring model generated using the combined data, which is unlikely to be feasible in real-world environments outside of this experiment.

[0049] A total of 12 scoring models were obtained and tested, including 10 local models generated independently at each site, one federated model developed by FedScore, and one integrated model generated based on integrated data from all sites, which would not be possible in real-world scenarios outside of this experiment. The AUC values ​​and confidence intervals (CIs) for each model for the test data from different sites are shown in Table 2. The AUC values ​​and 95% CIs for each score are individually plotted in Figure 3 to better illustrate the results in Table 2 in terms of model performance variability across all sites. The mean and standard deviation (SD) of the AUC values ​​for each model across all 10 test sets were also calculated and are shown in Table 2. Figure 4 provides corresponding information. As shown in both Figures 3 and 4, the federated scoring model outperformed all local scores in terms of stability and generalizability by achieving the lowest SD. The federated score also showed a sufficient average AUC for each site, indicating that the existing FedScore framework can generate reliable global clinical scores. The last row of Table 2 shows the average AUC values ​​of all local models at each site, and these values ​​are mostly exceeded by the AUC values ​​of the federated score. This suggests that for single sites, FedScore has the potential to yield a better scoring system than locally developed ones, especially when the local site sample size is insufficient.

[0050] [Table 2]

[0051] In Table 2, the following abbreviations apply: AUC, area under the curve; CI, confidence interval; SD, standard deviation; SBP, systolic blood pressure; DBP, diastolic blood pressure; SpO2, oxygen saturation measured by pulse oximetry; PACS, Patient Acuity Category Scale; ED, emergency department. Furthermore, aLocal model obtained via AutoScore for site 1;Variables selected in the model (in order of ranking): pulse rate, age, SBP, DBP, SpO2, respiration, day of the week, ED admission within the past year; b Local model obtained via AutoScore at site 2;Variables selected in the model (in order of ranking): SBP, DBP, pulse, age, SpO2, respiration, ED admission within the past year; c Local model obtained via AutoScore at site 3;Variables selected in the model (in order of ranking): pulse, age, SBP, DBP, SpO2, respiration, ED admission within the past year; d Local model obtained via AutoScore at site 4; selected variables in the model (in order of ranking): pulse, SBP, age, DBP, SpO2, respiration, ED admission within the past year, PACS triage category; e Local model obtained via AutoScore at site 5;Variables selected in the model (in order of ranking): SBP, pulse, DBP, age, SpO2, respiration, day of week, ED admission within the past year, shift time, PACS triage category; f Local model obtained via AutoScore at site 6;Variables selected in the model (in order of ranking): SBP, pulse, age, DBP, SpO2, respiration, ED admission within the past year; g Local model obtained via AutoScore at site 7;Variables selected in the model (in order of ranking): SBP, pulse, DBP, age, SpO2, respiration, ED admission within the past year; h Local model obtained via AutoScore at site 8;Variables selected in the model (in order of ranking): SBP, pulse, DBP, age, SpO2, respiration, day of the week, ED admission within the past year; i Local model obtained via AutoScore at site 9; variables selected in the model (in order of ranking): pulse rate, SBP, age, DBP, SpO2, respiration, day of week, ED admission within the past year, shift time, PACS triage category; jLocal model obtained via AutoScore for site 10;Variables selected in the model (in order of ranking): SBP, pulse, age, DBP, SpO2, respiration, ED admission within the past year, day of the week; k Association model obtained via FedScore;Variables selected in the model (in order of ranking): SBP, pulse rate, age, DBP, SpO2, respiration, ED admission within the past year, day of the week, shift time, PACS triage category; l Integrated model obtained via AutoScore; variables selected in the model (in order of ranking): SBP, pulse rate, age, DBP, SpO2, respiration, day of the week, ED admission within the past year.

[0052] The framework provided by the embodiments is extensible and flexible, considering that each scoring model can be modified or replaced to accommodate different clinical research questions. For example, the score derivation module may be tailored to produce ordinal results. FedScore fills a gap in existing medical machine learning applications, which lack an established method for generalizing a unified score across multiple sites. It also addresses the lack of a reproducible benchmarking method, particularly for more interpretable models.

[0053] Embodiments generate more interpretable scoring models due to the preference for using fewer variables and simpler model structures compared to black-box techniques, such as deep learning-based models. Conventional deep learning techniques also require larger amounts of data to generate more accurate models. In contrast, the disclosed FedScore system does not require the sharing of confidential data and allows multiple nodes / sites to work toward a federated scoring model. Because each site only has access to a limited amount of data, if each site pursues a deep learning-based model, the results at each site may be suboptimal due to the limited amount of data. The FedScore system addresses this challenge by generating a federated scoring model that provides a more interpretable scoring model and addresses the limitations associated with black-box machine learning models, such as deep learning models. Additionally, scoring models trained on data from multiple sites are more generalizable due to the diversity of data sources on which such models are trained as a whole. Thus, federated scoring models generated by embodiments are more generalizable to new settings, such as data from new clinical settings or data from different populations. The disclosed FedScore framework could serve as the foundation for a data science software platform that handles large-scale multi-center data analysis and risk scoring development.

[0054] Reference in this specification to any prior publication (or information derived from a prior publication) or to any publicly known matter is not, and shall not be construed as, an acknowledgment or admission or any form of suggestion that the prior publication (or information derived from a prior publication) or publicly known matter forms part of the common general knowledge in the field of development to which this specification pertains.

[0055] Throughout this specification and the appended claims, unless contradictory to the context, the word "comprise" and variations such as "comprises" and "comprising" are understood to imply the inclusion of a stated integer value or step or group of integer values ​​or steps, but are not to be understood to exclude any other integer value or step or group of integer values ​​or steps.

[0056] The scope of the present disclosure encompasses all changes, substitutions, variations, alterations, and modifications to the exemplary embodiments described or illustrated herein that would be understood by one skilled in the art. The scope of the present disclosure is not limited to the exemplary embodiments described or illustrated herein. Furthermore, although the present disclosure describes and illustrates each embodiment herein as including particular components, elements, features, functions, operations, or steps, any of these embodiments can include any combination or permutation of any of the components, elements, features, functions, operations, or steps described or illustrated anywhere herein that would be understood by one skilled in the art. Although the present disclosure describes or illustrates particular embodiments as offering certain advantages, a particular embodiment may offer none, some, or all of these advantages.

Claims

1. A plurality of nodes; a communication network enabling communication between a plurality of said nodes; Including, each node includes at least one processor and memory, the memory of each node including sensitive data and program code; the sensitive data includes a plurality of records; The program code generating a first set of scoring model coefficients based on confidential data accessible to each of said nodes; broadcasting the first set of scoring model coefficients to the other nodes; and the plurality of nodes is executable by the processor of each of the plurality of nodes, At least one of the nodes receiving contributing intermediate statistics from other said nodes; generating a federated scoring model based on the received contribution intermediate statistics and the first set of respective scoring model coefficients; A federated scoring system configured as follows:

2. The system of claim 1 , wherein each node is configured to generate a node-specific scoring model based on the first set of scoring model coefficients.

3. The system of claim 2 , further comprising a central server configured to evaluate each of the node-specific scoring models and the federated scoring model based on model parsimony statistics.

4. Each record contains multiple variables, and each node is determining a rank for each of the plurality of variables based on the relevance of each variable to a scoring result produced by a respective one of the scoring models; transmitting the rank of each of the plurality of variables to the central server. further configured as follows: The central server defining a global variable rank based on the ranks of a plurality of the variables received from a plurality of the nodes; transmitting the global variable rank to at least one of the plurality of nodes; The system of claim 3 , configured to:

5. The system of claim 4 , wherein the relevance of each of the plurality of variables to the scoring result is evaluated based on a model parsimony statistic or a model area under the curve statistic.

6. The system of claim 4 or claim 5, wherein the federated scoring model is generated based on the global variable ranks.

7. The system of claim 6 , wherein the federated model is generated by incorporating variables above a threshold into the global variable rank.

8. The system of any one of claims 4 to 7, wherein the node uses a random forest model to determine the rank of each of the plurality of variables.

9. The system of any one of claims 4 to 8, wherein the central server defines the global variable rank by averaging the ranks of a plurality of the variables received from each of a plurality of the nodes.

10. The system of any one of claims 1 to 9, wherein the scoring model is implemented using any one of a linear classification model, a logistic regression model, and a clinical decision support model.

11. The system of any one of claims 1 to 10, wherein each of a plurality of said nodes is configured to transmit its node-specific scoring model and scoring model performance data to said central server.

12. The system of claim 11 , wherein the central server is configured to receive the federated model from at least one of the nodes.

13. The system of claim 10 , wherein the central server is configured to transmit the federated model to at least a subset of the plurality of nodes.

14. The system of any preceding claim, wherein the variables comprise one or more continuous data variables, and each of the nodes is further configured to transform the continuous data variables into discrete variables.

15. The system of any one of claims 1 to 14, wherein at least one of the nodes is configured to process new clinical data using the federated model to generate a score.

16. The system of any one of claims 1 to 15, wherein the contributing intermediate statistics are calculated by each respective node based on confidential data accessible to the respective node.

17. providing a plurality of nodes, each node including at least one processor and a memory, the memory of each node including sensitive data and program code, the sensitive data including a plurality of records; providing a communications network enabling communication between a plurality of said nodes; generating a first set of scoring model coefficients based on confidential data accessible to each of said nodes; broadcasting the first set of scoring model coefficients to the other nodes; executing the program code by the processor of each of the plurality of nodes, such that receiving contribution scoring intermediate statistics from other said nodes; generating a federated scoring model based on the received contribution scoring intermediate statistics and the first set of respective scoring model coefficients; executing the program code by at least one of the nodes such that Association scoring methods, including:

18. 20. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of claim 17.

Images