Joint of scoring systems

Abstract

Systems and methods for joint scoring by a plurality of nodes, where each node includes sensitive data based on which a first set of scoring model coefficients is generated. A first set of scoring model coefficients is broadcast to the remaining nodes, and at least one node generates a joint scoring model based on the received contributive intermediate statistics and its respective first set of scoring model coefficients.

Description

Technical Field

[0001] The present disclosure generally relates to methods and systems for federation of scoring systems or scoring models. Background Art

[0002] The purpose of providing this background description is to generally present the context of the present disclosure. The contents of this background section are neither explicitly nor implicitly admitted to be prior art to the present disclosure.

[0003] Cross-institutional partnerships in research, including healthcare research, are becoming increasingly popular. As healthcare processes become computerized, a large amount of healthcare data is generated from individuals’ interactions with the healthcare system. As healthcare providers continue to further automate or computerize large portions of their services, the amount of data reflecting healthcare interactions or outcomes will continue to increase. The increasing amount of data provides a tremendous opportunity to develop and implement computational constructs such as machine learning (ML) models into the healthcare system. The increasing amount of healthcare interaction data provides an opportunity to develop and train more robust and accurate machine learning models. As more data becomes available for machine learning models, machine learning models can be trained to more accurately simulate different real-world situations.

[0004] However, data held by individual institutions are subject to restrictions, including privacy restrictions that limit the sharing or transfer of data to other entities or institutions. The inability to share data poses a significant barrier to individual institutions or data holders developing machine learning models, so there is a need to provide frameworks, systems, and methods to solve the problem, or at least provide alternatives to existing solutions. Summary of the invention

[0005] In one embodiment, the present disclosure provides a system for joint scoring, the system comprising:

[0006] Multiple nodes,

[0007] A communication network that enables communication between multiple nodes.

[0008] Each node includes at least one processor and memory, and the memory of each node includes sensitive data and program code.

[0009] Sensitive data includes multiple records;

[0010] wherein the program code is executed by a corresponding processor of each of the plurality of nodes to:

[0011] generating a first set of scoring model coefficients based on the sensitive data accessible to each node;

[0012] Broadcast the first set of scoring model coefficients to the rest of the nodes;

[0013] And wherein at least one node is configured as:

[0014] Receive contributing intermediate statistics from the remaining nodes;

[0015] A joint scoring model is generated based on the received contribution intermediate statistical data and its corresponding first set of scoring model coefficients.

[0016] In another embodiment, the present disclosure provides a method for joint scoring, comprising:

[0017] Provide multiple nodes, each node includes:

[0018] at least one processor and memory,

[0019] The storage of each node includes sensitive data and program code, and the sensitive data includes multiple records;

[0020] Provide a communication network to enable communication between multiple nodes,

[0021] The program code is executed by a respective processor of each of the plurality of nodes to:

[0022] generating a first set of scoring model coefficients based on the sensitive data accessible to each node;

[0023] Broadcast the first set of scoring model coefficients to the rest of the nodes;

[0024] Program code is executed by at least one of the nodes to:

[0025] Receive contributing intermediate statistics from the remaining nodes;

[0026] and

[0027] A joint scoring model is generated based on the received contributing intermediate statistics and their corresponding first set of scoring model coefficients.

[0028] Also disclosed are one or more non-transitory computer-readable storage media, the storage medium or media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the above-described method. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] By way of non-limiting example only, some embodiments of systems and methods for privacy-preserving joint execution of scoring models are described with reference to the accompanying drawings, in which:

[0030] Figure 1 A schematic block diagram of a joint scoring system is shown;

[0031] Figure 2 shows the data partitioning for experiments using the joint scoring system;

[0032] Figure 3 The performance of various scoring models on different nodes in the experiment is shown;

[0033] Figure 4 The combined performance of various scoring models including the pooled scoring model and the joint scoring model is shown;

[0034] FIG5 shows a parsimonious graph generated by the scoring model during the performance evaluation process in the experiment; and

[0035] Figure 6 It is a block diagram of the joint scoring system. DETAILED DESCRIPTION

[0036] Federated learning, also known as distributed learning or distributed algorithms, can address issues associated with data sharing restrictions by collectively training algorithms without exchanging data. In the context of healthcare data, embodiments of the federated scoring systems and methods disclosed herein can protect patient privacy by distributing model training to data owners and aggregating results across individual data owners / sites. Federated learning breaks down data silos and allows for faster development of much-needed scoring systems / models for analysis of healthcare data and decision-making based on healthcare data. In addition, the federated system disclosed herein is able to federate preferred interpretable models in clinical research.

[0037] Figure 1 and Figure 6 The system architecture of joint scoring is shown. First, refer to Figure 6 , system 600 is used for joint scoring. System 600 includes multiple nodes 630 that communicate with each other via a communication network 620. As shown in nodes 630(1) and 630(N) (representing that there are N nodes in the system or network 600), each node includes at least one processor 632 and a memory 634. The memory 634 includes sensitive data 638 and program code 636. The sensitive data 638 may include multiple records, such as patient data records.

[0038] The program code 636 can be executed by the corresponding processor 632 of each node 630 to generate a first set of scoring model coefficients based on the sensitive data accessible to the corresponding node. In general, there will be two cases: the node 630 can only access the sensitive data 638 stored in its own memory 634; nodes from a group of affiliated nodes (for example, a group of clinics operated by a particular company) can access sensitive data stored on all affiliated nodes. Regarding the coefficients, the example model used involves logistic regression, which is logit(Pr(Y=1|X))=β0+β1X1+β2X2+...Here p is the probability of responding Y=1 (Y can only be 0 or 1), and X1, X2... are predictors, such as gender, age, etc. Since there may be multiple predictors (X), the visualization can be multidimensional instead of two-dimensional. logit(Pr(Y=1|X)) will be the y-axis, but there can be multiple x-axes). So, the coefficient is the beta coefficient.

[0039] The instructions stored in the memory 634 also cause the processor 632 to broadcast the first set of scoring model coefficients to the remaining nodes 630 .

[0040] Each node 630 is also configured to generate a node-specific scoring model based on the first set of scoring model coefficients.

[0041] At least one of the nodes 630 is configured to receive contributing intermediate model statistics (also referred to as contributing intermediate statistics, etc.) from the remaining nodes 630 and discussed below. The node or nodes then generate a joint scoring model based on these intermediate statistics and their respective data. For example, the intermediate statistics are a component of equation (5) for any particular node.

[0042] Each of the plurality of nodes 630 is configured to transmit its node-specific scoring model and scoring model performance data to the central server 610. Since at least one node 630 generates a joint scoring model, the central server 610 is also configured to receive the joint model from one or more nodes 630 that create such a joint scoring model. In order to ensure that a consistent model is implemented on some or all nodes 630, the central server 610 is configured to transmit the joint model to at least a subset of the plurality of nodes 630. In the case where the central server 610 receives multiple joint scoring models, the central server 610 can select a specific joint scoring model to be deployed on all nodes 630. Model selection can be performed through a minimalist graph as shown in FIG5. In some embodiments, this step involves user selection, and in other embodiments, the selection is automatically performed based on the general criterion that adding new variables to the model does not increase the model performance above a predetermined threshold (an increase in the AUC value). The predetermined threshold can be set by the user. Variables can also be added based on domain common sense-this can be accessed by using natural language processing on publications in the relevant field to identify the most frequently mentioned variables, or based on the user's domain common sense (i.e., variables added by the user). These additional variables may not necessarily be completely consistent with the minimalist graph. For example, suppose in the parsimony plot, the model performance has reached 0.85 using the first five variables (high enough to stop adding new variables), one can include another variable based on domain common sense or publications in the field.

[0043] Some embodiments combine scoring systems across various nodes of a computer system. The scoring system can be a classification model that includes the definition of a series of calculations on input data that are performed to make predictions based on the input data. The scoring model can be implemented using various types of models, such as linear classification models, logistic regression models, and clinical decision support models. This series of calculations includes addition, subtraction, multiplication, and other calculations. These models are used to assess the risk of many serious diseases because they provide effective and interpretable predictions.

[0044] Table 1 shows examples of scoring systems.

[0045]

[0046] Table 1: Sleep apnea screening scoring system

[0047] Doctors can easily determine if a patient screens positive for obstructive sleep apnea by adding up the scores for the patient's age, diabetes, body mass index and gender. If the score is above the threshold, the patient will be referred to the clinic for diagnostic testing.

[0048] Traditional scoring systems are largely developed based on data from a single source. As a result, the training or sample data sets are often small or unrepresentative—for example, data from a wealthy neighborhood may have lower rates of adverse outcomes than data from a poor neighborhood. While it is possible to develop scoring systems on pooled data, the pooling process is time-consuming and difficult to achieve for privacy reasons. Figure 6 The framework shown is used to construct a scoring system in a joint manner to address these difficulties.

[0049] The disclosed systems and methods (also referred to as federated scoring) provide a method for building a federated scoring system that can be executed on multiple computer system nodes provided in various locations. These embodiments improve robustness and eliminate bias in medical research, especially research conducted with relatively small sample sizes. Figure 1 and Figure 6 The overall architecture of a system according to an embodiment is shown.

[0050] Joint Scoring Framework

[0051] When implementing clinical models and other models that require maintaining data privacy (such as financial records), users often consider parsimony as a key feature of the model. A model is considered parsimonious when it is sparse (i.e., it uses as few variables as possible) and has good predictive accuracy. Figure 6 A block diagram of a joint scoring system / framework 600 is shown. The joint scoring system of some embodiments may include a central server 610 and multiple nodes 630. Alternatively, specific nodes in the multiple nodes of the joint scoring system may perform the functions of the central server and the node or system / framework 600, thereby eliminating the need for a designated central server. As described above, each node 630 includes at least one processor 632 and a memory 634. The memory 634 includes program code 636 and sensitive data 638. The various nodes communicate with each other via a communication network 620. Sensitive data can only be accessed by the corresponding node.

[0052] In order to incorporate privacy requirements while achieving good parsimony and interpretability, the joint scoring framework consists of five modules: (1) joint variable ranking module; (2) joint variable transformation module; (3) joint score derivation module / scoring module; (4) model selection module and (5) model evaluation module. Depending on the architecture of the specific embodiment, some or all of these modules can be provided in a specific node, distributed between nodes or on a central server. Figure 6 In the illustrated embodiment, these modules are stored in memory 614 .

[0053] (1) Joint variable ranking module

[0054] In order to build a global model across multiple sites or nodes, some embodiments may pre-identify a set of unified variables as candidate variables so that ranking can be performed independently on each node 630. For example: Assume that sites A and B both use 0%, 25%, 50%, 75%, and 100% to cut variables. Due to data heterogeneity, their cutoff values ​​for the variable age may be different: A: (, 24], (24, 49], (49, 62], (62,) years old; B: (, 24], (24, 52], (52, 67], (67,) years old. This indicates that the population of site B is relatively old. In this case, a union cannot be performed because these categorical variables have different meanings and are not unified. Instead, when united, the two sites specify cuts that are reasonable for both sites, such as (, 24], (24, 50], (50, 60], ( 60,) years old, can then be joined. This unification may be due to collecting data from all contributing sites (such as the age of all patients) and separating based on statistical parameters such as percentiles as described above. In some embodiments, random forests can be used to perform variable ranking. In the joint scoring framework, variable ranking is first performed at each local site / node 630. Each node 630 then sends the variable ranking to the central server 610 which generates a global variable ranking. The global variable ranking can be generated by sorting the variables by the average ranking of each site.

[0055] The random forest of some embodiments may include a collection of random classification and regression trees. An important metric for a given variable in a random forest is the increase in the mean error of the tree when the observations of the particular variable are randomly permuted in out-of-bag samples. More specifically, the importance of a variable can be quantified in terms of the mean squared error for regression and the misclassification rate for classification. In a classification task, the Gini index for each node θ of a decision tree Θ is defined as:

[0056]

[0057] where p r is the node r th The fraction of training samples in the class, and for binary classification tasks, R = 2. The variable X m The importance of is the weighted total impurity reduction w(θ)ΔGini(θ) of all nodes. When averaged over all trees, this metric is calculated as:

[0058]

[0059] Where w(θ) is the proportion of samples arriving at node θ θ / N, ΔGini(θ) is the impurity reduction after the node θ splits, and v(θ) is the variable used in the split. In the joint scoring framework, variable ranking is first performed at each local site / node, and then a global variable ranking is created by rearranging the variables by their average rank at each site.

[0060] In some embodiments, the variable ranking at each node can be determined based on model parsimony statistics, such as a model parsimony graph, which shows the correlation of each variable or variable combination with the performance of the scoring model. In some embodiments, the area under the curve statistics of various models can be used to evaluate the variable ranking. The joint scoring model of some embodiments can consider variables above a predefined threshold (such as an importance threshold) in the global variable ranking. For example, assume that a random forest is used for importance measurement. After scaling the importance values ​​to 0 and 1, the following variable importance values ​​may be obtained: variable 1: 0.8, variable 2: 0.5, variable 3: 0.3, variable 4: 0.2, variable 5: 0.15, variable 6: 0.08... The threshold can be empirically set to 0.1, so that only the first five variables are selected. By doing so, the joint scoring model reduces the number of variables incorporated in the model and serves as a more interpretable model while maintaining its accuracy in a clinical setting-that is, the model becomes sparse and there is no significant loss in accuracy.

[0061] (2) Joint variable conversion module

[0062] After determining the global variable ranking, the joint scoring framework also converts continuous variables into categorical / discrete variables. For example, the age of a person may be in bands such as 0 to 20 years, 20 to 30 years, 30 to 45 years, 45 to 60 years, and 60 years and above. The maximum number of categories for this conversion can be predetermined (e.g., 5), and if the maximum value for a particular variable is exceeded, the categories can be combined to meet the maximum requirement. The global cutoff value / discrete interval for each continuous variable is calculated by averaging the k values ​​obtained for each site. The k value is used to cut the continuous variable into several categorical variables. For example, assuming that the variable age has only one 50% k value, and the 50% cutoff value of age is 50 years, the age variable will be converted from a continuous variable to two categorical variables: age (≤50) (true or false) and age (≥50) (true or false). After defining the discrete interval / cutoff value for each variable, transmitting the defined cutoff value to multiple nodes enables the nodes to process the continuous variable data in a uniform and standardized discrete manner. In some embodiments, the quantiles of the continuous variables are set to 0%, k1%, k2%, k3%, k4% and 100%, where the values ​​of k1, k2, k3, k4 are set to 5, 20, 80 and 95. By providing standardized discrete intervals for the continuous variables, the joint variable transformation improves the accuracy of the joint because the individual nodes observe a common set of discrete intervals when providing input to their respective scoring models.

[0063] (3) Joint scoring derivation module

[0064] Depending on clinical needs and context, the score derivation process can be flexibly adjusted to suit different clinical modeling purposes by combining appropriate machine learning models. For example, a logistic regression model may only support binary outcomes. By converting the logistic regression in module 3 to other suitable models, the framework can be extended to support survival outcomes and ordinal outcomes, etc. (i.e., non-binary outcomes). One step in the score derivation process includes generating a first set of scoring model coefficients based on sensitive data. The scoring model coefficients include various parameters of the scoring model. For example, in an embodiment where the scoring model is a linear model, the coefficients are linear parameters and intercepts, etc. Each node generates its own first set of scoring model coefficients because each node has access to different sensitive data. In general, the data does not overlap. Each node uses the first set of scoring model coefficients to define a node-specific scoring model. The node-specific scoring model is used as a candidate model for comparison with the joint scoring model.

[0065] The first set of scoring model coefficients may be broadcast to the remaining nodes, and each node may generate a set of contribution / intermediate scoring model coefficients based on the received first set of scoring model coefficients and the sensitive data accessible to each respective node. The contribution scoring model coefficients may then be sent / broadcast to the remaining nodes. One or more nodes may generate a joint scoring model based on their first set of scoring model coefficients and the contribution scoring model coefficients received from the remaining nodes. An example of joint scoring model generation is described with reference to the ODAL2 algorithm.

[0066] As another example, logistic regression is a common choice for modeling binary outcomes.Joint logistic regression can be implemented by some embodiments that require multiple iterations of logistic regression or logistic regression over one iteration (a one-shot approach).

[0067] The federated score is a privacy-preserving framework that provides a unified and robust scoring system across multiple sites without sharing sensitive data, such as sensitive medical data or other personal information. The federated score was tested using a clinical scoring model for 30-day mortality prediction using the emergency department data of Singapore General Hospital and simulations of 10 nodes / sites that did not exchange sensitive patient data during the experiment. The robustness and generalizability of the federated score were established by achieving a higher average area under the curve on the test data for each site with the minimum variance compared to the baseline score.

[0068] Experiments were conducted using ODAL2, a one-shot privacy-preserving distributed algorithm, for performing federated logistic regression as disclosed in “Learning from Electronic Health Records at Multiple Sites: A Communication-Efficient and Privacy-Preserving Distributed Algorithm,” by R. Duan et al., JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION, Vol. 27, No. 3, pp. 376-385, December 2019, DOI: 10.1093 / jamia / ocz199. An embodiment utilizes information from a local site / node with first-order (ODAL1) (first set of scoring model coefficients) and second-order (ODAL2) gradients (contributing scoring model coefficients) of the likelihood function from remote sites to construct an approximation of the global likelihood function that forms part of the federated scoring model. The data of the remote sites / nodes is not accessible when performing the logistic regression calculations. During the logistic regression process, the coefficients generated by each node are transmitted to a central server.

[0069] The coefficients in the global logistic regression model are generated by optimizing the likelihood function and then rounded to obtain a score based on each variable. A central server defines a scoring table to calculate the total score by adding all the scores. Both the upper limit number of the total score and the standardization of the score subdivisions can be adjusted to meet the needs of the intended clinical application.

[0070] The coefficients of the global logistic regression model can be obtained by optimizing the global likelihood function. Let x1, x2, ...x P-1 represents p-1 predictors, y represents the binary outcome, and the logistic regression model can be expressed as

[0071] logit(Pr(y=1|x))=x T β (3)

[0072] Where x=(1, x1, x2, ... x p-1 ) T , β is the vector of intercepts and coefficients, and logit(t) = log t / (1-t). Assume that there are If there are K identical and independently distributed observations at K sites / nodes, then the likelihood function (log likelihood ratio) of the global logistic regression obtained by pooling the data from all sites is

[0073]

[0074] By optimizing L(β), we can obtain the combined estimate

[0075] However, when data cannot be shared, it is impossible to calculate the pooled likelihood function. As contemplated by the embodiments, the approximation of the likelihood function is performed as described herein. For example, the ODAL2 algorithm applies the idea of ​​Taylor expansion and proposes to use the first and second order gradients of the LLR (log likelihood ratio) for approximation:

[0076]

[0077] here, is an initial value, is the LLR of the jth site (assuming local site j = 1), is the log-likelihood ratio exist The first gradient evaluated at is the first gradient of the log-likelihood ratio at site j, where

[0078] and is the second gradient of the log-likelihood ratio for site j.

[0079] When executing the ODAL2 algorithm, the initial value The (first set of scoring model coefficients) are first obtained from the results of the local logistic regression performed at each local site / node, for example based on the data represented in Table 1. Then, The joint scoring model is generated based on the received contribution scoring model coefficients and the node's first set of scoring model coefficients. After receiving the broadcasted first set of scoring model coefficients, the site / node can calculate To construct an alternative likelihood, which forms part of the joint scoring model. Depending on the configuration of the joint scoring framework, the joint scoring model can be generated by one or more nodes or a central server. In some embodiments, the joint scoring model can also take into account the global variable ranking generated by (1) the joint variable ranking module. For the sake of model simplicity and interpretability, variables with lower rankings may be discarded by the joint scoring model.

[0080] Global Beta Estimate is obtained by optimizing the alternative likelihood function. In some embodiments, Figure 1 As shown in Figure 1, two JSON files "site_beta.json" and "intermediate.json" are used to store the values ​​that need to be broadcast in this process. The information in these two shared files is aggregated and does not contain any patient-level information, which ensures data privacy. Part of the process of generating a joint scoring model is shown in Figure 1. Figure 1 shown.

[0081] In some embodiments, the coefficients in the joint logistic regression model (joint scoring model) are rounded to obtain the relevance score for each variable. A scoring table is created and a total score is calculated by adding all scores together. The upper limit number of the total score and the standardization of the score subdivisions can be adjusted to meet the needs of the intended clinical application.

[0082] (4) Model selection module

[0083] The scoring model trained at each node is a candidate that can be adopted by any or all other nodes to obtain the most accurate results. Model evaluation and selection can be performed using a parsimony graph generated by averaging the area under the curve over all sites / nodes. The parsimony graph represents the performance of the scoring model as a function of the number of variables incorporated into the model. Let i denote the site / node index, where i∈{1,2,...K}. A general model selection criterion can be obtained by maximizing Ψ m =∑w i φ i (p1, p2, p3, ...p m ) is defined, where w i is the weight of site i, φ iis the performance of the scoring model on the i-th validation set, and m is the predefined number of total variables to be included, which may be uniform across all sites. In some embodiments, the weights may be defined as = means that all sites are equally weighted. However, if the scoring performance on this site is considered more important than other sites, it is flexible to assign

[0084] It is also possible to add different constraints to the maximization task. For example, the total number of variables m cannot exceed the integer N. The variable set {p1, p2, ... p m} may also be constrained to meet predefined criteria required by the system. For example, the system of some embodiments may be configured to include a set of variables {x1, x2, ..x q}, where q≤m. Furthermore, Ψ can be maximized using a number of d variables less than m, as long as increasing the number of variables from d to m has an effect on Ψ: m -ψ d The change of |≤∈ has an acceptable impact, where the size of ∈ can be determined intuitively by the user.

[0085] The final selection of variables can be confirmed based on the joint model selected from the multiple scoring models at each node. The new scoring model can be refitted to any new data using step (2). The performance of the selected joint scoring model is verified on each test dataset of each site participating in the joint learning process. The selected joint scoring model can be transmitted to each node to allow the node to process new clinical data using the joint scoring model.

[0086] Notably, to maintain parsimony without losing accuracy, intercorrelated variables can be identified—e.g., variables that have essentially the same effect on patient outcomes and vary greatly in proportion to each other (e.g., within 5% of each other)—for example, for a particular sex, weight and height may vary greatly in proportion to each other (e.g., within 5%). Among the intercorrelated variables, the system can select the best variable for the joint model (e.g., the variable among the intercorrelated variables that is most strongly associated with an outcome such as 30-day mortality) based on each node capturing that variable in its sensitive data or capturing a different variable that is intercorrelated with the best variable. Each node that does not capture the best variable can then replace the relevant intercorrelated variable in the local model.

[0087] The resulting joint scoring model is interpretable. Unless the context dictates otherwise, interpretability means that the correlation between sensitive data and outcomes is explicit and can be explained from the model—for example, according to Table 1, age is strongly associated with sleep apnea in people over 60 years old. This is in contrast to machine learning models, which use hidden layers to identify features in the data, so the correlation between data and outcomes may not be obvious.

[0088] (5) Model evaluation module

[0089] Use the test datasets of each site participating in the federated scoring framework to verify the performance of the federated scoring model and / or the node-specific scoring model. Depending on the configuration of the federated scoring framework, model evaluation can be performed by designated nodes or a central server. The Ψ defined in step (4) m Afterwards, the overall average performance of the joint scoring model can be defined as: where μ i is the performance of the scoring model on the i-th test set; and As a measure of the variation in performance across sites, higher M1 values ​​and lower M2 values ​​indicate better performance and generalization of the score.

[0090] Clinical study design

[0091] A retrospective analysis was performed using emergency health record data from Singapore General Hospital extracted from the Singapore Health Electronic Health Intelligence System. An initial study cohort of 86,527 admissions was identified by selecting emergency department admissions to Singapore General Hospital between 2016 and 2017. After excluding patients under 18 years of age and those with missing values, a total of 80,613 admissions were included, which were then randomized into 10 mock sites with proportions of 4%, 5%, 7%, 9%, 10%, 11%, 12%, 13%, 14%, and 15%. Figure 2 Data partitioning for experiments using a machine learning model federation execution system is shown.

[0092] The outcome of the study was 30-day mortality, defined as death occurring within 30 days of emergency department admission. Candidate predictors included a total of 29 variables: demographic information, PACS triage category, shift time, day of the week (Friday, Monday, weekend, midweek), vital signs, comorbidities, and previous healthcare utilization, among others.

[0093] result

[0094] The analysis was performed for three groups (1) training ten local scores independently at each site (2) training a joint score using all sites without data sharing (3) generating a pooled score using pooled data, which is an ideal situation but is often not possible in real-world applications. Models were selected based on their corresponding parsimony plots, with a predefined criterion that the maximum number of variables in the model should not exceed 10 and that adding more variables should stop when there is no significant improvement in the AUC. The variables selected for each model are included in the footnotes of Table 2. Figure 5 shows a series of parsimony plots obtained for various scoring models. Panels (a)-(j) relate to local models generated at sites 1 to 10. Panel (k) relates to a joint scoring model generated at any one site / node. Panel (l) relates to a scoring model generated using pooled data, which is otherwise not feasible in real-world settings.

[0095] A total of 12 scoring models were obtained and tested, including 10 local models generated independently at each site, a joint model developed by federated learning, and a pooled model generated based on the pooled data of all sites, which is impossible in the real world. Table 2 lists the AUC values ​​and confidence intervals of each model on the test data of different sites. Figure 3 The AUC values ​​and 95% confidence intervals for each score are plotted separately in order to better illustrate the results of the model performance variation across all sites in Table 2. The mean and standard deviation of the AUC values ​​for each model in all 10 test sets were also calculated and listed in Table 2. Figure 4 The information is described accordingly. Figure 3 and Figure 4 As shown, the joint scoring model outperformed all local scores in terms of stability and generalizability, achieving the lowest standard deviation. The joint scoring also showed a satisfactory average AUC (area under the curve) for each site, indicating that the existing joint scoring framework can generate trustworthy global clinical scores. The bottom row of Table 2 shows the average AUC values ​​of all local models for each site, which are mainly exceeded by the AUC values ​​of the joint scoring. This shows that for a single site, the joint scoring has the potential to produce a better scoring system than the locally developed one, especially when the sample size of the local site is insufficient.

[0096]

[0097] Table 2: Performance comparison between the joint scoring model and the baseline model

[0098] In Table 2, the following abbreviations are used: AUC, area under the curve; CI, confidence interval; SD, standard deviation; SBP, systolic blood pressure; DBP, diastolic blood pressure; SpO2, oxygen saturation measured by pulse oximetry; PACS, patient acuity classification scale; ED, emergency department. In addition: Local model obtained by AutoCore on site 1 a ; Variables selected in the model (in rank order): pulse, age, systolic blood pressure, diastolic blood pressure, oxygen saturation measured by pulse oximetry, respiration, day of the week, emergency department admissions in the past year; Local model obtained via AutoCore on site 2 b ; Variables selected in the model (in rank order): systolic blood pressure, diastolic blood pressure, pulse, age, oxygen saturation measured by pulse oximetry, respiration, emergency department admission in the past year; Local model obtained via AutoCore on site 3 c ; Variables selected in the model (in rank order): pulse, age, systolic blood pressure, diastolic blood pressure, oxygen saturation measured by pulse oximetry, respiration, emergency department admission in the past year; Local model obtained via AutoCore on site 4 d ; Variables selected in the model (in rank order): pulse, systolic blood pressure, age, diastolic blood pressure, oxygen saturation measured by pulse oximetry, respiration, emergency department admission in the past year, PACS triage category; Local model obtained via AutoCore on site 5 e ; Variables selected in the model (in rank order): systolic blood pressure, pulse, diastolic blood pressure, age, oxygen saturation measured by pulse oximetry, respiration, day of the week, emergency department admission in the past year, shift length, PACS triage category; Local model obtained via AutoCore on site 6 f ; Variables selected in the model (in rank order): systolic blood pressure, pulse, age, diastolic blood pressure, oxygen saturation measured by pulse oximetry, respiration, emergency department admission in the past year; Local model obtained via AutoCore on site 7 g ; Variables selected in the model (in rank order): systolic blood pressure, pulse, diastolic blood pressure, age, oxygen saturation measured by pulse oximetry, respiration, emergency department admission in the past year; Local model obtained via AutoCore on site 8 h; Variables selected in the model (in rank order): systolic blood pressure, pulse, diastolic blood pressure, age, oxygen saturation measured by pulse oximetry, respiration, day of the week, emergency department admissions in the past year; Local model obtained via AutoCore on site 9 i ; Variables selected in the model (in rank order): pulse, systolic blood pressure, age, diastolic blood pressure, oxygen saturation measured by pulse oximetry, respiration, day of the week, emergency department admissions in the past year, shift length, PACS triage category; Local model obtained via AutoCore on site 10 j ; Variables selected in the model (in rank order): systolic blood pressure, pulse, age, diastolic blood pressure, oxygen saturation measured by pulse oximetry, respiration, emergency department admission in the past year, day of the week; Joint model obtained by joint score k ; Variables selected in the model (in rank order): systolic blood pressure, pulse, age, diastolic blood pressure, oxygen saturation measured by pulse oximetry, respiration, emergency department admission in the past year, day of the week, shift duration, PACS triage category; Pooled model obtained by AutoCore l ; Variables selected in the model (in rank order): systolic blood pressure, pulse, age, diastolic blood pressure, oxygen saturation measured by pulse oximetry, respiration, day of the week, emergency department admission in the past year.

[0099] Considering that each scoring model can be modified or replaced to adapt to different clinical research questions, the framework provided by the embodiment is extensible and flexible. For example, the score derivation module can be adjusted to generate sequential results. Joint scoring fills a gap in existing medical machine learning applications, which lack established methods to promote unified scoring across multiple sites. It also addresses the lack of repeatable benchmark methods, especially for more interpretable models.

[0100] Compared with black box techniques such as deep learning-based models, the scoring models generated by the embodiments are easier to interpret due to the tendency to use fewer variables and a more simplified model structure. Traditional deep learning techniques also require a larger amount of data to generate more accurate models. In contrast, the disclosed joint scoring system allows multiple nodes / sites to work towards a joint scoring model without sharing confidential data. Since each site can access a limited amount of data, if each site pursues a deep learning-based model, the results of each site may not be optimal due to the limited amount of data. The joint scoring system solves this challenge by providing a more interpretable scoring model and generating a joint scoring model, which solves the constraints associated with black box machine learning models (such as deep learning models). In addition, the scoring model trained based on data from multiple sites is more universal because there is a diversity of data sources for these models to be trained as a whole. Therefore, the joint scoring model generated by the embodiment is more suitable for new settings, such as data from new clinical settings or data from different populations. The disclosed joint scoring framework can be used as the basis of a data science software platform that handles large-scale, multi-center data analysis and risk score development.

[0101] Reference in this specification to any prior publication (or information derived therefrom) or any known matter is not, and should not be taken as, an admission, acknowledgement or any form of suggestion that the prior publication (or information derived therefrom) or known matter forms part of the common general knowledge in the technical field to which this specification relates.

[0102] In this specification and the claims that follow, unless the context requires otherwise, the word "comprise" and variations such as "comprises" will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps.

[0103] The scope of the present disclosure includes all changes, substitutions, variations, alterations, and modifications to the example embodiments described or shown herein that will be understood by those of ordinary skill in the art. The scope of the present disclosure is not limited to the example embodiments described or shown herein. In addition, although the present disclosure describes and illustrates the various embodiments herein as including specific components, elements, features, functions, operations, or steps, any of these embodiments may include any combination or permutation of any components, elements, features, functions, or steps described or illustrated anywhere herein that will be understood by those of ordinary skill in the art. Although the present disclosure describes or illustrates specific embodiments that provide specific advantages, specific embodiments may not provide any, some, or all of these advantages.

Claims

1. A system for joint scoring, the system comprising: Multiple nodes, A communication network that enables communication between multiple nodes. Each node includes at least one processor and memory, and the memory of each node includes sensitive data and program code. Sensitive data includes multiple records; wherein the program code is executed by a corresponding processor of each of the plurality of nodes to: generating a first set of scoring model coefficients based on the sensitive data accessible to each node; Broadcast the first set of scoring model coefficients to the rest of the nodes; And wherein at least one node is configured as: Receive contributing intermediate statistics from the remaining nodes; A joint scoring model is generated based on the received contribution intermediate statistical data and its corresponding first set of scoring model coefficients.

2. The system according to claim 1, characterized in that Each node is configured to generate a node-specific scoring model based on a first set of scoring model coefficients.

3. The system according to claim 2, characterized in that The system also includes a central server, wherein the central server is configured to evaluate each node-specific scoring model and the joint scoring model based on model parsimony statistics.

4. The system according to claim 3, characterized in that Each record includes multiple variables and each node is also configured as: determining a ranking of each variable in the plurality of variables based on a correlation of each variable with a scoring result generated by a corresponding scoring model; sending the ranking of each of the plurality of variables to a central server; Among them, the central server is configured as: defining a global variable ranking based on rankings of a plurality of variables received from a plurality of nodes; The global variable ranking is transmitted to at least one of the plurality of nodes.

5. The system according to claim 4, characterized in that The association of each variable with the score outcome was assessed based on model parsimony statistics or area under the model curve statistics.

6. The system according to claim 4 or 5, characterized in that: The joint scoring model is generated based on the global variable ranking.

7. The system according to claim 6, characterized in that The joint model is generated by merging variables above a threshold in the global variable ranking.

8. The system according to any one of claims 4 to 7, characterized in that The node determines the ranking of each of the multiple variables using a random forest model.

9. The system according to any one of claims 4 to 8, characterized in that The central server defines a global variable ranking by averaging the rankings of the plurality of variables received from each of the plurality of nodes.

10. The system according to any one of claims 1 to 9, characterized in that The scoring model is implemented using any of the linear classification model, logistic regression model, and clinical decision support model.

11. The system according to any one of claims 1 to 10, characterized in that Each of the plurality of nodes is configured to transmit its node-specific scoring model and scoring model performance data to the central server.

12. The system according to claim 11, characterized in that The central server is configured to receive the federated model from at least one of the nodes.

13. The system according to claim 10, characterized in that The central server is configured to transmit the federated model to at least a subset of the plurality of nodes.

14. The system according to any one of claims 1 to 13, characterized in that The variables include one or more continuous data variables, and each node is further configured to convert the continuous data variable into a discrete variable.

15. The system according to any one of claims 1 to 14, characterized in that At least one of the nodes is configured to process new clinical data using the joint model to generate a score.

16. The system according to any one of claims 1 to 15, characterized in that The contributing intermediate statistics are calculated by each corresponding node based on the sensitive data accessible to the corresponding node.

17. A method for joint scoring, comprising: Provide multiple nodes, each node includes: at least one processor and memory, The storage of each node includes sensitive data and program code, and the sensitive data includes multiple records; Provide a communication network to enable communication between multiple nodes, The program code is executed by a respective processor of each of the plurality of nodes to: generating a first set of scoring model coefficients based on the sensitive data accessible to each node; Broadcast the first set of scoring model coefficients to the rest of the nodes; Program code is executed by at least one of the nodes to: Receive contribution score intermediate statistics from other nodes; and A joint scoring model is generated based on the received contribution scoring intermediate statistics and their corresponding first set of scoring model coefficients.

18. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of claim 17.

Images