Method and system for joint learning of machine learning models

Abstract

Methods and systems for joint learning of machine learning models are disclosed, embodiments of which implement the steps of receiving (S10), at a model aggregator device (MAD), a local update (ML ') of a machine learning model (ML) and a parameterization (P) of local data (LTD) from a local site (LS) remote from the model aggregator device (MAD), the local update (ML ') is generated at the local site (LS) based on the local data (LTD); generating (S20), at the model aggregator device (MAD), a synthetic representation (SR) of the local data (LTD) based on the parameterization (P) using a generative AI function (GEN); evaluating (S30), at the model aggregator device (MAD), the local update (ML ') using the synthetic representation (SR) in order to obtain an evaluation result indicative of the performance of the local update (ML'); and updating (S40), at the model aggregator device (MAD), the machine learning model (ML) based on the evaluation result and the local update (ML ').

Description

Technical Field

[0001] Embodiments of the present invention relate to systems and methods for providing updated machine learning models in a distributed environment. In particular, embodiments of the present invention relate to federated learning of machine learning models to provide updated machine learning models. In particular, embodiments of the present invention relate to using updated machine learning models for medical data processing, and in particular for medical image data processing. Background Art

[0002] Machine learning methods and algorithms are widely used to generate insights and / or (predictive) computational models from data. Typically, data is brought into a processing unit (e.g., the cloud), which can run such methods and algorithms to train the models or generate insights. In this regard, machine learning methods have proven to be very versatile in a variety of application areas. For example, machine learning methods are used to support decision-making in autonomous driving. Similarly, the processing of physiological measurements (e.g., medical images) by automated systems to provide medical diagnoses relies on machine learning methods.

[0003] However, due to data privacy regulations, it's often impossible to introduce data to external processing units that can execute machine learning methods and algorithms but whose ownership differs from that of the data. In these cases, the data must always remain with the data owner. This situation often arises in healthcare, where the inherent sensitivity of patient data raises significant patient privacy concerns. Summary of the Invention

[0004] One way to solve this problem is to implement a distributed or federated learning solution. Here, a central machine learning model hosted at a model aggregator device (e.g., a central server unit) can be improved based on usage reported by many client units located at local sites. Thus, a central machine learning model that is easy to train, run, and deploy is distributed to local sites and executed locally. Each client can send local updates to the central server unit randomly, periodically, or on command. The local updates can summarize local changes to the machine learning model based on local data collected by the client. The model aggregator device can use the local updates to improve the machine learning model. In turn, the model aggregator device can upload to the client a modified machine learning model that has been learned based on the actual usage reported by the client. This enables clients to collaboratively learn and improve shared machine learning models without their local and potentially classified data being distributed outside the client unit.

[0005] One problem in this regard is that local updates must be evaluated or tested at the model aggregator device to determine whether they constitute an improvement. To do this, the model aggregator device needs to have test data on which to test modifications to the machine learning model. Typically, this test data is a fixed set that has been obtained in compliance with data privacy regulations. There are several disadvantages to centrally maintaining such a fixed set. First, such test data is difficult to obtain, and therefore the number of data instances is limited. Furthermore, there is no guarantee that centrally hosted test data represents real-world scenarios, and more importantly, such fixed data cannot adapt to concept drift that may occur in the field.

[0006] As an alternative, a decentralized approach has been proposed for evaluating models (e.g., see U.S. 2021 / 0 097439A1). Here, an aggregated model is sent to all clients to be evaluated on local test data. Each client downloads the model, performs inference, and then uploads the evaluation results. A central server aggregates the evaluation performance across all sites and then decides whether to retain or discard the model. This offers the advantage that the model is tested on real-world data, and the chance of overfitting is reduced because the test data changes over time. However, this approach also has disadvantages. First, clients need to reserve computing resources not only for training but also for testing. This may reduce the transition time between model update cycles and the actual use of the model at the client. Second, this inevitably leads to increased data traffic, as clients need to download the model and upload the results back to the model aggregator device. Third, local data will conceptually change over time. While this reduces the impact of overfitting, it can make it difficult to compare current models with historical models.

[0007] It is therefore an object of embodiments of the present invention to provide improved methods and systems for federated learning of machine learning functions. In particular, it is an object of embodiments of the present invention to provide systems and methods that allow for more efficient evaluation of machine learning functions in a distributed environment.

[0008] This object is solved by a method for federated learning of machine learning models, a system for federated learning of machine learning models, a corresponding computer program product and a computer-readable storage medium according to a main embodiment. Alternative and / or preferred embodiments are objects of dependent embodiments.

[0009] Hereinafter, the technical solution according to the present invention is described with respect to the claimed apparatus and with respect to the claimed method. Features, advantages, or alternative embodiments described herein may also be assigned to other claimed objects, or other claimed objects may be assigned to features, advantages, or alternative embodiments described herein. In other words, the embodiment of the method of the present invention may be improved by features described or claimed with respect to the apparatus. In this case, for example, the functional features of the method may be embodied by the target unit or element of the apparatus.

[0010] The technical solution will be described with respect to a method and a system for providing an updated machine learning function and also with respect to a method and a system for providing training or test data for updating a machine learning system. Features and alternative forms of implementation of the data structures and / or functions of the method and system for providing a machine learning function can be transferred to similar data structures and / or functions of the method and system for providing training or test data. In particular, similar data structures can be identified by the use of the prefix "training". Furthermore, the prediction function used in the method and system for providing information can in particular have been adjusted and / or trained and / or provided by the method and system for adjusting the prediction function.

[0011] According to one aspect, a computer-implemented method for federated learning of a machine learning model in a model aggregator device is provided. The method comprises a plurality of steps. One step is to receive, at the model aggregator device, a local update and a log file of the machine learning model from a local site remote from the model aggregator device, wherein the local update is generated (or provided) at the local site based on local data and the log file comprises a parameterization of the local data. Another step is to generate, at the model aggregator device, a synthetic representation of the local data based on the parameterization using a generative AI function. Another step is to evaluate the local update using the synthetic representation at the model aggregator device to obtain an evaluation result indicating the performance of the model update. Another step is to update the machine learning model at the model aggregator device based on the evaluation result and the local update.

[0012] According to another aspect, a computer-implemented method for federated learning of a machine learning model in a model aggregator device is provided. The method comprises a plurality of steps. One step is to receive, at the model aggregator device, a first local update of the machine learning model and a log file from a first local site remote from the model aggregator device, wherein the first local update is generated (or provided) at the local site based on local data and the log file comprises a parameterization of the local data. Another step is to generate, at the model aggregator device, a synthetic representation of the local data based on the parameterization using a generative AI function. Another step is to receive, at the model aggregator device, a second local update of the learned machine that is different from the first local update from a second local site remote from the model aggregator device and different from the first local site. Another step is to evaluate, at the model aggregator device, the second local update using the synthetic representation to obtain an evaluation result indicating the performance of the second model update. Another step is to update the machine learning model at the model aggregator device based on the evaluation result and the second local update.

[0013] The model aggregator device may be a central server unit that is configured to manage the federated learning of the machine learning model. For example, the model aggregator device may include a web server. In addition, the model aggregator device may include a cloud server or a local server. The model aggregator device may communicate data with one or more local sites. The model aggregator device may be configured to provide the machine learning model to the local site, and to obtain an updated machine learning model (local update) from the local site. The model aggregator device may also be configured to evaluate the local update, and to decide the integration of the features of the local update in the machine learning model based on the evaluation step. The model aggregator device may include an interface unit for facilitating data communication with the local site (e.g., via an Internet connection).

[0014] A local site may be conceived as a client or client unit in a federated learning network managed by a model aggregator device. In particular, a local site may comprise a local computer network comprising one or more computing units. A local site may, for example, be associated with an organization in which a machine learning model is to be deployed. In particular, a local site may be associated with a healthcare environment, organization, or facility, such as a hospital, a laboratory, a workplace, a university, or a consortium of one or more of the aforementioned. According to some examples, the model aggregator device is located outside of the local site and serves one or more of the local sites from the outside.

[0015] The machine learning model can be thought of as the master model in a federated learning scheme that is centrally managed by a model aggregator device.

[0016] Typically, a machine learning model is configured to provide a desired or predetermined type of output by processing a certain type of input data. Thus, a machine learning model mimics the cognitive functions of humans and other human minds. In particular, by training based on training data, a machine learning function can adapt to new environments and can detect and infer patterns. Other expressions for a machine learning model can be a trained function, a trained machine learning model, a trained mapping specification, a mapping specification with trained parameters, a function with trained parameters, an artificial intelligence-based algorithm, or a machine learning algorithm.

[0017] Typically, the parameters of the machine learning model can be adjusted through training to obtain model updates (e.g., in the form of local updates of the machine learning model at the model aggregator device or central updates). In particular, supervised training, semi-supervised training, unsupervised training, reinforcement learning, and / or active learning can be used. In addition, representation learning can be used. In particular, the parameters of the machine learning model can be iteratively adjusted through several training steps.

[0018] In particular, the machine learning model may include a neural network, a support vector machine, a decision tree, and / or a Bayesian network, and / or the trained function may be based on k-means clustering, Q-learning, a genetic algorithm, a transformer network, and / or association rules. In particular, the neural network may be a deep neural network, a convolutional neural network, or a convolutional deep neural network. Furthermore, the neural network may be an adversarial network, a deep adversarial network, and / or a generative adversarial network. Furthermore, the neural network may include a transformer network.

[0019] The local data may include (training) input data and, optionally, corresponding (training) output data. The training output data may be data that the machine learning function is expected to produce based on the input training data. The training output data may include a verified output. According to some examples, the verified output may be verified by a (human) expert at the local site. Notably, for unsupervised learning, output training data is not required.

[0020] The local data, or portions of the local data, may be subject to data protection obligations that restrict the transmission of training data outside the local site. Thus, according to some examples, the local data cannot be accessed from outside the local site. In particular, the local data cannot be accessed by the model aggregator device.

[0021] According to some examples, local data may be related to medical data of one or more patients. For example, local data may include laboratory test results and / or pathology data derived from pathology imaging and / or medical image data generated by one or more medical imaging devices (e.g., computed tomography, magnetic resonance imaging, angiography (or C-arm X-ray) systems, positron emission tomography systems, etc., and any combination thereof). In addition, local data may include supplementary information related to the patient, such as diagnostic reports, information about treatments administered, information about symptoms and treatment responses, health progress, etc. Such information may be provided, for example, via an electronic medical record (EMR). Local data may be stored locally in one or more databases at the local site. The database may be part of a hospital information system (HIS), a radiology information system (RIS), a clinical information system (CIS), a laboratory information system (LIS), and / or a cardiovascular information system (CVIS), a picture archiving and communication system (PACS), etc. From these databases, local data can be accessed locally to train machine learning models (and to periodically use machine learning models later after deployment). Local data may be subject to data privacy regulations that may prohibit local data from leaving the local site. The local data may specifically include a dataset with which the machine learning model can be trained, validated, and tested. According to some examples, the local data may include a dataset based on which local updates have been validated and / or tested. In other words, the local data may not include training data based on which actual further training of the machine learning model is performed. The dataset may include input data and associated output data, which can be used to evaluate the performance of the machine learning model during supervised learning. The output data may be a validation result corresponding to the input data. The output data may be generated and / or validated by a human based on the input data.

[0022] According to some examples, the local data includes multiple individual data items. Thus, each data item may include a training input data item and, optionally, a corresponding training output data item. For example, the training input data item may relate to a respective medical image dataset of a patient, and the training output data item may relate to a corresponding detection result. Thus, the local data may be conceived as a collection of multiple data items.

[0023] According to some examples, a local update of a machine learning model may include a machine learning model in which one or more parameters of the machine learning model have been adjusted (or changed or optimized), in particular based on local data. In particular, the one or more adjusted parameters may include one or more adjusted weights and / or adjusted hyperparameters of the machine learning model. "Generated at a local site based on local data" may include that the local update has been trained and / or validated and / or tested using local data. Specifically, the local data may be divided into training data, validation data, and test data. For the actual training (in the sense of adjusting the machine learning model to generate the local update), a backpropagation scheme may be used based on an appropriate cost function and using the training data. Based on the validation data, the best performing local update among several local updates may be selected at the local site. The specificity and sensitivity may then be determined at the local site based on the test data. According to some examples, the specificity and sensitivity may be included in a log file.

[0024] According to some examples, parameterization can be conceived as data minimization of the original local data. According to some examples, the parameterization can have a reduced information depth, in particular a reduced data size, when compared to the local data. The parameterization can be not subject to data protection obligations. The parameterization can include one or more parameters, such as numerical values or semantic expressions that characterize the local data. According to some examples, the parameterization does not include (original) local data and / or (only) excerpts of the local data. In particular, the parameterization can be configured to enable reconstruction of at least parts of the local data, in particular those parts that are not subject to data protection obligations.

[0025] According to some examples, the parameterization may be based on training input data or training input data and training output data. According to some examples, the parameterization may be based on (only) training output data. This is because the expected output encoded in the local training output data may already reflect the content of the training input data at a high level and may therefore provide a good basis for generating a synthetic representation. According to other examples, the parameterization may be based on the training output data as well as a corresponding excerpt of the training input data. To provide an example, the parameterization may include a description of a medical finding (which would be the training output data) and a fragment of a medical image dataset illustrating the medical finding.

[0026] To provide an example from the field of medical imaging, parameterization may not include the complete image data of a medical image dataset, but only certain key parameters. For example, this may include the imaging parameters and imaging modality used, image quality metrics, descriptions and locations of findings, image segments, etc. Notably, it may not include any data based on which a patient could be identified. This could obviously be demographic information about the patient, but it could also be more subtle information that could lead to the identification of, for example, the body shape or implants visible in the medical image.

[0027] According to some examples, the synthetic representation can be conceived as a recreation or reconstruction of the original local data based on parameterization. According to some examples, the synthetic representation is not subject to data protection obligations. However, according to some examples, the synthetic representation can include relevant features for evaluating and / or training machine learning models.

[0028] A generative AI function is a machine learning function or model configured to generate text, images, or other data based on input data. According to some examples, the input may be a parameterization, or a natural language prompt generated based on the parameterization. In other words, the generative AI function exchanges the parameterization for a synthetic representation. According to some examples, the generative AI function may include a transformer network, in particular a transformer-based (deep) neural network. According to some examples, the local data may include image data, the parameterization does not include image data, and the generative AI function is a parameterized-to-image function that outputs a synthetic version / representation of the image data. According to some examples, the generative AI function includes a visual transformer as described herein.

[0029] Generative AI functions can be trained based on paired parameterizations and corresponding data, particularly including image data. Thus, the parameterizations may have been extracted from the corresponding data in the same or similar manner as parameterizations would be extracted from local data. During training of the generative AI function, the corresponding data can serve as the ground truth against which the output of the generative AI function is compared.

[0030] According to some examples, generative AI functionality may include a transformer network. A transformer network is a neural network architecture that typically includes an encoder, a decoder, or both an encoder and a decoder. In some cases, the encoder and / or decoder are composed of several corresponding encoding and decoding layers, respectively. Within each encoding and decoding layer is an attention mechanism. The attention mechanism—sometimes referred to as self-attention—correlates data elements (e.g., words or pixels) within a series of data elements with other data elements within the series.

[0031] For a review of transformer networks, see Vaswani et al., “Attention Is All You Need,” arXiv:1706.03762, June 12, 2017, which is incorporated herein by reference in its entirety.

[0032] According to some examples, off-the-shelf generative AI functions such as DALL-E or Midjourney can be used. According to other examples, custom generative AI functions based on transformer architectures can be used, which are trained based on paired parameterizations and corresponding (real) data.

[0033] According to some examples, evaluation (another word is testing) can include testing whether the local update can perform well enough on unseen data. According to some examples, evaluation can include having the machine learning model process the synthetic representation and / or other test data available at the model aggregator device and comparing the results of the processing with expected results. The other test data can include one or more synthetic representations obtained from parameterization received from one or more local sites different from the local site.

[0034] The features work together to provide a synthetic representation of the local data to the model aggregator while ensuring that the data protection requirements of the local site are met. In addition, data minimization also reduces the amount of data that needs to be transmitted, which reduces latency in the system. This allows for more efficient federated learning workflows in distributed environments.

[0035] According to some examples, the step of updating the machine learning model at the model aggregator device includes aggregating local updates in the machine learning model. In this way, the master model can be continuously optimized.

[0036] According to some examples, the aggregation may include taking over one or more update parameters of the local update in the machine learning model. According to some examples, the update may be performed if the evaluation results indicate improved performance of the local update.

[0037] According to some examples, the parameterization is generated by applying a trained feature encoder to the local data. According to some examples, the feature encoder is provided to the local site by a model aggregator device. According to some examples, the parameterization includes encoding features identified by the feature encoder based on the local data. According to some examples, the encoder is different from and / or independent of the generative AI functionality. This may mean that the encoder has been trained independently of the generative AI functionality and / or includes a different architecture. According to some examples, generating the parameterization includes applying the feature encoder to each individual data item of the local data to generate a set of encoding features for each data item, and appending each set of encoding features to a parameterization / log file.

[0038] According to some examples, the log file may include a performance log of the updated machine learning function. The performance log may indicate how well the locally updated machine learning model performs on the local data at the local site. According to some examples, the step of evaluating includes additionally evaluating the local update based on the performance log. This allows for an improved evaluation of the local update. Alternatively, parameterization may be provided so that it is not included in the log file. In this case, no log file is generated / sent.

[0039] According to some examples, the log files are formatted according to the JSON standard. JSON stands for JavaScript Object Notation and is an open standard file format and data interchange format that uses human-readable text to store and send data objects consisting of attribute-value pairs and arrays. This improves interoperability of methods.

[0040] According to an aspect, parameterization includes parameterization of data for validating and / or testing the local update at the local site.

[0041] In other words, only the parameterization of the data used to verify and / or test the local updates is generated / sent. According to this aspect, the actual training data used to fine-tune (in the sense of adjusting) the machine learning model is not parameterized and / or no parameterization of such local data is sent. Therefore, the parameterization does not include the parameterization of the actual training data. Therefore, the synthetic representation is (only) a synthetic reconstruction of the dataset used to test and / or verify the local updates, and not a synthetic reconstruction of the training dataset.

[0042] This may have the advantage that bias may be reduced since the models are not systematically evaluated at the model aggregator device based on the information with which they have been trained.

[0043] According to one aspect, the generative AI functionality is configured to generate a synthetic representation based on a natural language prompt indicating the synthetic representation to be generated, and the generating step includes obtaining the natural language prompt based on parameterization and inputting the natural language prompt into the generative AI functionality to generate the synthetic representation.

[0044] Hints can be thought of as natural language descriptions of the synthetic representations to be generated.

[0045] Using prompts has the advantage that the method is easily compatible with off-the-shelf generative AI functions that typically require prompts as input.

[0046] According to some examples, the generation step includes inputting prompts along with parameterization. This has the advantage of providing the generative AI function with additional information for simulating synthetic representations.

[0047] According to some examples, prompts can be generated by a parser module configured to generate prompts based on parameterization. The parser can include a language decoder configured to generate natural language text based on the parameterization. The language decoder can include a transformer network. By using the parser module, workflows can be further automated and rendered more efficiently.

[0048] According to some examples, the method also includes modifying the natural language prompt to generate a modified natural language prompt, wherein the step of generating the synthetic representation includes inputting the modified prompt into the generative AI function to generate an additional synthetic representation, and including the additional synthetic representation in the synthetic representation.

[0049] Typically, the modified prompt may include different instructions and / or different content for the generative AI function.

[0050] Using modified hints, further enrichment of the training data can be achieved. At the same time, the modified hints can be used as ground truth for further synthetic representations. For example, the initial hint may specify that an X-ray image shows a lesion at a certain location in the patient's lungs. The modified hint may then specify a lesion at a (slightly) different location.

[0051] According to some examples, the method further includes adding the synthetic representation to an existing test dataset for testing the machine learning model at the model aggregator device to generate an extended test dataset, wherein, in the evaluating step, the local update is evaluated based on the extended test dataset.

[0052] In other words, synthetic data is appended to the test database of the model aggregator device. This can make the expanded test dataset more representative of real-world scenarios and make it more resilient to concept drift in the local site.

[0053] According to some examples, the method further comprises determining a data quality of the synthesized representation, wherein, in the step of evaluating, the local update is evaluated based on the data quality.

[0054] The data quality may comprise an indication of how good the simulation (ie the synthetic representation) of the local data at the model aggregator device actually is. In particular, the data quality may comprise an indication of how well the synthetic representation corresponds to the local data.

[0055] According to some examples, the step of evaluating includes checking whether the quality of the data satisfies a predetermined quality criterion, and evaluating the local update using the synthesized representation if the quality of the data satisfies the predetermined quality criterion.

[0056] According to some examples, the step of adding the synthesized data comprises adding the synthesized representation based on data quality. According to some examples, the step of adding the synthesized data comprises checking whether the data quality meets a predetermined quality criterion, and adding the synthesized data if it meets the predetermined quality criterion.

[0057] According to some examples, the parameterized representation comprises a plurality of independent data items (of the local data), and the step of generating a composite representation of the local data comprises generating a composite data item for each of the independent data items, such that the composite representation comprises the composite data item.

[0058] According to some examples, the step of determining the data quality includes determining a data quality metric for each synthesized data item.

[0059] According to some examples, the checking step includes checking whether a data quality metric of each synthesized data item satisfies a predetermined criterion. According to some examples, the evaluating step includes using the synthesized data item to evaluate whether the local update has a data quality metric that satisfies the predetermined criterion. According to some examples, the adding step includes adding the synthesized data item to existing test data whose data quality metric satisfies the predetermined criterion.

[0060] By determining data quality and using this information in locally updated evaluation and / or extension of central test datasets, we can ensure that inappropriate synthetic representations do not lead to unwanted effects.

[0061] According to an aspect, the step of determining the data quality of the synthetic representation comprises generating a reverse parameterization of the synthetic representation, comparing the reverse parameterization to the parameterization, and determining the data quality based on the step of comparing the reverse parameterization to the parameterization.

[0062] According to some examples, the step of comparing includes determining a difference or distance between the parameterization and the inverse parameterization.

[0063] According to some examples, the reverse parameterization is generated in substantially the same manner as the parameterization. According to some examples, the parameterization is generated on the local side by applying an encoder to the local data. According to some examples, the reverse parameterization is generated at the model aggregator device by applying the encoder or a copy of the encoder to the synthetic representation.

[0064] According to some examples, the step of generating the inverse parameterization includes generating an inverse parameterization for each synthesized data item, and the step of comparing includes comparing the inverse parameterization with the corresponding parameterization (or corresponding portion of the parameterization) to generate a quality metric for each data item.

[0065] Direct quality control of synthesized representations is difficult due to distributed environments. This is because the original training data typically must remain at the local site and is therefore unavailable for comparison with the synthesized representation. The proposed computation of inverse parameterization provides an elegant way to obtain readouts that can be directly compared with information uploaded to the model aggregator. This ensures the quality of the synthesized representation, which in turn allows for more efficient evaluation of machine learning functions in a federated learning setting.

[0066] According to one aspect, determining data quality of the synthesized representation includes generating a natural language summary based on the synthesized representation, comparing the natural language summary to a natural language prompt, and determining the data quality based on comparing the natural language summary to the natural language prompt.

[0067] According to some examples, the step of comparing includes determining a difference or distance between the natural language summary and the natural language prompt.

[0068] According to some examples, the summary is generated by applying a parser to the synthetic representation, the parser being configured to summarize and / or describe the data in natural language text. The parser may include a language decoder configured to generate natural language text based on the data, particularly image data. The language decoder may include a transformer network. According to some examples, the parser may be of the same type as the parser used to generate the prompt or a copy of the parser used to generate the prompt.

[0069] According to one aspect, the step of generating a natural language summary includes generating a natural language summary for each synthesized data item, and the step of comparing includes comparing the natural language summary with a corresponding natural language prompt to generate a quality metric for each data item.

[0070] Another quality control of the method may be performed by generating summaries based on the synthetic representation and comparing these summaries with hints used to trigger the creation of the synthetic representation.In addition to or as an alternative to the inverse parameterization based quality control, hint based quality control may be performed.

[0071] According to some examples, the local data includes a plurality of independent data items, and the parameterization includes: an item parameterization for each data item of each independent data item, and one or more statistical properties of the plurality of independent data items.

[0072] According to some examples, a statistical property may include one or more distributions of parameters of independent data items and / or statistical observables derived from the distributions. The statistical observables may be related to quantifiable properties of the corresponding distributions. According to some examples, the statistical observables may include mean, entropy, skewness, variance, etc.

[0073] According to some examples, one or more statistical properties are input into a generative AI function along with corresponding item-by-item parameterizations, and a synthetic representation is additionally generated based on the one or more statistical properties. In particular, the synthetic representation can be generated such that corresponding statistical properties of the synthetic representation correspond to statistical properties of the parameterizations.

[0074] Using statistical properties may have the benefit that the synthesized representation corresponds more accurately to the local data. In particular, it may be ensured that the synthesized representation shows the same statistics as the local data.

[0075] According to some examples, determining the data quality of the composite representation includes determining one or more corresponding statistical properties based on the composite representation and comparing the corresponding statistical properties with the statistical property. The corresponding statistical properties may include the same statistical observable as the statistical property.

[0076] In other words, quality control based on statistical tests can be implemented. This allows for an efficient way to supervise the quality of synthesized representations when the original training data is not available for direct comparison.

[0077] According to an aspect, the method further comprises generating a modified parameterization based on the parameterization, wherein, in the step of generating, the synthetic representation is additionally generated based on the modified parameterization.

[0078] According to some examples, the modified parameterization may include one or more values that are different from the parameterization. According to some examples, the modified parameterization is configured in such a way that, compared to the (original) parameterization, it results in a (slightly) different synthetic representation if processed by the generative AI function. For example, if one part of the parameterization lists findings of certain characteristics, the corresponding part in the modified parameterization may list findings with different characteristics or no findings at all. In particular, the latter is easy to implement, fail-safe, and can increase the number of normal, i.e., non-suspicious, samples in the synthetic data.

[0079] According to some examples, a modified parameterization can be generated using a trained function that has been configured to derive reasonable modifications to a (physically or medically) parameterization. According to some examples, the corresponding trained function can be trained by relying on the quality control described herein until the trained function is able to produce an acceptable modified parameterization.

[0080] In other words, parameterized data augmentation is performed. This, in turn, results in additional and more varied data for testing the machine learning model. Furthermore, perturbations can have the advantage that those datasets used for the actual adaptation of the machine learning model can be more easily used in the evaluation step without introducing too much bias.

[0081] According to some examples, data quality control measures described in conjunction with synthetic representations generated based on parameterizations may also be applied to synthetic representations generated based on modified parameterizations.

[0082] Specifically, the step of determining the data quality of the synthetic representation comprises determining one or more corresponding statistical properties based on the synthetic representation (generated based on the modified parameterization) and comparing the corresponding statistical properties with the statistical properties of the (original) parameterization. In this way, it can be checked whether the modified parameterization still leads to the same statistical data.

[0083] According to some examples, the local data includes multiple independent data items, and the parameterization includes an item parameterization for each independent data item, wherein the step of generating the modified parameterization includes generating the modified item parameterization, and the step of generating the synthetic representation includes generating a synthetic data item for each of the item parameterization and the modified item parameterization, wherein the synthetic representation includes the synthetic data item.

[0084] According to some examples, the local data includes protected information, particularly protected personal information, and the parameterization does not include the protected information.

[0085] According to some examples, protected information may be information that must not leave the local site.For example, local data may be subject to data protection regulations such as confidentiality agreements or legal regulations such as the General Data Protection Regulation of the European Union (EU).

[0086] Stripping away the protected information used to generate the parameterization allows this data to be distributed outside of the local site. This enables the use of this information at the model aggregator device to test machine learning models.

[0087] According to one aspect, the machine learning model is an image processing function configured to generate an image processing result based on image data, the local data includes training image data, the parameterization includes parameterization of the training image data, and the synthetic representation includes synthetic image data generated by the generative AI function based on the parameterization of the training image data.

[0088] Local data includes (training) image data and verified image processing results. For example, the image data may include images captured by a camera system or other image acquisition system. According to some examples, this may involve image data captured by a smartphone camera or a camera system installed in a car. The image processing results may include objects detected in the image data, such as people, text, cars, lane markings, or other objects.

[0089] According to some examples, the parameterization includes basic characteristics of the image data, such as resolution, color, noise level, acquisition system, etc. In addition, the parameterization may include information characterizing the content of the underlying image data. According to some examples, the information may include feature vectors or embeddings extracted by a correspondingly configured encoder. In addition, according to some examples, the information may include one or more semantic meanings and relationships of the content depicted in the image data. The parameterization may be tangible to a person (e.g., "the picture shows a person riding a bicycle") and / or (only) machine interpretable information, such as complex feature vectors. In addition, the parameterization may include information about verified image processing results (i.e., training output data). According to some examples, the parameterization of the image data may be the same as the parameterization of the image processing results.

[0090] According to some examples, the parameterization may have been generated by applying an encoder or feature encoder to the local data. According to some examples, the feature encoder may include a visual transformer. The visual transformer may be configured to decompose the input image into tiles and label them (extract representation vectors) before applying the labels to a standard transformer architecture. The visual transformer may include an attention mechanism configured to repeatedly transform the representation vectors of the image tiles to incorporate more and more semantic relationships between the image tiles in the image.

[0091] According to some examples, a visual transformer and / or a generative AI function can be obtained by training a masked autoencoder. The masked autoencoder comprises two visual transformers placed end to end. The first visual transformer receives image tiles with position encoding and outputs a vector representing each tile. The second visual transformer receives vectors with position encoding and again outputs image tiles. During training, both visual transformers are used. The image is cut into tiles. The second visual transformer obtains the encoded vectors and outputs a reconstruction of the complete image. During use, the first visual transformer can be used as an encoder and / or the second visual transformer can be used as a generative AI function. In this way, the encoder and the generative AI function complement each other through a design that enables seamless data processing with limited losses.

[0092] According to some examples, visual transformers and / or generative AI functions can be obtained or can be based on training a visual transformer (VQGAN). In the visual transformer (VQGAN), there is a discriminator and two visual transformer encoders. One encodes the image tiles into a list of vectors, one vector per tile. The other encodes the quantized vectors back into image tiles. The training objective is to try to make the reconstructed image (output image) faithful to the input image. The discriminator (usually a convolutional network, but other networks can also be used) tries to decide whether the image is the original real image or a reconstructed image with the help of the visual transformer.

[0093] This has the following advantages: After such a visual transformer VQGAN is trained, it can be used to encode any image into a list of symbols, and any list of symbols into an image. The list of symbols can be used to train a standard autoregressive transformer to autoregressively generate images. Furthermore, one can obtain a list of caption-image pairs, convert the image into a string of symbols, and train a standard GPT-style transformer. Then, at test time, one can give it only the image caption and have it autoregressively generate the image.

[0094] According to some examples, the synthetic image data is generated such that it resembles or mimics the native data as closely as possible.

[0095] By parameterizing and reconstructing image data, an efficient joint learning scheme is provided. Specifically, the method ensures data accessibility while protecting data privacy and reducing data traffic.

[0096] According to some examples, the parameterization does not include image data.

[0097] This can have the advantage of particularly effective data minimization.

[0098] According to some examples, the parameterization may include one or more image tiles extracted from the local data. In other words, the parameterization may include a subset of the image data in the local data. According to some examples, the tiles may correspond to image detection results. Specifically, the tiles may be cutouts of the image data. In other words, the parameterization may include only the most relevant image data, while less relevant portions of the image data are not included in the parameterization.

[0099] This may have the advantage that a more accurate synthetic representation of the image data may be made at the model aggregator device, while still allowing for appropriate data minimization.

[0100] According to some examples, the machine learning model is configured to generate an image processing result based on medical image data, the image processing result being selected from: a detection result of a medical finding in the medical image data, a classification of a medical finding in the medical image data, and / or a segmentation of the medical image data, and the training image data includes the medical image data.

[0101] Thus, the synthetic representation may comprise a synthetic reconstruction of the medical image data.

[0102] According to some examples, the medical image data includes a plurality of medical image data sets respectively showing body parts of a patient.

[0103] The medical image dataset may relate to a medical image study. The medical image dataset may relate to a three-dimensional dataset providing three dimensions in space or two dimensions in space and one dimension in time, a two-dimensional dataset providing two dimensions in space, and / or a four-dimensional dataset providing three dimensions in space and one dimension in time.

[0104] The medical image dataset may depict a body part of the patient in the sense that it contains three-dimensional image data of the body part of the patient. The medical image dataset may represent an image volume. The body part of the patient may be included in the image volume.

[0105] Medical image datasets include image data, for example, in the form of a two-dimensional or three-dimensional array of pixels or voxels. Such an array of pixels or voxels can represent intensity, absorption, or other parameters as a function of three-dimensional position and can be obtained, for example, by suitable processing of measurement signals obtained by a medical imaging modality.

[0106] A medical imaging modality corresponds to a system for generating or producing medical image data. For example, a medical imaging modality may be a computed tomography system (CT system), a magnetic resonance system (MR system), an angiography (or C-arm X-ray) system, a positron emission tomography system (PET system), an ultrasound imaging system, or the like. Specifically, computed tomography is a widely used imaging method that utilizes "hard" X-rays generated and detected by a special rotating instrument. The resulting attenuation data (also referred to as raw data) is presented by computer analysis software that generates a detailed image of the internal structure of a patient's body part. The resulting image set is called a CT scan, which can consist of multiple series of continuous images to present the internal anatomical structure in cross-sections perpendicular to the axis of the human body. Magnetic resonance imaging (MRI), to provide another example, is an advanced medical imaging technology that utilizes the effect of an effective magnetic field on the movement of protons. In an MRI machine, the detector is an antenna, and the signal is analyzed by a computer to create a detailed image of the internal structure of any part of the human body.

[0107] Therefore, the depicted patient's body part will typically include multiple anatomical bodies and / or organs (also denoted as compartments or anatomical structures). Taking a chest image as an example, a medical image dataset may show lung tissue, bones (e.g., rib cage), heart and aorta, lymph nodes, etc.

[0108] A medical image dataset may include multiple images or image slices. Slices may each show a cross-sectional view of an image volume. Slices may include a two-dimensional array of pixels or voxels as image data. The arrangement of slices in a medical image dataset may be determined by the imaging modality or by any post-processing scheme used.

[0109] Furthermore, the medical image dataset may be a two-dimensional pathology image dataset depicting tissue slices of a patient, ie, a so-called whole-slide image.

[0110] According to some examples, the medical image dataset may have been acquired at a local site.

[0111] Medical image data sets may be stored in a standard image format such as the Digital Imaging and Communications in Medicine (DICOM) format and stored in a memory or computer storage system at a local site, such as a picture archiving and communication system (PACS). Whenever DICOM is mentioned herein, it should be understood that DICOM refers to the "Digital Imaging and Communications in Medicine" (DICOM) standard, for example, according to the DICOM PS3.12020c standard (or any later or earlier version of the standard).

[0112] According to some examples, for each medical image dataset, the local dataset includes verified image processing results, in particular, verified detection results of medical findings in the corresponding medical image dataset, verified classification results of medical findings in the corresponding medical image dataset and / or segmentation of the corresponding medical image dataset.

[0113] According to some examples, the parameterization includes a validated image processing result or parameterization thereof. As appropriate, the parameterization may include an indication of the medical imaging modality (or modalities) with which the medical image data has been acquired, imaging parameters used when acquiring the medical image data, medical findings included in the medical image data, or a segmentation of an object included in the medical image data.

[0114] A medical finding may indicate a certain condition or pathology in a patient. The condition or pathology may be related to the patient's diagnosis.

[0115] The medical finding may be related to an anatomical structure that distinguishes the patient from other patients. The medical finding may be located within a different organ of the patient (e.g., within a lung of the patient, or within a liver of the patient) or between organs of the patient. In particular, the medical finding may also be related to a foreign body.

[0116] In particular, the medical finding may relate to a neoplasm (also denoted as a "tumor"), in particular a benign neoplasm, an in situ neoplasm, a malignant neoplasm, and / or a neoplasm of uncertain / unknown nature. In particular, the medical finding may relate to a nodule, in particular a lung nodule. In particular, the medical finding may relate to a lesion, in particular a lung lesion.

[0117] Classification may involve identifying the type of finding and / or providing a classification according to a plurality of predefined categories, such as benign or malignant.

[0118] According to some examples, segmentation can be directed to an organ, a finding, or other compartment of a patient's body part. According to some examples, the step of segmenting can mean obtaining a contour of the corresponding organ or compartment and / or delineating the organ or compartment from the rest of the image data.

[0119] The advantages of this method are particularly evident when applied to medical image data processing. This is because the medical environment is particularly regulated by restrictive data protection policies. Furthermore, there are strict regulations regarding the quality and validation of machine learning functions.

[0120] According to some examples, the parameterization does not include protected health information. Protected health information may include, among other things, personal information or other information about the patient that may lead to identification of the patient.

[0121] According to some examples, parameterization includes one or more occlusions of the medical image data, each centered around a finding depicted in the medical image data. Another term for an occlusion is a tile. The occlusion typically displays the finding and surrounding tissue rather than the entire medical image. This allows for the generation of a better synthetic representation that more closely reflects the original local data.

[0122] According to one aspect, the method further includes providing the updated machine learning model to a second local site that is different from the local site.

[0123] By providing updated machine learning models to other sites, knowledge gathered at one site through local model updates and centrally validated at the model aggregator device can be shared and distributed. At a second local site, additional local model updates can be generated based on the second local data. Additional local updates can be received at the model aggregator device along with parameterization of the second local data, and the process can restart for the additional local updates.

[0124] According to one aspect, a computer-implemented method for federated learning of machine learning models is provided. The method comprises a plurality of steps. A first step is to receive a machine learning model at a local site from a model aggregator device remote from the local site. Another step is to generate (or provide) a local update of the machine learning model at the local site using local data of the local site. Another step is to generate a parameterization of the local data at the local site. Another step is to send the local update and parameterization to the model aggregator device by the local site.

[0125] According to another aspect, a computer-implemented method for federated learning of a machine learning model is provided. The method comprises a plurality of steps. A first step is to receive a machine learning model (and optionally, generative AI functionality) at a local site from a model aggregator device remote from the local site. Further steps are to generate (or provide) a local update of the machine learning model at the local site using local data of the local site. Further steps are to generate a parameterization of the local data at the local site. Further steps are to generate a synthetic representation of the local data based on the parameterization using generative AI functionality at the local site. Further steps are to send the local update and the synthetic representation by the local site to the model aggregator device.

[0126] In other words, the above method is directed to client-side processing. According to the aspects and examples described herein, each step may be further detailed and combined with other features. In particular, the steps of the client-side processing may be combined with the steps of the server-side processing at the model aggregator device. The advantages described in conjunction with the other aspects and examples of this disclosure are also achieved through the corresponding configuration steps of the client-side processing.

[0127] According to one aspect, a computer-implemented method for providing a synthetic representation of local data at a local site to an aggregator device is provided. The method includes multiple steps. One step is to generate a parameterization of the local data at the local site. Another step is to send the parameterization from the local site to the aggregator device. Another step is to receive the parameterization at the aggregator device. Another step is to generate a synthetic representation of the local data based on the parameterization using generative AI functionality at the aggregator device. Another step is to provide the synthetic representation at the aggregator device.

[0128] According to an alternative aspect, a computer-implemented method for providing a composite representation of local data at a local site to an aggregator device is provided, the method comprising generating, at the local site, a composite representation of the local data using a generative AI function, and providing the composite representation from the local site to the aggregator device. Specifically, generating the composite representation at the local site may include generating a parameterization of the local data at the local site, and generating, at the local site, a composite representation of the local data based on the parameterization using the generative AI function.

[0129] The above method can provide a data privacy-preserving method for exchanging information. The aggregator device can be configured in a manner equivalent to the model aggregator device. In addition, the above method can be modified according to other examples and aspects described herein, and similar advantages can be achieved.

[0130] According to one aspect, a model aggregator device for federated learning of machine learning models is provided, the model aggregator device comprising a computing unit and an interface unit. The interface unit is configured to receive a local update and a log file of the machine learning model from a local site remote from the model aggregator device, wherein the local update is generated at the local site based on local data, and the log file includes a parameterization of the local data. The computing unit is configured to use a generative AI function to generate a synthetic representation of the local data based on the parameterization, use the synthetic representation to evaluate the local update to obtain an evaluation result indicating the performance of the model update, and update the machine learning model based on the evaluation result and the local update.

[0131] The computing unit can be implemented as a data processing system or as part of a data processing system. Such a data processing system may, for example, include a cloud computing system, a computer network, a computer, a tablet computer, a smartphone, etc. The computing unit may include hardware and / or software. The hardware may include, for example, one or more processors, one or more memories, and combinations thereof. One or more memories may store instructions for executing the method steps according to the present invention. The hardware may be configurable and / or operable by software. Typically, all units, subunits or modules may exchange data with each other, for example, at least temporarily, via a network connection or a corresponding interface. Therefore, the individual units may be located separately from each other. In addition, the computing unit may be configured as an edge device.

[0132] The interface unit may comprise an interface for exchanging data with one or more local clients, for example, via the Internet. The interface unit may also be adapted to interface with one or more users of the system, for example, by displaying processing results to the user (for example, in a graphical user interface).

[0133] The model aggregator device can be adapted to implement the method as described herein in its various aspects and examples for federated learning of machine learning functions. The advantages described in conjunction with the method aspects and examples can also be achieved by components of a correspondingly configured system.

[0134] According to one aspect, a local model update device for federated learning of a machine learning model is provided. The local model update device is located at a local site. The local model update device includes a local interface unit and a local computing unit. The local interface unit is configured to receive the machine learning model from a model aggregator device remote from the local site, and is configured to send parameterization of local data used to generate a local update at the local site and the local update of the machine learning model to the model aggregator device. The computing unit is configured to generate a local update of the machine learning model based on the local data at the local site, and is configured to generate the parameterization of the local data.

[0135] One or more of the local model update devices can be combined with the model aggregator device to form a system for federated learning of machine learning functions. The local computing unit can generally be configured in a manner equivalent to the computing unit. Similarly, the local interface unit can be configured in substantially the same manner as the interface unit.

[0136] According to another aspect, the present invention relates to a computer program product comprising a program element which, when loaded into the memory of a computing unit of a model aggregator device (or a local model update device) for federated learning of machine learning functions, causes the computing unit to perform one or more steps according to the above-mentioned method aspects and examples.

[0137] According to another aspect, the present invention relates to a computer-readable medium having program elements stored thereon, which can be read and executed by a computing unit of a model aggregator device (or a local model update device) for federated learning of machine learning functions, so as to perform steps according to one or more method aspects and examples when the program elements are executed by the computing unit.

[0138] Implementing the invention by means of a computer program product and / or a computer-readable medium has the advantage that already existing provisioning systems can be easily adapted by means of a software update in order to function as proposed by the invention.

[0139] A computer program product may be, for example, a computer program, or include another element adjacent to the computer program itself. This other element may be hardware, such as a storage device on which the computer program is stored, a hardware key for using the computer program, etc., and / or software, such as documentation or a software key for using the computer program. A computer program product may also include development materials, a runtime system, and / or a database or library. A computer program product may be distributed across several computer instances. BRIEF DESCRIPTION OF THE DRAWINGS

[0140] The characteristics, features, and advantages of the above-described invention and the manner in which they are achieved will become clearer and more easily understood from the following description of the embodiments, which will be described in detail with reference to the accompanying drawings. The following description does not limit the invention to the embodiments included. The same components, parts, or steps may be marked with the same reference numerals in different figures. In general, the figures are not drawn to scale. In the following:

[0141] Figure 1 schematically depicting an embodiment of a system for federated learning of machine learning functions, according to an embodiment;

[0142] Figure 2schematically depicts a method for federated learning of machine learning functions according to an embodiment;

[0143] Figure 3 schematically depicts an exemplary data flow diagram associated with a method for federated learning of machine learning functions according to an embodiment;

[0144] Figure 4 schematically depicts an exemplary data flow diagram associated with a method for federated learning of machine learning functions according to an embodiment;

[0145] Figure 5 schematically depicts a method for federated learning of machine learning functions according to an embodiment;

[0146] Figure 6 schematically depicts an exemplary data flow diagram associated with a method for federated learning of machine learning functions, according to an embodiment; and

[0147] Figure 7 An encoder-decoder transformer network according to an embodiment is schematically depicted. DETAILED DESCRIPTION

[0148] Figure 1 An example system 1 for federated learning of machine learning models ML in a distributed environment is depicted. The system may be capable of creating, training, updating, distributing, monitoring, and generally managing machine learning models ML in an environment comprising multiple local sites LS, LS-2, LS-3. The system 1 is adapted to perform methods according to one or more embodiments, for example, as described with respect to Figures 2 to 6 Further described.

[0149] System 1 includes a model aggregator device (MAD) and multiple clients located at different local sites (LS, LS-2, and LS-3). The model aggregator device (MAD) and the clients can interface via a network. The model aggregator device (MAD) is typically configured to control, coordinate, and manipulate the federated learning process in system 1. The local sites (LS, LS-2, and LS-3) can be associated with, for example, a clinical or medical environment, such as a hospital or hospital group, a clinic, or a workplace.

[0150] The machine learning model ML can be envisioned as a master model that is centrally managed by a model aggregator device MAD and distributed to local sites LS, LS-2, LS-3 and further trained at the local sites LS, LS-2, LS-3. The machine learning model ML can generally be configured to provide a medical diagnosis based on medical input data. This can include outcome prediction, detection of findings in medical image data, annotation of medical images (e.g., in terms of orientation or landmark detection), generation of medical reports, etc.

[0151] The model aggregator device MAD can be hosted on a server, which can be a cloud server or a local server. However, the model aggregator device MAD can also be implemented using any other suitable computing device. The model aggregator device MAD includes a computing unit CU and an interface unit IU. In addition, the model aggregator device MAD can access a central database CDB, which is configured to centrally store training data for evaluating machine learning models ML.

[0152] The computing unit CU may include one or more processors and a working storage device. The one or more processors may include, for example, one or more central processing units (CPUs), graphics processing units (GPUs) and / or other processing devices. The computing unit CU may also include a microcontroller or an integrated circuit. Alternatively, the computing unit CU may include a real or virtual computer group, such as a so-called "cluster" or "cloud". The working storage device may include one or more computer-readable media, such as a RAM for temporarily loading data (e.g., data from a database CDB or data uploaded from a local site LS, LS-2, LS-3). The working storage device may also store information that can be accessed by one or more processors for executing the method steps according to one or more embodiments described herein.

[0153] The interface unit IU may include any suitable components for interfacing with one or more networks, including, for example, a transmitter, a receiver, a port, a controller, or other suitable components. The model aggregator device MAD may exchange information with one or more local sites LS, LS-2, LS-3 via the interface unit IU. Any number of local sites LS, LS-2, LS-3 may be connected to the model aggregator device MAD via the interface unit IU.

[0154] The computing unit CU may include subunits SYNTH, AGGR, and MGMT. The subunit MGMT may be a management module or unit configured to control and manage the federated learning of the machine learning model ML in the system 1. Once a new updated version ML* is available, the subunit MGMT may trigger the distribution of the machine learning model ML to the local sites LS, LS-2, and LS-3, and initiate an update of the machine learning model ML in the system 1.

[0155] The subunit SYNTH can be conceived as a training data synthesizer. The subunit SYNTH is configured to generate a synthetic representation SR of the local data LTD based on the corresponding parameterization P. To this end, the subunit SYNTH can be configured to host and run a correspondingly configured generative AI function GEN.

[0156] The sub-unit AGGR can be considered as a model update unit. The sub-unit AGGR is configured to evaluate the local model update ML′ and aggregate the local update ML′ into the main machine learning model ML if the local update ML′ constitutes an improvement. To this end, the sub-unit AGGR can be configured to apply a cross-validation scheme.

[0157] The names of the various subunits SYNTH, AGGR, and MGMT are provided by way of example and are not intended to limit the present disclosure. Thus, the subunits SYNTH, AGGR, and MGMT may be integrated to form a single processing unit, or may be embodied by computer code segments configured to execute corresponding method steps running on a processor of a computing unit CU, etc. Each subunit SYNTH, AGGR, and MGMT may be individually connected to other subunits and / or other components of the system 1 that require data exchange to execute the method steps.

[0158] The central database CDB may be implemented as a cloud storage. Alternatively, the central database CDB may be implemented as a local or extended storage device, in particular within the premises of the model aggregator device MAD. The central database CDB is configured to store central training data CTD.

[0159] Each of the local sites LS, LS-2, and LS-3 includes a local model update device LMUD and a local database LDB. The local database LDB can be implemented as a local or extended storage device within the premises of the corresponding local site LS, LS-2, and LS-3. The local database LDB can store local (training) data LTD to be processed by the machine learning model ML.

[0160] The local data LTD may include, for example, a plurality of separate data items relating to clinical or medical issues. As an example, the data items may relate to laboratory test results and / or pathology data and / or medical imaging data, electronic medical records, and any combination thereof. The local data LTD may relate to the medical data of one or more patients. The local data LTD may have been generated at the corresponding local site LS, LS-2, LS-3. The local database LDB may be part of a hospital information system (HIS), a radiology information system (RIS), a clinical information system (CIS), a laboratory information system (LIS), and / or a cardiovascular information system (CVIS), a picture archiving and communication system (PACS), etc.

[0161] From the local database LDB, local data LTD can be accessed locally to train the machine learning model ML, and to use the machine learning model ML periodically later after deployment. Training can include adjusting the machine learning model and verifying and testing the adjusted machine learning model. Local data can be divided into training data, verification data, and test data. In order to train the machine learning model at the local site, a backpropagation algorithm can be used based on an appropriate cost function and using the training data. Based on the verification data, the best performing machine learning model can be selected from several machine learning models (with different hyperparameters, such as the number of layers, size, and number of cores, padding, etc.). Specificity and sensitivity can then be determined based on the test data.

[0162] In particular, the local data LTD cannot be accessed externally because the local data LTD may be subject to data protection regulations that may prohibit the local data LTD from leaving the local site LS, LS-2, LS-3. The local data LTD may include training input data and associated training output data, which can be used to evaluate the performance of the machine learning model ML during training. The output training data may be associated with verification results corresponding to the input training data. The output training data may be generated and / or verified by a human based on the input training data.

[0163] The local model updating device LMUD may include a local computing unit LCU and a local interface unit LIU. The local interface unit LIU may be configured in an equivalent manner to the interface unit IU and may include any suitable components for interfacing with the interface unit IU over a network such as the Internet.

[0164] The local computing unit LCU is configured to further train the machine learning model ML based on the local data LTD so as to provide a local update ML′ of the machine learning model ML. To this end, the local computing unit LCU may include a correspondingly configured training unit or module TRN. Furthermore, the local computing unit LCU may include a parameterization module or unit PAR configured to generate a parameterization P of the local data LTD. To this end, the parameterization unit PAR may be configured to host a correspondingly configured encoder function ENC. In order to ensure that privacy-sensitive information cannot be derived or inferred from the parameterization P, one or more encryption techniques, random noise techniques and / or other security techniques may be added by the parameterization unit PAR when generating the parameterization P. Both the local update ML′ and the parameterization P may be provided to the model aggregator device MAD via the local interface unit LIU.

[0165] The names of the different sub-units TRN and PAR are explained by way of example and are not intended to limit the present disclosure. Therefore, the sub-units TRN and PAR may be integrated to form a single processing unit, or may be embodied by computer code segments configured to execute corresponding method steps running on a processor of a local computing unit LCU or the like.

[0166] The local computing unit LCU can be any suitable type of computing device, such as a general-purpose computer, a special-purpose computer, a laptop computer, a local server system, or other suitable computing device. The local computing unit LCU may include a memory and one or more processors. The one or more processors may include, for example, one or more central processing units (CPUs), graphics processing units (GPUs), and / or other processing devices. The memory may include one or more computer-readable media and may store information accessible by one or more processors, including instructions that can be executed by one or more processors. The instructions may include instructions for local further training of the machine learning model ML and / or generation of parameterizations P.

[0167] In an alternative embodiment (not shown), it is also conceivable to provide the local sites LS, LS-2, and LS-3 with a module for generating a synthetic representation SR of the local data LTD based on the corresponding parameterization P. The synthetic representation SR would then be generated directly at the local sites LS, LS-2, and LS-3 and transmitted to the model aggregator device MAD. Also according to this alternative, data protection requirements can be complied with, since only the synthetic representation SR leaves the local sites LS, LS-2, and LS-3.

[0168] Figure 2 Describes a method for federated learning of machine learning models in a distributed environment. Figure 3 The corresponding data flow is shown in FIG.

[0169] Figure 4 The data flow associated with model aggregation at the model aggregator device MAD is shown. The method comprises several steps. The order of the steps does not necessarily correspond to the numbering of the steps, but may also vary between different embodiments of the present invention. In addition, individual steps or a series of steps may be repeated.

[0170] Steps C10 to C40 occur on the client side, i.e., at the respective local sites LS, LS-2, and LS-3, and may be performed by a local model update device LMUD. Steps S10 to S80 occur on the server side and may be performed by a model aggregator device MAD. According to the method of the present invention, each step may be performed independently. In other words, aspects of the present invention encompass methods that include only client-side method steps, while other aspects encompass methods that include only server-side steps. Furthermore, aspects of the present invention may also encompass methods that include both server-side and client-side steps.

[0171] At step C10, a machine learning model ML is received at the local side LS. The machine learning model ML may be a copy of a master model provided and managed by the model aggregator device MAD. The machine learning model ML is easily trained and is meant to be deployed at the local site LS according to a learning task. According to an embodiment, the learning task may include automated processing of medical image data for deriving a medical diagnosis. Specifically, the machine learning model may be configured to process medical image data of a patient in order to detect medical findings and / or classify medical findings. According to some examples, the medical image data may show a portion of a patient's torso, and the findings include lesions in the patient's lungs or liver. According to other examples, the medical image data includes a digital pathology image of the patient, and the findings are related to segmentation of the digital pathology image according to one or more tissue types.

[0172] At step C20, the machine learning model ML may be further trained based on the local data LTD at the local site LS. This results in a local update ML′ of the machine learning model ML. According to some examples, such training may occur on the fly (e.g., when a user at the local site views the processing results of the machine learning model ML). For example, a radiologist may reject or accept lesions found by the machine learning model ML. In addition, a radiologist may add lesions that the machine learning model did not find. According to other examples, a pathologist may modify the segmentation as provided by the machine learning model ML. The user input may be used as a true value for further optimization (i.e., training the machine learning model ML at the local sites LS, LS-2, LS-3). The true value together with the underlying local data may form the local data LTD.

[0173] At step C30 , a parameterization P of the local data LTD or a portion of the local data may be generated. In particular, only the portion of the local data used to validate the further trained machine learning model may be parameterized. Parameterization P may involve a data minimization step in which the local data LTD is stripped down to a version that still allows for synthesis or re-creation of the local data LTD at the model aggregator device MAD, but that does not contain any unnecessary information. In particular, parameterization P may not include any information subject to data protection regulations, such as personal information of the patient.

[0174] The parameterization P may include multiple feature values of the underlying medical image and, optionally, also include an image data excerpt / crop of the medical image. To provide an example, the parameterization P may read as follows: type: chest CT scan, bolus agent: xyz, modality: Siemens Medical CT scanner, model: 12345, kilovolt peak: xxx, milliampere-seconds: yyy, lung nodule 1: size: 11 mm, type: solid, location: left upper lobe, lung nodule 2: size: 16 mm, type: ground glass nodule (ggn), location: left lower lobe, etc. According to other examples, the parameterization P may have a more abstract form and may be provided in the form of an embedding that can be interpreted by the generative AI function GEN but not necessarily by a human user. In addition, the parameterization P may also include one or more statistical properties of the entire local data.

[0175] At step C30, the parameterization P may be generated by an encoder ENC or an autoencoder, which may be provided to the local site LS by the model aggregator device MAD together with the machine learning model ML. The encoder ENC may be considered as a counterpart to the generative AI function GEN and may be trained in conjunction with the generative AI function GEN as described herein.

[0176] At step C40 , the local update ML′ and the parameterization P are transmitted by the local model update device LMUD to the model aggregator device MAD.

[0177] At step S10 , the local update ML′ and the parameterization P are sequentially received at the model aggregator device MAD. At step S20 , a synthetic representation SR of the local data LTD is generated. To this end, a generative AI function GEN can be applied to the parameterization P. Following the above example, the synthetic representation SR again comprises a medical image, such as a radiological or pathological medical image, as a reconstruction of the corresponding image data at the local site LS.

[0178] Optionally, the generative AI function GEN can operate based on a natural language prompt. This prompt can be considered an instruction or control command for the generative AI function. This prompt can be generated in optional sub-step S21 based on the parameterization P. In a sense, step S21 can be considered a translation step that translates the parameterization P into a set of instructions based on which the generative AI function GEN can act.

[0179] At optional step S22 , a prompt is input into the generative AI function in order to trigger the generation of a synthetic representation SR.

[0180] At step S30, the local update ML' is evaluated using the synthesized representation SR - optionally together with other training already present in the central database CDB. The result of the processing may be provided in the form of an evaluation result.

[0181] Figure 4 The data flow diagram of the model evaluation and aggregation process is shown in Figure 4 As can be seen in , the model aggregator device MAD receives local updates ML' not only from one local site LS, but from a plurality of local sites LS, LS-1, LS-2. Likewise, the model aggregator device MAD may receive composite representations SR from different local sites LS, LS-1, LS-2.

[0182] In order to test, validate and ultimately obtain an updated master model ML*, a validation scheme may be used at sub-step S31. In particular, a cross-validation scheme may be used, according to which the available data, i.e. the synthetic representation SR and other appropriate training data available at the model aggregator device MAD, are divided into a plurality of complementary subsets or groups. The different available models (or their parameters) may be combined in a trail-fashion manner to generate a plurality of candidate model updates ML_tmp. Additional further training of these different combinations ML_tmp may be performed on a subset of the available data (referred to as the training set or training group), and testing may be performed on other subsets (referred to as the test set or group). In order to reduce variability, multiple rounds of cross-validation may be performed using different partitions of the training data, and the validation results may be combined (e.g., averaged) over the different partitions to give an estimate of the predictive performance of the ML_tmp of the corresponding machine learning model. If additional hyperparameters need to be optimized, a nested cross-validation scheme may be applied. Basically, these schemes rely on (1) internal cross-validation to tune hyperparameters and select the best hyperparameters, and (2) external cross-validation to evaluate the model trained with the best hyperparameters as selected by the internal cross-validation. The best model can be provided in the form of a (final) evaluation result.

[0183] Before using the synthesized representations SR in the evaluation of the machine learning model ML_tmp, they can be subjected to Figure 5 and Figure 6 Quality control as described.

[0184] Based on the model evaluation and aggregation results of step S30, the best performing variant can be used for an updated version of the main model and provided as a (global) update ML* of the machine learning model ML.

[0185] As from Figure 3 and Figure 4 As can be seen, the method may also include a step of adding the synthesized representation SR to the central database CDB. This may occur at optional step S50. Furthermore, before adding the synthesized representation SR to the central database CDB, a quality control step may be performed to determine whether the synthesized representation SR is of sufficient quality. In this regard, a combination of Figure 5 and 6 Instructions for the steps.

[0186] Finally, at optional step S60, the updated version ML* of the machine learning model ML may be pushed to one or more of the local sites LS, LS-1, LS-2 in order to provide a (globally) updated machine learning model ML*.

[0187] Figure 5 Describes optional substeps in a method for federated learning of machine learning models in a distributed environment. Figure 6 The corresponding data flow is shown in FIG. The method comprises several steps. The order of the steps does not necessarily correspond to the numbering of the steps, but may also vary between different embodiments of the present invention. In addition, individual steps or a series of steps may be repeated.

[0188] At step S70, a quality assessment of the synthesized representation SR is performed. Figure 2 In the workflow depicted in FIG, step S70 can, for example, follow step S20. In step S70, two alternative processes for quality control are included. One alternative process involves the generation of the reverse parameterization P′ (steps S71 and S72), and the other alternative process is based on the generation of the text summary SUM (steps S73 and S74). These two processes can be applied separately or in combination.

[0189] Specifically, at step S71 , a reverse parameterization P' of the synthetic representation SR may be generated. According to some examples, the same encoder ENC used to generate the parameterization P at the local site LS may be used.

[0190] Subsequently, at step S72 , the reverse parameterization P′ may be compared with the parameterization P. If, based on the comparison, the reverse parameterization P′ and the parameterization P sufficiently correspond, it may be assumed that the data quality of the composite representation SR is sufficient for further use (e.g., integration in a central database CDB and / or evaluation / aggregation of the local update ML′).

[0191] At sub-step S73, a text summary SUM of the synthetic representation SR may be generated. For example, the text summary may be automatically generated by applying another trained function independent of the generative AI functionality. In a medical setting, a visual transformer may be used that has been trained to analyze medical image data and incorporate findings into textual impressions (such as those found in medical reports).

[0192] At step S74, the summary SUM may be compared to the prompt.If the text summary SUM matches the prompt, this may be considered an indication of sufficient data quality for the synthesized representation SR.

[0193] Step S80 can be considered an optional data augmentation step. In step S80, other variations of the parameterization P can be generated by adjusting or perturbing the included values. For example, the size and position of the nodules may be slightly different. In addition, descriptions of additional nodules can be added, while descriptions of other nodules can be deleted. In other words, this results in a perturbed parameterization P_mod for the generative AI function. As appropriate, the perturbed features can be used to generate additional hints that differ from the original hints based on the unperturbed parameterization P.

[0194] According to other examples, the prompt PMT may also be directly perturbed / modified (if the prompt PMT is generated in the workflow). In this case as well, other versions of the input for use in the generative AI function GEN may be obtained.

[0195] This, in turn, results in an additional synthetic representation SR which can further increase the amount of data in the central database CDB. As such, the synthetic representation SR generated based on such perturbed input parameters can be subjected to the same process as described in connection with step S70 and Figure 6 Same quality control as shown.

[0196] According to some examples, the encoder ENC for generating the parameterized P can be related to the encoder part ENCP of the encoder-decoder transformer network, and the generative AI function GEN can be related to the decoder part DEC of the encoder-decoder transformer network. The encoder part can be configured to receive an image and output a feature encoding, while the decoder is configured to generate a synthetic representation of the input image based on the feature encoding. In other words, the encoder ENC and the generative AI function GEN can be regarded as visual transformers that operate in corresponding manners relative to each other. In this regard, the feature encoding connecting the two parts can be regarded as the parameterized P of the present invention.

[0197] That is to say, Figure 7 A schematic representation of an encoder-decoder transformer network according to an embodiment is shown. While using such a structure may have some advantages, such as end-to-end training, it should be noted that other configurations may also be feasible. In particular, the encoder ENC and the generative AI function GEN may also be independent of each other, as described elsewhere herein.

[0198] In short, the task of the encoder ENC is to map the input INPT (i.e., local data LTD, in particular a medical image) into a series of continuous representations (parameterizations P) which are then fed into the decoder GEN. The decoder GEN receives the output P of the encoder ENC and the decoder output OUTR at the previous iteration to generate an output OUT which is a synthesized representation SR of the input INPT, in particular a synthesized image.

[0199] The encoder ENC of this embodiment may include a stack of N=8 identical layers. For ease of reference, only one layer xN is shown in the figure. In addition, depending on the corresponding task, N can also be set to different values, and in particular, to a value greater than N=8. Each layer xN of the encoder ENCP includes two sublayers L1 and L3. The first sublayer L1 implements the so-called multi-head self-attention mechanism. Specifically, the first sublayer L1 can be configured to determine the degree of relevance of a particular image data element relative to other elements in the input INPT. This can be represented as an attention vector. To avoid any bias, multiple attention vectors can be generated for each word and fed into a weighted average to calculate the final attention vector for each word. The second sublayer L3 is a fully connected feedforward network, which can, for example, include two linear transformations with a rectified linear unit (ReLU) activated between them. The N=8 layers of the encoder ENC apply the same linear transformation to all elements in the input INPT, but each layer uses different weights and bias parameters to do so. Each sub-layer L1, L3 is followed by a normalization layer L2, which normalizes the sum calculated between the input fed into the corresponding sub-layer L1, L3 and the output generated by the corresponding sub-layer L1, L3 itself. In order to capture information about the relative positions of elements in the input INPT, a position encoding PE is generated based on the input embedding INPT-E before being fed into the layer xN. The position encoding PE has the same dimension as the input embedding INPT-E and can be generated using sine and cosine functions of different frequencies. The position encoding PE can then be simply added to the input embedding INPT-E in order to inject the position information PE. Typically, the input embedding INPT-E can be a representation of each image patch in the input INPT, typically in the form of a real-valued vector that encodes a pattern or other visual feature so that patches that are closer in vector space are considered similar. According to some examples, a convolutional neural network can be used to generate the input embedding INPT-E.

[0200] The decoder GEN of this embodiment may also include a stack of N=8 identical layers xN, each layer xN including three sublayers L4, L1, and L3, which may be followed by a normalization layer L2, as described in conjunction with the encoder ENC. For ease of reference, only one layer xN of the decoder GEN is shown in the figure. In addition, N can also be set differently depending on the corresponding task and, in particular, be greater than N=8. While the sublayers L1 and L3 of the decoder GEN correspond functionally to the corresponding sublayers L1 and L3 of the encoder ENC, the sublayer L4 receives the previous output OUTR of the decoder GEN (optionally transformed into a corresponding embedding and augmented with position information if the output is a synthesized image tile) and implements multi-head self-attention on it, thereby weighing the importance of each element of the previous output vector OUTR. Next, the value from the first sublayer L4 of the decoder DEC is input to the L1 sublayer of the decoder GEN. This sublayer L1 of the decoder GEN implements a multi-head self-attention mechanism similar to the multi-head self-attention mechanism implemented in the first sublayer L1 of the encoder ENC. On the decoder side, the multi-head mechanism receives the values from the previous decoder sublayer L4 and the output of the encoder ENC. This allows the decoder GEN to process all tiles in parallel. Like in the encoder ENC part, the output of the L1 sublayer is passed to the feedforward layer L2, which will form the output vector into something that can be easily accepted by another decoder block or linear layer. After all layers xN of the decoder DEC have been processed, the intermediate results are fed into the linear layer L5, which can be another feedforward layer. It is used to expand the size to the image format desired for the output OUT. Subsequently, the result is passed through the Softmax layer L6, which transforms the result into the final output.

[0201] The various embodiments or their aspects and features may be combined or interchanged with one another wherever it makes sense, without limiting or expanding the scope of the invention. Where applicable, the advantages described with respect to one embodiment of the invention are also advantageous for the other embodiments of the invention. Regardless of grammatical usage of the term, individuals with masculine or feminine identities are included within the term.

Claims

1. A computer-implemented method for federated learning of machine learning models (ML) in a model aggregator device (MAD), the method comprising: - receiving (S10) at the model aggregator device (MAD) a local update (ML′) of the machine learning model (ML) and a parameterization (P) of local data (LTD) from a local site (LS) remote from the model aggregator device (MAD), wherein the local update (ML′) is generated at the local site (LS) based on the local data (LTD), - generating (S20) a synthetic representation (SR) of said local data (LTD) based on said parameterization (P) using a generative AI function (GEN) at said model aggregator device (MAD), - evaluating (S30) said local update (ML') using said synthetic representation (SR) at said model aggregator device (MAD) so as to obtain an evaluation result indicative of the performance of said local update (ML'), and - at the model aggregator device (MAD), updating (S40) the machine learning model (ML) based on the evaluation result and the local update (ML').

2. The method according to claim 1, wherein - said parameterization (P) comprises parameterization of data for validating and / or testing said local update (ML') at said local site (LS).

3. The method according to any one of claims 1 or 2, wherein: - said generative AI functionality (GEN) being configured to generate said synthetic representation (SR) based on a natural language prompt indicating said synthetic representation (SR) to be generated, and - said step of generating (S20) comprises obtaining (S21) said natural language prompt based on said parameterization (P) and inputting (S22) said natural language prompt input in said generative AI function (GEN) in order to generate said synthesized representation (SR).

4. The method according to any one of the preceding claims, further comprising: - adding (S50) said synthetic representation (SR) to an existing test dataset accessible to said model aggregator device (MAD) for validating and / or testing said machine learning model (ML) so as to generate an extended test dataset, - wherein, in the step of evaluating (S30), the local update (ML') is evaluated based on the extended test dataset.

5. The method according to any one of the preceding claims, further comprising: - determining (S70) the data quality of said synthetic representation (SR), - wherein, in said step of evaluating (S30), said local update (ML') is evaluated based on said data quality.

6. The method according to claim 5, wherein: The step of determining (S70) the data quality of the synthetic representation (SR) comprises: - generating (S71) a reverse parameterization (P') of said synthetic representation (SR), - comparing said inverse parameterization (P') with said parameterization (P) (S72), and - determining (S70) said data quality based on a step of comparing said inverse parameterization (P') with said parameterization (P).

7. The method according to any one of claims 5 or 6 in combination with claim 3, wherein: The step of determining (S70) the data quality of the synthetic representation (SR) comprises: - generating (S73) a natural language summary (SUM) based on said synthetic representation (SR), - comparing the natural language summary (SUM) with the natural language prompt (S74), and - determining (S70) the data quality based on the step of comparing the natural language summary with the natural language prompt.

8. A method according to any one of the preceding claims, wherein - the local data (LTD) comprises a plurality of independent data items, and - said parameterization (P) comprises: - an item parameterization of said data item for each individual data item, and - one or more statistical properties of the plurality of independent data items.

9. The method according to any one of the preceding claims, further comprising: - generating (S80) a modified parameterization based on said parameterization (P), - wherein, in said step of generating (S20), said synthetic representation (SR) is additionally generated based on said modified parameterization.

10. A method according to any one of the preceding claims, wherein The local data (LTD) includes protected information, in particular protected personal information, and the parameterization (P) does not include the protected information.

11. The method according to any one of the preceding claims, wherein: - the machine learning model (ML) is an image processing function configured to generate an image processing result based on image data, - said local data (LTD) comprises training image data, - said parameterization (P) comprises a parameterization of said training image data, - said synthetic representation (SR) comprises synthetic image data generated by said generative AI functionality (GEN) based on said parameterization (P) of said training image data.

12. The method according to claim 11, wherein - the machine learning model (ML) is configured to generate an image processing result based on the medical image data, the image processing result being selected from: a detection result of a medical finding in the medical image data, a classification of a medical finding in the medical image data and / or a segmentation of the medical image data, and - The training image data comprises medical image data.

13. The method according to any one of the preceding claims, further comprising: - providing (S60) the updated machine learning model (ML') to a second local site (LS-2) different from said local site (LS).

14. A model aggregator device (MAD) for federated learning of a machine learning model (ML), the model aggregator device (MAD) comprising a computing unit (CU) and an interface unit (IU), wherein: The interface unit (IU) is configured to: - receiving (S10) a local update (ML′) of the machine learning model (ML) and a parameterization (P) of local data (LTD) from a local site (LS, LS-2) remote from the model aggregator device (MAD), wherein the local update (ML′) is generated at the local site (LS, LS-2) based on the local data (LTD), And wherein the computing unit (CU) is configured to: - generating (S20) a synthetic representation (SR) of said local data (LTD) based on said parameterization (P) using a generative AI function (GEN), - evaluating (S30) said local update (ML') using said synthetic representation (SR), so as to obtain an evaluation result indicative of the performance of said local update (ML′), and - Updating (S40) the machine learning model (ML) based on the evaluation result and the local update (ML').

15. A computer program product comprising program elements, which, when loaded into the memory of a computing unit (CU) of a model aggregator device (MAD) for federated learning of machine learning models (ML), cause the computing unit (CU) to perform the steps of the method according to any one of claims 1 to 13.

16. A computer-readable medium having program elements stored thereon, the program elements being capable of being read and executed by a computing unit (CU) of a model aggregator device (MAD) for federated learning of a machine learning model (ML) so as to perform the steps of the method according to any one of claims 1 to 13 when the program elements are executed by the computing unit (CU).

Images