Methods and systems for federated learning of a machine learning model

Abstract

Computer-implemented methods and corresponding systems for federated learning of a machine learning model are disclosed, the embodiments of which implement the following steps: - receiving (S10), at the model aggregator device (MAD), from a local location (LS) located remotely from the model aggregator device (MAD), a local update (ML') of the machine learning model (ML), wherein the local update (ML') was generated at the local location (LS) based on local data (LTD), and a parameterization (P) of the local data (LTD), - generating (S20), at the model aggregator device (MAD), a synthetic representation (SR) of the local data (LTD) based on the parameterization (P) using a generative AI function (GEN), - evaluating (S30), at the model aggregator device (MAD), the local update (ML') using the synthetic representation (SR) to obtain an evaluation result indicating the performance of the local update (ML'), and - updating (S40), at the model aggregator device (MAD), the machine learning model (ML) based on the assessment result and the local update (ML').

Description

[0001] Embodiments of the present invention relate to systems and methods for providing an updated machine learning model in a distributed environment. In particular, embodiments of the present invention relate to federated learning of a machine learning model to provide an updated machine learning model. In particular, embodiments of the present invention relate to the use of an updated machine learning model for medical data processing, and in particular, for medical image data processing.

[0002] Machine learning methods and algorithms are widely used to generate insights and / or (predictive) computational models from data. Typically, data is fed into a processing unit (such as a cloud) that can execute such methods and algorithms to train models or generate insights. In this respect, machine learning methods have proven highly versatile in various application areas. For example, they are used to support decision-making in autonomous driving. Similarly, machine learning methods are used to provide medical diagnoses by automated systems that process physiological measurements such as medical images.

[0003] However, due to data protection regulations, it is often not possible to feed data to an external processing unit capable of executing machine learning procedures and algorithms, but whose ownership differs from that of the data. In such circumstances, the data must often remain on-site with the data owner. This situation often arises in healthcare, where the inherent sensitivity of patient-related data raises important concerns regarding patient privacy.

[0004] One way to address this problem is to implement a distributed or federated learning scheme. Here, a central machine learning model hosted at a model aggregator device (such as a central server unit) can be improved based on the usage reported by many client units, each located at a local location. This distributes a fully trained, functioning, and deployable central machine learning model to the local locations and executes it locally. Each client can send a local update to the central server unit randomly, periodically, or on command. The local update can aggregate a local change to the machine learning model based on the local data collected by the client. The model aggregator device can use the local updates to improve the machine learning model.In return, the model aggregator device can then upload a modified machine learning model to the clients, implementing the learning modifications based on the actual usage reported by the client. This allows the clients to collaboratively learn and improve a shared machine learning model while their local and potentially classified data is not distributed outside the client units.

[0005] One problem in this regard is that the local updates must be evaluated or tested on the model aggregator to determine whether they represent an improvement. For this purpose, the model aggregator must have test data against which modifications to the machine learning model are tested. Typically, this test data is a fixed set obtained in compliance with data protection regulations. Maintaining such a fixed set centrally has several disadvantages. First, such test data is difficult to obtain, and therefore the number of data instances is limited. Furthermore, there is no guarantee that the centrally hosted test data is representative of real-world scenarios, and furthermore, such fixed data is not robust to concept drift, which can occur in the field.

[0006] As an alternative, a decentralized approach to model evaluation has been proposed (see, for example, US 2021 / 0 097 439 A1). Here, an aggregated model is sent to all clients to perform an evaluation on the local test data. Each client downloads the models, performs inference, and then uploads evaluation results. The central server aggregates the evaluation performance across locations and then decides whether to retain or discard the model. This has the advantage of testing the models on real data and reducing the possibility of overfitting, as the test data changes over time. However, this approach also suffers from disadvantages. First, clients must reserve computational resources not only for training but also for testing.This could reduce turnaround times for both model update cycles and actual model usage on clients. Second, it necessarily leads to increased data traffic, as clients must download the models and upload the results back to the model aggregator. Third, local data will conceptually change over time. While this reduces the effect of overfitting, it can make it difficult to compare current models with historical models.

[0007] It is therefore an object of embodiments of the invention to provide improved methods and systems for federated learning of a machine learning function. In particular, it is an object of embodiments of the present invention to provide systems and methods that enable more efficient evaluation of machine learning functions in a distributed environment.

[0008] This object is achieved by a method for federated learning of a machine learning model, a system for federated learning of a machine learning model, corresponding computer program products, and computer-readable storage media according to the main claims. Alternative and / or preferred embodiments are the subject of the dependent claims.

[0009] In the following, the technical solution according to the present invention is described with reference to the claimed devices and with reference to the claimed methods. Features, advantages, or alternative embodiments described herein may equally be assigned to other claimed subject matter, and vice versa. In other words, claims addressing the inventive method may be enhanced by features described or claimed with reference to the devices. In this case, for example, functional features of the method are embodied by objective units or elements of the device.

[0010] The technical solution is described both with reference to methods and systems for providing an updated machine learning function and with reference to methods and systems for providing training or test data for updating a machine learning system. Features and alternative forms of embodiments of data structures and / or functions for methods and systems for providing machine learning functions can be transferred to analogous data structures and / or functions for methods and systems for providing training or test data. Analogous data structures can be identified in particular by using the prefix "training". Furthermore, the prediction functions used in methods and systems for providing information can in particular have been adapted and / or trained and / or provided by methods and systems for adapting prediction functions.

[0011] According to one aspect, a computer-implemented method for federated learning of a machine learning model in a model aggregator device is provided. The method comprises a plurality of steps. One step is directed to receiving, at the model aggregator device, from a local location located remotely from the model aggregator device, a local update of the machine learning model and a log file, wherein the local update was generated (or provided) at the local location based on local data, and the log file comprises a parameterization of the local data. Another step is directed to generating, at the model aggregator device, a synthetic representation of the local data based on the parameterization using a generative AI function.Another step is aimed at evaluating the local update at the model aggregator using the synthetic representation to obtain an evaluation result indicating the performance of the model update. Another step is aimed at updating the machine learning model at the model aggregator based on the evaluation result and the local update.

[0012] According to another aspect, a computer-implemented method for federated learning of a machine learning model in a model aggregator device is provided. The method comprises a plurality of steps. One step is directed to receiving, at the model aggregator device, from a first local location located remotely from the model aggregator device, a first local update of the machine learning model and a log file, wherein the first local update was generated (or provided) at the local location based on local data, and the log file includes a parameterization of the local data. Another step is directed to generating, at the model aggregator device, a synthetic representation of the local data based on the parameterization using a generative AI function.Another step is directed to receiving, at the model aggregator device, from a second local location located remotely from the model aggregator device and different from the first local location, a second local machine learning update that is different from the first local update. Another step is directed to evaluating, at the model aggregator device, the second local update using the synthetic representation to obtain an evaluation result indicating the performance of the second model update. Another step is directed to updating, at the model aggregator device, the machine learning model based on the evaluation result and the second local update.

[0013] The model aggregator device may be a central server unit configured to administer the federated learning of the machine learning model. For example, the model aggregator device may comprise a web server. Furthermore, the model aggregator device may comprise a cloud server or a local server. The model aggregator device may be in data communication with one or more local locations. The model aggregator device may be configured to provide the machine learning model to the local locations and to receive updated machine learning models from the local locations (local updates). The model aggregator device may further be configured to evaluate the local updates and to decide on the integration of features of the local update into the machine learning model based on the evaluation step.The model aggregator device may comprise an interface unit for enabling data communications with the local sites, for example via an Internet connection.

[0014] The local site may be considered a client or client unit in the federated learning network administered by the model aggregator device. In particular, the local sites may comprise a local computer network comprising one or more computing units. The local sites may, for example, refer to an organization in which the machine learning model is to be deployed. In particular, the local sites may refer to healthcare environments, organizations, or facilities, such as hospitals, laboratories, medical practices, universities, or connections of one or more of the aforementioned. According to some examples, the model aggregator device is located outside the local sites and serves one or more of the local sites from the outside.

[0015] The machine learning model can be viewed as a master model in a federated learning scheme that is centrally administered by the model aggregator device.

[0016] In general, a machine learning model is designed to produce a desired or predetermined output by processing a certain type of input data. In doing so, a machine learning model mimics cognitive functions that humans associate with other human thoughts. Specifically, by training based on training data, the machine learning function can adapt to new circumstances, detecting and extrapolating patterns. Other terms for machine learning model include trained function, trained machine learning model, trained mapping specification, mapping specification with trained parameters, function with trained parameters, artificial intelligence-based algorithm, or machine learning algorithm.

[0017] In general, parameters of a machine learning model can be adjusted through training to obtain a model update (e.g., in the form of a local update or a central update of the machine learning model at the model aggregator device). In particular, supervised training, semi-supervised training, unsupervised training, reinforcement learning, and / or active learning can be used. Furthermore, representation learning can be used. In particular, the parameters of the machine learning model can be adjusted iteratively through multiple training steps.

[0018] In particular, the machine learning model may comprise a neural network, a support vector machine, a decision tree, and / or a Bayesian network, and / or the trained function may be based on k-means clustering, Q-learning, genetic algorithms, a transformer network, and / or association rules. In particular, a neural network may be a deep neural network, a convolutional neural network, or a convolutional deep neural network. Furthermore, a neural network may be an adversarial network, a deep adversarial network, and / or a generative adversarial network. Furthermore, a neural network may comprise a transformer network.

[0019] The local data may include (training) input data and, optionally, corresponding (training) output data. The training output data may be data that the machine learning function is expected to produce based on the input training data. The training output data may include verified outputs. According to some examples, the verified outputs may be verified by a (human) expert at the local site. It should be noted that unsupervised learning does not require output training data.

[0020] The local data, or parts of the local data, may be subject to a data protection requirement that restricts the transmission of training data outside the local site. Thus, according to some examples, the local data cannot be accessed from outside the local site. In particular, the model aggregator device cannot access the local data.

[0021] According to some examples, the local data may relate to medical data of one or more patients. For example, the local data may include laboratory test results and / or pathology data derived from pathology imaging, and / or medical image data generated by one or more medical imaging devices, such as computed tomography devices, a magnetic resonance system, an angiography system (or C-arm x-ray system), a positron emission tomography system, or the like, and any combination thereof. Furthermore, the local data may include supplemental information regarding a patient, such as diagnostic reports, information about administered treatments, information about symptoms and treatment responses, health history, and the like. Such information may be provided, for example, via an electronic health record (EHR).The local data can be stored locally in one or more databases at the local sites. The databases can be part of hospital information systems (HIS), radiology information systems (RIS), clinical information systems (CIS), laboratory information systems (LIS), and / or cardiovascular information systems (CVIS), picture archiving and communication systems (PACS), or the like. From these databases, the local data can be accessed locally for the training of machine learning models (and the subsequent regular use of the machine learning models after deployment). The local data may be subject to data protection regulations that prohibit the local data from leaving the local sites. In particular, the local data may include datasets with which a machine learning model can be trained, validated, and tested.According to some examples, the local data may include datasets based on which the local update was validated and / or tested. In other words, the local data may not include training data based on which actual further training of the machine learning model was performed. Datasets may include input data and associated output data that can be used to evaluate the performance of a machine learning model during supervised learning. The output data may be verified results corresponding to the input data. The output data may be generated and / or verified by a human based on the input data.

[0022] According to some examples, the local data comprises a plurality of individual data elements. Thus, each data element may comprise a training input data element and, optionally, a corresponding training output data element. For example, a training input data element may refer to a single medical image data set of a patient, and a training output data element may refer to a corresponding detection result. Thus, the local data may be considered a set comprising the plurality of data elements.

[0023] According to some examples, the local update of the machine learning model may include the machine learning model wherein one or more parameters of the machine learning model have been adjusted (or modified or optimized), in particular based on the local data. In particular, the one or more adjusted parameters may include one or more adjusted weights and / or adjusted hyperparameters of the machine learning model. "Generated at the local site based on local data" may include that the local update has been trained and / or validated and / or tested using the local data. In particular, the local data may be divided into training data, validation data, and test data.For actual training (in the sense of adapting the machine learning model to generate a local update), a backpropagation scheme based on a suitable cost function and using the training data can be used. Based on the validation data, the best-performing local update can be selected from among several local updates at the local site. Specificity and sensitivity can then be determined at the local site based on the test data. According to some examples, specificity and sensitivity can be included in the log file.

[0024] According to some examples, the parameterization may be considered a data minimization of the original local data. According to some examples, the parameterization may have a reduced information depth compared to the local data, and in particular a reduced data size. The parameterization may not be subject to data protection obligations. The parameterization may comprise one or more parameters, such as numerical values ​​or semantic expressions, that characterize the local data. According to some examples, the parameterization does not comprise the (original) local data and / or comprises (only) excerpts of the local data. In particular, the parameterization may be designed to enable reconstruction of at least parts of the local data, and in particular those parts that are not subject to data protection obligations.

[0025] According to some examples, the parameterization may be based on the training input data or the training input data and the training output data. According to some examples, the parameterization may be based (only) on the training output data. This is because the expected output encoded in the local training output data may already reflect the contents of the training input data at a higher level, and thus provide a good basis for generating the synthetic representations. According to other examples, the parameterization may be based on training output data and corresponding excerpts of the training input data. To provide an example, the parameterization may include a description of a medical finding (which would be the training output data) and a snippet of a medical image dataset showing the medical finding.

[0026] To provide an example from the field of medical imaging, parameterization may not include the full image data of a medical image dataset, but only certain key parameters. For example, this may include the imaging modality and imaging parameters used, image quality metrics, locations and descriptions of findings, image snippets, and so on. It should be noted that it may not include data based on which patient identification might be possible. Trivially, this may include demographic information about the patient, but also more subtle features that could lead to identification, such as body shapes or implants visible in the medical image.

[0027] According to some examples, the synthetic representation can be considered a restoration or reconstruction of the original local data based on the parameterization. According to some examples, the synthetic representation is not subject to data protection obligations. Furthermore, according to some examples, the synthetic representation can include the relevant characteristics for evaluating and / or training the machine learning model.

[0028] The generative AI function is a machine learning function or machine learning model configured to generate text, images, or other data based on input data. According to some examples, the input may be the parameterization or a natural language prompt generated based on the parameterization. In other words, the generative AI function exchanges the parameterization for the synthetic representation. According to some examples, the generative AI function may comprise a transformer network, in particular a transformer-based (deep) neural network. According to some examples, the local data may comprise image data, the parameterization does not comprise the image data, and the generative AI function is a parameterization-to-image function that outputs a synthetic version / representation of the image data.According to some examples, the generative AI function includes a vision transformer as described herein.

[0029] The generative AI function can be trained based on pairs of parameterizations and corresponding data, including, in particular, image data. This allows the parameterizations to be extracted from the corresponding data in the same or a similar way as the parameterization extracted from the local data. During training of the generative AI function, the corresponding data can be used as a ground truth against which the output of the generative AI function is compared.

[0030] According to some examples, the generative AI function may include a transformer network. A transformer network is a neural network architecture that generally includes an encoder, a decoder, or both an encoder and a decoder. In some cases, the encoders and / or decoders consist of multiple corresponding encoding and decoding layers, respectively. Within each encoding and decoding layer is an attention mechanism. The attention mechanism, sometimes referred to as self-attention, relates data items (such as words or pixels) within a series of data items to other data items in the series.

[0031] For an overview of transformer networks, see Vaswani et al., “Attention Is All You Need,” in arXiv: 1706.03762, June 12, 2017, the contents of which are incorporated herein by reference in their entirety.

[0032] According to some examples, commercially available generative AI functions can be used, such as DALL-E or Midjourney. According to other examples, custom-built generative AI functions based on a Transformer architecture can be used, which are trained based on pairs of parameterizations and corresponding (real-world) data.

[0033] According to some examples, assessing (another word is testing) may include testing whether the local update is capable of achieving sufficiently good results on unseen data. According to some examples, assessing may include allowing the machine learning model to process the synthetic representation and / or other test data available at the model aggregator device and comparing the result of the processing with the desired result. The other test data may include one or more synthetic representations obtained from parameterizations received from one or more local sites that are different from the local site.

[0034] These features work synergistically to provide the model aggregator with synthetic representations of the local data while ensuring that the privacy requirements of the local sites are met. Furthermore, data minimization also reduces the amount of data that needs to be transferred, reducing latency in the system. This enables a more efficient federated learning workflow in distributed environments.

[0035] According to some examples, the step of updating the machine learning model at the model aggregator device includes aggregating the local update in the machine learning model. This allows a master model to be successively optimized.

[0036] According to some examples, aggregating may include incorporating one or more updated parameters of the local update into the machine learning model. According to some examples, updating may be performed if the evaluation result indicates increased performance of the local update.

[0037] According to some examples, the parameterization is generated by applying a trained feature encoder to the local data. According to some examples, the feature encoder is provided to the local site by the model aggregator device. According to some examples, the parameterization comprises encoded features as identified by the feature encoder based on the local data. According to some examples, the encoder is different from and / or independent of the generative AI function. This may mean that the encoder was trained independently of the generative AI function and / or comprises a different architecture. According to some examples, generating the parameterization comprises applying the feature encoder to each individual data item of the local data to generate a set of encoded features for each data item, and appending each set of encoded features to the parameterization / log file.

[0038] According to some examples, the log file may include a performance log of the updated machine learning function. The performance log may indicate how well the machine learning model performed with local updating on the local data at the local site. According to some examples, the evaluating step additionally includes evaluating the local update based on the performance log. This allows for an improved evaluation of the local update. Alternatively, the parameterization itself may be provided without inclusion in a log file. In this case, no log file is generated / transmitted.

[0039] According to some examples, the log file is formatted according to the JSON standard. JSON stands for JavaScript Object Notation and is an open standard file format and data interchange format that uses human-readable text to store and transmit data objects consisting of attribute-value pairs and arrays. This can improve the interoperability of the process.

[0040] In one aspect, the parameterization consists of parameterizing the data used to validate and / or test the local update at the local site.

[0041] In other words, only a parameterization of the data used to validate and / or test the local update is generated / transmitted. The actual training data used to fine-tune (in the sense of adapting) the machine learning model is not parameterized, and / or no parameterization of such local data is transmitted according to this aspect. Thus, the parameterization does not include any parameterization of this actual training data. Consequently, the synthetic representation is (only) a synthetic reconstruction of the datasets used to test and / or validate the local update, but not of the training datasets.

[0042] This may have the advantage of reducing bias because the model(s) are systematically not evaluated at the model aggregator based on information on which they were trained.

[0043] According to one aspect, the generative AI function is configured to generate the synthetic representation based on a natural language prompt specifying the synthetic representation to be generated, and the generating step comprises obtaining the natural language prompt based on the parameterization and inputting the natural language prompt to the generative AI function to generate the synthetic representation.

[0044] A request can be viewed as a natural language description of the synthetic representation to be generated.

[0045] The advantage of using prompting is that the method is easily compatible with commercially available generative AI functions, which often require prompts as input.

[0046] According to some examples, the generation step involves entering the prompt along with parameterization. This has the advantage of providing the generative AI function with additional information to simulate the synthetic representation.

[0047] According to some examples, the prompts may be generated by a parser module configured to generate prompts based on parameterizations. The parser may include a speech decoder configured to generate natural language text based on the parameterization. The speech decoder may include a transformer network. Using a parser module may further automate and make the workflow more efficient.

[0048] According to some examples, the method further comprises modifying the natural language prompt to generate a modified natural language prompt, wherein the step of generating the synthetic representation comprises inputting the modified prompt to the generative AI function to generate another synthetic representation, and including the another synthetic representation in the synthetic representation.

[0049] In general, the modified prompt may include different content and / or different instructions for the generative AI function.

[0050] Modified prompts can be used to further expand the training data. At the same time, the modified prompt can serve as ground truth for further synthetic representation. For example, an initial prompt can specify that an X-ray image shows a lesion at a certain location in the patient's lung. Then, the modified prompt can specify a lesion at a (slightly) different location.

[0051] According to some examples, the method further comprises adding the synthetic representation to an existing test dataset for testing the machine learning model at the model aggregator device to generate an augmented test dataset, wherein the step of evaluating comprises evaluating the local update based on the augmented test dataset.

[0052] In other words, the synthetic data is appended to the model aggregator's test database. This can make the extended test dataset more representative of real-world scenarios and more resilient to concept drift at local sites.

[0053] According to some examples, the method further comprises determining a data quality of the synthetic representation, wherein in the step of assessing, the local update is assessed based on the data quality.

[0054] Data quality may include an indication of how well the simulation of the local data at the model aggregator device (i.e., the synthetic representation) actually is. In particular, data quality may include an indication of how well the synthetic representation corresponds to the local data.

[0055] According to some examples, the step of assessing comprises checking whether the data quality meets a predetermined quality criterion, and using the synthetic representation to assess the local update if the data quality meets the predetermined quality criterion.

[0056] According to some examples, the step of adding the synthetic data comprises adding the synthetic representation based on the data quality. According to some examples, the step of adding the synthetic data comprises checking whether the data quality meets a predetermined quality criterion and adding the synthetic data if it meets the predetermined quality criterion.

[0057] According to some examples, the parameterization represents a plurality of independent data items (the local data), and the step of generating a synthetic representation of the local data comprises generating a synthetic data item for each of the independent data items, such that the synthetic representation comprises the synthetic data items.

[0058] According to some examples, the step of determining data quality includes determining a data quality measure for each synthetic data element.

[0059] According to some examples, the step of checking comprises checking whether the data quality measure of each synthetic data element meets the predetermined criterion. According to some examples, the step of evaluating comprises using synthetic data elements to evaluate the local update whose data quality measure meets the predetermined criterion. According to some examples, the step of adding comprises adding synthetic data elements to the existing test data whose data quality measure meets the predetermined criterion.

[0060] By determining data quality and using this information when assessing local updates and / or extensions of the central test dataset, it can be ensured that inappropriate synthetic representations do not lead to undesirable effects.

[0061] According to one aspect, the step of determining a data quality of the synthetic representation comprises generating a backward parameterization of the synthetic representation, comparing the backward parameterization with the parameterization, and determining the data quality based on the step of comparing the backward parameterization with the parameterization.

[0062] According to some examples, the step of comparing includes determining a difference or distance between the parameterization and the reverse parameterization.

[0063] According to some examples, the reverse parameterization is generated in substantially the same way as the parameterization. According to some examples, the parameterization is generated at the local side by applying an encoder to the local data. According to some examples, the reverse parameterization is generated at the model aggregator device by applying the encoder or a copy of the encoder to the synthetic representation.

[0064] According to some examples, the step of generating the reverse parameterization comprises generating a reverse parameterization for each synthetic data item, and the step of comparing comprises comparing the reverse parameterizations with the respective parameterization (or the respective part of the parameterization) to generate a quality measure for each data item.

[0065] Due to the distributed environment, direct quality control of the synthetic representation is difficult. This is because the original training data typically must remain at the local sites and is therefore not available for comparison with the synthetic representation. Here, the proposed computation of backward parameterization provides an elegant way to obtain a reading that can be directly compared with the information uploaded to the model aggregator. This ensures the quality of the synthetic representation. In turn, this enables more efficient assessment of machine learning capabilities in a federated learning environment.

[0066] According to one aspect, the step of determining a data quality of the synthetic representation comprises generating a natural language summary based on the synthetic representation, comparing the natural language summary to the natural language prompt, and determining the data quality based on the step of comparing the natural language summary to the natural language prompt.

[0067] According to some examples, the comparing step includes determining a difference or distance between the natural language summary and the natural language prompt.

[0068] According to some examples, the summary is generated by applying a parser to the synthetic representation, wherein the parser is configured to summarize and / or describe data into natural language text. The parser may comprise a speech decoder configured to generate natural language text based on data, particularly image data. The speech decoder may comprise a transformer network. According to some examples, the parser may be of the same type as, or a copy of, the parser used in generating the prompt.

[0069] In one aspect, the step of generating the natural language summary comprises generating a natural language summary for each synthetic data item, and the step of comparing comprises comparing the natural language summaries to the respective natural language prompt to generate a quality measure for each data item.

[0070] By generating summaries based on the synthetic representations and comparing these summaries with the prompts used to trigger the generation of the synthetic representation, another quality control of the process can be performed. Quality control based on the prompts can be performed in addition to, or as an alternative to, quality control based on backward parameterization.

[0071] According to some examples, the local data comprises a plurality of independent data elements, and the parameterization comprises, for each independent data element, an element parameterization of each data element and one or more statistical properties of the plurality of independent data elements.

[0072] According to some examples, the statistical properties may include one or more distributions of parameters of the independent data items and / or statistical observables derived from the distributions. A statistical observable may refer to quantifiable properties of a corresponding distribution. According to some examples, a statistical observable may include mean values, entropy, skewness, variance, and so on.

[0073] According to some examples, the one or more statistical properties are input to the generative AI function along with the one or more respective element-wise parameterizations, and the synthetic representation is additionally generated based on the one or more statistical properties. In particular, the synthetic representation may be generated such that the corresponding statistical properties of the synthetic representation correspond to the statistical properties of the parameterization.

[0074] Using statistical properties can have the advantage of making the synthetic representation more accurate to the local data. In particular, it can ensure that the synthetic representation shows the same statistics as the local data.

[0075] According to some examples, the step of determining a data quality of the synthetic representation comprises determining one or more corresponding statistical properties based on the synthetic representation and comparing the corresponding statistical properties to the statistical properties. The corresponding statistical properties may include the same statistical observables as the statistical properties.

[0076] In other words, quality control based on a statistical test can be implemented. This provides an efficient way to monitor the quality of the synthetic representation in a scenario where the original training data is not available for direct comparison.

[0077] According to one aspect, the method further comprises generating a modified parameterization based on the parameterization, wherein in the generating step the synthetic representation is additionally generated based on the modified parameterization.

[0078] According to some examples, the modified parameterization may include one or more values ​​that differ from the parameterization. According to some examples, the modified parameterization is designed in such a way that it results in (slightly) different synthetic representations, when processed with the generative AI function, compared to the (original) parameterization. For example, if one section of the parameterization specifies a finding with certain characteristics, the corresponding section in the modified parameterization may specify a finding with different characteristics or no finding at all. The latter, in particular, is easy to implement, fail-safe, and can increase the number of normal, i.e., non-suspicious, samples in the synthetic data.

[0079] According to some examples, the modified parameterization may be generated using a trained function designed to derive (physically or medically) meaningful modifications for the parameterization. According to some examples, a corresponding trained function may be trained by building upon the quality control described herein until the trained function is capable of generating acceptable modified parameterizations.

[0080] In other words, a data augmentation of the parameterization is performed. In turn, this leads to additional and more diverse data for testing the machine learning model. Furthermore, perturbation can have the advantage that the data sets used for the actual fitting of the machine learning model can be more easily used in the evaluation step without introducing too much bias.

[0081] According to some examples, the data quality control measures described in connection with the synthetic representation generated based on the parameterization may also be applied to the synthetic representation generated based on the modified parameterization.

[0082] In particular, the step of determining the data quality of the synthetic representation involves determining one or more corresponding statistical properties based on the synthetic representation (generated based on the modified parameterizations) and comparing the corresponding statistical properties with the statistical properties of the (original) parameterization. This allows checking whether the modified parameterizations still lead to the same statistics.

[0083] According to some examples, the local data comprises a plurality of independent data elements, and the parameterization comprises an element parameterization for each independent data element, wherein the step of generating a modified parameterization comprises generating modified element parameterizations, and the step of generating the synthetic representation comprises generating a synthetic data element for both the element parameterization and the modified element parameterization, wherein the synthetic representation comprises the synthetic data elements.

[0084] According to some examples, the local data includes protected information, in particular protected personal information, and the parameterization does not include the protected information.

[0085] According to some examples, the protected information may be information that is not permitted to leave the local site. For example, the local data may be subject to a privacy policy, such as a confidentiality agreement, or a legal requirement such as the EU General Data Protection Regulation.

[0086] Deleting the protected information used to generate the parameterization allows such data to be distributed outside the local site. This allows this information to be used to test the machine learning model on the model aggregator device.

[0087] According to one aspect, the machine learning model is an image processing function configured to generate an image processing result based on image data, the local data comprises training image data, the parameterization comprises a parameterization of the training image data, the synthetic representation comprises synthetic image data generated by the generative AI function based on the parameterization of the training image data.

[0088] The local data includes (training) image data and verified image processing results. For example, the image data may include images captured by a camera system or other image acquisition systems. According to some examples, this may refer to image data captured by a smartphone camera or a camera system attached to a car. The image processing result may refer to an object detected in the image data, such as people, text, cars, lane markings, or other objects.

[0089] According to some examples, the parameterization includes basic characteristics of the image data, such as resolution, color, noise level, acquisition system, and so on. Furthermore, it may include information characterizing the content of the underlying image data. According to some examples, this information may include feature vectors or embeddings extracted by an appropriately configured encoder. Furthermore, according to some examples, the information may include one or more semantic meanings and relationships for what is represented in the image data. The parameterization may be human-comprehensible (e.g., "the image shows a person riding a bicycle") and / or may be (only) machine-interpretable information, such as complex feature vectors. Furthermore, the parameterization may include information about the verified image processing result (i.e., the training output data).According to some examples, the parameterization of the image data may be identical to the parameterization of the image processing result.

[0090] According to some examples, the parameterization may have been generated by applying an encoder or feature encoder to the local data. According to some examples, the feature encoder may comprise a vision transformer. The vision transformer may be configured to decompose input images into patches and tokenize them (extracting representation vectors) before applying the tokens to a standard transformer architecture. The vision transformer may comprise an attention mechanism configured to repeatedly transform representation vectors of image patches to incorporate more and more semantic relationships between image patches in an image.

[0091] According to some examples, the vision transformer and / or the generative AI function can be obtained by training a masked autoencoder. A masked autoencoder comprises two vision transformers concatenated. The first takes image patches with position encoding and outputs vectors representing each patch. The second takes vectors with position encoding and outputs image patches again. During training, both vision transformers are used. An image is sliced ​​into patches. The second vision transformer takes the encoded vectors and outputs a reconstruction of the full image. During use, the first vision transformer can be used as an encoder, and / or the second vision transformer can be used as a generative AI function. Thus, the encoder and generative AI function are complementary by design, enabling seamless data processing with bounded losses.

[0092] According to some examples, the vision transformer and / or the generative AI function can be obtained by training a vision transformer (VQGAN) or can be based on it. In a vision transformer (VQGAN), there are two vision transformer encoders and a discriminator. One encodes patches of an image into a list of vectors, one for each patch. Another encodes the quantized vectors back to image patches. The training goal attempts to make the reconstruction image (the output image) faithful to the input image. The discriminator (usually a convolutional network, but other networks are possible) attempts to decide whether an image is an original real image or an image reconstructed by the vision transformer.

[0093] This has the advantage that, once such a vision transformer (VQGAN) is trained, it can be used to encode any image into a list of symbols, and any list of symbols into an image. The list of symbols can be used to train a standard autoregressive transformer to autoregressively generate an image. Furthermore, a list of caption-image pairs can be taken, the images converted into strings of symbols, and a standard GPT-style transformer trained. Then, at test time, only an image caption can be given, and the image can be autoregressively generated.

[0094] According to some examples, the synthetic image data is generated to resemble or mimic the local data as much as possible.

[0095] By parameterizing and reconstructing image data, an efficient federated learning scheme can be provided. In particular, the method ensures data accessibility while safeguarding privacy and reducing data traffic.

[0096] According to some examples, the parameterization does not include image data.

[0097] This can have the advantage of particularly efficient data minimization.

[0098] According to some examples, the parameterization may include one or more image patches extracted from the local data. In other words, the parameterization may include a subset of the image data in the local data. According to some examples, the patches may correspond to the image detection results. In particular, a patch may be a section of the image data. In other words, the parameterization may include only the most relevant image data, while less relevant parts of the image data are not included in the parameterization.

[0099] This may have the advantage of allowing a more accurate synthetic representation of the image data to be created at the model aggregator device while still allowing for appropriate data minimization.

[0100] According to some examples, the machine learning model is configured to generate an image processing result based on medical image data, wherein the image processing result is selected from: a detection result of a medical finding in medical image data, a classification of a medical finding in medical image data, and / or a segmentation of medical image data, and the training image data comprises medical image data.

[0101] Accordingly, the synthetic representation may include a synthetic reconstruction of the medical image data.

[0102] According to some examples, the medical image data includes a plurality of medical image data sets, each showing a body part of a patient.

[0103] The medical image dataset may refer to a medical image study. The medical image dataset may refer to three-dimensional datasets, which provide three dimensions in space or two dimensions in space and one dimension in time, two-dimensional datasets, which provide two dimensions in space, and / or four-dimensional datasets, which provide three dimensions in space and one dimension in time.

[0104] The medical image dataset may represent a patient's body part in the sense that it contains three-dimensional image data of the patient's body part. The medical image dataset may represent an image volume. The patient's body part may be contained in the image volume.

[0105] The medical image dataset comprises image data, for example, in the form of a two- or three-dimensional array of pixels or voxels. Such arrays of pixels or voxels may represent intensity, absorption, or other parameters as a function of three-dimensional position and may be obtained, for example, by appropriate processing of measurement signals obtained with a medical imaging modality.

[0106] A medical imaging modality is a system used to generate or produce medical image data. For example, a medical imaging modality can be a computed tomography (CT) system, a magnetic resonance imaging (MR) system, an angiography system (or C-arm X-ray system), a positron emission tomography (PET) system, an ultrasound imaging system, or the like. In particular, computed tomography is a widely used imaging technique and utilizes "hard" X-rays generated and detected by a specially rotating instrument. The resulting attenuation data (also called raw data) is presented by analytical computer software that produces detailed images of the internal structure of the patient's body parts.The resulting sets of images are called CT scans and can represent numerous series of sequential images to present internal anatomical structures in cross-sections perpendicular to the axis of the human body. Magnetic resonance imaging (MRI), to provide another example, is an advanced medical imaging technique that utilizes the effect of a magnetic field on the movement of protons. In MRI machines, the detectors are antennas, and the signals are analyzed by a computer, producing detailed images of the internal structures in any section of the human body.

[0107] Accordingly, the imaged patient body part will generally include a variety of anatomies and / or organs (also referred to as compartments or anatomical structures). For example, a chest image may depict lung tissue, bones (e.g., the rib cage), the heart and aorta, lymph nodes, and more.

[0108] The medical image dataset may comprise a plurality of images or image slices. The slices may each show a cross-sectional view of the image volume. The slices may comprise a two-dimensional array of pixels or voxels as image data. The arrangement of slices in the medical image dataset may be determined by the imaging modality or by any post-processing scheme used.

[0109] Furthermore, the medical image dataset may include a two-dimensional pathological image dataset, i.e. a so-called whole-slide image representing a tissue section of the patient.

[0110] According to some examples, the medical image dataset may have been captured at the local site.

[0111] The medical image dataset can be stored in a standard image format, such as the Digital Imaging and Communications in Medicine (DICOM) format, and in a local memory or computer storage system, such as a picture archiving and communication system (PACS). Whenever DICOM is mentioned herein, it should be understood that this refers to the Digital Imaging and Communications in Medicine (DICOM) standard, e.g., according to the DICOM PS3.1 2020c standard (or any later or earlier version of this standard).

[0112] According to some examples, the local data set comprises for each medical image data set a verified image processing result, in particular a verified detection result of a medical finding in the respective medical image data set, a verified classification result of a medical finding in the respective medical image data set and / or a segmentation of the respective medical image data set.

[0113] According to some examples, the parameterization includes the verified image processing result or a parameterization thereof. As the case may be, the parameterization may include an indication of the medical imaging modality (or modalities) used to acquire the medical image data, imaging parameters used in acquiring the medical image data, a medical finding included in the medical image data, and a segmentation of an object included in the medical image data.

[0114] A medical finding may indicate a specific condition or pathology of the patient. The condition or pathology may be relevant to the patient's diagnosis.

[0115] A medical finding can refer to an anatomical structure that distinguishes the patient from other patients. Medical findings can be located within different organs of the patient (e.g., within a patient's lungs or within a patient's liver) or between the patient's organs. In particular, a medical finding can also refer to a foreign body.

[0116] In particular, a medical finding may refer to a neoplasm (also referred to as a "tumor"), in particular a benign neoplasm, a neoplasm in situ, a malignant neoplasm, and / or a neoplasm of uncertain / unknown behavior. In particular, a medical finding may refer to a nodule, in particular a lung nodule. In particular, a medical finding may refer to a lesion, in particular a lung lesion.

[0117] Classification may refer to identifying a type of finding and / or providing a classification according to a plurality of predefined classes such as benign or malignant.

[0118] According to some examples, segmentation may be targeted to an organ, a finding, or another compartment of a patient's body part. According to some examples, the segmentation step may involve obtaining an outline of the respective organ or compartment and / or delimiting the organ or compartment from the rest of the image data.

[0119] The advantages of the method become particularly apparent when applied to medical image processing. This is because the medical environment, in particular, is regulated by restrictive data protection guidelines. At the same time, there are strict guidelines regarding the quality and validation of machine learning functions.

[0120] According to some examples, the parameterization does not include protected health information. Protected health information may specifically refer to personal information or other information about the patient that could lead to the patient's identification.

[0121] According to some examples, the parameterization includes one or more patches of the medical image data, each created around a finding represented in the medical image data. Another word for patch can be patch. A patch will generally display the finding and surrounding tissue, but not the entire medical image. This allows for better synthetic representations to be created that more closely reflect the original local data.

[0122] In one aspect, the method further comprises providing the updated machine learning model to a second local location different from the local location.

[0123] By deploying the updated machine learning model to other locations, the knowledge gathered at one location through a local model update and centrally verified at the model aggregator device can be shared and distributed. At the second local location, another local model update can be generated based on second local data. The additional local update can be received at the model aggregator device along with a parameterization of the second local data, and the process can start again for the additional local update.

[0124] According to one aspect, a computer-implemented method for federated learning of a machine learning model is provided. The method comprises a plurality of steps. A first step is directed to receiving, at a local site, a machine learning model from a model aggregator device located remotely from the local site. Another step is directed to generating (or providing), at the local site, a local update of the machine learning model using local data of the local site. Another step is directed to generating, at the local site, a parameterization of the local data. Another step is directed to transmitting, by the local site, the local update and the parameterization to the model aggregator device.

[0125] According to another aspect, a computer-implemented method for federated learning of a machine learning model is provided. The method comprises a plurality of steps. A first step is directed to receiving, at a local site, from a model aggregator device located remotely from the local site, a machine learning model (or optionally the generative AI function). Another step is directed to generating (or providing), at the local site, a local update of the machine learning model using local data of the local site. Another step is directed to generating, at the local site, a parameterization of the local data. Another step is directed to generating, at the local site, a synthetic representation of the local data based on the parameterization using a generative AI function.A further step is aimed at transmitting, through the local site, the local update and the synthetic representation to the model aggregator device.

[0126] In other words, the above method is directed toward client-side processing. The steps can be further detailed and combined with other features according to the aspects and examples described herein. In particular, the client-side processing steps can be combined with server-side processing steps at the model aggregator device. The advantages described in connection with the other aspects and examples of the disclosure are also realized by the appropriately configured client-side processing steps.

[0127] According to one aspect, a computer-implemented method for providing a synthetic representation of local data at a local location to an aggregator device is provided. The method comprises a plurality of steps. One step is directed to generating, at the local location, a parameterization of the local data. Another step is directed to transmitting the parameterization from the local location to the aggregator device. Another step is directed to receiving the parameterization at the aggregator device. Another step is directed to generating, at the aggregator device, a synthetic representation of the local data based on the parameterization using a generative AI function. Another step is directed to providing the synthetic representation to the aggregator device.

[0128] According to an alternative aspect, a computer-implemented method for providing a synthetic representation of local data at a local location to an aggregator device is provided, the method comprising generating, at the local location, a synthetic representation of the local data using a generative AI function and providing the synthetic representation from the local location to the aggregator device. In particular, generating the synthetic representations at the local locations may comprise generating, at the local location, a parameterization of the local data and generating, at the local location, a synthetic representation of the local data based on the parameterization using the generative AI function.

[0129] The above methods can provide a privacy-preserving manner for exchanging information. The aggregator device can be configured in a manner equivalent to the model aggregator device. Furthermore, the above method can be modified according to the other examples and aspects described herein and can provide similar advantages.

[0130] According to one aspect, a model aggregator device for federated learning of a machine learning model is provided, wherein the model aggregator device comprises a computing unit and an interface unit. The interface unit is configured to receive a local update of the machine learning model and a log file from a local location remote from the model aggregator device, wherein the local update was generated at the local location based on local data, and the log file comprises a parameterization of the local data.The computing unit is configured to generate a synthetic representation of the local data based on the parameterization using a generative AI function, evaluate the local update using the synthetic representation to obtain an evaluation result indicating the performance of the model update, and update the machine learning model based on the evaluation result and the local update.

[0131] The computing unit can be implemented as a data processing system or as part of a data processing system. Such a data processing system can comprise, for example, a cloud computing system, a computer network, a computer, a tablet computer, a smartphone, and / or the like. The computing unit can comprise hardware and / or software. The hardware can comprise, for example, one or more processors, one or more memories, and combinations thereof. The one or more memories can store instructions for carrying out the method steps according to the invention. The hardware can be configurable by the software and / or operable by the software. In general, all units, subunits, or modules can at least temporarily exchange data with one another, e.g., via a network connection or corresponding interfaces. Consequently, individual units can be arranged remotely from one another.

[0132] Furthermore, the computing unit can be designed as an edge device.

[0133] The interface unit may comprise an interface for exchanging data with one or more local clients, e.g., via the Internet. The interface unit may further be adapted to interface with one or more users of the system, e.g., by displaying the processing result to the user (e.g., in a graphical user interface).

[0134] The model aggregator device may be adapted to implement the methods described herein in their various aspects and examples for federated learning of a machine learning function. The advantages described in connection with the method aspects and examples may also be realized by the appropriately configured system components.

[0135] According to one aspect, a local model update device for federated learning of a machine learning model is provided. The local model update device is located at a local site. The local model update device comprises a local interface unit and a local computing unit. The local interface unit is configured to receive a machine learning model from a model aggregator device located remotely from the local site and to transmit a local update of the machine learning model and a parameterization of the local data used to generate the local update at the local site to the model aggregator device. The computing unit is configured to generate the local update of the machine learning model at the local site based on the local data and to generate the parameterization of the local data.

[0136] One or more of the local model update devices may be combined with the model aggregator device to form a system for federated learning of the machine learning function. The one or more local computing units may be generally configured in a manner equivalent to the computing unit. Likewise, the one or more local interface units may be configured in substantially the same manner as the interface unit.

[0137] According to another aspect, the present invention is directed to a computer program product comprising program elements that cause a computing unit of a model aggregator device (or a local model update device) for federated learning of a machine learning function to perform the steps according to one or more of the above method aspects and examples when the program elements are loaded into a memory of the computing unit.

[0138] In another aspect, the present invention is directed to a computer-readable medium storing program elements that are readable and executable by a computing unit of a model aggregator device (or a local model update device) for federated learning of a machine learning function to perform the steps according to one or more method aspects and examples when the program elements are executed by the computing unit.

[0139] The realization of the invention by means of a computer program product and / or a computer-readable medium has the advantage that already existing provisioning systems can be easily adapted by software updates in order to operate as proposed by the invention.

[0140] The computer program product may, for example, be a computer program or comprise another element in addition to the computer program as such. This other element may be hardware, e.g., a storage device in which the computer program is stored, a hardware key for using the computer program, and the like, and / or software, e.g., documentation or a software key for using the computer program. The computer program product may further comprise development material, a runtime system, and / or databases or libraries. The computer program product may be distributed among multiple computer instances.

[0141] Both the characteristics, features, and advantages of the invention described above, as well as the manner in which they are achieved, will become clearer and more understandable in light of the following description of embodiments, which are described in detail with reference to the figures. This following description does not limit the invention to the embodiments included. The same components, parts, or steps may be designated by the same reference numerals in different figures. In general, the figures are not drawn to scale. In the following: Fig. 1 schematically illustrates an embodiment of a system for federated learning of a machine learning function according to an embodiment; Fig. 2 schematically illustrates a method for federated learning of a machine learning function according to an embodiment; Fig. 3 schematically illustrates an example data flow diagram associated with a method for federated learning of a machine learning function according to an embodiment; Fig. 4 schematically illustrates an example data flow diagram associated with a method for federated learning of a machine learning function according to an embodiment; Fig. 5 schematically illustrates a method for federated learning of a machine learning function according to an embodiment; Fig. 6 schematically illustrates an exemplary data flow diagram associated with a method for federated learning of a machine learning function according to one embodiment; and Fig. 7 schematically illustrates an encoder-decoder-transformer network according to one embodiment.

[0142] Fig. 1 illustrates an exemplary system 1 for federated learning of a machine learning model ML in a distributed environment. The system may be capable of generating, training, updating, distributing, monitoring, and generally administering the machine learning model ML in an environment comprising a plurality of local locations LS, LS-2, LS-3. The system 1 is adapted to perform the method according to one or more embodiments, e.g., as further described with reference to Fig. 2 to 6.

[0143] System 1 comprises a model aggregator device MAD and a plurality of clients, each located at different local locations LS, LS-2, LS-3. The model aggregator device MAD and the clients can be linked via a network. The model aggregator device MAD is generally designed to control, coordinate, and direct the federated learning procedures in System 1. The local locations LS, LS-2, LS-3 can, for example, refer to clinical or medical environments, such as hospitals or hospital groups, clinics, or medical practices.

[0144] The machine learning model (ML) can be viewed as a master model that is centrally administered by the model aggregator (MAD) and distributed to the local locations (LS, LS-2, LS-3) and further trained there. The machine learning model (ML) can generally be designed to provide a medical diagnosis based on medical input data. This can include outcome prediction, the detection of findings in medical image data, the annotation of medical images (e.g., with regard to orientation or landmark detection), the generation of medical reports, and the like.

[0145] The model aggregator device MAD can be hosted on a server, which can be a cloud server or a local server. However, the model aggregator device MAD can also be implemented using one or more other suitable computing devices. The model aggregator device MAD comprises a computing unit CU and an interface unit IU. Furthermore, the model aggregator device MAD can have access to a central database CDB, which is designed to centrally store training data for evaluating the machine learning model ML.

[0146] The computing unit CU may comprise one or more processors and a memory. The one or more processors may, for example, comprise one or more central processing units (CPUs), graphics processing units (GPUs), and / or other processing devices. The computing unit CU may further comprise a microcontroller or an integrated circuit. Alternatively, the computing unit CU may comprise a real or virtual group of computers, such as a so-called 'cluster' or 'cloud'. The memory may comprise one or more computer-readable media, such as a RAM for temporarily loading data, e.g., data from the database CDB or data uploaded from the local sites LS, LS-2, LS-3.The memory may further store information accessible to the one or more processors to perform method steps according to one or more embodiments described herein.

[0147] The interface units IU may include any suitable components for interfacing with one or more networks, including, for example, transmitters, receivers, ports, controllers, or other suitable components. The model aggregator device MAD may exchange information with one or more local sites LS, LS-2, LS-3 via the interface units IU. Any number of local sites LS, LS-2, LS-3 may be connected to the model aggregator device MAD via the interface units IU.

[0148] The computing unit CU can comprise subunits SYNTH, AGGR, and MGMT. The subunit MGMT can be a management module or a management unit designed to control and administer the federated learning of the machine learning model ML in System 1. The subunit MGMT can trigger the distribution of the machine learning model ML to the local sites LS, LS-2, and LS-3 and initiate the update of the machine learning model ML in System 1 as soon as a new, updated version ML* is available.

[0149] The SYNTH subunit can be considered a training data synthesizer. The SYNTH subunit is designed to generate synthetic representations SR of the local data LTD based on appropriate parameterizations P. For this purpose, the SYNTH subunit can be designed to host and execute an appropriately configured generative AI function GEN.

[0150] The AGGR subunit can be considered a model update unit. The AGGR subunit is designed to evaluate local model updates ML' and, if they represent an improvement, to aggregate the local updates ML' into the master machine learning model ML. For this purpose, the AGGR subunit can be designed to apply a cross-validation scheme.

[0151] The designation of the various subunits SYNTH, AGGR, MGMT is to be interpreted as exemplary and not limiting the disclosure. Accordingly, the subunits SYNTH, AGGR, MGMT can be integrated to form a single processing unit, or they can be embodied by computer code segments designed to execute the corresponding method steps running on a processor or the like of the computing unit CU. Each subunit SYNTH, AGGR, MGMT can be individually connected to other subunits and / or other components of the system 1, whereby data exchange is required to perform the method steps.

[0152] The central database CDB can be implemented as cloud storage. Alternatively, the central database CDB can be implemented as local or distributed storage, particularly within the premises of the model aggregator device MAD. The central database CDB is designed to store central training data CTD.

[0153] Each of the local sites LS, LS-2, and LS-3 includes a local model update device (LMUD) and a local database (LDB). The local database (LDB) can be implemented as a local or distributed storage facility within the premises of the respective local site (LS, LS-2, and LS-3). The local database (LDB) can store the local (training) data (LTD) to be processed by the machine learning model (ML).

[0154] The local data LTD may include a number of individual data elements relating, for example, to a clinical or medical problem. As an example, the data elements may relate to laboratory test results and / or pathological data and / or medical imaging data, electronic medical records, and any combination thereof. The local data LTD may relate to medical data of one or more patients. The local data LTD may have been generated at the respective local sites LS, LS-2, LS-3. The local database LDB may be part of hospital information systems (HIS), radiology information systems (RIS), clinical information systems (CIS), laboratory information systems (LIS), and / or cardiovascular information systems (CVIS), a picture archiving and communication system (PACS), or the like.

[0155] The local LTD data can be accessed locally from the local database EDB for training the ML machine learning model and for subsequent regular use of the ML machine learning model after deployment. Training can include adapting the machine learning model and validating and testing the adapted machine learning model. The local data can be divided into training data, validation data, and test data. The backpropagation algorithm can be used to train the machine learning model at the local sites based on a suitable cost function and using the training data. Based on the validation data, the best-performing machine learning model can be selected from several machine learning models (with different hyperparameters, e.g., number of layers, size and number of kernels, padding, etc.).The specificity and sensitivity can then be determined based on the test data.

[0156] In particular, the local data LTD cannot be accessed from the outside, as the local data LTD may be subject to data protection regulations that prohibit the local data LTD from leaving the local sites LS, LS-2, and LS-3. The local data LTD may include training input data and associated training output data, which can be used to evaluate the performance of a machine learning model (ML) during training. The output training data may refer to verified results corresponding to the input training data. The output training data may be generated and / or verified by a human based on the input training data.

[0157] The local model update device LMUD may comprise a local computing unit LCU and a local interface unit LIU. The local interface unit LIU may be configured in an equivalent manner to the interface unit IU and may include any suitable components for interfacing with the interface unit IU via a network, such as the Internet.

[0158] The local computing unit LCU is designed to further train the machine learning model ML based on the local data LTD in order to provide a local update ML' of the machine learning model ML. For this purpose, the local computing unit LCU can comprise a correspondingly configured training unit or a correspondingly configured training module TRN. Furthermore, the local computing unit LCU can comprise a parameterization module or a parameterization unit PAR, which is designed to generate a parameterization P of the local data LTD. For this purpose, the parameterization unit PAR can be designed to host a correspondingly configured encoder function ENC.To ensure that no privacy-sensitive information can be derived or inferred from the parameterization P, one or more encryption techniques, random noise techniques, and / or other security techniques can be added by the parameterization unit PAR after the generation of the parameterizations P. Both the local update ML' and the parameterization P can be provided to the model aggregator device MAD via the local interface unit LIU.

[0159] The designation of the various subunits TRN, PAR is to be interpreted as exemplary and not limiting the disclosure. Accordingly, the subunits TRN, PAR can be integrated to form a single processing unit, or they can be embodied by computer code segments designed to execute the corresponding method steps running on a processor or the like of the local processing unit LCU.

[0160] Local Computing Units (LCUs) may be any suitable type of computing device, such as a general-purpose computer, a special-purpose computer, a laptop, a local server system, or other suitable computing device. The Local Computing Units (LCUs) may include one or more processors and memory. The one or more processors may, for example, include one or more central processing units (CPUs), graphics processing units (GPUs), and / or other processing devices. The memory may include one or more computer-readable media and may store information accessible by the one or more processors, including instructions executable by the one or more processors. The instructions may include instructions for locally further training the machine learning model (ML) and / or generating parameterizations (P).

[0161] In an alternative embodiment (not shown), it would also be conceivable to provide the local locations LS, LS-2, LS-3 with modules for generating synthetic representations SR of the local data LTD based on corresponding parameterizations P. Then, the synthetic representation SR would be generated directly at the local locations LS, LS-2, LS-3 and transmitted to the model aggregator device MAD. Data protection requirements can also be met according to this alternative, since only the synthetic representation SR leaves the local locations LS, LS-2, LS-3.

[0162] Fig. 2 shows a method for federated learning of a machine learning model in a distributed environment. Corresponding data streams are stored in Fig. 3. Additionally, Fig. 4 Data streams associated with model aggregation at the model aggregation device MAD. The method comprises several steps. The order of the steps does not necessarily correspond to the numbering of the steps, but may also vary between different embodiments of the present invention. Furthermore, individual steps or a sequence of steps may be repeated.

[0163] Steps C10-C40 take place on the client side, i.e., at the respective local sites LS, LS-2, LS-3, and can be executed by the local model update device LMUD. Steps S10-S80 take place on the server side and can be executed by the model aggregator device MAD. According to methods of the present invention, the steps can be executed separately. In other words, aspects of the invention cover methods that include only client-side method steps, while other aspects cover methods that include only server-side steps. Furthermore, aspects of the invention can also cover methods that include both server-side and client-side steps.

[0164] In step C10, a machine learning model ML is received at the local location LS. The machine learning model ML can be a copy of the master model provided and administered by the model aggregator device MAD. The machine learning model ML is fully trained and is to be deployed at the local location LS according to the learned task. According to embodiments, this learned task can include automated processing of medical image data to derive a medical diagnosis. In particular, the machine learning model can be configured to process medical image data of a patient to detect and / or classify medical findings. According to some examples, the medical image data can show parts of the patient's upper body, and the findings can include lesions in the patient's lungs or liver.According to other examples, the medical image data comprises digital pathology images of the patient, and the findings relate to a segmentation of the digital pathology image according to one or more tissue types.

[0165] At step C20, the machine learning model ML can be further trained based on the local data LTD at the local locations LS. This results in a local update ML' of the machine learning model ML. According to some examples, such training can occur on-the-fly, e.g., when a user at the local location reviews the processing results of the machine learning model ML. For example, a radiologist can reject or accept lesions found by the machine learning model ML. Further, the radiologist can add lesions not found by the machine learning model. According to other examples, a pathologist can modify a segmentation provided by the machine learning model ML. The user inputs can be used as ground truth for further optimization, i.e., training, of the machine learning model ML at the local locations LS, LS-2, LS-3.The ground truth together with the underlying local data can form the local data LTD.

[0166] At step C30, a parameterization P of the local data LTD or a portion of the local data may be generated. In particular, only the portion of the local data used to validate the further trained machine learning model may be parameterized. The parameterization P may refer to a data minimization step in which the local data LTD is mined to a version that still allows the synthesizing or reconstructing of the local data LTD at the model aggregator device MAD, but that does not contain unnecessary information. In particular, the parameterization P may not include information subject to data protection regulations, such as personal information of a patient.

[0167] The parameterization P may include a plurality of characteristic values ​​of the underlying medical image and, optionally, image data extracts of the medical image. To provide an example, the parameterization P may be as follows: Type: Chest CT Scan, Bolus Agent: xyz, Modality: Siemens Healthineers CT Scanner, Model Number: 12345, Kilovoltage Peak: xxx, Milliampereseconds: yyy, Lung Nodule 1: Size: 11 mm, Type: solid, Location: upper left lung lobe, Lung Nodule 2: Size: 16 mm, Type: ggn, Location: lower left lung lobe, etc. According to other examples, the parameterization P may have a more abstract form and may be provided in the form of embeddings that can be interpreted by the generative AI function GEN, but not necessarily by a human user. Further, the parameterization P may also include one or more statistical properties of the entire local data.

[0168] At step C30, the parameterization P can be generated by an encoder ENC or autoencoder, which can be provided to the local site LS by the model aggregator device MAD together with the machine learning model ML. The encoder ENC can be considered a counterpart of the generative AI function GEN and can be trained in conjunction with the generative AI function GEN, as described herein.

[0169] At step C40, the local update ML' and the parameterization P are transmitted by the local interface units LMUD to the model aggregator device MAD.

[0170] In step S10, the local update ML' and the parameterization P are received in return at the model aggregator device MAD. In step S20, a synthetic representation SR of the local data LTD is generated. For this purpose, the generative AI function GEN can be applied to the parameterization P. According to the above example, the synthetic representation SR again comprises a medical image, such as a radiological or pathological medical image, as a reconstruction of the corresponding image data at the local location LS.

[0171] Optionally, the generative AI function GEN can operate based on a natural language prompt. The prompt can be viewed as an instruction or control command for the generative AI function. Such a prompt can be generated in the optional substep S21 based on the parameterization P. In a way, step S21 can be viewed as a translation step for translating the parameterization P into a set of instructions on the basis of which the generative AI function GEN can operate.

[0172] In the optional step S22, the request is entered into the generative AI function to trigger the generation of the synthetic representation SR.

[0173] In step S30, the synthetic representation SR is optionally used together with other training data already present in the central database CDB to evaluate the local update ML'. The result of the processing can be provided in the form of an evaluation result.

[0174] A data flow diagram of a model evaluation and aggregation process is shown in Fig. 4. As shown in Fig. As can be seen in Figure 4, the model aggregator device MAD receives not only local updates ML' from one local location LS, but from a plurality of local locations LS, LS-1, LS-2. Similarly, the model aggregator device MAD can receive synthetic representations SR from different local locations LS, LS-1, LS-2.

[0175] To test, validate, and ultimately arrive at an updated master model ML*, a validation scheme can be used at substep S31. In particular, a cross-validation scheme can be used, according to which the available data, i.e., the synthetic representation SR and other appropriate training data available at the model aggregator device MAD, are partitioned into a plurality of complementary subsets or folds. Different available models (or their parameters) can be combined according to the trail principle to generate a plurality of candidate model updates ML_tmp. Additional further training of these different combinations ML_tmp can be performed on a subset of the available data (called the training set or training fold), and a test is performed on the other subset (called the test set or test fold).To reduce variability, multiple cross-validation runs can be performed using different partitions of the training data, and the validation results are combined (e.g., averaged) across the different partitions to yield an estimate of the predictive performance of the respective machine learning model ML_tmp. If additional hyperparameters need to be optimized, nested cross-validation schemes can be applied. These essentially rely on (1) inner cross-validation to tune the hyperparameters and select optimal hyperparameters, and (2) outer cross-validation, which is used to evaluate the model trained using optimized hyperparameters, as selected by inner cross-validation. The best model can be provided as a (final) evaluation result.

[0176] Before synthetic representations SR are used in the evaluation of the machine learning model ML_tmp, they may be subject to quality control, as described in connection with the Fig. 5 and Fig. 6 described.

[0177] Based on the model evaluation and aggregation result of step S30, the variant with the best performance can be adopted for the updated version of the master model and provided as the (global) update ML* of the machine learning model ML.

[0178] As can be seen from the Fig. 3 and Fig. 4, the method may further comprise a step of adding the synthetic representation SR to the central database CDB. This may take place in optional step S50. Furthermore, the addition of synthetic representations SR to the central database CDB may be preceded by quality control steps to determine whether the synthetic representation SR has sufficient quality. In this respect, the steps associated with the Fig. 5 and Fig. 6 are explained.

[0179] Finally, in optional step S60, the updated version ML* of the machine learning model ML can be introduced at one or more of the local locations LS, LS-1, LS-2 to provide the (globally) updated machine learning model ML*.

[0180] Fig. 5 shows optional substeps in a procedure for federated learning of a machine learning model in a distributed environment. Corresponding data streams are shown in Fig. 6. The method comprises several steps. The order of the steps does not necessarily correspond to the numbering of the steps, but may also vary between different embodiments of the present invention. Furthermore, individual steps or a sequence of steps may be repeated.

[0181] In step S70, a quality assessment of the synthetic representation SR is performed. In the Fig. In the workflow illustrated in Figure 2, step S70 can, for example, follow step S20. Step S70 contains two alternative quality control processes. One involves the generation of a backward parameterization P' (steps S71 and S72), the other is based on the generation of a text summary SUM (steps S73 and S74). The two processes can be applied separately or in combination.

[0182] In particular, at step S71, a reverse parameterization P' of the synthetic representation SR may be generated. According to some examples, the same encoder ENC that was used to generate the parameterization P at the local site LS may be used.

[0183] Subsequently, in step S72, the backward parameterization P' can be compared with the parameterization P. If, based on the comparison, the backward parameterization P' and the parameterization P sufficiently match, it can be assumed that the data quality of the synthetic representation SR is sufficient for further use (e.g., integration into the central database CDB and / or the assessment / aggregation of the local update ML').

[0184] In substep S73, a text summary SUM of the synthetic representation SR can be generated. For example, the text summary can be generated automatically by applying yet another trained function that is independent of the generative AI function. In a medical context, vision transformers can be used that have been trained to analyze medical image data and consolidate the findings into textual impressions (such as those found in medical reports).

[0185] In step S74, the summary SUM can be compared with the request. If the text summary SUM matches the request, this can be considered an indication of sufficient data quality of the synthetic representation SR.

[0186] Step S80 can be considered an optional data augmentation step. In step S80, further variants of the parameterization P can be generated by slightly modifying or perturbing the contained values. For example, the size and location of nodes can be slightly varied. Furthermore, descriptions of additional nodes can be added, while the descriptions of other nodes can be deleted. In other words, this results in perturbed parameterizations P_mod for the generative AI function. As the case may be, the perturbed features can be used to generate additional prompts that differ from the original prompt based on the non-perturbed parameterization P.

[0187] According to other examples, the PMT prompt can also be perturbed / modified directly (if a PMC prompt is generated in the workflow). In this case, additional versions can also be obtained for input to the generative AI function GEN.

[0188] In turn, this leads to additional synthetic representations SR, which can further increase the amount of data in the central database CDB. Of course, the synthetic representations SR generated based on such perturbed input parameters can be subject to the same quality controls as described in connection with step S70 and in Fig. 6 shown.

[0189] According to some examples, the encoder ENC used to generate the parameterization P may refer to an encoder portion ENCP of an encoder-decoder-transformer network, and the generative AI function GEN may refer to the decoder portion DEC of an encoder-decoder-transformer network. The encoder portion may be configured to take images and output feature encodings, while the decoder may be configured to generate a synthetic representation of the input images based on the feature encodings. In other words, the encoder ENC and the generative AI function GEN can be viewed as vision transformers that operate in a reciprocal manner with respect to each other. In this regard, the feature encodings connecting the two portions can be viewed as the parameterization P of the present invention.

[0190] That means, Fig.Figure 7 shows a schematic representation of an encoder-decoder-transformer network according to one embodiment. While using such a structure may have some advantages, such as end-to-end training, it should be noted that other configurations may also be possible. In particular, the encoder ENC and the generative AI function GEN may also be independent of each other, as described elsewhere herein.

[0191] In brief, the task of the encoder ENC is to map an input INPT, i.e., the local data LTD, specifically a medical image, to a sequence of continuous representations, the parameterizations P, which are then fed to a decoder GEN. The decoder GEN receives the output P of the encoder ENC along with the decoder output OUTR from a previous iteration to generate an output OUT, which is a synthesized representation SR of the input INPT, specifically a synthetic image.

[0192] The encoder ENC of this embodiment may comprise a stack of N = 8 identical layers. For ease of reference, only one layer xN is shown in the drawing. Furthermore, N may be set to different values ​​according to the respective task, and in particular to values ​​greater than N = 8. Each layer xN of the encoder ENCP comprises two sublayers L1 and L3. The first sublayer L1 implements a so-called multihead self-attention mechanism. Specifically, the first sublayer L1 may be configured to determine how relevant a specific image data element is with respect to other elements in the input INPT. This may be represented as an attention vector. To avoid any bias, multiple attention vectors may be generated per word and fed into a weighted average to calculate the final attention vector for each word.The second sublayer L3 is a fully connected feedforward network, which can, for example, comprise two linear transformations with a rectified linear unit (ReLU) activation in between. The N = 8 layers of the encoder ENC apply the same linear transformations to all elements in the input INPT, but each layer uses different weight and bias parameters. Each sublayer L1, L3 is followed by a normalization layer L2, which normalizes the sum calculated between the input fed to the respective sublayer L1, L3 and the output generated by the respective sublayer L1, L3 itself. To capture information about the relative positions of the elements in the input INPT, position encodings PE are generated based on the input embeddings INPT-E before being fed to the xN layers.The position encodings PE have the same dimension as the input embeddings INPT-E and can be generated using sine and cosine functions with different frequencies. Then, the position encodings PE can simply be summed with the input embeddings INPT-E to inject the position information PE. The input embeddings INPT-E can, as usual, be a representation of each image patch in the input INPT, typically in the form of a real-valued vector encoding the pattern or other visual characteristics, so that patches closer in the vector space are expected to be similar. According to some examples, a convolutional neural network can be used to generate the input embeddings INPT-E.

[0193] The decoder GEN of this embodiment can also comprise a stack of N = 8 identical layers xN, each comprising three sublayers L4, L1, L3, which can be followed by a normalization layer L2, as explained in connection with the encoder ENC. For ease of reference, only one layer xN of the decoder GEN is shown in the drawing. Furthermore, N can also be set differently, in particular greater than N = 8, depending on the respective task. Although the sublayers L1 and L3 of the decoder GEN correspond in functionality to the respective sublayers L1 and L3 of the encoder ENC, the sublayer L4 receives the previous output OUTR of the decoder GEN (which is optionally transformed into corresponding embeddings and augmented with position information if the output is a synthesized image patch) and implements multihead self-attention through its weighting of the importance of individual elements of the previous output vector OUTR.Subsequently, the values ​​from the first sublayer L4 of the decoder DEC are input to the L1 sublayer of the decoder GEN. This sublayer L1 of the decoder GEN implements a multihead self-attention mechanism similar to that implemented in the first sublayer L1 of the encoder ENC. On the decoder side, this multihead mechanism receives the values ​​from the previous decoder sublayer L4 and the output of the encoder ENC. This allows the decoder GEN to process all patches in parallel. As in the encoder ENC part, the output of the L1 sublayers is fed into a feedforward layer L2, which causes the output vectors to form into something easily acceptable by another decoder block or linear layer. After all layers xN of the decoder DEC have been processed, the intermediate result is fed into a linear layer L5, which can be another feedforward layer.It is used to expand the dimensions into an image format expected for the output OUT. The result is then passed through a softmax layer L6, which transforms the result into a final output.

[0194] Wherever appropriate, individual embodiments or their individual aspects and features may be combined or interchanged without limiting or expanding the scope of the present invention. Advantages described with reference to one embodiment of the present invention are also advantageous for other embodiments of the present invention, wherever applicable. Regardless of the grammatical usage of the term, the term includes persons of male, female, or other gender identities. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 2021 / 0 097 439 A1

[0006] Cited non-patent literature

[0000] Vaswani et al. “Attention Is All You Need,” in arXiv: 1706.03762, June 12, 2017

[0031]

Claims

[1] A computer-implemented method for federated learning of a machine learning (ML) model in a model aggregator device (MAD), the method comprising: - receiving (S10), at the model aggregator device (MAD), from a local location (LS) located remotely from the model aggregator device (MAD), a local update (ML') of the machine learning model (ML), wherein the local update (ML') was generated at the local location (LS) based on local data (LTD), and a parameterization (P) of the local data (LTD), - generating (S20), at the model aggregator device (MAD), a synthetic representation (SR) of the local data (LTD) based on the parameterization (P) using a generative AI function (GEN), - evaluating (S30), at the model aggregator device (MAD), the local update (ML') using the synthetic representation (SR) to obtain an evaluation result indicating the performance of the local update (ML'), and - updating (S40), at the model aggregator device (MAD), the machine learning model (ML) based on the assessment result and the local update (ML'). [2] The method according to claim 1, wherein - the parameterization (P) consists of a parameterization of the data used to validate and / or test the local update (ML') at the local site (LS). [3] A method according to any one of claims 1 or 2, wherein: - the generative AI function (GEN) is designed to generate the synthetic representation (SR) based on a natural language request specifying the synthetic representation (SR) to be generated, and - the generating step (S20) comprises obtaining (S21) the natural language prompt based on the parameterization (P) and inputting (S22) the natural language prompt into the generative AI function (GEN) to generate the synthetic representation (SR). [4] Method according to one of the preceding claims, further comprising - adding (S50) the synthetic representation (SR) to an existing test dataset that the model aggregator device (MAD) can access to validate and / or test the machine learning model (ML) to generate an extended test dataset, - wherein in the step of evaluating (S30) the local update (ML') is evaluated based on the extended test data set. [5] Method according to one of the preceding claims, further comprising - Determining (S70) a data quality of the synthetic representation (SR), - wherein in the step of judging (S30) the local update (ML') is judged based on the data quality. [6] The method of claim 5, wherein the step of determining (S70) a data quality of the synthetic representation (SR) comprises: - generating (S71) a backward parameterization (P') of the synthetic representation (SR), - comparing (S72) the backward parameterization (P') with the parameterization (P), and - Determining (S70) the data quality based on the step of comparing the backward parameterization (P') with the parameterization (P). [7] Method according to one of claims 5 or 6, in combination with claim 3, wherein the step of determining (S70) a data quality of the synthetic representation (SR) comprises: - generating (S73) a natural language summary (SUM) based on the synthetic representation (SR), - comparing (S74) the natural language summary (SUM) with the natural language request, and - Determining (S70) the data quality based on the step of comparing the natural language summary with the natural language prompt. [8] Method according to one of the preceding claims, wherein - the local data (LTD) includes a large number of independent data elements, and - the parameterization (P) includes: - for each independent data element, an element parameterization of the data element, and - and one or more statistical properties of the plurality of independent data elements. [9] Method according to one of the preceding claims, further comprising - generating (S80) a modified parameterization based on the parameterization (P), - wherein in the generating step (S20) the synthetic representation (SR) is additionally generated based on the modified parameterization. [10] Method according to one of the preceding claims, wherein the local data (LTD) comprise protected information, in particular protected personal information, and the parameterization (P) does not comprise the protected information. [11] A method according to any one of the preceding claims, wherein: - the machine learning (ML) model is an image processing function designed to generate an image processing result based on image data, - the local data (LTD) includes training image data, - the parameterization (P) comprises a parameterization of the training image data, - the synthetic representation (SR) comprises synthetic image data generated by the generative AI function (GEN) based on the parameterization (P) of the training image data. [12] The method of claim 11, wherein - the machine learning model (ML) is designed to generate an image processing result based on medical image data, wherein the image processing result is selected from the following: a detection result of a medical finding in medical image data, a classification of a medical finding in medical image data and / or a segmentation of medical image data, and - the training image data includes medical image data. [13] A method according to any one of the preceding claims, further comprising: - Providing (S60) the updated machine learning model (ML') to a second local site (LS-2) different from the local site (LS). [14] Model aggregator device (MAD) for federated learning of a machine learning (ML) model, wherein the model aggregator device (MAD) comprises a computing unit (CU) and an interface unit (IU), wherein the interface unit (IU) is designed to: - receiving (S10), from a local location (LS, LS-2) located remotely from the model aggregator device (MAD), a local update (ML') of the machine learning model (ML), wherein the local update (ML') was generated at the local location (LS, LS-2) based on local data (LTD), and a parameterization (P) of the local data (LTD), and wherein the computing unit (CU) is designed to: - generating (S20) a synthetic representation (SR) of the local data (LTD) based on the parameterization (P) using a generative AI function (GEN), - evaluating (S30) the local update (ML') using the synthetic representation (SR) to obtain an evaluation result indicating the performance of the local update (ML'), and - Updating (S40) the machine learning (ML) model based on the assessment result and the local update (ML'). [15] A computer program product comprising program elements that cause a computing unit (CU) of a model aggregator device (MAD) for federated learning of a machine learning model (ML) to perform steps of the method according to any one of claims 1 to 13 when the program elements are loaded into a memory of the computing unit (CU). [16] A computer-readable medium in which program elements are stored that are readable and executable by a computing unit (CU) of a model aggregator device (MAD) for federated learning of a machine learning model (ML) to perform steps of the method according to any one of claims 1 to 13 when the program elements are executed by the computing unit (CU).

Images