Method and system for operating a technical apparatus

Abstract

The invention relates to a computer-implemented method for operating a technical apparatus by means of a system, wherein the following steps are carried out: a) providing and initialising an initial model on the basis of artificial intelligence and storing as a global model; b) transmitting the global model to the at least one client and storing as a respective local model; c) training the respective local model with provided training data of the respective client and calculating first gradients on the behaviour of model weights of the respective local model during the training; d) calculating respective importance metrics of the local model on the basis of the first gradients, by the respective client; e), transmitting the model parameters of the respective local model with the respective importance metrics, which lie within a predetermined range of values, from the respective client to the server; f) aggregating the previously received model parameters of the respective clients with the global model to form an aggregated model, and removing those model parameters in the aggregated model for which no respective importance metrics are present; g) storing the aggregated model as a global model; h) calculating second gradients on the behaviour of model weights of the aggregated model, and checking whether the second gradient has reached a specified convergence criterion; if not, continuing with step b), otherwise continuing with step i); i) operating the technical apparatus using the local model.

Description

[0001] Method and system for operating a technical device

[0002] The invention relates to a computer-implemented method for operating a technical device by a system comprising a server and at least one client with a respective connected technical device.

[0003] The invention relates to a system and a computer program product.

[0004] A foundation model (FM) is a model based on artificial intelligence (AI) that is trained with broad data so that it can be applied to a wide variety of use cases.

[0005] Foundation models have transformed AI and enabled prominent chatbots and generative AI. Foundation models are general-purpose technologies that can support a wide variety of use cases.

[0006] The creation of fundamental models is often very resource-intensive, with the most expensive models costing hundreds of millions of dollars to pay for the underlying data and calculations.

[0007] Federated learning (FL) proves to be a practical solution for easily providing large amounts of data, and therefore enables basic models to utilize rich data from a wider range of sources.

[0008] This improved access to data leads to the development of more accurate and efficient systems that are better tailored to the specific needs and preferences of users in different scenarios.

[0009] Furthermore, the integration of FL with FM makes the use of private data more accountable and offers individuals the opportunity to trace the sources of their data.

[0010] On the other hand, the generation of basic models usually requires a high level of computing, storage and communication capacity.

[0011] Communicating large amounts of model parameters, for example in large language models such as "Large Language Model Meta AI", or "LLaMA" for short, with up to 70 billion parameters, can also be unsuitable in practice in FM-FL scenarios.

[0012] Furthermore, the question of data protection when using data, as well as the ownership of the underlying data, is not always clarified.

[0013] In the current state of the art, a communication strategy for federated learning often involves the exchange of model updates between a central server and the participating devices while maintaining privacy and minimizing data transmission.

[0014] Publication US 2024 / 155025 A1 describes a so-called “Importance-Aware Federated Edge Learning” (FEEL) system, in which a problem for maximizing learning efficiency is formulated based on the relationship between loss reduction and gradient norm by jointly considering the allocation of communication resources and data selection, thereby reducing training latency and improving learning accuracy.

[0015] A typical communication strategy in federated learning is, for example:

[0016] 1) Initialization: The central server initializes a global model and participating devices (clients) download the global model.

[0017] 2) Model training on devices: Each device trains the model locally with its own data, and the local training uses the downloaded global model as a starting point.

[0018] 3) Local model updates: After local training, each device calculates the model updates, usually in the form of gradients, based on its local data.

[0019] 4) Aggregation on the central server: the devices upload their model updates to the central server and the central server aggregates these updates to obtain a refined global model.

[0020] 5) Model distribution: the updated global model is then sent back to the devices for the next round of local training.

[0021] 6) Iterations: steps 2 to 5 are repeated iteratively for several rounds until the model converges.

[0022] The challenge in communicating with basic models in federated learning arises from the considerable size of these models, which leads to high data transmission costs, limitations in network bandwidth, and increased communication latency.

[0023] It is therefore an object of the invention to provide a solution that improves the communication requirements when applying FL to FM.

[0024] The problem according to the invention is solved by a computer-implemented method for operating a technical device by a system comprising a server and at least one client with a respective connected technical device, wherein the following steps are performed: a) Provision and initialization of an initial model based on artificial intelligence, and storage as a global model, by the server, b) Transfer of the global model to the at least one client, and storage as the respective local model, c) Training of the respective local model with training data provided by the respective client, and calculation of first gradients over the behavior of model weights of the respective local model during training, d) Calculation of respective importance metrics of the local model based on the first gradients by the respective client.e) Transmitting the model parameters of the respective local model, along with their respective importance metrics within a predefined range, from each client to the server; f) Aggregating the previously received model parameters from each client with the global model to form an aggregated model, and removing those model parameters from the aggregated model for which no respective importance metrics are available, by the server; g) Storing the aggregated model as a global model, by the server; h) Calculating second gradients over the behavior of model weights of the aggregated model, and checking whether the second gradient has met a predefined convergence criterion; if not, proceeding with step b), otherwise proceeding with step i), by the server; i) Operating the technical device with the local model, by the respective client.

[0025] Reducing communication costs in a very large parameter model, for example with seven billion parameters, depends, like any specific scenario, on factors such as the model architecture, the distribution of the training data, and the degree of parameter updates required.

[0026] Strategies such as transmitting only important weights, quantization and compression techniques can help minimize communication overhead.

[0027] The approach involves a modified FL algorithm in which each client calculates the weighting significance and only transmits important weights to the server.

[0028] The importance or materiality values ​​or metrics can be determined using the L 2 -norm, especially for calculating the magnitude of gradients for each weighting. The Euclidean norm, standard norm, or L 2-norm is a vector norm commonly used in mathematics.

[0029] In two- and three-dimensional Euclidean space, the Euclidean norm corresponds to the intuitive length or magnitude of a vector and can be calculated using the Pythagorean theorem.

[0030] More generally, the Euclidean norm is also defined for real and complex vector spaces of arbitrary finite dimension and is then the norm derived from the standard scalar product.

[0031] As a norm induced by a scalar product, it possesses, in addition to the three norm axioms, a number of other properties, such as the validity of the Cauchy-Schwarz inequality and the parallelogram equation, as well as invariance under unitary transformations.

[0032] For orthogonal vectors, the Euclidean norm itself satisfies a more general form of the Pythagorean theorem.

[0033] Concepts such as the Euclidean distance and the Euclidean topology are derived from the Euclidean norm, and it can be generalized to infinite-dimensional vector spaces, for example to sequence spaces by the £ 2 -Norm and on functional spaces through the L 2 -Standard.

[0034] These assessments help to determine the importance of each weighting for the contribution to the model updates.

[0035] Only weights with importance values ​​above a certain threshold are transmitted to the central server, which reduces communication effort by transmitting only the relevant information.

[0036] In other words, the server only contains importance or materiality values ​​or metrics from those clients whose corresponding values ​​exceed a predetermined threshold.

[0037] All clients with their respective local models, with importance or materiality values ​​or metrics below the threshold, are removed from an aggregated, global model on the server.

[0038] The threshold parameter in the "communicate_important_weights" function controls which weights are considered important. Adjusting this threshold affects the rarity of the communicated weights.

[0039] Local training on each device is followed by the aggregation of only the important weights on the central server, i.e., in the global model. This enables decentralized model updates while simultaneously focusing on key information. This FL algorithm emphasizes the selective communication of important weights and strikes a balance between minimizing communication costs and maintaining the overall performance of the model.

[0040] Adjustments to hyperparameters, such as a threshold for determining importance, can be made based on specific requirements and characteristics of the problem domain.

[0041] The initial model based on artificial intelligence includes weights and structural information in the form of a neural network.

[0042] The server is connected to the respective clients via a data connection in order to transfer or exchange models.

[0043] Training the respective local model with provided training data from the respective client can be done, for example, by labeling sensor data accordingly and marking permissible or impermissible features in the sensor data.

[0044] Calculating first and / or second gradients over the behavior of model weights of the respective local model involves analyzing changes in individual model weights over an observation period or across training sessions.

[0045] Calculating importance metrics of the local model based on the first gradients can be done, for example, by classifying model weights that do not change or change only slightly during training as not "important", which can be expressed by a corresponding importance metric.

[0046] The importance metrics can then be checked to see if they lie within a specified range of values.

[0047] Aggregating the model parameters with the global model to form an aggregated model can be done, for example, by including the "important" model weights in the global model and replacing corresponding previous values ​​of the global model.

[0048] Model parameters for which no respective importance metrics are available can be removed from the aggregated model.

[0049] The first and second gradients can converge, meaning they approach a final value, where the final value can represent a model weight and be calculated using a predefined convergence criterion. The client can operate the technical device with the local model by, for example, reading sensor data for the device and analyzing it with the model to make a prediction about its future behavior.

[0050] However, the operation of the technical device with the local model by the respective client can also be carried out in such a way that, for example, the device is controlled individually by taking special operating conditions into account.

[0051] In a further development of the invention, it is provided that in step f) the aggregated model has fewer model weights than the global model.

[0052] Reducing the model size reduces the complexity of the process in terms of the resources required.

[0053] In a further development of the invention, it is provided that during the aggregation in step f) an average is calculated of the received model parameters of the respective clients.

[0054] This improves the model accuracy for a general application.

[0055] The problem according to the invention is also solved by a system for operating a technical device, comprising a server and at least one client with a respective connected technical device, wherein the system is configured to execute the method according to the invention.

[0056] The problem according to the invention is also solved by a computer program product with machine-readable instructions stored therein which, when executed by the system according to the invention, cause it to execute the method according to the invention.

[0057] The following algorithms and figures describe the invention in detail using an exemplary embodiment. The algorithms and figures show

[0058] Fig. 1 shows a first embodiment of the invention in the form of a pseudo-code,

[0059] Fig. 2 shows a second embodiment of the invention in the form of a pseudocode,

[0060] Fig. 3 shows a simplified block diagram for a Vision-FM segmentation,

[0061] Fig. 4 shows a system for carrying out the method according to the invention.

[0062] Fig. 5 shows an example of a flowchart for the method according to the invention. Fig. 1 shows a first embodiment of an algorithm according to the invention in the form of a pseudo-code.

[0063] This example demonstrates how the method according to the invention can be implemented in a simple and efficient manner.

[0064] Fig. 2 shows a second embodiment of an algorithm according to the invention in the form of a pseudo-code, wherein an improvement of the algorithm is carried out by applying a more powerful FM.

[0065] An extension of the previous algorithm using the so-called “Low Rank Adaptation”, or “LoRA” for short, a tuning technique that utilizes the performance of advanced FMs for user-defined tasks and datasets to reduce the burden on system resources or excessive “costs”, i.e., to reduce and improve system complexity.

[0066] The function “low_rank_adaptation” applies the singular value decomposition (SVD) to the model weights and retains only the singular values ​​and vectors of the top “rank” to obtain an approximation of the low rank.

[0067] •The function “communicate_important_weights_with_low_rank” includes a low-rank adjustment before the important weights are selected and communicated.

[0068] The rank parameter in the function "low_rank_adaptation" controls the desired rank of the low-rank approach.

[0069] Fig. 3 shows a simplified block diagram for a Vision-FM segmentation, i.e. an application of the FM FL strategy to vision transformer models such as “Segment Anything” (SAM).

[0070] The largest part of the model is the image encoder l_ENC, namely a VIT-based image encoder ("Vision Transformer"), which calculates the image embeddings IMG_EMB using the attention to patches of an image IMG.

[0071] Using a DS sampling method, masks M are fed to a lightweight mask decoder LWM_DEC.

[0072] The lightweight mask decoder LWM_DEC is controlled via a prompt encoder P_ENC, which is configured using points P, boxes B and texts T.

[0073] The lightweight mask decoder LWM_DEC delivers valid masks V_M with confidence values ​​CS1-CS3. Relative positional embedding is used.

[0074] This approach can be used to train or refine the image encoder of a SAM-like model, for example to fine-tune the encoder specifically for industrial fault detection data.

[0075] Fig. 4 shows an embodiment of a system with a server S, two clients C1, C2 and each connected devices TD1 , TD2.

[0076] The system is designed to execute the method according to the invention.

[0077] Fig. 5 shows an embodiment of a flowchart of the method according to the invention.

[0078] The procedure for operating a technical device through the system is at least partially computer-implemented.

[0079] The following process steps are performed: a) Provisioning and initializing an initial model based on artificial intelligence, and storing it as a global model, by the server S, b) Transferring the global model to at least one client C1, C2, and storing it as a respective local model, c) Training the respective local model with training data provided by the respective client C1, C2, and calculating initial gradients over the behavior of the model weights of the respective local model during training, d) Calculating the respective importance metrics of the local model based on the initial gradients by the respective client C1, C2, e) Transmitting the model parameters of the respective local model with the respective importance metrics, which lie within a predefined range of values, from the respective client C1, C2 to the server S, f) Aggregating the previously received model parameters of the respective clients C1, C2 ...02. Convert the global model to an aggregated model, and remove those model parameters in the aggregated model for which no respective importance metrics exist, by the server S, g) Save the aggregated model as a global model, by the server S, h) Calculate second gradients over the behavior of model weights of the aggregated model, and check whether the second gradient has reached a predefined convergence criterion; if not, continue with step b, otherwise continue with step i, by the server S, i) Operate the technical device TD1, TD2 with the local model, by the respective client C1, 02.,

[0080] In the first training round, training data based on the initial model with its model structure (the first model structure) is applied. The aggregated model in step f) has fewer model weights than the global model; it is therefore a simplified, reduced model with a changed model structure.

[0081] This means that the respective training data for the clients must also be adapted to the new model structure, namely the second model structure.

[0082] The second model structure can change after each training round. During aggregation in step f), an averaging of the received model parameters of the respective clients 01 and 02 is optionally performed.

[0083] Regardless of the grammatical gender of a particular term, persons with male, female or other gender identities are included.

[0084] Reference symbol list:

[0085] B Box

[0086] C1. C2 Client

[0087] CS1-CS3 confidence score

[0088] DS sampling (English: "down sample") l_ENC image encoder

[0089] IMG image

[0090] IMG_EMB Image embedding

[0091] LWM_DEC Lightweight mask decoder A

[0092] M Mask

[0093] P point

[0094] P_ENC prompt encoder

[0095] S Server

[0096] T Text

[0097] TD1, TD2 technical device

[0098] V_M valid masks

Claims

Patent claims 1. A computer-implemented method for operating a technical device through a system comprising a server (S) and at least one client (C1, O2) with a respective connected technical device (TD1, TD2), wherein the following steps are performed: a) Provisioning and initializing an initial model based on artificial intelligence, and storing it as a global model, by the server (S); b) Transferring the global model to the at least one client (O1, O2), and storing it as a respective local model; c) Training the respective local model with training data provided by the respective client (O1, O2), and calculating initial gradients over the behavior of model weights of the respective local model during training; d) Calculating respective importance metrics of the local model based on the initial gradients by the respective client (O1, O2).e) Transmitting the model parameters of the respective local model with their respective importance metrics, which lie within a predefined range of values, from the respective client (01, 02) to the server (S); f) Aggregating the previously received model parameters of the respective clients (01, 02) with the global model to form an aggregated model, and removing those model parameters in the aggregated model for which no respective importance metrics are available, by the server (S); g) Storing the aggregated model as a global model, by the server (S); h) Calculating second gradients over the behavior of model weights of the aggregated model, and checking whether the second gradient has met a predefined convergence criterion; if not, continuing with step b), otherwise continuing with step i), by the server (S); i) Operating the technical device (TD1, TD2) with the local model, by the respective client (01, 02)., 2. A method according to the preceding claim, wherein in step f) the aggregated model has fewer model weights than the global model.

3. Method according to one of the preceding claims, wherein in the aggregation in step f) an averaging of the received model parameters of the respective clients (01, 02) is carried out.

4. System for operating a technical device (TD1, TD2), comprising a server (S) and at least one client (01, 02) with a respective connected technical Device (TD1, TD2), wherein the system is configured to perform the method according to any one of the preceding claims.

5. Computer program product with machine-readable data stored therein. Instructions which, when executed by the system according to one of the preceding claims, cause it to execute the method according to one of the preceding claims

Images