Computer-implemented method and system for operating a technical device in a client-server system

Abstract

Computer-implemented method for operating a technical device (TD1) by way of a system, wherein the following steps are carried out: a) providing a trained base model (BM) for operation of the technical device (TD1-TD3) to the server (S), b) providing a predefined range of values for an admissible error in the base model (BM) to the server (S), c) relaying the base model (BM) from the server (S) to the first client (GI), d) training (TR) the base model (BM) by way of the first client (GI), and relaying it as a first client model (CM1) to the server (S), e) calculating a first statistical error indicator for the first client model (CM1) by way of the server (S), f) checking (PR) whether the first statistical error indicator is within the predefined range of values by way of the server (S), I. if it is, then aggregating the first client model (CM1) into the base model (BM) as a global model (GM), II. if it is not, notifying (FB) the first client (GI) that the first client model (CM1) has not been used in the model formation, g) providing the global model (GM) to the first client (GI) as a local model (LM), and operating the technical device (TD1) using the local model (LM).

Description

[0001] Computer-implemented method and system for operating a technical device in a client-server system. The invention relates to a computer-implemented method and a system for operating a technical device by a system comprising a server and a first client, with a model based on artificial intelligence. The invention further relates to a computer program product. Federated Learning (FL for short) is a machine learning technique in which an algorithm, such as a neural network, is trained using local data samples across multiple decentralized edge devices as clients or servers, without exchanging the data itself. This enables data anonymity, which facilitates collaboration between different industrial partners because no raw data needs to be shared, but anonymized data or abstractions thereof can. Thus, machine learning models (ML for short) can be trained jointly.without disclosing the data. However, industry participants are not always familiar with the requirements of machine learning and data. This can result in a client having very little impact on the overall performance of the model. For example, if a client has too little data, they could try to increase it using a data augmentation technique, which may be insufficient. This can subsequently degrade the overall performance of the model. Furthermore, the data quality of individual clients may be poor or not match the data distribution of other clients. Therefore, it is important to identify the low-performing client and provide the client with helpful feedback that they are currently a poor performer and that there are difficulties with the dataset. Ideally, potential sources of error should be highlighted to allow the client toto improve its respective data. One source of error for a low-performing client is a client that does not have enough data to participate in federated learning and specifies an incorrect dataset size. For example, a server may require clients participating in federated learning to have a dataset size of at least 100 elements, or test datasets. The client can either provide false information about the size of its dataset or simply expand the data and synthetically create a larger number to meet the server requirements and participate in the system. This can lead to the client's model not being a good fit to the corresponding client data. In unwanted "model poisoning," a client intentionally inputs incorrect data into the model.so that the federation process is disrupted and a negative knowledge transfer occurs. This degrades the model on the server and all participants, negatively impacting all participants. In an equally undesirable "men-in-the-middle" attack, outsiders can conduct so-called "poisoning" attacks if the communication link between a client and the central node or server is vulnerable by intercepting the communication between some users and the central node. The attacker can then send malicious model updates to the central node on behalf of legitimate users, or detect the use of potentially manipulated data. Currently, most of the literature focuses on the detection of malicious updates (also called "poisoning"), which is solved using the cosine similarity of data.or removing the malicious updates via an anomaly detection model. Furthermore, embeddings of other models or their respective design flaws are also used. Sometimes a second model is trained and sent (such as an autoencoder). However, this requires access to the data of a client or a second model. Therefore, the object of the invention is to identify and exclude clients with poor performance in a federated system, thereby improving an overall model. The object of the invention is achieved by a method of the type mentioned above, wherein the following steps are carried out: a) providing a trained base model for the operation of the technical device to the server, b) providing a predefined value range for a permissible error of the base model to the server, c) transmitting the base model from the server to the first client,and providing at least one first parameter for the operation of the device to the first client, and adopting the at least one first parameter as at least one operating parameter, d) training the base model for the at least one operating parameter by the first client and transmitting it as a first client model to the server, e) calculating a first statistical error indicator for the first client model by the server, f) checking whether the first statistical error indicator is within the predefined value range by the server, I. If yes, then aggregating the first client model into the base model as a global model, II. If no, notifying the first client with the first statistical error indicator that the first client model was not used in the formation of the global model,and performing an analysis of operating parameters of the technical device based on the first client model by the first client, determining at least one second parameter for the operation of the device, which differs from the at least one second parameter, and adopting the at least one second parameter as the at least one operating parameter, and continuing in step d), g) providing the global model to the first client as a local model, and operating the technical device with the local model. This achieves the detection of a low-performing client in a simple and efficient manner and provides helpful feedback on the data requirements for the respective client in order to guide the client to improve its own data quality in the global context, i.e., through the predefined value range for a permissible error of the base model.where the server represents the global context. A client typically has limited resources for calculating statistical parameters, but a server has these capabilities. A client's resources can generally be limited not only in their computing power, but also in their temporal availability, since the resources often have to be shared with other system components of the client for a variety of tasks, and a lengthy statistical calculation could potentially block these system components for an unacceptably long time. It is advantageous if the quality of the first client model is not performed by the first client, because the server has far better resources for implementing the statistical error indicator compared to the client's generally rather limited resources. Furthermore, it is particularly advantageouswhen the quality of the first client model is checked in the context of the global model and, in particular, in comparison with other clients. After providing feedback to the client, the client performs an analysis of the operating parameters of the technical device using the first client model to determine the cause of the inadequate data quality in detail. This can be done by comparing the original base model with the first client model and inferring the cause of the inadequate data quality based on the difference, such as model weights, between the two models. Based on the analysis of the operating parameters, the first client can make appropriate adjustments to the current operating parameters, which should lead to improved operation of the device and can subsequently contribute to an improved client model.which, in turn, can lead to a further improved global model upon achievement of the quality target in step f). At least one provided first operating parameter is initially applied in step d) when training the base model by the first client. After analyzing the at least one first operating parameter in step f), the first client can, based on the at least one operating parameter and using the first statistical error indicator, determine at least one second operating parameter with which the device is controlled, and the base model is retrained by the first client with the at least one second operating parameter in step d). The system therefore only uses and aggregates client models,which meet predetermined requirements. Calculating a statistical error indicator can be done by applying one or more statistical functions over time or across multiple clients, i.e., by determining quantitative information using methods for analyzing empirical data. This procedure makes it possible within a FL system, based on quality specifications for a global model, to exclude individual clients from federated learning if their data quality is insufficient, for example, for individual cases, temporarily or permanently, in order not to adversely affect the overall quality of the global model. At the same time, the client whose data quality has been identified as insufficient can receive appropriate feedback, for example, to give them the opportunity toto improve the data quality and to be considered again in the global model in a subsequent round of federated learning. The training of the base model by the first client can, for example, be carried out using data collected by the first client during operation of the connected technical device. The method therefore provides that the first client can react autonomously to feedback regarding the data quality from the server and, if necessary, initiate necessary measures to improve the data quality. The goal is therefore not only to identify clients with insufficient model quality and not to consider them when aggregating them into a global model. In addition, the goal is to inform the client that their model could not be considered in the global context due to its quality, and that the client also receives information aboutwhy the client model was inadequate in the global context. This gives the client the opportunity to improve their client model in order to be considered in future aggregation rounds, because the more clients can be considered in the global model, the better and more accurate the global model becomes. In a further development of the invention, it is provided that the first statistical error indicator is formed by a mean or a confidence interval of parameters of the first client model. The system therefore only uses and aggregates client models that meet predetermined requirements, for example, regarding the data quality of the associated clients. In a further development of the invention, it is provided that the first and / or the second statistical error indicator is calculated using a Dice similarity coefficient, an accuracy,a precision or an F-measure. The system therefore only uses and aggregates client models that meet predetermined requirements, for example, regarding similarity, accuracy, and / or precision, etc. In a further development of the invention, it is provided that the system further comprises at least two second clients, and the following steps are carried out: a1) transmitting the base model from the server to each of the at least two second clients, a2) training the base model by each of the at least two second clients, and transmitting the respective trained models as a respective second client model to the server, a3) calculating a respective second statistical error indicator for the respective second client model by the server, a4) calculating the value range for the permissible error of the base model from step b) from the at least two second statistical error indicators,by the server. This makes it easy to determine the predetermined value range from data from other clients. It is particularly advantageous if the quality of the first client model is tested in comparison with other clients. For example, the first client may have only limited training data or may not be sufficiently well configured for optimal operation of the connected device, while a second client may have more comprehensive, better training data. The first client often cannot determine whether it is using sufficiently good training data in relation to other clients. Using the feedback received, the data quality can be analyzed by the respective client and improved so that the respective client model is taken into account in the global model in the future, which further improves the global model.because the basis of the global model could be expanded by another client model with sufficient data quality. The training of the base model by the respective second client can be carried out, for example, using data collected by the respective second client during operation of the respective connected technical device. In a further development of the invention, it is provided that the at least two second statistical error indicators are each formed by a mean value or a confidence interval of parameters of the respective second client model. In a further development of the invention, it is provided that the at least two second client models are aggregated with the provided base model from step a). The object of the invention is also achieved by a system for operating a technical device, comprising a server, a first client, with a model based on artificial intelligence,wherein the steps of the method according to the invention are carried out. In a further development of the invention, it is provided that at least two second clients are further included, and the steps of the method according to the invention are carried out. The object of the invention is also achieved by a computer program product with machine-readable instructions stored therein, which, when executed by a processing unit, cause the processing unit to carry out the method according to the invention. The invention is explained in more detail below with reference to an exemplary embodiment illustrated in the accompanying drawings. The drawings show: Fig. 1 an example of a client-server system, Fig. 2 schematically the sequences of the method according to the invention, Fig. 3 an exemplary embodiment of a flowchart of the method according to the invention, Fig. 4 a representation of value ranges of client models,Fig. 5 shows an embodiment of a pseudo-program code for the method according to the invention. Fig. 6 shows an embodiment of a pseudo-program code for detecting clients with low performance. Fig. 1 shows an example of a client-server system with a server S and three connected clients C1-C3, each of which is connected to a technical device TD1-TD3. The respective technical devices TD1-TD3 are operated with an ML model. The ML model is based on federated learning, which takes place between the server S and the clients C1-C3. Figs. 2 and 3 schematically show the sequences of the method according to the invention and a corresponding flowchart based on the system of the previous figure. The following steps are carried out by the system: a) Providing a trained base model BM for the operation of the technical device TD1-TD3 to the server S.b) Providing a predefined value range for a permissible error of the base model BM to the server S, c) Transmitting the base model BM from the server S to the first client C1, and providing at least one first parameter for the operation of the technical device TD1 to the first client C1, and adopting the at least one first parameter as at least one operating parameter, d) Training TR of the base model BM for the at least one operating parameter by the first client C1, and transmitting it as the first client model CM1 to the server S, e) Calculating DET of a first statistical error indicator for the first client model CM1 by the server S, f) Checking PR whether the first statistical error indicator lies within the predefined value range by the server S, I. If yes, then aggregating the first client model CM1 into the Basic model BM as global model GM, II. If no,Notifying the first client C1 with the first statistical error indicator that the first client model CM1 was not used in the formation of the global model, and performing an analysis of operating parameters of the technical device TD1 based on the first client model CM1 by the first client C1, determining at least one second parameter for the operation of the device TD1, which differs from the at least one second parameter, and adopting the at least one second parameter as the at least one operating parameter, and continuing in step d), g) providing the global model GM to the first client C1 as a local model LM,and operating the technical device TD1 with the local model LM. The first statistical error indicator can be formed by a mean or a confidence interval of parameters of the first client model CM1. The first and / or the second statistical error indicator can each be determined using a Dice similarity coefficient, an accuracy, a precision, or an F-measure (F1 score). The Dice score, also called Dice similarity coefficient, is a measure of the similarity between two data sets. Accuracy is generally a measure of the closeness of some measurements to a specific value, while precision is the closeness of the measurements to each other. In addition, the following steps are carried out: a1) Transmitting the base model BM from the server S to the two second clients C2, C3, a2) Training TR of the base model BM by one of the two second clients C2, C3,and transmitting the respective trained models as a respective second client model CM1, CM2 to the server S, a3) calculating a respective second statistical error indicator for the respective second client model CM1, CM2 by the server S, a4) calculating the value range for the permissible error of the base model BM from step b) from the at least two second statistical error indicators by the server S. The two second statistical error indicators can each be formed by a mean value or a confidence interval of parameters of the respective second client model. The two second client models CM1, CM2 are aggregated with the provided base model BM from step a). Fig. 4 shows a representation for value ranges of client models. Several client models are formed over time. Statistical error indicators can be calculated for the parameters of these client models,for example, a median M with a fluctuation range of the standard deviation STD, which is multiplied by a constant C. A client model that, for example, has an F-measure F1 that lies outside the fluctuation range, i.e., a predetermined value range, is recognized as insufficient for further use and aggregation in a global model and is therefore not used. The associated client from which the client model originates is informed of this rejection. Fig. 5 shows an exemplary embodiment of a pseudo-program code for the inventive method. A mean value M, a standard deviation STD of model data, and the number of clients k are provided. A client, here as ^^, ^ referred to, registers for federated learning in a communication round, here as ^^ ௧ with the server S. The server S sends the base model BM, here as ^^௧ designated, to the client ^^ ^ . The client ^^ ^ trains the model ^^ ௧ with a data set. Then the client sends ^^ ^ the trained model ^^ ௧ା^ to the server S. In the server, the detection starts with the program code shown, whether the model of the client ^^ ^ has sufficient performance. Fig. 6 shows an exemplary embodiment of a pseudo-program code for detecting clients with low performance. Client models ^^ ௧ ^ the client ^^ ^,…,^ in the communication round ^^ ௧ provided and a recognition parameter ^^ is determined. Regardless of the grammatical gender of a particular term, persons with male, female, or other gender identities are included.

[0002] List of reference symbols: BM basic model C constant C1-C3 client CM1-CM3 client model DET detection of low-performing clients F1 F-measure FB feedback K number of clients LM local model PR test M median S server STD standard deviation T time TD1-TD3 technical device TR model training on the client

Claims

1. A computer-implemented method for operating a technical device (TD1) by a system comprising a server (S) and a first client (C1) with a model based on artificial intelligence, wherein the following steps are carried out: a) providing a trained basic model (BM) for the operation of the technical device (TD1-TD3) to the server (S), b) providing a predefined value range for a permissible error of the basic model (BM) to the server (S), c) transmitting the basic model (BM) from the server (S) to the first client (C1), and providing at least one first parameter for the operation of the technical device (TD1) to the first client (C1), and adopting the at least one first parameter as at least one operating parameter, d) training (TR) the basic model (BM) for the at least one operating parameter by the first client (C1),and transmitting as a first client model (CM1) to the server (S), e) calculating a first statistical error indicator for the first client model (CM1) by the server (S), f) checking (PR) whether the first statistical error indicator is within the predefined value range by the server (S), I. If yes, then aggregating the first client model (CM1) into the base model (BM) as a global model (GM), II. If no, notifying the first client (C1) with the first statistical error indicator that, the first client model (CM1) was not used in the formation of the global model, and performing an analysis of operating parameters of the technical device (TD1) based on the first client model (CM1) by the first client (C1), determining at least one second parameter for the operation of the technical device (TD1) which differs from the at least one second parameter, and adopting the at least one second parameter as the at least one operating parameter, and continuing in step d) g) providing the global model (GM) to the first client (C1) as a local model (LM), and operating the technical device (TD1) with the local model (LM), wherein the first statistical error indicator is formed by a mean value or a confidence interval of parameters of the first client model (CM1). 2.Method according to one of the preceding claims, wherein the first statistical error indicator is determined using a Dice similarity coefficient, an accuracy, a precision or an F-measure.

3. Method according to one of the preceding claims, wherein the system further comprises at least two second clients (C2, C3), and the following steps are carried out: a1) transmitting the base model (BM) from the server (S) to each of the at least two second clients (C2, C3), a2) training (TR) the base model (BM) by each of the at least two second clients (C2, C3), and transmitting the respective trained models as a respective second client model (CM1, CM2) to the server (S). a3) Calculating a respective second statistical error indicator for the respective second client model (CM1, CM2) by the server (S). a4) Calculating the value range for the permissible error of the base model (BM) from step b) from the at least two second statistical error indicators by the server (S).

4. The method according to the preceding claim, wherein the second statistical error indicator is determined using a Dice similarity coefficient or an F-measure.

5. The method according to one of claims 3 or 4, wherein the at least two second statistical error indicators are each formed by a mean value or a confidence interval of parameters of the respective second client model.

6. The method according to one of claims 3 to 5, wherein the at least two second client models (CM1, CM2) are aggregated with the provided base model (BM) from step a). 7.System for operating a technical device (TD1), comprising a server (S), a first client (C1), with a model based on artificial intelligence, wherein the method steps according to one of claims 1 to 2 are carried out.

8. System according to the preceding claim, further comprising at least two second clients (C2, C3), and the method steps according to one of claims 3 to 6 are carried out.

9. Computer program product with machine-readable instructions stored therein which, when executed by a processing unit, cause the processing unit to carry out the method according to one of claims 1 to 6.

Images