Method and system for operating a technical apparatus

Abstract

Proposed is a computer-implemented method for operating a technical apparatus by means of a client-server system having a server and clients that are connected to respective technical apparatuses, said method comprising the following steps: a. providing operational data from the operation of the technical apparatus of each client, and generating operational meta-information relating to the operation of the technical apparatus from the operational data using a provided, trained feature extraction model, by means of the server; b. providing a trained base model to the server, and applying the operational data and the meta-information relating to each client to the base model and obtaining associated operational features, by means of the server; c. calculating the similarity between the operational features of the respective clients in the form of a similarity value; d. assigning the client to a particular client group from a set of client groups, each client group from a set of client groups being defined by means of an associated specified value range for the similarity value of the particular client group; e. determining the largest client group, which has the most clients, from a set of client groups; f. forming the average similarity value from the respective similarity values of the clients in the largest client group; g. defining a personalization factor for each client on the basis of the average similarity value, which expresses a similarity of the particular client to the average similarity value of the associated largest client group; h. generating global model parameters for each client on the basis of the associated personalization factor and the difference between the associated operational data and the global model parameters; i. updating the base model using the global model parameters; j. continuing with step c) until a predetermined criterion for the repetition of steps c) - i) has been satisfied, and otherwise continuing with step k); and k. operating the technical apparatus with the updated base model.

Description

[0001] Method and system for operating a technical device

[0002] The invention relates to a computer-implemented method for operating a technical device through a client-server system comprising a server and clients, each of which is connected to respective technical devices.

[0003] Operating a device using a model based on artificial intelligence or machine learning is an important application in industry, with increasing use of large language models (LLM).

[0004] A foundation model (FM) is a machine learning (ML) model that is trained with very comprehensive data, i.e., a very large dataset, so that it can be applied to a wide variety of use cases.

[0005] Basic models have transformed artificial intelligence (AI) and drive prominent generative AI applications such as ChatGPT.

[0006] Most approaches focus on centralized data to train business models.

[0007] In an example internet scenario, data is collected via a crawler and stored in a central storage location.

[0008] The Foundation model is typically trained in a cloud environment, such as OpenAI and Microsoft Azure.

[0009] In an industrial environment, data may be distributed across different locations and cannot be shared or collected in a central storage location due to legal restrictions as well as practical limitations, such as limited network bandwidth in factories.

[0010] In the state of the art, distributed training of AI is already used to solve the aforementioned problems, for example, to avoid having to copy data to a central storage location, which can be stored on local storage.

[0011] A conventional sequential technique based on federated learning (FL) can also be used to train basic models when, for example, devices have limited computing resources or when network bandwidth is restricted.

[0012] However, data characteristics such as "IIDness" play an important role in the applicability of sequential versus parallel federated learning. In probability theory and statistics, a collection of random variables is independent and identically distributed if each random variable has the same probability distribution as the others and is all independent of each other.

[0013] This property was first defined in statistics and is used in various fields such as data mining and signal processing.

[0014] Statistics often deals with random samples, which can be thought of as a set of objects that are selected at random.

[0015] More formally, it is "a sequence of independent, identically distributed random data points" (HD for short).

[0016] In other words, the terms random sample and HD are equivalent.

[0017] In statistics, "random sample" is the typical terminology, but in probability it is more common to say "HD", where "HDness" expresses the numerical value or degree of HD, and thus "high HDness" stands for a high value for HD, and "low HDness" stands for a low value for HD.

[0018] Identically distributed means that there are no general trends - the distribution does not fluctuate and all elements in the sample come from the same probability distribution.

[0019] Independent means that the sample elements are independent events; in other words, they are not related to each other in any way.

[0020] Knowing the value of one variable provides no information about the value of the other, and vice versa.

[0021] In a high-definition industrial image dataset with high HDness, each image representing a specific industrial component, such as bolts or screws, is independent of others and comes from the same distribution, thus ensuring consistent statistical properties for all images.

[0022] In an industrial image dataset without HD, i.e., low HDness, images collected from different factories or production lines may show correlations due to variations in manufacturing processes or equipment, leading to differences in the statistical properties between subsets of images.

[0023] To train basic models, scenarios with low I-D data must also be mastered, with the degree of HDness determining which training method for distributed learning can be applied. Additionally, the available bandwidth between the system units of an AI system is a significant and limiting factor when exchanging large ML models and / or training data between the server and the connected clients.

[0024] It is therefore an object of the invention to provide a method which allows improved model training for basic models with a distributed system architecture, in particular for systems with low bandwidth between the system units, and training data with both low IIDness and high IIDness.

[0025] It is advantageous if data anonymity is supported, meaning that no raw data is distributed within the system's network.

[0026] The object of the invention is achieved by a computer-implemented method for operating a technical device through a client-server system comprising a server and clients, each connected to respective technical devices, and the following steps: a. Providing operational data from the operation of the technical device of the respective client, and generating operational meta-information about the operation of the technical device from the operational data using a provided, trained feature extraction model, by the server; b. Providing a trained base model to the server, and applying the operational data and the operational meta-information of the respective client to the base model and obtaining respective operational features, by the server; c. Calculating the similarity between the operational features of the respective clients in the form of a similarity value; d.Assigning clients to a respective client group from a set of client groups, where each client group is defined by a predefined range of similarity values ​​for that client group; e. Determining the largest client group from a set of client groups, which contains the most clients; f. Calculating the average similarity value from the respective similarity values ​​of the clients in the largest client group; g. Setting a personalization factor for each client based on the average similarity value, which expresses the similarity of the respective client to the average similarity value of the largest client group; h. Generating global model parameters for each client based on the respective personalization factor and the difference between the respective operational data and the global model parameters; i.Update the base model using the global model parameters, j. Continue at step c) until a predetermined criterion for repeating steps c) - i) is met, otherwise continue with step k), k. Operate the technical device with the updated base model.

[0027] The invention makes it advantageous to achieve personalized updates of model parameters.

[0028] This approach uses the similarity between clients within the largest group to adjust model fittings and improve model performance and convergence in federated learning.

[0029] Furthermore, a dynamic personalization factor can be advantageously introduced, which is used to improve the overall model by specifically weighting data from individual clients more heavily, or to increase the model accuracy.

[0030] By dynamically defining the personalization factor based on average similarity values, the method adjusts the influence of each client's local dataset on global model updates, enabling flexible, fine-grained, and detailed adjustments to accommodate different data distributions in federated environments.

[0031] A major advantage of this approach is that it addresses sequential bias accumulation in sequential federated learning (SFL).

[0032] If the order of device updates is not carefully considered, distortions that accumulate during sequential updates can negatively impact the overall performance of the model.

[0033] For example, early updates of devices with skewed data distributions can dominate the model, leading to poor generalization.

[0034] The "Dynamic Personalization" factor solves this problem in an advantageous and simple way, allowing the model to be continuously adapted to current changes or influences during the operation of the technical device, thus maintaining or improving accuracy.

[0035] In this context, operational meta-information about the operation of the technical device refers to data that describes, for example, the correct operation or operation in impermissible operating ranges in order to carry out corresponding anomaly detection or operational status monitoring, or data to describe the technical device itself, its structure, operating status or its operating environment.

[0036] This data can be obtained, for example, from raw data such as sensor data or image data, whereby, for example, resizing, normalization or other transformations of the raw data can be carried out to prepare for subsequent further processing of multiple datasets by the application of one or more ML models.

[0037] The technical device, for example a pump, a production machine, a robot, etc., can use integrated sensors to collect device data such as operating voltages, temperatures or vibrations, and analyze this device data using a monitoring device with the help of an artificial intelligence-based model, for example to monitor the ongoing operation, and the device is controlled by an artificial intelligence-based model, for example in the form of control signals which are generated by the model from a control device.

[0038] The technical device can, for example, be an inspection device for a product produced by a production machine, whereby imaging sensors are used to detect the product, analyze its external condition, and control the production machine accordingly.

[0039] In a further development of the invention, it is provided that the provision in step a) is carried out by the respective client.

[0040] This allows clients to process and provide the data themselves, thus relieving the server and reducing the amount of data that needs to be transferred over the network, as well as making it easy and advantageous to comply with requirements for confidentiality or secure data transmission.

[0041] The operational characteristics can also be referred to as feature vectors, which were extracted, for example, from sensor images by applying a corresponding ML model as inference, and the ML model was trained using provided sensor images.

[0042] In a further development of the invention, steps c) - i) are performed by the server. This ensures that the data can be processed in a simple and efficient manner according to the invention.

[0043] In a further development of the invention, it is provided that the similarity is the cosine similarity.

[0044] This ensures that the similarity in the present context can be determined in a simple and efficient manner.

[0045] In a further development of the invention, it is provided that the value ranges of the individual client groups from a set of client groups do not overlap.

[0046] This ensures that the groups have a high degree of differentiation and that models formed from them can exhibit a high degree of accuracy.

[0047] In a further development of the invention, it is provided that a high respective personalization factor results in a high weighting of the respective model parameters in the global model.

[0048] This ensures that the respective personalization factor can be taken into account in a simple and efficient way when determining models.

[0049] In a further development of the invention, it is provided that the predetermined criterion is a number of repetitions of steps c) - i), the achievement of a desired level of accuracy, or the observation of a minimal improvement in the global model performance.

[0050] This ensures that group formation can be carried out in a simple and efficient manner.

[0051] The problem according to the invention is also solved by a client-server system for operating a technical device, wherein the system comprises a server and clients, each of which is connected to the respective technical devices, and the method according to the invention is carried out.

[0052] In a further development of the invention, it is provided that the respective technical device has a device for recording the model accuracy during the operation of the technical device, which is configured to record the model accuracy in comparison with currently recorded operating parameters of the technical device.

[0053] The problem according to the invention is also solved by a computer program product with machine-readable instructions stored therein, which, when executed by the client-server system according to the invention, cause it to execute the method according to the invention. The invention is described in more detail in the following drawings with reference to an exemplary embodiment. The figures show

[0054] Fig. 1 Flowcharts with two embodiments of the method according to the invention,

[0055] Fig. 2 shows an example of a first part of the procedure in the form of a pseudo-code,

[0056] Fig. 3 shows an example of a second part of the procedure in the form of a pseudo-code,

[0057] Fig. 4 shows an example of a third part of the procedure in the form of a pseudo-code,

[0058] Fig. 5 shows an example of a system on which the method according to the invention can be carried out.

[0059] Fig. 1 shows on the left side of the figure a first embodiment with a flowchart of the inventive method 100 with ten method steps 110 — 200.

[0060] On the right side of the figure, a second embodiment is shown with a flowchart of the method according to the invention.

[0061] Procedure 100 can be implemented, for example, through the following steps.

[0062] The first step 110 involves extracting features from images provided by clients.

[0063] The images are first subjected to preprocessing, which may include resizing, normalization, or other transformations to standardize them for feature extraction.

[0064] Fig. 2 shows an embodiment of the first step 110 of the method according to the invention, in which features are extracted from the preprocessed images using a pretrained CNN model (Convolutional Neural Network).

[0065] These functions capture high-level representations of the image content and enable comparisons between images and subsequent clustering based on their similarity.

[0066] The subsequent second step 120 involves calculating the cosine similarity between the feature vectors extracted from images of different clients.

[0067] Fig. 3 shows an embodiment of the second step 120 of the method according to the invention, wherein the similarity of the operating characteristics of clients C1-C3 is calculated.

[0068] Each client's dataset is represented by a set of feature vectors, and the cosine similarity is calculated pairwise between these vectors. This similarity measure quantifies the degree of similarity between the datasets of different clients and provides a basis for grouping clients with similar datasets.

[0069] Fig. 4 shows an embodiment of the third step 130 of the method according to the invention, in which clients are grouped based on the cosine similarity values ​​calculated in step 120.

[0070] In the fourth step 140 according to Fig. 1, a predefined similarity threshold is applied to determine when clients are considered similar enough to be grouped together.

[0071] In the fifth step, 150 groups are iteratively assigned based on their pairwise similarity values, with each group representing a group of clients with sufficiently similar datasets.

[0072] The grouping process continues until all clients have been assigned to the appropriate groups, forming related groups based on the similarity of their data sets.

[0073] In the sixth step 160, the largest group is identified, i.e., among the groups, the largest group overall - with the most assigned clients - is determined, which were formed in step 130.

[0074] In step seven, 170, the average similarity rating between the client and their "peer" clients, i.e., clients within the same group, is calculated for each client.

[0075] In step eight 180, the global model parameters are personalized based on the client's local data set and the average similarity value.

[0076] In step nine, 190, the global model is updated with the personalized parameters from the client.

[0077] In step 200, the sequential iteration is repeated through all clients in the group until one or more predefined convergence criteria are met.

[0078] The convergence criteria are criteria that determine when the iteration should be terminated by the client.

[0079] This could be, for example, reaching a specific number of iterations, achieving a desired level of accuracy, or observing a minimal improvement in the overall model performance. The model's accuracy level can be determined when the model is used in the operation of a technical device by means of a suitable instrument, whereby the absolute accuracy of the model or the relative change in accuracy during iterative improvements of the model can be used.

[0080] After convergence, the final global model represents the aggregated knowledge learned from all customers within the largest cluster, personalized based on their local datasets and similarities.

[0081] Personalizing the global model parameters is possible because each client in the context of federated learning has their own local dataset, and the global model is updated based on the aggregated knowledge of all clients.

[0082] When a client participates in the training, they calculate the difference between their local data set and the current global model parameters.

[0083] This difference represents the gradient of the loss function with respect to the parameters, which is calculated for each client i, where the gradient of the loss function L with respect to the parameters 0 is calculated using the respective local data set Dj.

[0084] This gradient – ​​denoted as V0L(Di,0) – represents the direction and magnitude of the change required to minimize the loss function while updating the parameters from the current global model state towards the client dataset Dj.

[0085] The difference between the client's local data set and the current global model parameters is defined by the gradient D-0=V0L(Di,0) and captures how the global model needs to be changed to better match the client's local data set.

[0086] In other words, the difference provides information about updates needed to adapt the global model to the characteristics of the clients' data set.

[0087] The personalization factor is used to include personalized updates based on the client's data set and the personalization factor Ai, and to scale the gradient using the personalization factor Aj.

[0088] Based on the average similarity value calculated for each client, the personalization factor Aj can be defined according to this value. The personalization factor Aj scales the impact of the client's local dataset on updating the global model parameters.

[0089] A higher value for average similarity leads to larger personalization factors, indicating a stronger influence of the client's local dataset on parameter updates.

[0090] An update equation determines the update of the global model parameters 0 using the scaled gradient 0j=0+Ai-(Dj- 0) .

[0091] This equation combines the current global model parameters with personalized updates based on the client's dataset and the personalization factor, and generates the personalized model parameters 0j specific to client i.

[0092] The method according to the invention can therefore also be described by a further, second embodiment using the following steps: a. Providing operational data from the operation of the technical device TD1-TD3 of the respective client C1-C3, and generating operational meta-information about the operation of the technical device TD1-TD3 from the operational data using a provided, trained feature extraction model, by the server S, b. Providing a trained base model FM to the server S, and applying the operational data and the meta-information of the respective client C1-C3 to the base model FM and obtaining respective operational features, by the server S, c. Calculating the similarity between the operational features of the respective clients C1-C3 in the form of a similarity value, d.Assigning clients C1-C3 to a respective client group from a set of client groups, where each client group is defined by a predefined range of similarity values ​​for the respective client group; e. Determining the largest client group from a set of client groups, which has the most clients (i.e., the highest number of clients); f. Calculating the average similarity value from the respective similarity values ​​of the clients in the largest client group; g. Determining a personalization factor for each client based on the average similarity value, which expresses the similarity of the respective client to the average similarity value of the respective largest client group; h.i. Generate global model parameters for each client based on the respective personalization factor and the difference between the respective operational data and the global model parameters, i. Update the base model (FM) using the global model parameters, j. Continue at step c) until a predetermined criterion for repeating steps c) - i) is met, otherwise continue at step k), k. Operate the technical device (TD1) with the updated base model.

[0093] The provision in step a) can be carried out by the respective client C1-C3.

[0094] Steps c) - i) can be executed by server S.

[0095] The similarity is the cosine similarity.

[0096] The value ranges of the individual client groups from a set of client groups preferably do not overlap.

[0097] A high personalization factor results in a high weighting of the respective model parameters during generation or updating in the global model.

[0098] The predetermined criterion of a number of repetitions of steps c) - i) can be advantageous for achieving a desired level of accuracy or observing a minimal improvement in global model performance.

[0099] In comparison to the first embodiment mentioned above, the individual process steps can now be compared to the second embodiment.

[0100] Steps a) and b) can therefore correspond to the previously described procedure step 110.

[0101] Step c) can correspond to the previously described procedure step 120.

[0102] Step d) can correspond to the previously described process steps 130, 140 and 150.

[0103] Step e) can correspond to the previously described procedure step 160.

[0104] Step f) can correspond to the previously described procedure step 170.

[0105] Steps g) and h) can correspond to the previously described procedure step 180.

[0106] Step i) can correspond to the previously described procedure step 190.

[0107] Step j) can correspond to the previously described process step 200. Fig. 5 shows an example of a system on which the method according to the invention can be carried out.

[0108] The client-server system SYS shown has one server S and three clients C1-C3.

[0109] Each client C1-C3 is connected to individual technical devices TD1-TD3. Each technical device TD1-TD3 may include a device for recording model accuracy during operation, which is configured to record the model accuracy in comparison with currently recorded operating parameters of the technical device TD1-TD3 during its operation.

[0110] Regardless of the grammatical gender of a particular term, persons with male, female or other gender identities are included.

[0111] Reference symbol list

[0112] 100 procedures

[0113] 110-200, a) - k) Procedural steps

[0114] C1-C3 Client S Server

[0115] SYS Client-Server System

[0116] TD1-TD3 technical device

Claims

Patent claims 1. A computer-implemented method for operating a technical device (TD1) through a client-server system (SYS) with a server (S) and clients (C1-C3), each connected to their respective technical devices (TD1-TD3), and performing the following steps: a. Providing operational data from the operation of the technical device (TD1-TD3) to the respective client (C1-C3), and generating operational meta-information about the operation of the technical device (TD1-TD3) from the operational data using a provided, trained feature extraction model, by the server (S); b. Providing a trained base model to the server (S), and applying the operational data and meta-information of the respective client (C1-C3) to the base model and obtaining the respective operational features, by the server (S); c. Calculating the similarity between the operational characteristics of the respective clients (C1-C3) in the form of a similarity value, i.e.Assigning clients (C1-C3) to a respective client group from a set of client groups, where each client group is defined by a predefined range of similarity values ​​for the respective client group, e. Determining the largest client group from a set of client groups, which has the most clients, f. Calculating the average similarity value from the respective similarity values ​​of the clients in the largest client group, g. Setting a personalization factor for each client based on the average similarity value, which expresses the similarity of the respective client to the average similarity value of the largest client group, h. Generating global model parameters for each client based on the respective personalization factor and the difference between the respective operational data and the global model parameters, i.Update the base model using the global model parameters, j. Continue at step c) until a predetermined criterion for repetitions of steps c) - i) is met, otherwise continue with step k), k. Operate the technical device (TD1) with the updated base model.

2. Method according to the preceding claim, wherein the provision in step a) is carried out by the respective client (C1-C3).

3. Method according to any of the preceding claims, wherein steps c) - i) are performed by the server (S).

4. Method according to any of the preceding claims, wherein the similarity is the cosine similarity.

5. Method according to one of the preceding claims, wherein the value ranges of the individual client groups from a set of client groups do not overlap.

6. Method according to one of the preceding claims, wherein a high respective personalization factor results in a high weighting of the respective model parameters in the global model.

7. Method according to any of the preceding claims, wherein the predetermined criterion is a number of repetitions of steps c) - i), the achievement of a desired level of accuracy, or the observation of a minimal improvement in global model performance.

8. Client-server system (SYS) for operating a technical device (TD1) with a server (S) and clients (C1-C3), each of which is connected to respective technical devices (TD1-TD3), and the method according to one of the preceding claims is carried out.

9. System (SYS) according to the preceding claim, wherein the respective technical device (TD1-TD3) has a device for recording the model accuracy during the operation of the technical device (TD1-TD3), which is configured to record the model accuracy in comparison with currently recorded operating parameters of the technical device (TD1-TD3).

10. Computer program product comprising machine-readable instructions stored therein which, when executed by the client-server system (SYS) according to one of the two preceding claims, cause the latter to execute the method according to one of the preceding claims.

Images