Method and system for inspecting or operating a product

Abstract

The invention relates to a computer-implemented method for inspecting or operating a first product (TD1), the following steps being carried out: a) transmitting a provided global model (GM) of the first product (TD1) or the operation thereof from the server (S) to the first client (C1) and second client (C2), b) capturing a first data set (DS1) of the first product (TD1) or the operation thereof with first product parameters of the first product (TD1) as respective first subsets (C1.1-C1.3), and generating and training respective first local models (LM1.1-LM1.3) with the respective first subsets (C1.1-C1.3), c) capturing a respective second data set (DS2) of the respective second product (TD2) or the operation thereof with respective second product parameters of the respective second product (TD2) as respective second subsets (C2.1-C2.3), and generating and training respective second local models (LM2.1-LM2.3) with the respective second subsets (C2.1-C2.3), d) performing a similarity analysis between models of the first and the second local models (LM1.1-LM1.3, LM2.1-LM2.3), and assigning these similar models to a common model group (G1-G3), e) training a global detail model (M1) on the basis of the model of the common model group (G1-G3) and transmitting same to the first client (C1), f) applying the detail model (M1) to the first product (D1) for inspection or operation thereof, by the first client (C1).

Description

[0001] Method and system for inspecting or operating a product

[0002] The invention relates to a computer-implemented method and a system for inspecting or operating a first product, as well as a computer program product.

[0003] Nowadays, artificial intelligence is used in industrial environments to visually inspect manufactured products or to monitor or control devices in operation.

[0004] The principle of federated learning (FL) can be used to quickly and easily apply models based on artificial intelligence, using client-server systems.

[0005] However, it has been shown that when using artificial intelligence, insufficient model accuracy can occur due to multimodal data distributions from clients.

[0006] Model performance can be poor if, for example, FL is applied to clients with data from multiple underlying data distributions and sources, as no other similar client is suitable for collaboration, since a multimodal data distribution may be due to each client.

[0007] This is particularly the case with industrial customers where multiple data sources, such as sensors on production machines, are used in a factory-wide database that is used for local training in the FL context.

[0008] In the prior art, the use of a Gaussian mixture model for each client is known, as well as the shared use of the resulting mean (p) and sigma (o) parameters with the server, which in turn aggregates them and calculates a globally common multimodal distribution on this basis.

[0009] Customers personalize the resulting global FL model based on the difference to the globally aggregated multimodal distribution.

[0010] Publication WO 2022 / 258524 A1 describes a method for inspecting or operating a product using artificial intelligence, in which the underlying model is generated using federated learning, but still does not achieve sufficient model accuracy, especially with multiple clients whose data quality varies greatly or whose data are heterogeneous.

[0011] However, the known approaches do not lead to sufficient model accuracy. It is therefore an object of the invention to further improve the model accuracy during the inspection of a product or the operation of a device, particularly for clients with heterogeneous data or heterogeneous probability distributions of the data.

[0012] The problem according to the invention is solved by a computer-implemented method for inspecting or operating a first product, comprising a system with a server, a first client assigned to the first product, and at least one second client assigned to a respective second product, and performing the following steps: a) transmitting a provided global model of the first product or its operation from the server, which model is defined on the basis of artificial intelligence in the form of model weights and a model structure, to the first client and the at least one second client; b) acquiring a first data set of the first product or its operation with at least two first product parameters of the first product as respective first subsets of the first data set with an associated corresponding acquisition means; and, starting from the global model,c) generate and train at least two respective first local models with their respective first subsets, and transmit these at least two respective first local models to the server in the form of model weights and a model structure, by the first client; c) capture a respective second data set from the respective second product or its operation with at least two respective second product parameters of the respective second product as respective second subsets of the respective second data set with an associated corresponding capture device, and, starting from the global model, generate and train at least two respective second local models with their respective second subsets, and transmit these at least two respective second local models to the server in the form of model weights and a model structure, by the respective second client.d) Performing a similarity analysis between models of the first and second local models, and if a similarity exists between a model of the first local models and a model of the second local models, assigning these similar models to a common model group, by the server; e) Training at least one global detail model based on the model of the common model group and transferring it to the first client, by the server; f) Applying the detail model to the first product for inspection or operation, by the first client.

[0013] The invention provides for creating subsets of clients (sub-clients) based on a given original set of clients with a multimodal distribution.

[0014] Each generated subclient is then assigned to a cohort or group with other subclients that share similar data distributions.

[0015] This allows similar clients to contribute together to their own model, which has improved model accuracy.

[0016] The respective subsets represent independent clients that, while relating to the same product or device, may have differences in its manufacture, particularly in the environment or operating hours of the relevant equipment or operating personnel, or similar factors, which can be reflected in different data or datasets, and therefore datasets of varying quality.

[0017] Therefore, the first data set differs from the respective second data sets in the statistical composition of their data, such as in the data distribution or quality.

[0018] Therefore, it is advantageous to favor those data or datasets when training a global model which have higher data quality and which exhibit favorable and similar probability distributions.

[0019] Furthermore, by selectively choosing data or datasets, it is possible to personalize the resulting model, for example to configure production or operations according to individual strengths or aspects, such as particularly fast and cost-effective production, or operation of a device in colder or warmer environments, or particularly reliable operation.

[0020] These specific adjustments in the sense of personalization can be made by ensuring that a model has a particular accuracy in the desired aspect, as explained above.

[0021] The invention therefore provides that “virtual” sub-clients are created that are specific to underlying entities, such as production lines or seasons, such as separate models per month.

[0022] The specific models can be used for inference services of the respective entities or during the respective season. Therefore, potentially weak models are not applied generally; for example, a global factory model is not used for a single production line.

[0023] Furthermore, client splitting increases the likelihood that other sub-clients, for example from other factories, have a similar data distribution.

[0024] Therefore, cohort or group formation can further increase the chance of a high-performing personalized model.

[0025] When forming groups or cohorts, a clustering approach can be used to aggregate new clients into cohorts with other similar clients.

[0026] Statistical aspects of the client datasets, such as mean, standard deviation, etc., can be applied as characteristics for grouping.

[0027] Furthermore, it can be advantageous to split clients with respect to the original data source, if possible, for example by splitting data from factory clients into subsets for more accurate production line clients.

[0028] Based on this new client design, model performance can be further improved through personalization.

[0029] A product can refer to an object to be analyzed or inspected, which is detected by optical or other electronic or magnetic sensors, for example in its external appearance.

[0030] A means of recording product properties can be a recording device which is set up to record the manufactured product with sensors, for example the external appearance or geometric, mechanical, electrical, magnetic properties of the product.

[0031] Additionally, the product can also refer to a technical device to be analyzed or inspected during its operation, which is recorded by optical or other electronic or magnetic sensors, for example with an operating voltage, power consumption, vibrations, temperature, rotational movements, etc., which are recorded by operating data during operation.

[0032] A means of recording the operating parameters of a technical device can be a recording device designed to detect the operation of the device using sensors, for example, mechanical, electrical, magnetic, or thermal operating parameters of the device or its environment. An operating parameter can therefore be defined by a corresponding measured quantity, which can be determined using the recording device and its sensor or sensors.

[0033] The detection device may have a computing device with a processor and a memory.

[0034] It is clear that several different operating parameters can be recorded and used for model generation and model training, either individually or in combination with other operating parameters.

[0035] Servers and clients each have a computing device with a processor and memory.

[0036] The artificial intelligence-based model can be a neural network, which is described in terms of model weights and a model structure.

[0037] In neuroscience, a neural network is defined as any number of interconnected neurons that, as part of a nervous system, form a connection geared towards specific functions.

[0038] The individual connections are expressed by model weights, and the structure of the network as a model structure.

[0039] In statistics, particularly in multivariate statistics, similarity and distance measures can be used to measure the similarity between different objects, with similarity measures typically being used for nominal or ordinal scaled variables and distance measures being used for metric scaled variables (i.e., interval and ratio scales).

[0040] For example, a correlation function can also be used for a similarity analysis.

[0041] When checking for similarity between a model of the first local models and a model of the second local models, several common model groups can also be formed, with a separate global detail model being created and trained for each of these common model groups.

[0042] The application of similarity analysis between the models allows for variable prioritization or weighting of the clients used, thereby enabling dynamic adjustment of the global detail model during aggregation if a client's data develops unfavorably and the corresponding local model no longer meets the quality requirements or expectations. In a further development of the invention, the method for inspecting the first product is provided, and the first and at least one second product are each manufactured by a production machine. The inspection of each product is carried out by a respective technical inspection device configured to capture the manufactured product as a data set, preferably by a visual inspection device, and particularly preferably by an optical camera.

[0043] A product can refer to an object that is detected by optical or other electronic or magnetic sensors, for example in its external appearance, in order to control correct production and to sort out defective products.

[0044] The invention provides a federated approach which ensures data protection for the data of individual clients by transmitting only model weights and model structure between clients and server for the respective models.

[0045] The aforementioned training course therefore refers to a procedure for inspecting a first product.

[0046] In a further development of the invention, it is provided that for the respective manufactured product, the respective two first product parameters of the first product and the respective two second product parameters of the respective second product are determined by the respective technical inspection means, which are configured to record the respective manufactured product as a respective data set, preferably by position data or orientation data or operating parameters of the inspection means.

[0047] Operating parameters of the inspection device can include, for example, the position of the product on a transport device, such as a conveyor belt, as well as settings of the inspection device itself, such as optical settings of a camera, such as focus, exposure time, etc.

[0048] The aforementioned training course therefore refers to a procedure for inspecting a first product.

[0049] In a further development of the invention, the method for operating the first product is provided, and the first and at least one second product are each technical devices, with each technical device being connected to the respective client for operation by the respective client. Additionally or alternatively, the product can also refer to a technical device during its operation, which is monitored by optical or other electronic or magnetic sensors, for example, by operating voltage, power consumption, vibrations, temperature, rotational movements, etc., which are recorded by operating data or operating parameters during operation.

[0050] The aforementioned training therefore refers to a procedure for operating a technical device as a first product.

[0051] In a further development of the invention, it is provided that the respective data set describes a permissible or impermissible operating mode of the respective technical device.

[0052] For example, a permissible continuous operating mode can be defined for a device, which is to be monitored using the method according to the invention in order to carry out predictive maintenance, as well as certain fault operating modes in which a shortened maintenance interval must be provided.

[0053] The aforementioned training therefore refers to a procedure for operating a technical device as a first product.

[0054] In a further development of the invention, it is provided that the respective data set describes an operating parameter of the respective technical device.

[0055] To apply an inference, a specific operating parameter of the device can be used, through operating data collected during the operation of the device.

[0056] The aforementioned training therefore refers to a procedure for operating a technical device as a first product.

[0057] In a further development of the invention, it is provided that the operating parameter of the respective technical device is an operating voltage or a current consumption or a power consumption or an operating temperature or an ambient temperature or an electric field strength or a magnetic field strength or a computing load or a storage load.

[0058] To apply inference, a specific operating parameter of the device can be used, for example, operating voltage, power consumption, vibrations, temperature or rotational movements of the device, which are recorded by operating data during the operation of the device.

[0059] The aforementioned further development thus relates to a method for operating a technical device as the first product. In a further development of the invention, it is provided that the similarity analysis is carried out by having the first client calculate a statistical distribution function for the first subsets, and by having the respective second client calculate a distribution function for the second subsets, and by transmitting the respective distribution functions to the server, which then compares the distribution functions with each other for similarity.

[0060] This makes it easy to identify similar subsets in the captured data set and transmit them to the server anonymously.

[0061] This ensures that no sensitive manufacturing or operational data is disclosed to the server, which is extremely beneficial for data protection and the safeguarding of trade secrets of production facilities.

[0062] In a further development of the invention, it is provided that the similarity analysis is carried out by the respective client providing the respective meta-information of the respective operating parameter of the respective product for the respective subset via the respective acquisition means, and the respective meta-information is transmitted from the respective client to the server, and the meta-information is compared with each other for similarity by the server.

[0063] In this context, metadata or metainformation refers to descriptive information about the subsets, i.e., excluding the numerical data of the recorded operating parameters themselves, which may also be cryptic, i.e., simple labels or names for groups or subsets.

[0064] This makes it easy to identify similar subsets in the captured data set and transmit them to the server anonymously.

[0065] This ensures that no sensitive manufacturing or operational data is disclosed to the server, which is extremely beneficial for data protection and the safeguarding of trade secrets of production facilities.

[0066] In a further development of the invention, it is provided that the similarity analysis is determined on the basis of respective predetermined, overlapping boundary areas.

[0067] This allows similarity analysis to be carried out in a simple way, for example by predefining permissible limit ranges for certain operating modes.

[0068] A boundary range can be defined, for example, by a lower and upper limit, where these limits can be variable via an operating parameter, i.e., defined by a mathematical function such as a distribution. Depending on the mathematical function or distribution underlying the applied similarity analysis, a boundary range can be defined by whether a calculated similarity value lies within or outside the lower and upper limits.

[0069] The problem according to the invention is also solved by a system for operating a technical device comprising a server, a connected first client and at least one connected second client, and the first client is connected to the technical device, wherein the system is configured to execute the method according to the invention.

[0070] The problem according to the invention is also solved by a computer program product with machine-readable instructions stored therein which, when executed by the system according to the invention, cause it to execute the method according to the invention.

[0071] The invention is described in detail in the following figures using an exemplary embodiment. The figures show

[0072] Fig. 1 shows a simplified block diagram with individual process steps of an embodiment of the invention.

[0073] Fig. 2 is a flowchart with an embodiment of the invention according to the invention.

[0074] Fig. 1 shows an embodiment of the invention as a simplified block diagram and describes a method for inspecting or operating a first product TD1.

[0075] The procedure is applied to a system with a server S, a first client C1 and a second client C2.

[0076] The first client C1 is assigned to a first product TD1.

[0077] The second client C2 is assigned to a second product TD2.

[0078] The following process steps are performed: a) Transmission of a provided global model GM of the first product TD1 from server S, which model is defined based on artificial intelligence in the form of model weights and a model structure, to the first client C1 and the second client C2, b) Acquisition of a first data set DS1 of the first product TD1 with three first product parameters of the first product TD1 as respective first subsets C1.1-C1.3 of the first data set DS1 with a corresponding associated acquisition tool, and, starting from the global model GM, generation and training of three respective first local models LM1.1-LM1.3 with the respective first subsets C1.1-C1.3, and transmission of the three respective first local models LM1.1-LM1.3. to the server S in the form of model weights and a model structure, by the first client C1, c) Acquisition of a respective second data set DS2 of the respective second product TD2 or its operation with respective at least two second product parameters of the respective second product TD2 as respective second subsets C2.1-C2.3 of the respective second data set DS2 with a connected corresponding acquisition means, and, starting from the global model GM, generation and training of three respective second local models LM2.1-LM2.3 with the respective second subsets 01.1-C1.3, and transmission of the at least two respective second local models LM2.1-LM2.3 to the server S in the form of model weights and a model structure, by the respective second client 02, d) Performance of a similarity analysis between models of the first and the second local models LM1.1-LM1.3, LM2.1-LM2.3, and if there is a similarity between a model of the first local models LM1.1-LM1.3 and a model of the second local models LM2.1-LM2.3, assign these similar models to a common model group G1-G3, by the server S, e) train at least one global detail model M1 based on the model of the common model group G1-G3 and transfer it to the first client 01, by the server S, f) apply the detail model M1 to the first product D1 for inspection or its operation, by the first client 01.

[0079] The first and second products are each manufactured by a production machine.

[0080] The product can be inspected using a technical inspection device designed to capture the manufactured product, such as an optical camera (visual quality inspection, or VQI).

[0081] The respective product parameters are provided by respective sensor means or detection means, which are included by the respective client 01, 02, and are set up to detect the condition or product characteristics of the manufactured product TD1, TD2.

[0082] A means of recording product properties can be a recording device which is set up to record the manufactured product with sensors, for example the external appearance or geometric, mechanical, electrical, magnetic properties of the product, such as by a camera for the external appearance of the product.

[0083] The formation of subsets 01.1-C1.3, C2.1-C2.3 in steps b) and c) can also be called "splitting" SP and is carried out by the respective client C1 , C2.

[0084] The model training MT of the first, second and third global detail models M1-M3 in step e) is performed by the server S.

[0085] The similarity analysis is performed by calculating a statistical distribution function D1.1-D1.3, D2.1-D2.3 by the first client C1 for the first subsets 01.1-01.3, and by the respective second client C2 for the second subsets C2.1-C2.3.

[0086] The respective distribution functions D1.1-D1.3, D2.1-D2.3 are transmitted to the server s.

[0087] The distribution functions D1.1-D1.3 and D2.1-D2.3 are compared for similarity by server S, for example by applying a correlation function.

[0088] The similarity analysis can be performed by having the respective client C1, C2 provide the respective meta-information of the respective operating parameter of the respective product TD1, TD2 for the respective subset 01.1-01.3, C2.1-C2.3 via the respective acquisition device, and by having the respective meta-information be transferred from the respective client C1, C2 to the server S, and by having the server S compare the meta-information with each other for similarity.

[0089] Similarity analysis can be determined based on respective predetermined, overlapping boundary areas.

[0090] Fig. 2 shows an embodiment of the invention in the form of a flowchart.

[0091] The procedure for inspecting or operating a first product TD1 is computer-implemented in at least one step.

[0092] Regardless of the grammatical gender of a given term, persons of male, female, or other gender identities are included. Reference list:

[0093] 01, 02 Client

[0094] C1.1-C1.3, C2.1-C2.3 subset

[0095] DS1, DS2 Data set D1.1-D1.3, D2.1-D2.3 Distribution function

[0096] G1-G3 Group

[0097] GM global model

[0098] LM1-LM3 local model

[0099] M1-M3 Detail Model MT Model Training

[0100] S Server

[0101] SP Formation of subsets (English: "splitting")

[0102] TD1, TD2 technical device

Claims

Patent claims 1. A computer-implemented method for inspecting or operating a first product (TD1), comprising a system with a server (S), a first client (C1) associated with the first product (TD1), and at least one second client (C2) associated with a respective second product (TD2), and performing the following steps: a) transmitting a provided global model (GM) of the first product (TD1) or its operation from the server (S), which model is defined based on artificial intelligence in the form of model weights and a model structure, to the first client (01) and the at least one second client (02); b) capturing a first data set (DS1) of the first product (TD1) or its operation with at least two first product parameters of the first product (TD1) as respective first subsets (01.1-01.2).3) of the first data set (DS1) with an associated corresponding acquisition tool, and, starting from the global model (GM), generate and train at least two respective first local models (LM1.1-LM1.3) with the respective first subsets (01.1-01.3), and transmit the at least two respective first local models (LM1.1-LM1.3) to the server (S) in the form of model weights and a model structure, by the first client (01), c) Acquire a respective second data set (DS2) of the respective second product (TD2) or its operation with respective at least two second product parameters of the respective second product (TD2) as respective second subsets (02.1-02.3) of the respective second data set (DS2) with an associated corresponding acquisition tool, and, starting from the global model (GM), generate and train at least two respective second local models (LM2.1-LM2.3) with the respective second subsets (02.1-02.3), and transmit the at least two respective second local models (LM2.1-LM2.3) to the server (S) in the form of model weights and a model structure, by the respective second client (02), d) Perform a similarity analysis between models of the first and the second local models (LM1.1-LM1.3, LM2.1-LM2.3), and if there is a similarity between a model of the first local models (LM1.1-LM1.3) and a model of the second local models (LM2.1-LM2.3), assign these similar models to a common model group (G1-G3), by the server (S),. e) Training at least one global detail model (M1) based on the model of the common model group (G1-G3) and transferring it to the first client (01) by the server (S), f) Applying the detail model (M1) to the first product (D1) for inspection or operation by the first client (01).

2. Method according to claim 1, wherein the similarity analysis is performed by the first client (01) calculating a statistical distribution function (D1.1-D1.3, D2.1-D2.3) for the first subsets (01.1-01.3) and by the respective second client (01) calculating a statistical distribution function (D1.1-D1.3, D2.1-D2.3) for the second subsets (01.1-01.3), and transmitting the respective distribution functions (D1.1-D1.3, D2.1-D2.3) to the server (S), and comparing the distribution functions (D1.1-D1.3, D2.1-D2.3) with each other for similarity by the server (S).

3. A method according to one of the preceding claims, wherein the method is provided for inspecting the first product (TD1), and the first and at least one second product (TD1, TD2) are each products manufactured by a production machine, and the inspection of each product (TD1, TD2) is carried out by a respective technical inspection means which is configured to capture each manufactured product (TD1, TD2) as a respective data set (DS1, DS2), preferably by a visual inspection device, particularly preferably an optical camera.

4. Method according to the preceding claim, wherein for the respective manufactured product (TD1 , TD2) the respective two first product parameters of the first product (TD1) and the respective two second product parameters of the respective second product (TD2) are determined by the respective technical inspection means, which is configured to record the respective manufactured product (TD1 , TD2) as a respective data set (DS1 , DS2), preferably by position data or orientation data or operating parameters of the inspection means.

5. Method according to claim 1 or 2, wherein the method is provided for operating the first product (TD1), and the first and the at least one second product are each a technical device (TD1, TD2), and the respective technical device (TD1, TD2) is connected to the respective client (01, C2) for operation by the respective client (01, 02).

6. Method according to the preceding claim, wherein the respective data set (DS1 , DS2) describes a permissible or impermissible operating mode of the respective technical device (TD1 , TD2).

7. Method according to one of claims 5 or 6, wherein the respective data set (DS1 , DS2) describes an operating parameter of the respective technical device (TD1 , TD2).

8. Method according to the preceding claim, wherein the operating parameter of the respective technical device (TD1 , TD2) is an operating voltage or a current consumption or a power consumption or an operating temperature or an ambient temperature or an electric field strength or a magnetic field strength or a computing load or a storage load.

9. Method according to claim 2, and one of claims 5 to 8, wherein the similarity analysis is carried out by the respective client (C1 , C2) providing the respective meta-information of the respective operating parameter of the respective product (TD1 , TD2) for the respective subset (C1.1-C1.3, C2.1-C2.3) by means of the respective acquisition means, and the respective meta-information is transmitted from the respective client (01, C2) to the server (S), and the meta-information is compared with each other for similarity by the server (S).

10. Method according to the preceding claim, wherein the similarity analysis is determined based on respective predetermined, overlapping boundary regions.

11. System for operating a technical device with a server (S), a connected first client (01) and at least one connected second client (02), and the first client (01) is connected to the technical device (TD1), wherein the system is configured to perform the method according to one of the preceding claims.

12. Computer program product comprising machine-readable instructions stored therein which, when executed by the system according to any of the preceding claims, cause the system to execute the method according to any of the preceding claims.

Images