TRAINING MACHINE LEARNING MODELS FOR DATA-DRIVEN DECISION MAKING

Description

[0001] In particular, the present disclosure relates to a method and system for training machine learning models, to a computer program product, and to a machine learning model.

[0002] US 2019 / 0118443 A1 discloses a state determination device comprising a primary determination learning model that has learned a basic feature of a state of a manufacturing device from a state variable obtained from a manufacturing operation of a product of the manufacturing device, and a secondary determination learning model that has learned a state of the manufacturing device from the state variable obtained from a predetermined operation pattern set in advance.

[0003] US 2019 / 0286075 A1 discloses a method for operating a substrate processing system comprising the steps of: receiving a plurality of training data sets, storing a plurality of machine learning models, storing a plurality of physical process models, receiving a selection of a machine learning model from the plurality of machine learning models and a selection of a physical process model from the plurality of physical process models, generating an implemented machine learning model according to the selected machine learning model, calculating a characterizing value for each training spectrum in each set of training data, thereby generating a plurality of characterizing training values, each characterizing training value being associated with one of the plurality of training spectra,Training the implemented machine learning model using the plurality of characterizing training values ​​and the plurality of training spectra to generate a trained machine learning model, and forwarding the trained machine learning model to a control system of the substrate processing system.

[0004] Currently, humans often classify certain data for decision-making purposes, e.g., to identify faulty components that require replacement, or for the regular evaluation of key performance indicators. However, such decision-making is often time-consuming, and in some use cases, data-driven decision-making within short response times is necessary. Furthermore, special analysis tools can be developed for a system to be monitored; however, this is usually very complex and requires laborious adaptation when changes are made to the monitored system. It is possible to train an artificial intelligence (AI) with data and a classification of this data, e.g., by providing several photos, each with a bicycle and the label "bicycle," as well as additional photos without a bicycle and the label "no bicycle." However, the learning process of such an AI is typically lengthy, because, for example,In many cases, several thousand data sets are used, and AI is often prone to errors, which limits the possible areas of application.

[0005] The object of the present invention is to enable improved automated, data-driven decision making.

[0006] According to one aspect, a method for training machine learning models for data-based decision making according to claim 1 is provided.

[0007] This is based on the realization that, when data forms the basis for decision-making, measured values ​​at specific time intervals allow particularly precise conclusions to be drawn about the condition of the measuring device or a component monitored by the measuring device. Multiple selections and classifications of the same data, e.g., performed by different users, can lead to different, yet potentially meaningful, results. Particularly when the measuring device is a gas turbine engine or a piston engine, for example, certain signatures in the data can indicate a deteriorating condition of the measuring device or a component monitored or monitorable by it.By training the machine learning models based on the selection of the respective selected part of the data together with the respective classification data unit, a significantly improved quality of decision-making is possible using machine learning models trained in this way, as well as a comparison of the various classifications. The trained machine learning models also allow for extensive automation. Furthermore, the amount of data required to train the machine learning models can be reduced. This can be particularly advantageous for applications in which only a limited amount of data is available for training, which would be insufficient for conventionally trained machine learning models. Furthermore, it is possible to adapt the method to a wide variety of different use cases, e.g.by only capturing data related to the respective decision-making process, as well as the classification data units and selected parts. Adapting the machine learning models to different use cases beyond training is not necessary.

[0008] For example, the classification comprises two or three decision options. For example, during classification, a choice is made between two answers (e.g., A or B) or between three answers (e.g., A, B, or C). The classification corresponds, for example, to the result of a yes / no decision or a decision between the options A (e.g., yes), B (e.g., no), and C (e.g., "unknown" or "undefined"). The classification data units each comprise, for example, the specification "yes" or "no" or another specification of positive or negative, e.g., 1 or 0. The data comprise, for example, a large number of discrete values ​​at different points in time. Optionally, some or all steps of the procedure are carried out multiple times, e.g., iteratively. The selected part of the data can be evaluated to obtain key figures (e.g., a minimum and / or a maximum and / or a standard deviation).Optionally, the machine learning model is trained using one or more of these metrics. The selected portion of the data is optionally enriched with user metadata (e.g., the time required for selection) and / or with additional information entered by the user (e.g., a classification and identification of specific features in the acquired data). The selected portion of the data forms a dataset.

[0009] The machine learning models can, for example, be used as digital assistants for data-driven decision-making using artificial intelligence.

[0010] The measuring device(s) is / are (each) designed in the form of a sensor for measuring a physical quantity, e.g. for measuring a temperature, a speed, a rotational speed or a pressure. For example, the computer(s) receive / receive several chronologically successive measured values ​​from one or more measuring devices. Each selected part of the data corresponds, for example, to one or more time windows which is / are smaller (at most exactly as large) than the total period spanned by the recorded data. Optionally, multi-dimensional data (e.g. two-dimensional) is generated from data (in particular in the form of time series data) from several measuring devices, whereby the selected part of the data is selected, for example, from the multi-dimensional data.

[0011] To train machine learning models, properties of the selected dataset are extracted in the form of one or more parameters (e.g., a maximum value, a minimum value, a median, a mean value, a variance, or the like), and training is performed based on these parameters. At least one of the parameters can be a statistical parameter.

[0012] Each of the machine learning models implements artificial intelligence and can, in particular, be used in the form of an assistant to support a user, e.g., a human specialist. Each machine learning model can be or include a classification model. In particular, each of the machine learning models can be or include an artificial neural network. Furthermore, each of the machine learning models can include a supervised learning model, a decision tree model, a random forest model, a support vector machine, a k-nearest neighbor model, and / or an XGBoost model, or others. Optionally, data with continuous variables are processed using a regression model.

[0013] The method optionally further comprises providing, by means of the one or more computers, the data on at least one interface, in particular multiple times, in particular for display to multiple users. In this case, it can be provided that the classification data units received by the one or more computers relate to the data provided multiple times on at least one interface. The interface (or each of multiple interfaces) can be used to classify the data, and the user can be enabled to select the selected part of the data. This can be done, for example, by providing additional measuring devices specifically intended for training the respective machine learning model and / or by classification by a human user.

[0014] Data from multiple measurement devices can be provided simultaneously at the interface, further improving the quality of training.

[0015] Optionally, the data displays measured values ​​from one or more machines, in particular one or more engines, e.g., piston engine(s) (especially diesel engine(s)), for example, from one or more gas turbines. Especially with gas turbines, it is often desirable to detect a deteriorating condition of a measuring device or a component monitored by a measuring device as early as possible, which is made possible by machine learning models trained in this way.

[0016] The machine learning models can be trained on a corresponding classification data unit after each provision. This enables a constantly optimized training state. Alternatively, the machine learning models can be trained with classification data units on different data and associated selected data segments, for example, as soon as a predetermined number of classification data units have been provided. This enables efficient training, for example, with limited computing capacity.

[0017] When capturing the data obtained by the one or more measuring devices, it can be provided that the data is selected from a large number of data sets (e.g., stored on a data storage device), wherein a comparison with a threshold value and / or a prediction of a further machine learning model is used for the selection. This further machine learning model can be configured to select the data that promises the best possible training of the machine learning model to be trained.

[0018] For each of the machine learning models, a prediction accuracy can be determined, e.g. using a validation dataset.

[0019] The prediction accuracies are optionally displayed on an interface.

[0020] One or more of the machine learning models can be selected (and selected) via the interface.

[0021] The machine learning model is calculated from multiple machine learning models, based on parameters from or for the individual machine learning models. The higher-level machine learning model enables particularly precise classification. The higher-level machine learning model can be calculated using, for example, a "bagging" or "ensemble learning" algorithm.

[0022] The individual machine learning models (e.g. their parameters) are weighted with different weighting factors to calculate the higher-level machine learning model.

[0023] Optionally, the weighting factors are determined by determining the prediction accuracy for each machine learning model using a validation dataset. The validation dataset includes, for example, data and classification data. This enables a significantly increased classification accuracy by the higher-level machine learning model. Furthermore, by selecting suitable training datasets for the higher-level machine learning model, so-called overfitting can be avoided.

[0024] A method is further provided for classifying data.

[0025] The method comprises providing a higher-level machine learning model generated according to the method described herein. Furthermore, the method comprises classifying, by one or more computers, (further) data acquired by one or more measuring devices using the higher-level machine learning model. This classification is possible with particularly high precision.

[0026] A result of the classification may be displayed to a user on a display, and at least one input command from the user may be captured.

[0027] The method for classifying data may further comprise the following step: generating, by the one or more computers and based on the classification of the data, a data record indicating the performance of maintenance work. The data record may be transmitted via a communication interface to automatically trigger the performance of the maintenance work.

[0028] The method of classifying data according to the method described above enables maintenance work to be carried out according to the generated data set.

[0029] According to one aspect, a computer program product according to claim 11 is provided.

[0030] The or a computer program product may comprise instructions that, when executed by one or more processors, cause the one or more processors to perform the method for training machine learning models and / or the method for classifying data according to any embodiment described herein.

[0031] According to one aspect, a (higher-level) machine learning model (e.g., an artificial neural network) is provided, generated according to the method for training machine learning models according to any embodiment described herein.

[0032] According to one aspect, a system for training machine learning models according to claim 13 is provided.

[0033] Optionally, the system comprises an interface. The interface may comprise a display section for displaying the acquired data and / or for receiving a selection of at least one selected portion of the data and / or a classification section for receiving at least one classification data unit.

[0034] Optionally, the system comprises at least one gas turbine engine or a piston engine or other machine, wherein the one or more measuring devices are arranged, for example, on that machine (e.g., on the gas turbine engine). It can be provided that the machine (the gas turbine engine) is movable relative to the one or more computers. For example, the one or more computers are (permanently) stationed on the ground.

[0035] Embodiments will now be described by way of example with reference to the figures, in which: Figure 1 shows an aircraft in the form of an airplane with multiple gas turbine engines; Figure 2 shows a side sectional view of a gas turbine engine; Figure 3 shows a system for training a machine learning model; Figure 4 shows details of the system according to Figure 3 ; Figures 5 to 8 show various examples of measurement data; Figures 9 to 11 show various views of an interface of the system according to Figure 3 ; Figure 12 a method for training a machine learning model; Figure 13 an interface of the system according to Figure 3 ; and Figure 14 Details of the system according to Figure 3 .

[0036] Figure 1 shows an aircraft 8 in the form of an airplane. The aircraft 8 includes several gas turbine engines 10.

[0037] Figure 2represents one of the gas turbine engines 10 of the aircraft 8 with a main axis of rotation 9. The gas turbine engine 10 comprises an air inlet 12 and a fan 23 which generates two air flows: a core air flow A and a bypass air flow B. The gas turbine engine 10 comprises a core 11 which receives the core air flow A. The core engine 11 comprises, in axial flow order, a low-pressure compressor 14, a high-pressure compressor 15, a combustor 16, a high-pressure turbine 17, a low-pressure turbine 19, and a core exhaust nozzle 20. An engine nacelle 21 surrounds the gas turbine engine 10 and defines a bypass duct 22 and a bypass exhaust nozzle 18. The bypass airflow B flows through the bypass duct 22. The fan 23 is attached to and driven by the low-pressure turbine 19 via a shaft 26 and an epicyclic planetary gear 30.

[0038] In operation, the core airstream A is accelerated and compressed by the low-pressure compressor 14 and passed into the high-pressure compressor 15 where further compression occurs. The compressed air exhausted from the high-pressure compressor 15 is passed into the combustor 16, where it is mixed with fuel and the mixture is combusted. The resulting hot combustion products then propagate through the high-pressure and low-pressure turbines 17, 19, thereby driving them before being expelled through the nozzle 20 to provide some thrust. The high-pressure turbine 17 drives the high-pressure compressor 15 through a suitable connecting shaft 27. The fan 23 generally provides the majority of the thrust. The epicyclic planetary gear 30 is a reduction gear.

[0039] It should be noted that the terms "low-pressure turbine" and "low-pressure compressor," as used herein, may be construed to mean the lowest-pressure turbine stage and the lowest-pressure compressor stage, respectively (i.e., not including fan 23) and / or the turbine and compressor stages connected by the lowest-speed connecting shaft 26 in the engine (i.e., not including the transmission output shaft that drives fan 23). In some writings, the "low-pressure turbine" and "low-pressure compressor" referred to herein may alternatively be known as the "intermediate-pressure turbine" and "intermediate-pressure compressor." When using such alternative nomenclature, fan 23 may be referred to as a first compression stage or lowest-pressure compression stage.

[0040] Other gas turbine engines to which the present disclosure may be applied may have alternative configurations. For example, such engines may have an alternative number of compressors and / or turbines and / or an alternative number of connecting shafts. As another example, the Figure 2The gas turbine engine shown has a split-flow nozzle 20, 22, meaning that the flow through the bypass duct 22 has its own nozzle separate from and radially outward from the engine core nozzle 20. However, this is not limiting, and any aspect of the present disclosure may also apply to engines in which the flow through the bypass duct 22 and the flow through the core 11 are mixed or combined before (or upstream of) a single nozzle, which may be referred to as a mixed-flow nozzle. One or both nozzles (whether mixed or split-flow) may have a fixed or variable area. Although the example described relates to a turbofan engine, the disclosure may be applied to any type of gas turbine engine, such as an open-rotor (where the fan stage is not surrounded by an engine nacelle) or a turboprop engine.

[0041] The geometry of the gas turbine engine 10 and components thereof is or are defined by a conventional axis system having an axial direction (aligned with the rotational axis 9), a radial direction (in the direction from bottom to top in Figure 2 ) and a circumferential direction (perpendicular to the view in Figure 2 ). The axial, radial, and circumferential directions are perpendicular to each other.

[0042] A plurality of measuring devices are arranged on the gas turbine engine 10, of which a plurality of measuring devices 60-62 arranged at different locations on the gas turbine engine 10 in the form of sensors, specifically temperature sensors for measuring temperatures, are shown here as an example.

[0043] Figure 3shows a system 50 with a machine learning model 51, specifically a plurality of machine learning models 51, which in this case represent a plurality of instances of the same machine learning model, and for training the machine learning models 51. The system 50 comprises one (optionally a plurality of) computers 52 with a memory 53. At least one machine learning model 51 is stored on the memory 53 (or a separate memory), in particular the plurality of machine learning models 51 are stored or storable. The computer 52 is communicatively coupled to the measuring devices 60-62 in order to acquire data, specifically measurement data therefrom. Alternatively or additionally, the computer 52 is communicatively coupled to at least one input means in order to receive maintenance data and / or status data therefrom. This maintenance and / or status data can, for example, be entered by a user at the input means.For the sake of simplicity, we will always refer to measurement data below, but alternatively or additionally, this may also refer to maintenance and / or status data, or data in general.

[0044] The machine learning models 51 are designed for machine learning and, in the present example, include a random forest and / or an artificial neural network.

[0045] Instructions 54 are stored in memory 53 which, when executed by a processor 55 (or multiple processors) of computer 52, cause the processor 55 (or multiple processors) to perform the following steps: Acquiring measurement data obtained by means of one or more measuring devices 60-62 of the system 50 (e.g., via an engine control unit); receiving a plurality of classification data units relating to the measurement data; receiving, in connection with each of the classification data units, a portion of the measurement data selected, in particular, by a human operator; and training each machine learning model 51 (in particular each instance) based on at least one classification data unit and the respectively associated selected portion of the measurement data.

[0046] The system 50 further comprises additional machine learning models 56 and 57, which will be explained in more detail below. Furthermore, the system 50 comprises interfaces 81, 84, which in the present example are designed as graphical user interfaces (GUIs) and can be displayed on a display 80, e.g., in the form of a screen. The interfaces 81, 84 will also be explained in more detail below.

[0047] Based on the trained machine learning models, 51 further measurement data can then be classified to make data-driven decisions, such as triggering maintenance work. Different training methods can lead to different results.

[0048] The instructions 54 are part of a computer program product which causes the processor 55 to Figure 12 to perform the method shown. Memory 53, for example, is a non-volatile memory.

[0049] The processor 55 includes, for example, a CPU, a GPU and / or a tensor processor.

[0050] The computer 52 is stationed on the ground and the gas turbine engine 10 is movable relative thereto.

[0051] Figure 4 shows further details of the System 50.

[0052] Measurement data from the measuring devices 60-62 are stored in a database 100 in the form of a plurality of time series and as raw data. The time series originate, for example, from multiple flights of the gas turbine engine 10, from the multiple gas turbine engines 10 of the aircraft 8, and / or from gas turbine engines 10 of multiple aircraft 8 (or, in general, from multiple aircraft). Transmission from the measuring devices 60-62 to the database 100 occurs, for example, via a data cable or wirelessly, for example, via GSM or another mobile communications standard.

[0053] Optionally, the data stored in database 100 is processed and stored in another database 101, which may also involve a transient data flow. For example, uninteresting data can be omitted to simplify further processing.

[0054] Optionally, the measurement data is further processed and stored in another database 102 to analyze the measurement data for suitable time series. This analysis takes place in block 117. For example, threshold monitoring can be used, whereby measurement data within a time window around a threshold violation is selected as a candidate.

[0055] In block 117, the machine learning model 56 can be applied, which selects suitable candidates, each with a time series from a measuring device 60-62 or with several time series (in particular spanning the same period) from several of the measuring devices 60-62 and is therefore referred to below as the selection model 56. The selection model 56 is, for example, an unsupervised machine learning model, e.g., dbscan, k means clustering, or PCA, or a script that extracts data based on defined rules. The selection model 56 stores the selected candidates or pointers thereto in a database 110. The machine learning model 56 can, for example, be implemented by a computer program that, for example, performs appropriate comparisons with the measured values. Alternatively or additionally, the computer program implements a physical model with which the measured values ​​are compared.

[0056] An import script retrieves these candidates from the database 102 (or the database 101) in block 118 and provides them (optionally via another database 106) to a block 111.

[0057] In block 111, a classification data unit and a selected portion of the measurement data of the respective candidate are recorded for all or some of the candidates. The classification data units indicate a classification of the candidate into one of several predefined classes. The classification data units and / or the selected portions of the measurement data are provided, for example, by additional sensors that have been additionally attached to the gas turbine engine 10 to generate the candidates, or by a selection by one or more users. This selection is made, for example, via interface 81.

[0058] The classification data units and selected parts of the candidates are stored in a database 108 and provided to a block 112. In block 112, one instance of the machine learning model 51 is trained per user based on the classification data units and selected parts of the candidates provided by the user. For this purpose, properties of the selected part of the measurement data are extracted in the form of parameters. Optionally, the extracted parameters and / or values ​​calculated therefrom, e.g., ratios of two parameters, are the input parameters for training. Examples of such parameters are described below in connection with Figure 7 be explained.

[0059] Training can be performed iteratively, e.g., for each candidate. The trained instance is stored in a database 107. The trained instance is then provided to block 111, so that a (continuously improving) prediction for the classification of the next candidate can be provided during training.

[0060] Multiple instances of the machine learning model 51 are generated and trained, whereby the classification and selection of the selected parts in block 111 can be performed in different ways, e.g., by different users. Instead of or in addition to trained instances of the machine learning model 51, multiple sets of input parameters can also be stored.

[0061] The components primarily responsible for training the multiple instances of the machine learning model 51 are in Figure 4highlighted by a dashed box and can be implemented as a separately stored software module.

[0062] The data stored in the database 108 are provided to a block 113, which can also access the database 107. In block 113, the (optional) higher-level machine learning model 57 is generated. The higher-level machine learning model 57 optionally corresponds to the machine learning model 51, but is trained, for example, with the (optionally weighted and / or selected) input parameters from the multiple instances of the machine learning model 51. For example, in block 113, an interface 84 (see Figure 13) in the form of a graphical user interface that displays details of the generation of the higher-level machine learning model 57 to a user and / or provides the user with options for influencing the generation, for example, to select and / or change those instances of the machine learning model 51 that are included in the generation of the higher-level machine learning model 57.

[0063] When generating the higher-level machine learning model 57, the available candidates can be divided into a training dataset and a validation dataset. The training dataset is used, for example, to generate the higher-level machine learning model 57 (e.g., by using this dataset to train the instances of the machine learning model 51, which are then used to calculate the higher-level machine learning model 57).

[0064] As already mentioned, the individual instances of the machine learning model 51 (and / or their input parameters) are optionally weighted with different weighting factors to calculate the higher-level machine learning model 57. The weighting factors are determined, for example, by determining a prediction accuracy and / or error for each of the instances of the machine learning model 51 based on the validation data set.

[0065] Optionally, a number of incorrect classifications, a number of classifications, a duration of the classifications, a time interval between individual classifications and / or a number of possible changes to the classifications are used for weighting.

[0066] The validation dataset is used alternatively or additionally to calculate a precision of the higher-level machine learning model 57.

[0067] According to one variant, a data set of n (e.g., 20) data sets is retained in a loop, the instances of the machine learning model 51 are trained on n-1 data sets, the higher-level machine learning model 57 is calculated, and the result for the retained data set is evaluated. This can be run n times, and the accuracy of the higher-level machine learning model 57 can be calculated from the total performance of all n runs.

[0068] The higher-level machine learning model 57 and / or its input parameters is / are stored in a database 109 (which is stored, for example, in the memory 53).

[0069] In the optional block 114, the generation of the higher-level machine learning model 57 is displayed on a user interface.

[0070] Database 103 contains the data from database 102 to which optional selection or correction scripts have been applied. Alternatively, only database 102 is provided instead of databases 102 and 103.

[0071] In block 115, the higher-level machine learning model 57 is applied to the measurement data from database 103 (or 102) to classify the measurement data. The results of the classification from block 115 are stored in a database 104, optionally also data from database 103 (or 102).

[0072] The analysis model 56 can exchange data with the higher-level machine learning model 57 via the database 103, e.g. to exclude certain time series data from classification.

[0073] In block 116, data-driven decisions are made, e.g., triggering the execution of maintenance work. For example, based on the classification, it was determined that one of the measuring devices 60-62 or a component of the gas turbine engine 10 (or generally a device monitored by the system 50) is defective and needs to be replaced. Optionally, a message indicating the decision is generated and transmitted, e.g., by email.

[0074] Optionally, the data underlying the decisions are stored in a database 105. Databases 100 to 104 (which may also be logical steps through a data flow) are optionally part of an Engine Equipment Health Management (EHM) system of the gas turbine engine 10. Database 105 may, for example, be located on the ground. It should also be noted that databases 100, 101, 102, 103, 104, and / or 105 (optionally all databases) may have separate physical storage locations or, alternatively, may be databases of a logical architecture, where, for example, several or all of the databases have the same physical storage location.

[0075] Figure 5shows exemplary measurement data 70 in the form of time series data. A large number of measured values ​​are plotted against time. Specifically, the measured data indicate a (first) temperature difference, which can be determined using two spaced-apart measuring devices of a machine, in this case a diesel engine (alternatively, for example, analogously using one or two of the measuring devices 60-62) in the form of temperature sensors, and which has been determined here.

[0076] Figure 6 shows further measurement data 70 in the form of time series data. Here, too, a large number of measured values ​​are plotted against time, over the same period as the measurement data of the Figure 5 . Specifically, the measurement data of the Figure 6a (second) temperature difference, which can be determined by means of two measuring devices of the machine, in this case the diesel engine (alternatively, for example, analogously by one or two of the measuring devices 60-62) in the form of temperature sensors and has been determined here, namely a different pair of measuring devices 60-62 than in Figure 5 .

[0077] Furthermore, Figure 6 a selected portion 71 of the measurement data 70 is illustrated. The selected portion 71 comprises conspicuous areas of the measurement data. When measurement data is conspicuous depends on the specific application. In the present example, temperature difference values ​​below a certain limit and strong fluctuations in the values ​​are conspicuous. The selected portion 71 generally comprises one or more temporal subsections of the measurement data 70 (along the x-axis). Optionally, the selected portion 71 also includes a restriction along the y-axis.

[0078] Figure 7 illustrates exemplary parameters that can be calculated from an exemplary selected part 71 of measurement data 70.

[0079] The parameters can be, for example, a maximum value, a minimum value, a median, a mean value, a variance, the sum of the squared individual values, the length of the selected part in the time direction, an autocorrelation or a parameter derived from it, the number of values ​​above or below the mean value, the longest time interval above or below the mean value, the sum of the slope sign changes, a slope, a standard deviation and / or a number of peaks. Some of these parameters are in Figure 7highlighted graphically. One or more, e.g., all, of these parameters can be used as input parameters for training the corresponding machine learning model 51. Furthermore, ratios of the aforementioned parameters can be formed and used as input parameters for training, e.g., mean / variance, length / sum of squared individual values, or other ratios.

[0080] Figure 8 illustrates that time series data from multiple measuring devices can optionally be plotted multidimensionally (here, two-dimensionally), so that the selected portion 71 of the measured data 70 can be selected multidimensionally. For example, several correlated measured data sets can exhibit particularly clear anomalies, which can then be selected particularly easily and precisely. For example, one point in the multidimensional representation corresponds to several different measured values ​​at the same time.

[0081] Optionally (especially in the selected part) clusters of data points are determined in the multidimensional representation and, for example, their distances to each other and / or sizes, e.g. radii and / or the number of data points contained therein, are determined.

[0082] Figure 9 shows an interface 81 designed as a GUI. The interface 81 can be displayed, for example, in a browser. The interface 81 comprises several display sections 82 (in the example shown, five display sections 82, although an interface 81 with only one or generally with more than one display section 82 is also conceivable) and a classification section 83. In the example shown, the classification section 83 is shown in a configuration that provides two alternative input options (in this case, "yes / no"). Optionally, several classes or parameters (e.g., 0 to 100) can also be selectable, e.g., displayed and used in the form of a controller.

[0083] Each display section 82 shows measured data acquired by one or more measuring devices against the time axis (in the same time window). In the example shown, a selection option is provided next to each display section 82, by means of which the respective X-axis parameter and the respective Y-axis parameter of the display section 82 can be selected. According to Figure 9 For the first display section 82 from the top, the parameters are as follows Figure 5 and for the second display section 82 from the top according to Figure 6 The other display sections 82 show a measured speed, an operating time determined by a time measurement, and a nitrogen oxide concentration, each plotted against time. Alternatively, a measured value can also be plotted against a measured value other than time, e.g., according to Figure 8 .

[0084] A user has already selected a selected part 71 of the measurement data 70 because it appeared to be conspicuous with regard to a possible damage to the machine, for example damage to a specific component of the machine (e.g. valve damage, e.g. an exhaust valve of an internal combustion engine).

[0085] After selecting the selected part 71, the classification section 83 is activated. Once the classification section 83 is activated, the user can enter a classification. In the example shown, the user would enter that damage is likely, which can be seen in the selected part 71 of the measurement data 70.

[0086] Figure 10shows the interface 81 with another candidate with additional measurement data (e.g., from another machine, in particular of the same type, e.g., a different engine or motor, e.g., a gas turbine engine). No selected part 71 has yet been selected. Because classifications of previous candidates have already been performed, the corresponding machine learning model 51 could already be trained and therefore already provides an estimate according to which, in the case shown, it is more likely that no damage is detectable in the measurement data now displayed.

[0087] Optionally, the probabilities for positive, false positive, negative and false negative are calculated and given in a matrix, e.g. in the form of a so-called "confusion matrix".

[0088] Figure 11shows a state in which a sufficient set of candidates has been classified by the user (or by other means). The first candidate is displayed again as an example, and the machine learning model 51 is now so well trained that it indicates damage with a significantly higher probability than no damage (e.g., valve damage).

[0089] A graph next to classification section 83 shows an overall increasing trend versus the number of classified candidates, indicating the prediction accuracy, and an overall decreasing trend, indicating the prediction error. After approximately 25 candidates, the accuracy is already over 80%, with the error being well below 0.1.

[0090] As soon as the user selects a selected part 71, the machine learning model 51 calculates the corresponding probabilities with respect to this selected part 71.

[0091] The classification of a sufficient set of candidates is possible within a few minutes and allows the training of a machine learning model 51 with a surprisingly precise prediction for many use cases. The classification is performed multiple times by different users, so that several trained machine learning models 51 are provided. These can provide predictions of varying quality based on different classifications by the users. For example, the best machine learning model 51 can be selected. The precision can be significantly increased again by calculating the higher-level machine learning model 57. The quality of the prediction models can be determined either based on ground truth or, optionally, if ground truth is not available, by an expert, and / or optionally purely data-driven by comparison with the majority of prediction models.

[0092] Figure 12shows a method for classifying measurement data, comprising the following steps: Step S1: Providing a trained higher-level machine learning model 57.

[0093] For this purpose, for example, a method for training the machine learning models 51 is carried out, comprising the steps S10 to S14: Step S10: Acquisition, by the one or more computers 52, of measurement data 70 obtained by means of one or more measuring devices 60-62, wherein the measurement data 70 is acquired in particular in the form of time series data and in particular indicates measured values ​​from one or more gas turbines 10. Optionally, when acquiring the measurement data 70 obtained by means of the one or more measuring devices 60-62, the measurement data 70 is selected from a plurality of measurement data, wherein a prediction of the further machine learning model 56 is used for the selection. Step S11 (optional): Provision, by means of the one or more computers 52, of the measurement data 70 at the interface 81, wherein at the interface 81, measurement data 70 from several measuring devices 60-62 is provided, in particular simultaneously.Step S12: Receiving, by the one or more computers 52, classification data units relating to the measurement data 70, wherein the classification data units received by the one or more computers 52 optionally relate to the measurement data 70 provided at the interface 81. Step S13: Receiving, by the one or more computers 52 and for each of the classification data units, a selected part 71 of the measurement data 70. Step S14: Training, by means of the one or more computers 52, a plurality of machine learning models 51 based on the classification data units and the selected parts 71 of the measurement data 70, wherein the machine learning models 51 comprise, for example, an artificial neural network. The machine learning models 51 can, for example,be trained after each provision of classification data or be trained with classification data units with respect to different measurement data 70 and associated selected parts 71 of measurement data 70 as soon as a predetermined number of classification data units has been provided. .

[0094] Steps S10 to S14 are optionally performed multiple times for different (candidates of) measurement data 70, whereby the accuracy of the prediction of the trained machine learning models 51 can be further improved.

[0095] In this case, several machine learning models 51, e.g., several instances of the same type of machine learning model 51, are trained (e.g., by having the above steps performed by several users each), and a higher-level machine learning model 57 is calculated from the several machine learning models 51 (instances), wherein the individual instances of the machine learning model 51 are weighted, e.g., with different weighting factors to calculate the higher-level machine learning model 57. The weighting factors are determined, in particular, by determining a prediction accuracy for each of the machine learning models 51 using a validation data set.

[0096] Step S2 comprises classifying, by the one or more computers 52, measurement data 70 acquired by the one or more measuring devices 60-62 using the higher-level machine learning model 57.

[0097] The optional step S3 comprises generating, by the one or more computers 52 and based on the classification of the measurement data 70, a data record indicating performance of maintenance work.

[0098] Figure 13 shows the previously mentioned interface 84. Interface 84 displays the performance of several (here, for example, five) machine learning models 51 trained differently, namely by different users. For this purpose, interface 84 displays a matrix 86 in the form of a "confusion matrix" as well as a diagram 87 that shows the progression of the prediction accuracy and the error in the manner described above. Furthermore, interface 84 displays additional information, such as the proportion of data already classified and a weighting factor calculated, for example, based on the prediction accuracy. The weighting factors are each a real number between 0 and 1.

[0099] The interface 84 comprises several selection sections 85, each in the form of a checkbox. Using these selection sections, a user can specify which of the machine learning models 51 (more precisely, the parameters of which of the machine learning models 51) should be included in the calculation of the higher-level machine learning model 57. The higher-level machine learning model 57 then serves as the "gold standard" for classifying further data.

[0100] Fig. 14shows details of a generation of the higher-level machine learning model 57. For each user, a set of features is generated based on the at least one selected part 71 of data, block 119. Based on this, a classification performance with respect to this data is determined, block 120. In parallel, reference data is extracted, block 122, optionally stored in a database 125, and a set of features of the reference data is generated, block 123. A classification performance is also determined with respect to the reference data, block 124. Based on the classification performance, a weighting factor is determined, block 121. In this way, it can be determined how precisely a user has selected features to detect a signature, e.g. an anomaly in the data.

[0101] For each potential signature (e.g., anomaly) in the data, the probability of a correct positive detection is then determined using each machine learning model 51. These probabilities are weighted with the weighting factors to determine the probability of the higher-level machine learning model 57. If this probability exceeds a certain value, e.g., 0.5, the signature is classified as a positive result, e.g., as a detected anomaly.

[0102] In particular, it should be noted that instead of the gas turbine engine 10, another machine, in particular generally an engine and / or power unit, e.g. a piston engine, can also be used. List of reference symbols

[0103] 8Aircraft 9Main axis of rotation 10Gas turbine engine 11Core engine 12Air inlet 14Low-pressure compressor 15High-pressure compressor 16Combustion device 17High-pressure turbine 18Bypass nozzle 19Low-pressure turbine 20Core nozzle 21Engine nacelle 22Bypass duct 23Fan 24Stationary support structure 26Shaft 27Connecting shaft 30Gearbox 50System for training a machine learning model 51Machine learning model 52Computer 53Memory 54Instructions 55Processor 56Machine learning model (selection model) 57High-level machine learning model 60-62Measuring device 70Data (measurement data) 71Selected part 80Display 81Interface 82Display section 83Classification section 84Interface 85Selection section 86Matrix 87Diagram 100-110, 125Database 111-124Block ACore airflow BBypass airflow

Claims

1. Method for training machine learning models (51,57), comprising: - recording (S10), by means of one or more computers (52), data (70) obtained by means of one or more measuring devices (60-62), each in the form of a sensor for measuring a physical quantity, in the form of time series data, wherein the data (70) comprise information on maintenance processes or conditions; - receiving (S12), by means of the one or more computers (52), a plurality of classification data units relating to the data (70); characterized by - receiving (S13), by the one or more computers (52) and for each classification data unit, a selected part (71) of the data (70); and - training (S14), by means of the one or more computers (52), a plurality of machine learning models (51), each based on at least one of the classification data units and the corresponding at least one selected part (71) of the data (70), wherein - the plurality of machine learning models (51) represent a plurality of instances of the same machine learning model, wherein - the selected parts (71) of the data (70) are provided by a selection by a plurality of users via one or more interfaces (81), wherein one instance of the machine learning model (51) is trained per user, wherein - a higher-level machine learning model (57) is calculated from parameters of the plurality of machine learning models (51), and - the individual machine learning models (51) are weighted with different weighting factors to calculate the higher-level machine learning model (57), wherein - the higher-level machine learning model (57) is trained to classify, by means of the one or more computers (52), other data recorded by means of the one or more measuring devices (60-62) and to generate a data set, by means of the one or more computers and based on the classification of the data, which indicates that maintenance work has been carried out.

2. Method according to claim 1, wherein properties of each selected part (71) of the data (70) are extracted in the form of parameters and the training (S14) of the plurality of machine learning models (51) is carried out based on these parameters.

3. Method according to claim 1 or 2, further comprising: - a plurality of provision (S11), by means of the one or more computers (52), of the data (70) on at least one interface (81) for display for multiple users.

4. Method according to any of the preceding claims, wherein the data (70) indicate measured values from one or more machines, in particular one or more gas turbines (10).

5. Method according to any of the preceding claims, wherein a prediction accuracy is determined for each of the machine learning models (51).

6. Method according to claim 5, wherein the prediction accuracies are displayed on an interface (84).

7. Method according to claim 6, wherein one or more of the machine learning models (51) are selectable and selected via the interface (84).

8. Method according to claim 5, wherein the weighting factors are determined based on the prediction accuracies.

9. Method for classifying data (70), comprising: - providing (S1) a higher-level machine learning model (57) calculated according to the method according to any of the preceding claims; - classifying (S2), by means of one or more computers (52), data (70) recorded by means of one or more measuring devices (60-62) and / or at least input means using the higher-level machine learning model (51).

10. Method according to claim 9, further comprising: - generating (S3), by the one or more computers (52) and based on the classification of the data (70) and / or at least one input command, a data set indicating that maintenance work has been carried out.

11. Computer program product comprising instructions (54) that, when executed by one or more processors (55), cause the one or more processors (55) to perform the following steps: - recording data (70) obtained by means of one or more measuring devices (60-62), each in the form of a sensor for measuring a physical quantity, in the form of time series data, wherein the data (70) comprise information on maintenance processes or conditions; - receiving a plurality of classification data units relating to the data (70); characterized by - receiving a selected part (71) of the data (70) for each of the classification data units; and - training a plurality of machine learning models (51), each based on at least one of the classification data units and the at least one corresponding selected part (71) of the data (70), wherein the plurality of machine learning models (51) represent a plurality of instances of the same machine learning model, wherein - the selected parts (71) of the data (70) are provided by a selection by one or more users via one or more interfaces (81), wherein one instance of the machine learning model (51) is trained per user, wherein - a higher-level machine learning model (57) is calculated from parameters of the plurality of machine learning models (51), and - the individual machine learning models (51) are weighted with different weighting factors to calculate the higher-level machine learning model (57), wherein - the higher-level machine learning model (57) is trained for classification, by the one or more computers (52), of further data recorded by means of the one or more measuring devices (60-62), and for generating a data set, by the one or more computers and based on the classification of the data, which indicates that maintenance work has been carried out.

12. Machine learning model (57) provided with the method according to one of claims 1 to 8.

13. System (50) for training machine learning models (51), comprising one or more processors (55) and a memory (53) on which instructions (54) are stored, which, when executed by the one or more processors (55), cause the one or more processors (55) to perform the following steps: - recording data (70) obtained by means of one or more measuring devices (60-62), each in the form of a sensor for measuring a physical quantity, in the form of time series data, wherein the data (70) comprise information on maintenance processes or conditions; - receiving a plurality of classification data units relating to the data (70); characterized by - receiving a selected part (71) of the data (70) for each of the classification data units; and - training a plurality of machine learning models (51), each based on at least one of the classification data units and the at least one corresponding selected part (71) of the data (70), wherein the plurality of machine learning models (51) represent a plurality of instances of the same machine learning model, wherein - the selected parts (71) of the data (70) are provided by a selection by one or more users via one or more interfaces (81), wherein one instance of the machine learning model (51) is trained per user, wherein - a higher-level machine learning model (57) is calculated from parameters of the plurality of machine learning models (51), and - the individual machine learning models (51) are weighted with different weighting factors to calculate the higher-level machine learning model (57), wherein - the higher-level machine learning model (57) is trained for classification, by the one or more computers (52), of further data recorded by means of the one or more measuring devices (60-62), and for generating a data set, by the one or more computers and based on the classification of the data, which indicates that maintenance work has been carried out.