Computer-implemented method for training a neural network and method for operating the trained network , computer program product, computer system, and use of a compute r to obtain an industrial machine parameter indicator

Abstract

An industrial machine may lack sensors for specific parameters, so a computer uses a neural network to virtualize the missing sensors. The computer trains the neural network to provide parameter indices for further process parameters of the industrial machine through the following steps: receiving a measurement time series with historical measurement data from a reference machine; obtaining a transformation rule by processing the time series into a feature series that is invariant to the domain differences of the reference machine; transforming the time series using the transformation rule; receiving univariate time series of further process parameters; and training the neural network with the feature series at the input and the univariate time series at the output.

Description

Technical Field

[0001] Regarding a computer system, method, and computer program for an industrial production process, particularly relating to the generation of a virtual sensor used in an industrial machine to simulate the sensor and to simulate the measurement data of process parameters of the industrial machine performing these processes. Prior Technology

[0002] Operating industrial machinery and simultaneously measuring its process parameters are intertwined. Measurements include those directly related to the ongoing process within the machine, such as temperature, pressure, sound, irradiance (as readings from an infrared camera), light reflection from surfaces, gas concentrations, and other physical properties or phenomena. However, measurements can also involve process parameters indirectly related to the process, such as measuring the physical properties of materials entering or leaving the machine (generated by the process).

[0003] In this sense, industrial machines are machines under observation.

[0004] Furthermore, process parameters can be differentiated between locations inside and outside the machine. For example, temperature distribution can be measured at different parts of the machine.

[0005] In a more general sense, industrial machinery is the machine that performs industrial processes. For example, a machine can be a chemical reactor, a metallurgical furnace, a vessel, or an engine. More specifically, considering the operating principle, a furnace can be a blast furnace. In a very simple sense, a blast furnace receives ore and coke, along with hot air (via tuyeres), and supplies molten metal.

[0006] Measurement is the primary purpose of sensors associated with machines. For example, a machine may be equipped with a temperature sensor. Such a sensor can be implemented as a thermometer, a PT100 resistance temperature detector, or other types of temperature sensor.

[0007] It is known that measurement results can be provided by at least two measurement modes: • Through sensors associated with the machine, and • Through the interaction between human operators and machines.

[0008] In the blast furnace example, sensors are distributed in multiple locations; and additionally, the operator typically collects samples of the molten material. In short, the operator opens part of the blast furnace hearth to extract the molten material (via an opening machine, this step is called tapping). The material then flows into a channel (called casting), and the operator can inspect the sample for chemical composition, temperature, appearance, etc.

[0009] Most measurement results are converted into measurement data. Regarding data modalities, measurement data can have various formats with varying degrees of complexity; for example, such as: • Scalars of temperature or pressure • Vectors relating to the motion modes of machine parts • Matrices of images from surfaces, etc.

[0010] The measurement data will be processed by one or more computers. The computers can process the measurement data to obtain indicators about the machine's operating status. Based on the measurement data and indicators, the operator can modify the operating parameters. For example, the sampling mentioned above may indicate that the temperature is insufficient. Along with other data, the computer will instruct the operator to add more coke to the furnace (coke quantity is an operating parameter).

[0011] In many other scenarios, computers can work directly with machines, acting as process controllers (e.g., control loops for temperature parameters).

[0012] However, measurement data is not always available for various reasons or limitations. Some of these warrant further attention:

[0013] A particular sensor may be damaged, or the data connection to the computer may fail.

[0014] The distribution of sensors may differ between machines at different stages of their development, even if these machines are essentially the same type. Continuing with the example of furnaces, they may be built individually or in very small series. Furnace fleets are therefore heterogeneous. Thus, furnaces with the same operating principles (e.g., thermal conditioning via coke loading and / or pulverized coal injection) and the same materials (e.g., iron ore, coke) will still be different (e.g., in volume). Such differences between machines may also reveal a lack of measurement data for each machine.

[0015] The role of human-machine interaction. Other technical issues involve potential dangers or hazards to the operator. Opening the furnace for the sampling mentioned above should be limited to a few occasions. Furthermore, some measurements may be necessary during critical furnace operations, such as measuring the temperature of the molten material during casting. Human involvement in measurements should be minimized so that the operator can focus entirely on furnace operation.

[0016] Some of these limitations come in combinations: old-fashioned analog thermometers require an operator to read the data, while digital thermometers do not.

[0017] If the relationships between measured data are known, a computer can calculate indicators for the machine's operating state. For example, a computer in a car can advise the driver on gear shifting. However, in many cases, the relationships between measured data are too complex to be modeled using mathematical equations.

[0018] Using neural networks is one option. Networks receive measurement data and provide metrics. In short, a network consists of so-called neurons arranged in multiple layers. The connections between neurons are controlled by so-called weights. The setting of these weights in a network is called configuration.

[0019] However, the network needs to be trained (to set weights and other network parameters). Again, in a simplified way, training involves feeding historical measurement data into the network.

[0020] However, historical data may not be available for a particular machine, especially when there is no suitable counterpart or peer. This lack of historical data is particularly pronounced for machines manufactured at different stages of development (such as furnaces).

[0021] Further background information can be found in the following papers:

[0022] LI XIANG ET AL: "Diagnosing Rotating Machines With Weakly Supervised Data Using Deep Transfer Learning", IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, IEEE SERVICE CENTER, NEW YORK, NY, US, vol. 16, no. 3, 8 July 2019 (2019-07-08)), pages 1688-1697. This paper deals with data-driven machine diagnostics. The challenge lies in the insufficient availability of training data for the machine in the target domain. However, training data from multiple source domains can be used as an alternative. This paper explains a method of adapting training data through transfer learning.

[0023] LI XIANG ET AL: "Domain generalization in rotating machinery fault diagnostics using deep neural networks", NEUROCOMPUTING, ELSEVIER, AMSTERDAM, NL, vol. 403, 8 May 2020 (2020-05-08), pages 409-420. This paper also deals with data-driven machine diagnostics. The task is to extract domain-general or domain-invariant features, and this paper explains the method using domain adversarial networks.

[0024] LIAO YIXIAO ET AL: "Deep Semisupervised Domain Generalization Network for Rotary Machinery Fault Diagnosis Under Variable Speed", IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, IEEE, USA, vol. 69, no. 10, 6 May 2020 (2020-05-06), pages 8064-8075. This paper deals with rotating machinery and the use of neural networks to detect faults. However, labeled training data is available for a single rotational speed, but not for different speeds. Nevertheless, this challenge is overcome by using a Wasserstein generative adversarial network. Summary of the Invention

[0025] Industrial machines operate with multiple process parameters, but they may lack appropriate sensors for specific parameters, or measurement data for those parameters may be unavailable for other reasons. Neural networks provide parameter metrics corresponding to the process parameters within the machine. This network has been trained based on historical data from a reference machine. At least one of the reference machines (the source reference) has a sensor for that process parameter.

[0026] The reference machine is different, and in terms of machine learning, it belongs to a different domain. Therefore, training involves transfer learning, or more precisely, unsupervised domain adaptation.

[0027] By employing a computer-implemented method, a neural network is trained to enable it to later process multivariate measurement time series data. The measurement data represents instances of specific process parameters of an industrial machine. The trained network is then trained to provide parameter indices for further process parameters of the industrial machine.

[0028] In the first receiving step, the computer receives a first multivariate measurement time series having historical measurement data from a first reference machine, and receives a second multivariate measurement time series having historical measurement data from a physically different second reference machine.

[0029] In the acquisition step, the computer processes the first and second multivariate measurement time series to obtain a set of transformation rules. These rules enable the converter module to convert the first and second multivariate measurement time series into first and second multivariate feature time series, respectively. The domain differences between the multivariate feature time series and the first and second reference machines remain unchanged.

[0030] In the transformation step, the computer transforms the first multivariate measurement time series into the first multivariate feature time series by applying transformation rules to the converter module.

[0031] In the second receiving step, the computer receives a univariate measurement time series with further process parameters from the first reference machine.

[0032] In the training step, the computer trains the neural network with a first multivariate feature time series at the network input and a univariate measurement time series at the network output.

[0033] In some embodiments, the computer may repeatedly obtain a set of transformation rules by the following steps: using the initial set of transformation rules, transforming a first multivariate measurement time series into a first multivariate feature time series, and transforming a second multivariate measurement time series into a second multivariate feature time series; using the first and second feature time series as sources for the first and second multivariate measurement time series by the discriminator module; modifying the initial set of transformation rules until the discriminator module can no longer distinguish the data source.

[0034] In some embodiments, the computer may use a discriminator module that determines that it is no longer able to discern whether the data source is being manipulated until the adversarial loss reaches its maximum value.

[0035] In some embodiments, the computer may repeat the steps of acquiring and training the conversion rules; wherein the converter module further receives network loss provided by the trained neural network, thereby repeating the stopping process to minimize network loss.

[0036] In some embodiments, the combination of the converter module and the discriminator module serves as a generative adversarial network operation.

[0037] In some embodiments, the computer performs method steps for receiving multivariate measurement time series, obtaining a set of transformation rules, transforming, and training a neural network, since the data selector module has determined the variables and their correlation with further process parameters.

[0038] In some embodiments, the data selector module is trained by a selector trainer module, which, during the training of the neural network, uses minimizing the loss function as a criterion to enable the computer to repeatedly perform the acquisition, transformation, and training steps.

[0039] In some embodiments, after performing data modality adaptation of the measurement data, the computer performs the function of receiving first and second multivariate measurement time series with historical data.

[0040] In some embodiments, the computer may perform data modality adaptation by at least one of the following steps: analyzing measurement data having sound samples by classifying sound samples into categories, and analyzing measurement data having images by classifying images into categories.

[0041] In some embodiments, the computer repeatedly adapts to different targets, wherein the target is selected while minimizing the loss function during the training of the neural network.

[0042] In some embodiments, the computer uses a neural network, specifically a regressive neural network.

[0043] In some embodiments, the measurement data are related to process parameters, which are physical properties selected from the following: temperature, pressure, chemical composition of materials inside the machine, visual appearance of parts of the industrial machine, and sound emitted by the industrial machine.

[0044] In some embodiments, the industrial machine is selected from chemical reactors, metallurgical furnaces, containers, and engines.

[0045] In some embodiments, a computer-implemented method for operating the neural network for training includes the steps of receiving a multivariate measurement time series having measurement data of an industrial machine under observation; converting the multivariate measurement time series into a multivariate feature time series; and operating the neural network to provide parametric indicators.

[0046] In some embodiments, the operation of the neural network occurs in the following situations: sensor failure, data connection failure, or minimizing human involvement in the measurement.

[0047] In some embodiments, a computer program product, when loaded into the memory of a computer system and executed by at least one processor of the computer system, causes the computer system to perform the steps of a computer-implemented method.

[0048] In some embodiments, a computer system includes a plurality of computer-implemented modules that, when executed by the computer system, can perform the steps of a computer-implemented method. Alternatively, from a different perspective, the computer system performs the steps of a computer-implemented method.

[0049] Furthermore, the computer used in the implementation method functions as a vision sensor to acquire parameter indicators representing the parameters of the industrial machine. Simple Explanation of the Diagram

[0050] [Figure 1] illustrates an industrial machine with multivariate measurement time series and a neural network configured to explain the coding conventions; [Figure 2] illustrates the industrial machines and computer modules used in the preparation phase to obtain a set of conversion rules; [Figure 3] illustrates the reference industrial machine, converter module, and network being trained during the training phase; [Figure 4] illustrates the industrial machine of Figure 1 and the converter module of Figure 3 during the operation phase to provide parameter indicators; [Figure 5] illustrates a flowchart of a method for training a neural network and a method for using the network during the operation of an industrial machine; [Figure 6] illustrates an optional modal adaptor module that implements modal adaptation of optional measurement data; [Figure 7] illustrates an optional data selector module that can optionally perform the selection of relevant measurement data; [Figure 8] illustrates an example of use, where the industrial machine is a blast furnace; and [Figure 9] illustrates an example of a general-purpose computer device that can be used with the techniques described herein. Implementation

[0051] [Writing Conventions] (Writing conventions) [Overview] []

[0052] Figure 1 illustrates an industrial machine 123 with a multivariate measurement time series 223 and a neural network 373 with a configuration 283 to explain the writing conventions.

[0053] Although only symbolically, the accompanying drawing also illustrates industrial machine 123 performing an industrial process with process parameters 173 / 183. The process parameters can be divided into two groups: • Measurement parameter 183 (or x parameter), and • Non-measurement parameter 173 (or z parameter, at least one).

[0054] In the sense that the measurement data enters the multivariate measurement time series 223 of the machine 123, the attribute “measurement” is related to the specific industrial machine 123.

[0055] For at least one process parameter z, measurement data is missing (i.e., unavailable or unusable). In principle, parameter z is a measurable parameter, but the data is unavailable. The output of neural network 373 provides a parameter index Z', which is a representation of that specific process parameter z. The parameter index Z' can be considered an "artificial measurement" or artificial or virtual measurement data. In other words, the computer simulates the sensor.

[0056] The parameter Z' can be used as a status indicator for machine 123 (Z' indicates the status), or Z' can be used as an input variable for a computer or other module to determine the machine status (Z' is a factor of the status). This description focuses on calculating Z'.

[0057] For the (human) operator 193 of the industrial machine 123, the neural network 373 appears as a virtual sensor for the output parameter index Z'. In theory and ideally, index Z' would represent physical reality just like a real sensor (hardware sensor, physical sensor).

[0058] In real-world, non-ideal conditions, the accuracy of the parameter index Z' (calculated by neural network 373) can be defined as the difference between the index Z' from neural network 373 and the measurement data Z from appropriate hardware sensors.

[0059] This instruction explains the method for determining Z' (Figures 1-5) and will explain the optional measurements associated with it to improve accuracy (Figures 6-7).

[0060] Industrial machine 123 (or industrial equipment) is a machine that continuously performs an industrial process. Although examples have been mentioned above, this description focuses on the example of machine 123 being a blast furnace.

[0061] Based on this blast furnace example, the process parameter z can be the temperature of the molten material. There is no directly available measurement (as described above, due to missing temperature sensors or other reasons), but the parameter index Z' provides a calculated value. Z' can be further processed into state indicators (e.g., whether the furnace is ready for casting due to temperature and whether other criteria permit casting).

[0062] In order for neural network 373 to provide Z', several methodological steps, including machine learning, need to be performed beforehand. For convenience, these steps are explained in the context of stages. This description distinguishes the components active during the preparation stage**1 (see Figure 2), training stage**2 (see Figure 3), and operation stage**3 (see Figure 4). In other words, throughout this description, the reference numerals labeled **1 / **2 / **3** represent similar but different components in these stages. For example, machine 121 in Figure 2 and machine 123 in Figure 4 are physically the same machine, but operate as a reference machine (or as a machine under observation / supervision).

[0063] This description distinguishes between phases **1 / **2 / **3 based on the network (which provides the parameter metric Z'). From this perspective, training occurs in phase **2**.

[0064] However, other training loops may also exist: • The preparation phase **1 may optionally include a training loop (see Figure 5, a repeating loop with sub-steps in rule 421). • Training can also be applied to modal adaptation (see Figure 6), with loops on **1 and **2. Further training cycles can be applied at a larger time granularity because some steps in **1 and **2 are repeated (see Figure 7). • By including network loss as a parameter, further training loops can be applied to train the discriminator and transformer.

[0065] The process parameters to be measured (or at least simulated) will be given in lowercase letters, such as “x” or “z” (symbolized on the left side of machine 123).

[0066] The attached figure illustrates this with respect to parameters x_1 (which could be the temperature at a specific location inside the machine, e.g., at the top) and x_2 (e.g., the temperature at a different location, at the bottom). There are N process parameters {x_n}N and at least one further parameter z. The underscore “_” is a placeholder.

[0067] Measurement data are given in uppercase letters, such as X or Z. The parameter Z' is indicated by an apostrophe (') because it represents actual or artificial measurement data.

[0068] The uppercase convention also applies to intermediate data "Y" (or "feature data", unmeasured data, data with no output).

[0069] [Time Series] []

[0070] Data (measurement data, intermediate data) can be obtained in the form of time series, that is, a sequence of data values ​​with time-ordered indices at subsequent time points.

[0071] The attached figure uses short notions (rectangle 223) and introduces the time series using matrices.

[0072] The symbol {X1 ... XM} represents a single (i.e., univariate) time series with measured data elements Xm (or simply "elements"). Elements Xm from time point 1 to time point M are: X1, X2, ..., Xm, ...XM (i.e., "measurement time series"). The label m is the time point label. Time point m is followed by time point (m+1), usually at equal intervals Δt.

[0073] An example is the temperature increase over M time points: {1400°C ... 1500°C}. Those skilled in the art can preprocess the data values, for example, to normalize them to [0,1] or {0.2 ... 1}. The format of the measurement data is not limited to scalars or vectors; {X1 ... XM} can also represent a sequence of M images taken from time point 1 to time point M. In the case of a blast furnace, the images can be taken from inside the tuyeres of the blast furnace or from the immediate environment of the blast furnace. Images are acquired to detect anomalies.

[0074] The symbol {{X1 ... XM}}N represents a multivariate time series with a data element vector {X_m}N from time point 1 to time point M (reference numeral 223 above the matrix in Figure 1). The vector has a cardinality N (the number of variables, i.e., the process parameters available for that measurement), meaning that at any time point from 1 to M, N data elements are available. The matrix indicates that the variable label n is the row label (from x_1 to x_N).

[0075] For example, a single time series of temperature can be accompanied by a single time series of pressure, measurement data on the chemical composition of materials, and so on.

[0076] Those skilled in the art will understand that this description is simplified. The actual number of variables N can reach or even exceed several thousand. Time series are not ideal. Occasionally, elements may be missing, but those skilled in the art can adapt to such situations.

[0077] The choice of time interval Δt and the number of time points M depends on the process. The total duration of the time series Δt*M (i.e., the window size) corresponds to the process parameter rotation that takes the longest time. For example, a furnace takes several hours to process materials, so Δt*M is chosen accordingly.

[0078] The interval Δt can be conveniently selected within the range of 1 to 60 minutes, and Δt*M can be conveniently selected within the range of 4 to 24 hours. The example of a blast furnace can be applied with Δt = 15 minutes and M = 32, thus corresponding to an 8-hour work cycle.

[0079] Since the time point tm specifies the time used for network processing, some measurement data can be preprocessed. For example, a temperature sensor may provide data every minute, but for Δt = 15 minutes, some data can be discarded, averaged over Δt, or otherwise preprocessed.

[0080] Because data can originate from different industrial machines, data sources are sometimes indicated by numbers such as 1, 2, etc., before {} or {{...}}. In the example in the attached figure, 2{{...}} indicates a multivariate time series from machine 1, 2, or 3.

[0081] As will be explained, the computer applies a transformation from X to Y. The resulting values ​​can also be multivariate time series. The cardinality is given by K.

[0082] [Figure Labels] []

[0083] As used herein, reference numeral 1** refers to an industrial machine in the real world (physical world), reference numeral 2** refers to data (such as configuration 283), reference numeral 3** refers to hardware and computer-implemented modules (such as neural networks 373); and reference numeral 4** or 5** refers to method steps (see Figure 5).

[0084] [calculator] []

[0085] The symbol “computer” (singular, without figure mark) represents computing functionality or the functionality of a module implemented in a computer (such as a processing unit, which is a time-shared resource of such a unit). This functionality can be distributed across different physical computers, and the modules implemented in the computer can be distributed across these different computers.

[0086] The accompanying drawings also illustrate a computer program or computer program product. When the computer program product is loaded into the computer's memory and executed by at least one processor of the computer, the computer program product performs the steps of a computer-implemented method. In other words, the program provides instructions to the module.

[0087] The accompanying diagrams, from different perspectives, illustrate modules of a computer system comprising multiple computer-implemented modules, which, when executed by the computer system, perform the steps of the computer-implemented methods. Industrial machines are not considered computer-implemented modules.

[0088] [Machine Learning Tools and Their Configuration] []

[0089] The accompanying diagram also illustrates neural network 373 with neurons (circle symbols) arranged in layers (neurons along a vertical line). Neural network 373 is an example of a machine learning tool. Several other modules described in this paper also utilize machine learning.

[0090] Such a network has two basic descriptors: • The set of hyperparameters, and • Configuration.

[0091] hyperparameters • Define the mathematical function to be performed by the neuron (e.g., summing the weighted inputs and applying a threshold or other function). • Define the function performed by the combination of neurons (e.g., choice of kernel function, between convolution kernels, Gaussian kernel, etc.). • Define the arrangement of neurons in layers and the interconnections between layers (e.g., full connections between layers, such as between the first and second layers in the example).

[0092] Hyperparameters are typically described in this field as network architecture. No training is performed on the hyperparameters.

[0093] Configuration 283 defines the weighted data transmission from neuron to neuron and defines the bias (or offset) value for each neuron. Configuration 283 is obtained through training.

[0094] Neural network 373 is illustrated here only as an example. Figures 6 and 7 illustrate further tools (such as adapting data modalities, selecting relevant data, and data transformers), and the architecture and configuration conventions also apply to these tools.

[0095] Domain differences between machines []

[0096] Industrial machine 123 (Figure 1) is not the only machine for which this measurement data is available. Other industrial machines are also available for this measurement data (hereinafter referred to as "equivalent machines"). Equivalent machines operate according to the same principles, but may differ in some aspects.

[0097] Given this difference, industrial machines can be considered to belong to the first and second domains.

[0098] One of the equivalent machines can provide additional measurement data to train the network. This specification refers to industrial machines 111 / 112 as the source (providing real measurement data Z for parameter z) and machine 123 as the target (because the network calculates metric Z'). The equivalent machine providing data for training is also called the "reference machine." Both the source and target machines are reference machines because both can provide the measurement data used in preparation phase **1. One of the source and target machines (the source machine) has a sensor, while the other (the target machine) does not. The source machine is the source of the data used in training phase **2.

[0099] Domain differences between reference machines are characterized by different operating modes. Such operational differences arise from different machine parameterizations or from different environments affecting the underlying processes of machine operation.

[0100] Taking blast furnaces as an example again, two identical machines process the same metal, use the same materials (such as ore and coke), have sensors in similar locations, use the same parameters, and so on.

[0101] However, differences exist: the first furnace may be larger than the second (size difference) or the furnace may have a slightly different shape, resulting in slightly different measurements, such as differences in the volume of molten material, melting time, etc. Size differences are significant in furnaces with different heights (see, for example, Figure 8).

[0102] Domain differences result in different measurement data for the same equivalent parameters.

[0103] Furthermore, domain differences are the result of applying different measurement methods.

[0104] Computer applications require data conversion due to domain differences.

[0105] Let's briefly refer to music to provide an illustrative example. Both an organ pipe and a blast furnace produce sound from moving air. A longer organ pipe plays at a lower frequency compared to a shorter one. However, human listeners have learned to recognize melodies, which means sequences of tones with different relative frequencies. Melody is invariant with respect to absolute frequencies.

[0106] Since the sound from the furnace can be used as measurement data, the converter separates the melody from the pitch. In other words, a properly adapted converter can receive two time series 1{X1 ... XM} and 2{X1 ... XM} from the first and second furnaces and provide a melody (domain invariant) that can be further processed (such as by comparison).

[0107] [Data Modal] []

[0108] For the sake of brevity, this description initially assumes that the measured data have the same data mode. It is assumed that the measured data in the time series contain only scalars. Optional modality adaptation will be discussed in conjunction with Figure 6.

[0109] [Determine parameter indicators] [Z] [' correlation] []

[0110] For the sake of further brevity, this description initially assumes that all parameters x_1 to x_N can affect parameter z. Therefore, the computer processes all these parameters. Optionally, relevant differences may be considered, and this description will explain the details in conjunction with Figure 7.

[0111] [Steps to generate configuration] []

[0112] In the following description, the steps performed by one or more computers to obtain the configuration are explained. The accompanying drawings are described in order of the steps. However, it is also convenient to begin with an overview of Figures 2 through 4 from the end.

[0113] During its operation (see Figure 4), the industrial machine 123 provides measurement data to the neural network 373, which in turn provides parameter index Z'.

[0114] Neural network 373 can provide Z' because: • Configuration 283 was previously acquired during the training phase (Figure 3) by processing historical measurement data, which is substantially complete (with data for z) but originates from industrial machine 112 (appropriate data for z is available for industrial machine 112). More specifically, time series 212 and 272 include common measurement data 1 {{X1 ... XM}}N and further data 1 {Z1 ... ZM}. Although the neural network 373 with configuration 283 cannot directly process the measurement time series 223 with data 2{{X1 ... XM}}N, the converter 323 converts the time series 223 with data 2{{X1 ... XM}}N into an intermediate time series 243 2{{Y1 ... YM}}K.

[0115] Converter 323 can provide intermediate time series 243 because: • During the preparation phase (see Figure 2), the preamble 311 / 321 of converter 323 has been configured to convert the time series {{X1 ... XM}}N into the intermediate time series {{Y1 ... YM}}K. • The configuration of converters 311 / 321 has been completed by discriminator 361, which optimizes the common conversion rules so that the intermediate time series are no longer specific to the domain differences between physically different industrial machines 111 and 121.

[0116] It is possible to train network 372 to obtain configuration 283 (see Figure 3) because: The network being trained can receive historical measurement data (from an equivalent machine, in this case, from machine 112) because converter 322 can transform the data. There is no need to transform the time series 1{Z1 ... ZM}.

[0117] [Source of measurement data] []

[0118] To keep the illustrations simple, Figures 2 through 4 show the measurement data from the sensors. However, some measurement data can also be obtained via human-machine interfaces, etc. For convenience, the sensors will be referred to as the measurement data they provide: X1 ... XN, and Z (in the small rounded square symbol).

[0119] Figure 2 illustrates the industrial machines and computer modules in the preparation phase **1 to obtain a set of (421) conversion rules.

[0120] Industrial machines 111 and 121 each have sensors X1 to XN (common sensors, as they provide data for common process parameters). Industrial machine 111 has a sensor Z (shown in bold) to provide a univariate measurement time series 211 1{{X1 ... XM}}N, but machine 121 does not have such a sensor.

[0121] The information measurement data Z will be "exchanged" between peer machines. This "exchange" is not direct, but occurs during rule establishment. Industrial machine 111 is the "source" and machine 121 is the "target". Industrial machine 111 is the only machine equipped with sensor Z.

[0122] [Parameter equivalence] []

[0123] The process parameter xn is available in both reference machines (1xn for the source and 2xn for the target, see the matrices in Figure 1). For example, both machines have the temperature parameters for the upper part of the furnace. The process parameter x(n+1) is also available in both machines, for example, for the temperature of the lower part.

[0124] In other words, process parameters that are available in both reference machines are equivalent parameters. Equivalent parameters are labeled with the suffix n to indicate the equality of variables.

[0125] Equivalent parameters provide equivalent measurement data.

[0126] Sensor Xn of industrial machine 111 (“source”) and sensor Xn of machine 121 (“target”) are equivalent because the resulting measurement data points to the same parameters; for example, both are temperature sensors in the upper part. Sensor Xn of industrial machine 111 and sensor X(n+1) of machine 121 are not equivalent.

[0127] The parameter is singular (e.g., temperature at the upper part), but the measured data is plural (e.g., two furnaces).

[0128] [Sensor equivalence] []

[0129] The sensors of different machines that provide measurement data on equivalent parameters are called equivalent sensors. Equivalent sensors do not necessarily operate on the same principles.

[0130] For example, both the (source) sensor Xn and the (target) sensor Xn provide the furnace top gas temperature. Xn (the source machine) is implemented through an invasive method (moving a rod with an attached sensor above the furnace charge), and Xn (the target machine) is implemented through a sound transmitter and receiver above the furnace charge (i.e., above the furnace charge).

[0131] [Historical Data] []

[0132] Generally, training machine learning tools (such as networks) requires historical data. The operation of converters 311 / 321 begins by receiving the following items. • A multivariate measurement time series 211 (1{{X1 ... XM}}N) with historical measurement data from reference industrial machine 111, and • A second multivariate measurement time series 221,2{{X1 ... XM}}N with historical measurement data from a physically different second reference machine 121.

[0133] Although the accompanying figures illustrate the receiving process with arrows, those skilled in the art will store the time series data in a database. No further explanation is required herein.

[0134] However, simply feeding {{X1 ... XM}}N into the tool being trained (such as a transformer used to learn rules) is insufficient for M*N data elements (see the matrix in Figure 1). Two options exist:

[0135] First, historical data can come from multiple physical industrial machines 111 / 112 (not just from two machines, but from many). This option is only indicated by the repeating dashed boxes. However, in practical situations (heterogeneous groups), such a method is not available for furnaces.

[0136] Second, the physical industrial machines 111 / 112 repeatedly provide data. This second option is preferred.

[0137] Figure 2 illustrates multiple {{...}}N (and {{...}}K) using two rounded rectangles (overlapping). The figure also illustrates the arrow Q. Further symbolization is {{{...}}}Q.

[0138] Q is the number of multivariate time series measurements. It is convenient to assume that all machines have the same Q. However, this is not always possible in practice. A machine may be "younger" (with a smaller Q) than its equivalent machine (with a larger Q).

[0139] Continuing with the example above, Δt*M = 8 hours, the machine will collect Q = 3*365 multivariate measurement time series over a year. In other words, Q can be on the order of 1000 per year. It is not necessary to consider all available data. Sometimes the machine may be in maintenance or repair mode, so historical measurement data will not be available.

[0140] An 8-hour time slot is convenient for illustration. However, computers can apply window slots T_WINDOW to compute characteristics of multivariate time series. Such a window slot can be defined as a moving time window with a duration of several hours (e.g., 8 hours). Window slots can overlap (T_OVERLAP) and move over data collected over longer time periods (e.g., a year). A convenient relationship between T_WINDOW and T_OVERLAP is, for example, T_WINDOW > T_OVERLAP (or even equal). It is convenient to set T_WINDOW around 6 to 10 hours, with T_OVERLAP between 1 and 3 hours.

[0141] [Removing Domain Differences from Historical Data] []

[0142] Despite the equivalence mentioned earlier (in terms of parameters and sensors), historical data still reflects domain differences.

[0143] The computer obtains a set of transformation rules 251 (in Figure 2) / 252 (in Figure 4) by processing multivariate measurement time series 211 and 221 (step 421 in Figure 5). Transformation rules 251 / 252 enable converters 311 / 321 to convert multivariate measurement time series 211, 221 ({{..}}) into multivariate feature time series 231 and 241, respectively. These feature time series 1{{Y1 ... YM}}K and 2{{Y1 ... YM}} are invariant to the domain differences of the reference industrial machine 111 / 121.

[0144] In other words, the domain-invariant features have been extracted.

[0145] As mentioned earlier, the time series was measured using multiple multivariate measures (see Q), and the transformation did not change that.

[0146] Transformers 311 / 321 are illustrated here as time series 211 is transformed into time series 231, and time series 221 is transformed into time series 241. Whether the transformations are performed in parallel (as shown in the diagram) or serially is irrelevant. Both transformers use common rules 251 / 252.

[0147] Rules are acquired through training. No human expert involvement is required. In other words, domain adaptation (using features rather than measurement data) is unsupervised. Expert annotations or labels are not required. Optionally, experts can manually define some rules based on existing expert knowledge. Manually defining some rules further guarantees the convergence of training.

[0148] [Conversion Rules] []

[0149] Since the network has been trained with training data from different domains, transfer learning is involved.

[0150] In short, there exists a transformation that converts the first and second multivariate measurement time series 211 and 221 into the first and second multivariate feature time series 231 and 241, respectively. The domain differences of the multivariate feature time series 231 and 241 are invariant with respect to the reference industrial machines 111 and 121.

[0151] In other words, the transformation extraction is no longer domain-specific. The measurement time series can be considered to belong to the first space (or input space), and the feature time series can be considered to belong to the second space (or feature space). The input space is sensitive to domain differences, while the feature space is not.

[0152] Assume that there is a causal relationship between spaces X and Y, which represent a continuous space of regression.

[0153] The marginal distributions in the two domains (source=s, target=t) are similar: 𝑝s(𝑦𝑠|T(𝑥𝑠)) ≈ 𝑝𝑡(𝑦𝑡|T(𝑥𝑡)), where 𝑦𝑠 is the output data in the source domain ("s") and 𝑦𝑡 is the output data in the target domain ("t").

[0154] The conversion rule follows the principle of Y = T(X). More specifically, converters 311 and 321 operate according to the time interval Δt: at t=1, converter 311 converts data vector 1{X1}N to data vector 1{Y1}K and converter 321 converts data vector 2{X2}N to data vector 2{Y2}K. Next, at t=2, it converts 1{X2}N to 1{Y2}K and 2{X2}N to 2{Y2}K, and so on until t=M.

[0155] Since the variables of {{X1 ... XM}}N are not necessarily the same as the variables of {{Y1 ... YM}}K, the transformation can change the cardinality of the variables (usually N > K).

[0156] Although the transformation rule is introduced here for operations with data at time tm (vertically, processing the data vector over time interval tm), alternatively, the rule can be extended to process data from previous time points (m-1), (m-2), etc. Such cyclic networks are familiar to those skilled in the art.

[0157] Since the conversion has been described in this field, those skilled in the art can choose from existing conversion options.

[0158] For convenience, the following further paper is proposed: · Hochreiter, et al, “Long Short-Term Memory”, in Neural ComputationNovember 1997, https: / / doi.org / 10.1162 / neco.1997.9.8.1735 (1997) · Ashish Vaswani et al, “Attention Is All You Need“, Advances in Neural Information Processing Systems 30, NIPS 2017, (2017) · Anastasia Borovykh, et al, “Conditional Time Series Forecasting with Convolutional Neural Networks”, arXiv:1703.04691 (2017) · Aaron van den Oord, et al., “WaveNet: A generative model for raw audio”, arXiv:1609.03499, (2016).

[0159] Features for multimodal, multivariate time series are extracted using algorithms such as, but not limited to, recurrent neural networks (Hochreiter), which may be combined with convolutional neural networks, or transformers (Vaswani), or Borovykh, van den Oord.

[0160] The transformation function T can be identified by minimizing the dissimilarity between two class conditional distributions: 𝑝𝑠(𝑡(𝑥𝑠)|𝑦𝑠) ≈ 𝑝𝑡(𝑡(𝑥𝑡)|𝑦𝑡).

[0161] The transformation function φ() used for defining domain-invariant features is often found by training adversarial deep learning models, where the transformation converges to generate features in which the source or target domain is indistinguishable from each other, and the loss of the regression model predicting the output is minimized.

[0162] The following paper provides further background information: Y. Ganin, et al., “Domain-adversarial training of neural networks,” Journal of Machine Learning Research, vol. 17, no. 59, pp. 1–35, (2016).

[0163] [Example of training a converter] []

[0164] This instruction assumes that the converter has already obtained rules 251 / 252. The following describes alternative methods for obtaining rules by applying adversarial (deep) learning.

[0165] The method can be summarized into the following steps, which are presented as sub-steps of the step of obtaining rule 421 (see Figure 5). Obtaining the set of transformation rules 251 / 252 of 421 can be performed by a computer, which repeats the following steps:

[0166] In transformation step 431, and using the set of preliminary transformation rules, the computer transforms the (first) multivariate measurement time series 211 into the (first) multivariate feature time series 231, and transforms the (second) multivariate measurement time series 221 into the (second) multivariate feature time series 241. Details have been described above.

[0167] In discrimination step 441, the computer uses discriminator 361 to distinguish (or “discriminate”) the sources of the first and second feature time series 231, 241 as originating from (first) and (second) multivariate measurement time series 211, 221. During the initial repetitions, discriminator 361 can “easily” distinguish them. Figure 2 illustrates this with a graph showing the learning time (total time used for repetitions). On the left (early learning), the discriminator is able to distinguish: X relative to Y comes from 1 (first machine / domain) or from 2 (second machine / domain). On the right, the discriminator cannot.

[0168] In modification step 451, the computer modifies the set of preliminary transformation rules (or modifies the currently repeating rules).

[0169] The computer repeats steps 431, 441, and 451 until the discriminator 361 can no longer distinguish the data source (i.e., from 1 or 2). To determine that it can no longer distinguish the data source, the discriminator 361 may calculate the adversarial loss ADLOSS. The discriminator 361 will continue to operate (and allow rule changes) until ADLOSS reaches its maximum value.

[0170] Transformation rules 251 / 252 become the final rules that will be applied in subsequent steps.

[0171] In this particular approach, rules are acquired through a learning process with seemingly contradictory learning objectives.

[0172] In other words, the combination of converters 311 / 321 and discriminator 361 is trained (with changes to the rule settings) to classify (or “predict”) whether features are related to the source machine or the target machine.

[0173] In other words, the same encoder (or converter, in which rules are prepared in repetition) is shared between the source domain and the target domain (or the first and second reference machines), and it is expected that the encoder will extract domain-invariant representations with the help of an adversarial discriminator (i.e., discriminator 361).

[0174] For further details, please refer to the following paper. · Zhang, et al., “Domain-Invariant Adversarial Learning for Unsupervised Domain Adaption”, arXiv:1811.12751, (2018) · Xie, et al., “Learning Semantic Representations for Unsupervised Domain Adaptation@, Proceedings of the 35th International Conference on Machine Learning, PMLR 80:5423-5432, (2018) · Avisek Lahiri, et al., “Unsupervised Domain Adaptation for Learning Eye Gaze from a Million Synthetic Images: An Adversarial Approach”, In Proceedings of the 11th Indian Conference on Computer Vision, Graphics and Image Processing (2018)

[0175] Adversarial training is further outlined in Goodfellow, Ian J., Pouget-Abadie, Jean, Mirza, Mehdi, Xu, Bing, Warde-Farley, David, Ozair, Sherjil, Courville, Aaron C., and Bengio, Yoshua. Generative adversarial nets. NIPS, 2014.

[0176] The acquisition of rules through learning methods belongs to the preparation phase**1. This differs from the training network, which will be explained later.

[0177] Figure 3 illustrates the reference industrial machine 112, the converter module 322 (hereinafter referred to as "converter 322"), and the (trained) network 372 during the training phase**2.

[0178] Transformer 322 transforms the multivariate measurement time series 212 (step 452 in Figure 5) into a multivariate feature time series 323. Time series 212 may be the same as time series 211 (see Figure 2). Transformer 322 applies transformation rule 252 (see rule 251 in Figure 2, when rule 251 is finally completed). As symbolized by multiple overlapping rectangles, transformer 322 processes multiple multivariate measurement time series (see Q as described in Figure 2).

[0179] The computer receives from reference machine 112 (step 462 in Figure 5) a univariate measurement time series 272 (i.e., 1{Z1 ... ZM}) of measurement data (Z) with further process parameters (z), and typically multiple series also exist. In this sense, industrial machine 112 acts as a reference to provide measurement data from actual measurements (such as from sensors X1 to XM and from sensor Z).

[0180] Network 372 is trained with a first multivariate feature time series (232) at the input and a univariate measurement time series (272) at the output (step 472 in Figure 5).

[0181] Those skilled in the art can train the network using appropriate hyperparameters, optimization objectives, etc. As an example, training is complete when the loss function reaches a specific predefined value. During training, the intermediate calculated value LOSS (or "network loss") will change and eventually reach a minimum, thus allowing the weights and biases to be configured. In other words, LOSS is a control parameter used for looping processes within the network.

[0182] LOSS can also be used to control loops outside of network 372. The attached diagram illustrates the intermediate calculated value LOSS that can be provided by network 372. As will be explained with reference to Figure 7, selector trainer 380 can use LOSS in further loops. In other words, LOSS can also be a control parameter used for one or more loops outside the network process.

[0183] In addition, LOSS can be provided to the converter as a parameter to optimize the conversion rules (see Figure 2).

[0184] Figure 4 illustrates the industrial machine 123 (of Figure 1) and converter module 323 (see Figure 3) during operation phase **3, used to provide parameter index Z'. Figure 4 is an updated version of Figure 1, for which details have been explained above.

[0185] The computer receives (see step 513 in Figure 5) a multivariate measurement time series 223 (also labeled 2{{X1 ... XM}}N) containing measurement data for the industrial machine 123. Here, the machine 123 is under observation without sensors to measure parameter z.

[0186] Converter 323 converts the multivariate measurement time series 223 into a multivariate feature time series 243 (also 2{{Y1 ... YM}}K).

[0187] The network receives multivariate feature time series 243 and provides parameter indicators Z'. The neural network 373 is illustrated with configuration 283.

[0188] Note that, unlike Figures 2 through 4, the time series is a singular time series (i.e., for a specific job rotation, etc.).

[0189] Figure 5 illustrates a flowchart of a computer-implemented method 400 for training neural networks 372 / 373 and a flowchart of a computer-implemented method 500 for using the network during operation of industrial machine 123.

[0190] Method 400 is a computer-implemented method for training neural network 373. Neural network 373 is being trained for later processing of multivariate measurement time series (see 223 in Figure 1) with measurement data representing instances of specific process parameters (see 183 in Figure 1) of an industrial machine (see 123 in Figure 1), and thus for providing parameter indices Z' for further process parameters (see 173, z in Figure 1) of the industrial machine.

[0191] In receiving step 411, the computer receives a first multivariate measurement time series with historical measurement data from a first reference machine, and receives a second multivariate measurement time series (1{{X1 ... XM}}N) and 2{{X1 ... XM}}N with historical measurement data from a physically different second reference machine (see left side of Figure 2).

[0192] In step 421, the computer obtains a set of transformation rules 251 / 252 by processing the first and second multivariate measurement time series. These rules enable the converters (see 311 / 321 in Figure 2) to convert the first and second multivariate measurement time series into first and second multivariate feature time series, respectively (see 231 and 241 in Figure 2, also labeled 1{{Y1 ... YM}}K and 2{{Y1 ... YM}}K). The domain differences of the multivariate feature time series remain unchanged for the first and second reference machines (see 111 and 121, Figure 2). (The two reference machines belong to different domains, with details already explained.)

[0193] In transformation step 452, the computer transforms the first multivariate measurement time series into the first multivariate feature time series by applying transformation rules to the converter (322, Figure 3).

[0194] In the receiving step (462), the computer receives a univariate measurement time series of measurement data Z with further process parameters z from the first reference machine (see 112 in Figure 3).

[0195] In training step 472, neural network 372, the computer trains the network with a first multivariate feature time series at the network input and with a univariate measurement time series at the network output.

[0196] Figure 5 also illustrates optional details (dashed boxes) for obtaining the set of transformation rules for 421. The computer repeats steps 431, 441, and 451.

[0197] In transformation step 431, using a set of transformation rules as preliminary rules, the computer transforms the first multivariate measurement time series into a first multivariate feature time series, and transforms the second multivariate measurement time series into a second multivariate feature time series.

[0198] In the identification step 441, the computer runs the discriminator 361 to distinguish (or identify) the sources of the first and second characteristic time series as the first and second multivariate measurement time series.

[0199] In modification step 451, the computer modifies the set of preliminary conversion rules.

[0200] When the discriminator (361) can no longer distinguish the data source, the repeating steps stop (i.e., REPEAT UNTIL or WHILE loop).

[0201] The details of the steps in method 400 are explained within the context of this description.

[0202] Method 500 for using a computer to utilize a network during the operation of industrial machinery is illustrated in the accompanying drawings and referenced to Figure 4.

[0203] In receiving step 513, the computer receives a multivariate measurement time series with measurement data of an industrial machine under observation; in conversion step 523, the computer converts the multivariate measurement time series into a multivariate feature time series; and in operation step 533, the computer operates neural network 373 to provide parameter index Z'.

[0204] Note again that the computers used for methods 400 and 500 are physically different computers. The common data will be configuration 283 (the results of training).

[0205] [Improve accuracy] []

[0206] As briefly mentioned above, the accuracy of the indicator Z' corresponds to the difference in the measured data Z (if available). Simply put, lower difference implies higher accuracy.

[0207] This document will now explain two methods that can improve accuracy. One method (adaptation) increases the number of variables being processed, while the other method (selection) reduces the number of variables.

[0208] At first glance, the two methods may seem contradictory, but they are related. For convenience, this explanation will introduce the two methods separately, and then discuss their relationship.

[0209] [Data Modality Adaptation] []

[0210] So far, this specification has explained the methods for measuring data in time series containing only scalars, where {X1 ... XM} is a sequence of M consecutive scalar values. It doesn't matter whether the computer represents the scalars as integers or real numbers.

[0211] This document now discusses how to adapt measurement data in other modalities for processing.

[0212] Figure 6 illustrates an optional modal adaptor module 393 (or simply "adaptor") that optionally performs modal adaptation of the measurement data. The figure illustrates the adaptor 393 during operation phase **3 by way of example, but the adaptor 393 may also operate in other phases.

[0213] The adapter 393 receives a multivariate measurement time series 213 (also labeled {{X1 ... XM}}) from an industrial machine (illustrated as machine 113 by way of example, but the data source “1” or “2” is irrelevant). The term “receive” corresponds to method steps 411 and 513 (see Figure 5). The adapter 393 may be implemented accordingly as part of a module that performs the receiving.

[0214] A multivariate measurement time series {{X1 ... XM}}N includes univariate measurement time series {X1 ... XM}, which are sequences of scalar or non-scalar elements. In the example, the univariate time series {X1 ... XM}n should contain the non-scalar element Xn.

[0215] As a side note, in many cases, 1{Z1 ... ZM} can simply be a scalar. However, this is not necessary, so 1{Z ... ZM} can also be a vector, etc.

[0216] The adaptor 393 passes time series with scalars essentially unchanged to other modules. These time series do not even have to pass through the adaptor.

[0217] Adaptor 393 processes the nonscalar element Xn individually, but according to the rules used for each n. The attached figure highly symbolically illustrates the nonscalar element Xn. Thus, the computer can process this element and assign it a scalar (by classification), given here as Xn~. The "~" merely indicates that the assignment has occurred.

[0218] In the first scenario, the non-scalar elements are images (i.e., matrices of pixels indicating colors). For example, the image shows machine inspection holes captured at intervals Δt (the exposure time of the image is negligible).

[0219] In the blast furnace example, an inspection is conducted using a tuyer to allow an observer to see the flames inside the furnace. The flames appear regularly during operation. Very simply, an image showing flames can be encoded as 1, and an image without flames can be encoded as 0. The resulting time series can be {..}~ = {0, 0, 1, 1, 1, 0 ... 0, 0}~, and the computer will continue processing (through subsequent steps, such as obtaining rule 421, or transformation 523, see Figure 5, or even receiving 513).

[0220] In the second scenario, the non-scalar element is a sound sequence. For example, Xn could be an audio recording from a microphone sensor with a duration of Δt = 15 minutes. Those skilled in the art can apply appropriate sound processing, for example, by sampling the sound at a frequency of 20 kHz, thereby generating 15 * 60 * 20,000 audio samples per second.

[0221] The adaptor 393 can then process these millions of samples into a single scalar. For example, the scalar could indicate: a pitch increase during Δt, a pitch decrease during Δt, a constant pitch during Δt, a pitch increase and decrease during Δt (as in the example), and so on.

[0222] The adaptor 393 can be implemented by a previously trained neural network, potentially under the supervision of a human expert. For example, the adaptor 393 could have been trained using labeled images of flames (in a blast furnace), labeled sound sequences (taken from a blast furnace), etc. Alternatively, the adaptor 393 can be trained unsupervised to recognize abstract representations of images. This ensures the maximization of the image's information content. This process then reduces the image sequence to several scalar sequences.

[0223] Assigning a scalar does not necessarily mean that the measurement data is correlated with the index Z'. The attached figure shows that adaptation can use networks that have been trained differently, so for the same input Xn, the scalars can be different Xn~ and Xn~~. "6" and "7" in the attached figure are for illustration only. The consequences of having different scalars for the same input will be discussed below.

[0224] [Relevance Selection] []

[0225] Some process parameters are unrelated to parameter z. In other words, the entire process parameter set (x_1 to x_N, with a cardinality of N variables) can be divided into... • The first subset of the H process parameters x_1 to x_H related to parameter z (“related parameter set”), and • The second subset of (NH) process parameters that are not related to parameter z.

[0226] In music, for example, during further excursions, it doesn't matter whether the melody is played as a "largo" or an "allegro," at least to the listener the melody is the same.

[0227] Ideally, neural network 373 has been trained to simply ignore measurements of parameters related to the second subset. In real-world training scenarios, some measurements (the second subset) can still contribute to the metric Z'.

[0228] It is important to note that training can be conducted without the involvement of human experts. However, such supervision is not necessary during training.

[0229] Industrial machine experts can identify relevant parameter sets based on experience. Only measurement data with relevance identified by the expert will be entered into the computer. The expert's involvement here will be a one-time task.

[0230] For example, the energy consumption of a furnace may not affect the temperature (exemplary z parameter), and the consumption data can be ignored (by not processing measurements about consumption).

[0231] In addition to manual pre-selection, and as an alternative, correlation identification can be performed by a computer by selecting measurement data. An example will be explained below.

[0232] Figure 7 illustrates an optional data selector module 383 (or simply "selector"), which optionally performs the selection of relevant measurement data. The figure at the top illustrates the selector 383 during operation phase **3, and illustrates its conditioning during phases **1 and **2 by repeating some method steps from Figure 5.

[0233] In operation, selector 383 receives multivariate measurement time series 223 (see Figures 1 and 3, also labeled {{X1 ... XM}}N) and forwards modified multivariate measurement time series 223 (also labeled {{X1 ... XM}}H) to converter 323 (see Figure 4).

[0234] The modification is a selection (i.e., a positive selection from N variables to H variables in {{...}}H, where H < N). In other words, this corresponds to the selection of the relevant parameter set (but at the measurement data level). The attached figure illustrates this selection by showing (compared to the machine) fewer sensor symbols at the output of selector 383.

[0235] Performing method 500 with a reduced measurement dataset (see Figure 5) can improve the accuracy of providing the metric Z'. This method may seem contradictory: the selector blocks some data to be processed by the network, yet accuracy is improved. However, this selection picks out (sorts out) 373 measurement data that the neural network may not be able to ignore and that may potentially reduce accuracy. Again, note the unsupervised nature of training the network (see Figures 2-4).

[0236] Selector 383 can be configured through a learning process that uses the loss function of network 372 / 373 as a value to minimize. This method is symbolized by selector trainer module 380 (or simply "selector trainer").

[0237] Selector trainer 380 communicates with other modules, such as network 372 (receive training, receive loss), and performs the methods symbolized by the flowchart.

[0238] During repetition, until the intermediate value LOSS (in other words, the loss function) has a minimum value (“minimum loss”, “is”), the selector trainer 380 changes the (initial) selection ({{...}}H ∈ {{...}}N) and allows the component to perform method 400 (see Figure 4), which means (optionally) receiving 411, acquiring rules 421, transforming 452, (optionally) receiving 462, and training 472. Receiving is optional because the received data remains unchanged.

[0239] In other words, the initial choice of variable {{...}}H is replaced by another choice until the loss is minimized for the final choice.

[0240] This explanation is simplified, and those skilled in the art can implement the method accordingly. For example, there is no need to repeat receiving step 462 (because the selection 1{Z1 ... ZM} will not be deselected).

[0241] Once the loss function reaches its minimum, the operation explained in Figures 1 through 5 is performed with {{...}}H instead of {{...}}N (i.e., during operation phase **3, as explained with reference to the upper part of this figure).

[0242] Optionally, the operation of selector 383 can be enhanced by selecting some variables of the measurement time series and by combining or merging these variables into a result measurement time series. For example, selector 383 can select {...}1, {...}2, and {...}3 (such as temperatures from 3 locations) and combine them into a new univariate measurement time series (which will be part of {{...}}H).

[0243] Those skilled in the art can perform this, for example, through the selection and combination of principal component analysis.

[0244] [discuss] []

[0245] The modules that perform data modality adaptation (see Figure 6) and relevance selection (see Figure 7) operate independently of each other in phase **3.

[0246] However, to save computational resources, it is convenient to allow the data selector 383 to operate before the modality adaptor 393, at least during operation phase **3. For example, it is not necessary to process the audio recording into a scalar (see Figure 6) and then have the scalar filtered by the selector 383.

[0247] While "to-scalar-adaptation" allows the network to process more data, some adaptations may introduce errors. By assigning scalars, the adaptor can be applied to networks trained differently. Some results may be correlated with Z', while others may not. For example, analyzing a sound sequence as a category such as rising / falling / constant pitch may not be correlated with Z', but analyzing it as other categories such as oscillating / stable sound may indeed be correlated with Z'. In this sense, the differently acquired scalars Xn~ and Xn~~ (see Figure 6) can be seen as correlation selection (Figure 7).

[0248] [Report on network loss] []

[0249] This explanation now temporarily returns to Figures 2-3. As already explained, the transformation rules were obtained in preparation phase **1, thereby providing the transformer with feature time series that are invariant to domain differences. The criterion is adversarial loss (ADLOSS, see Figure 2).

[0250] To further improve accuracy, network loss augmentation transition rules (LOSS, see Figure 3) can be considered. Therefore, the computer can operate in a loop including steps **1 of the preparation phase** and **2 of the training phase** (i.e., as initial training).

[0251] In other words, the steps of acquiring the set of transformation rules 421 and training the neural network 372 are repeated. Transformers 311 / 321 further receive the network loss (LOSS) provided by the trained network. To minimize the network loss LOSS, the repetition can be stopped.

[0252] To illustrate this alternative method, Figure 2 shows the reception of LOSS with dashed lines. LOSS is minimized when ADLOSS is maximized. Other conventions may be applied by those skilled in the art.

[0253] [Usage Example] []

[0254] Figure 8 illustrates examples of blast furnace usage where industrial machines 111, 121, and 123 are examples. Multiple temperature parameters (symbolized by different values ​​ranging from 200°C to 2000°C) and other process parameters (such as pressure values, chemical composition of the materials, etc.) are present.

[0255] The domain difference is illustrated by two examples: the size difference is symbolized by the fact that furnace 111 is smaller than furnaces 121 and 123. Different measurement methods are applied to furnaces 111 and 121 for the temperature of the molten material (hot metal temperature): a remote sensor 151 (camera symbol) at furnace 111 produces a time series of samples available in Δt intervals (e.g., 15 minutes). Measurements performed manually by the operator of furnace 121 produce temperature values ​​available at larger intervals (e.g., 90 minutes), as mentioned in the background section.

[0256] In this exemplary use case, virtual sensor 153 acquires an index Z' (in terms of temperature) with a value of Δt. Although illustrated by a camera symbol with a small, dashed line, this virtual sensor represents a computer performing method 500 (see Figure 5). In other words, the figures generally illustrate the use of a computer performing (or executing) method 500 (and trained according to method 400) to acquire a parameter index Z' representing parameters of an industrial machine. A computer using method 500 can be considered to be using a virtual sensor.

[0257] Furnace 121 and Furnace 123 are physically identical, but Furnace 123 will be operated in the future, having been trained with a network 372 / 373. The accompanying diagram does not show the operator collecting samples for Furnace 123, and therefore indicates that this use case helps minimize human involvement in the measurement, allowing the operator to better focus on furnace operation. In other words, Furnace 121 can be upgraded to Furnace 123 via additional virtual sensors.

[0258] Using virtual sensors provides the opportunity to compare existing industrial machines by measuring process parameters, since data would otherwise be lost.

[0259] [Using manually measured data as training data] []

[0260] The indicator Z' can be used as training data or as supplementary historical data.

[0261] [General-purpose computer] []

[0262] Figure 9 illustrates examples of general-purpose computer devices that can be used with the techniques described herein. Figure 9 is a diagram illustrating examples of general-purpose computer device 900 and general-purpose mobile computer device 950 that can be used with the techniques described herein. General-purpose computer device 900 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. General-purpose computer device 900 may correspond to computer system 100 of Figure 1. Computer device 950 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, driver assistance systems or onboard computers in vehicles (e.g., vehicles 401, 402, 403, see Figure 1), and other similar computing devices. For example, a user (e.g., a blast furnace operator) may use computer device 950 as a front end to interact with computing device 900. The components, their connections and relationships, and their functions shown herein are intended to be exemplary only and are not intended to limit the implementation of the inventions described and / or claimed in this document.

[0263] Computing device 900 includes a processor 902, memory 904, storage device 906, a high-speed interface 908 connected to memory 904 and high-speed expansion port 910, and a low-speed interface 912 connected to low-speed expansion port 914 and storage device 906. Each component 902, 904, 906, 908, 910, and 912 is interconnected using multiple buses and may be mounted on a common motherboard or otherwise. Processor 902 can process instructions for execution within computing device 900, including instructions stored in memory 904 or storage device 906, to display graphical messages for a GUI on an external input / output device, such as a display 916 coupled to high-speed interface 908. In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and memory types, may be used as appropriate. Furthermore, multiple computing devices 900 may be connected, each providing a portion of the necessary operation (e.g., as a server group, blade server group, or multiprocessor system).

[0264] Memory 904 stores information within computing device 900. In one embodiment, memory 904 is one or more static random access memories. In another embodiment, memory 904 is one or more non-volatile memories. Memory 904 may also be another form of computer-readable medium, such as a disk or optical disc.

[0265] Storage device 906 provides large-capacity storage for computing device 900. In one embodiment, storage device 906 may be or includes computer-readable media such as floppy disk devices, hard disk devices, optical disk devices, or magnetic tape devices, flash memory or other similar solid-state storage devices, or device arrays, including devices in a storage area network or other configuration. The computer program product may be tangibly implemented in a message carrier. The computer program product may also include instructions that, when executed, perform one or more methods, such as those described above. The message carrier is a computer or machine-readable medium, such as memory 904, storage device 906, or memory on processor 902.

[0266] A high-speed controller 908 manages bandwidth-intensive operations for computing device 900, while a low-speed interface 912 manages less bandwidth-intensive operations. This functional allocation is merely exemplary. In one embodiment, the high-speed controller 908 is coupled to memory 904, a display 916 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 910 that accepts various expansion cards (not shown). In another embodiment, the low-speed interface 912 is coupled to storage device 906 and a low-speed expansion port 914. The low-speed expansion port may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet) and may be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, or, for example, to networked devices, such as a switch or router, via a network adapter.

[0267] As shown in the accompanying drawings, computing device 900 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 920 or as multiple servers in a group. It can also be implemented as part of a rack-mount server system 924. Alternatively, it can be implemented in a personal computer, such as a notebook computer 922. Alternatively, components from computer device 900 can be combined with other components in mobile devices (not shown), such as computer device 950. Each of such devices can contain one or more of computer devices 900, 950, and the entire system can consist of multiple computer devices 900, 950 communicating with each other.

[0268] In addition to other components, computer device 950 includes, in particular, a processor 952, memory 964, input / output devices (such as a display 954), a communication interface 966, and a transceiver 968. Computer device 950 may also be provided with storage devices (such as microdrives or other devices) to provide additional storage. Each of components 950, 952, 964, 954, 966, and 968 is interconnected using multiple buses, and some components may be mounted on a common motherboard or, as appropriate, in other ways.

[0269] Processor 952 can execute instructions within computer device 950, including instructions stored in memory 964. The processor can be implemented as a chipset comprising individual and multiple analog and digital processors. For example, the processor can provide coordination for other components of computer device 950, such as control of the user interface, applications running on computer device 950, and wireless communication of device 950.

[0270] Processor 952 can communicate with the user via control interface 958 and display interface 956 connected to display 954. For example, display 954 may be a thin-film transistor liquid crystal display (TFT-LCD) or an organic light-emitting diode (OLED) display or other suitable display technology. Display interface 956 may include suitable circuitry for driving display 954 to present graphics and other information to the user. Control interface 958 can receive commands from the user and translate those commands for submission to processor 952. Additionally, an external interface 962 may be provided to communicate with processor 952 to enable near-field communication between computer device 950 and other devices. For example, external interface 962 may be provided for wired communication in some embodiments, or for wireless communication in others, and multiple interface interfaces may be used.

[0271] Memory 964 stores information within computer device 950. Memory 964 may be implemented as one or more computer-readable media, one or more static random access memories, or one or more non-volatile memories. Extended memory 984 may also be provided and connected to computer device 950 via an extended interface 982, which may include a Single In Line Memory Module (SIMM) card interface. Such extended memory 984 may provide additional storage space for device 950, or may store application programs or other information for computer device 950. Specifically, extended memory 984 may include instructions for performing or supplementing the processes described above, and may also include security messages. Thus, for example, extended memory 984 may function as a security module for computer device 950 and may be programmable with instructions that allow secure use of device 950. Furthermore, secure applications and additional messages, such as immutable identification information, may be provided via a SIMM card.

[0272] As discussed below, for example, the memory may include flash memory and / or NVRAM memory. In one embodiment, the computer program product is tangibly embodied in a message carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The message carrier is a computer-readable or machine-readable medium, such as memory 964, extended memory 984, or memory on processor 952, which may be received, for example, on transceiver 968 or external interface 962.

[0273] Computer device 950 can communicate wirelessly via communication interface 966, which may include digital signal processing circuitry if necessary. Communication interface 966 can provide communication under various modes or protocols, particularly such as GSM voice calls, SMS, EMS or MMS messages, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, etc. For example, such communication can occur via radio frequency transceiver 968. Additionally, short-range communication can occur, such as using Bluetooth, Wi-Fi, or other transceivers (not shown). Furthermore, GPS (Global Positioning System) receiver module 970 can provide computer device 950 with additional navigation and location-related wireless data, which may be used by applications running on computer device 950, as appropriate.

[0274] The computer device 950 can also communicate audibly using an audio codec 960, which can receive spoken messages from a user and convert them into usable digital messages. Similarly, the audio codec 960 can generate audible sounds for the user, such as through a speaker in, for example, the handset of the computer device 950. Such sounds may include sounds from voice telephone calls, recorded sounds (e.g., voice messages, music files, etc.), and sounds generated by applications operating on the computer device 950.

[0275] As shown in the accompanying drawings, the computer device 950 can be implemented in several different forms. For example, it can be implemented as a cellular phone 980. It can also be implemented as part of a smartphone 982, a personal digital assistant, or other similar mobile device.

[0276] The various implementations of the systems and technologies described herein can be achieved using digital electronic circuits, integrated circuits, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation as one or more computer programs that are executable and / or interpretable on a programmable system, including at least one programmable processor, at least one input device, and at least one output device. The programmable processor may be a dedicated or general-purpose processor coupled to receive data and instructions from and transfer data and instructions to a storage system.

[0277] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or in assembly / machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, apparatus, and / or device (e.g., disk, optical disc, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0278] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including sound, speech, or tactile input.

[0279] The systems and technologies described herein can be implemented in computing devices that include back-end components (e.g., as a data server), middleware components (e.g., an application server), or front-end components (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such back-end, middleware, or front-end components. Components of the system can be interconnected via any form of digital data communication or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0280] A computing system may include clients and servers. Clients and servers are generally remote to each other and typically interact via a communication network. The relationship between clients and servers is established by computer programs running on the respective computers and having client-server relationships with each other.

[0281] Several embodiments have been described. However, it will be understood that various modifications can be made without departing from the spirit and scope of the invention.

[0282] Furthermore, the logical flow depicted in the accompanying drawings does not require a specific order or sequence of events to achieve the desired result. Additionally, other steps may be provided, or steps may be removed from the described flow, and other components may be added to or removed from the described system. Accordingly, other embodiments are within the scope of the appended claims.

[0283] 111, 112, 121, 123: Industrial machines 113: Machine 173: Non-measured parameters 183: Measurement parameters 193: Operator 211: First Multivariate Measurement Time Series 212, 213, 223, 243: Multivariate Feature Time Series 221: Second Multivariate Measurement Time Series 231, 232: First Multivariate Feature Time Series 241: Second Multivariate Feature Time Series 251: Conversion Rules 272: Univariate Measurement Time Series 283: Configuration 311, 321, 322, 323: Converters 361: Discriminator 372, 373: Neural Networks 380: Selector Trainer 383: Selector 393: Adaptor 400, 500: Method 900, 950: Computer equipment 902: Processor 904: Memory 906: Storage devices 908: High-speed interface 910: High-speed expansion port 912: Low-speed interface 914: Low-speed expansion port 916, 954: Monitors 920: Standard Server 922: Notebook computer 924: Rack-mounted server system 952: Processor 956: Monitor Interface 958: Control Interface 960: Audio Codec 962: External Interface 964: Memory 966: Communication Interface 968: Transceiver 970: GPS receiver module 980: Cellular Phone 982: Smart Phone z: parameter Z': Indicator 411, 421, 431, 441, 451, 452, 462, 472: Steps 513, 523, 533: Steps

Claims

1. A computer-implemented method (400) for training a neural network (373), wherein, The network (373) is trained for later processing of a multivariate measurement time series (223) with measurement data representing an instance of a specific process parameter (183) of an industrial machine (123), and thus provides a parameter index (Z') for further process parameters (173, z) of the industrial machine (123), the method (400) comprising: receiving (411) a first multivariate measurement time series (211, 221) having historical measurement data from a first reference machine (111) and a second multivariate measurement time series (221) having historical measurement data from a physically different second reference machine (121); By processing the first multivariate measurement time series and the second multivariate measurement time series, a set of transformation rules (251 / 252) is obtained (421). These transformation rules (251 / 252) enable a converter module (311 / 321) to convert the first multivariate measurement time series and the second multivariate measurement time series (211, 221) into a first multivariate feature time series and a second multivariate feature time series (231, 241), respectively. The multivariate feature time series is invariant to the domain differences of the first reference machine and the second reference machine (111, 121). The converter module (322) applying the transformation rules (251 / 252) converts the first multivariate measurement time series (211, 221) into the first multivariate feature time series (232) (452). Receive (462) a univariate measurement time series (272) of measurement data (Z) with the further process parameters (173, z) from the first reference machine (112); and train (472) the neural network (372) with the first multivariate feature time series (232) at the input and with the univariate measurement time series (272) at the output.

2. The method (400) as described in claim 1, wherein obtaining (421) the set of transformation rules (251 / 252) is performed by the computer in a repetitive manner by the following steps: using the initial set of transformation rules, transforming (431) the first multivariate measurement time series (211) into a first multivariate feature time series (231), and transforming (431) the second multivariate measurement time series (221) into a second multivariate feature time series (241); identifying (441) the source of the first and second feature time series (231, 241) as the first and second multivariate measurement time series (211, 221) by the discriminator module (361); and modifying (451) the initial set of transformation rules until the discriminator module (361) is no longer able to identify the data source (1, 2).

3. The method as described in claim 2, wherein, The discriminator module (361) determines that it can no longer distinguish the data source by operating until the adversarial loss reaches its maximum value.

4. The method as described in claim 2, wherein, The steps of repeatedly acquiring (421) the set of transformation rules and training (472) the neural network (372) are repeated; wherein the converter module (311 / 321) further receives the network loss (LOSS) provided by the trained neural network (372), such that the repetition stops to minimize the network loss.

5. The method (400) as described in request item 2, wherein, The combination of the converter module (311 / 321) and the discriminator module (361) serves as a generative adversarial network operation.

6. The method (400) as described in request item 1, wherein, The method steps of receiving (411) the multivariate measurement time series (211, 221), obtaining (421) the set of transformation rules, transforming (452) and training (472) the neural network are performed on the multivariate measurement time series because the data selector module (383) has determined that the variable is related to the further process parameter (173, z).

7. The method (400) as described in request item 6, wherein, The data selector module (383) is trained by the selector trainer module (380), which, during the training (482) of the neural network, minimizes the loss function, causing the computer to repeat the method steps of acquiring, transforming, and training.

8. The method (400) as described in claim 1, wherein, Receiving (411) the first and second multivariate measurement time series (211, 221) with historical data is performed after data modality adaptation of the measurement data.

9. The method (400) as described in request item 8, wherein, This data modality adaptation is performed by steps selected from the following: analyzing measurement data with sound samples by classifying sound samples into categories, and analyzing measurement data with images by classifying images into categories.

10. The method (400) as described in request item 9, wherein, The adaptation is repeated for different targets, wherein the target is selected while minimizing the loss function during the training (482) of the neural network.

11. The method (400) as described in request item 1, wherein, The neural network (372) is a regression neural network.

12. The method (400) as described in request item 1, wherein, The measurement data is related to process parameters, which are physical properties selected from the following: temperature, pressure, chemical composition of materials inside the machine, visual appearance of parts of the industrial machine, and sound emitted by the industrial machine.

13. The method (400) as described in request item 1, wherein, The industrial machinery is selected from chemical reactors, metallurgical furnaces, containers, and engines.

14. A computer-implemented method (500) for operating the neural network (373) trained according to the method (400) of claims 1 to 13, the method (500) for operating the neural network comprising the steps of: receiving (513) a multivariate measurement time series (223) having measurement data of an industrial machine (123) under observation; converting (523) the multivariate measurement time series (223) into a multivariate feature time series (243); and operating (533) the neural network (373) to provide the parameter index (Z').

15. The method (500) as described in request item 14, wherein, The neural network is operated under conditions selected from the following: sensor failure, data connection failure, and minimizing human involvement in the measurement.

16. A computer program product, when loaded into the memory of a computer system and executed by at least one processor of the computer system, causes the computer system to perform the steps of a computer-implemented method according to any one of the preceding claims.

17. A computer system comprising a plurality of computer-implemented modules, wherein when executed by the computer system, the modules perform the steps of the computer-implemented method as described in any one of claims 1 to 15.

18. Use of a computer that performs the method described in claim 14 or 15 to obtain a parameter index (Z') representing a parameter (z) of an industrial machine (123).

Images