Ontology-based data evaluation in industrial plant

Abstract

Embodiments of the present disclosure relate to ontology-based data evaluation in an industrial plant. A computer-implemented method for evaluating at least one record of data, in particular measurement and / or plant specification data, characterizing at least one component of an industrial plant, and / or an operating state of at least one component of the industrial plant, the method comprises the steps of: converting, by a trained encoder, at least one record of data into a representation in an embedding space; comparing the representation with representations of semantic entities from a semantic information model in the same embedding space, where each semantic entity carries a semantic meaning with respect to the structure, construction, and / or function of the industrial plant, and the semantic information model comprises relationships between the semantic entities; and evaluating the semantic meaning of the record of data about the structure, construction, and / or function of the industrial plant from the result of the comparison.

Description

Technical Field

[0001] The present invention relates to the evaluation of data in industrial plants, in particular measurement and / or plant specification data. Such evaluation is very important both during the engineering phase and during normal plant operation. Background Art

[0002] Both during the engineering phase (process and automation P&A engineering) and during normal operation of an industrial plant, a large amount of data of various modalities is generated. For example, the I / O ports of a device and their connections can be listed in an I / O list. The devices and their interconnections can be shown in a process and instrumentation (P&ID) diagram. Control narratives and other documents can include detailed information about the functionality of individual entities (such as controllers). This can be enhanced, for example, by images and videos taken in an industrial plant. When the plant is operating, the process control system can record a large amount of measurement data, and the behavior of the plant can also be monitored through video recordings, audio recordings, etc.

[0003] The engineering and operational monitoring of a plant currently involves a large amount of manual activity. The reason is that not all information is available in a machine-understandable form that allows it to be put together. Instead, the information flows are independent, in a multitude of different forms and formats, and are not related to each other. Summary of the invention

[0004] It is therefore an object of the present invention to facilitate the automatic evaluation of a record of data characterizing at least one component of an industrial plant and / or its operating state.

[0005] This object is achieved by a computer-implemented method for evaluating at least one record of data according to the first independent claim and by a computer-implemented method for training an encoder according to the second independent claim. Further advantageous embodiments are specified in the respective dependent claims.

[0006] The invention provides a computer-implemented method for evaluating at least one record of data, in particular measurement and / or plant specification data, the at least one record of data characterizing at least one component of an industrial plant and / or its operating state. The data may be of any type or multiple suitable modalities. A non-exhaustive list of examples of data modalities generated during engineering and / or normal operation of an industrial plant includes:

[0007] An image of pixels in a regular grid assigning values ​​of at least one measured quantity (e.g., a physical quantity, or a symbol or text);

[0008] Assigning values ​​of at least one measured quantity to a point cloud of unconnected points in space;

[0009] a time series of values ​​of at least one measured quantity, in particular an audio signal;

[0010] textual narratives, especially narratives of control in industrial factories;

[0011] List of input / output relationships for industrial plants; and

[0012] Process and instrumentation diagrams of industrial plants or other component schemes.

[0013] In particular, a record of data may include one or more values ​​that are to be viewed together when evaluating the data. For example, in a time series of measured values, each value may be considered as one record of data, but any portion of the time series or even the entire time series may also be considered as one record of data. In another example, all pixel values ​​of an image belong to one and the same record of data.

[0014] During the method, a trained encoder converts at least one record of data into a representation in an embedding space. In this embedding space, regardless of the nature of the original data, the representation is a fixed arrangement of numerical values ​​that always has the same number of values, such as an embedding vector or tensor. In other words, the embedding space can be used to unify measurement data of different modalities.

[0015] The representation of the records of the data is compared with the representation of the semantic entities from the semantic information model in the same embedding space. In particular, the semantic information model may include concepts, relationships, rules, and other domain knowledge about the composition and / or operation of the industrial plant. The semantic information model can be viewed as a semantic entity graph of the industrial plant as a formal means of representing domain knowledge. Each such semantic entity carries a semantic meaning about the structure, construction and / or function of the industrial plant. The semantic information model includes the relationships between the semantic entities. The "domain" may include the industrial plant itself and its engineering and operation. An example of a semantic information model is an ontology.

[0016] For example, a semantic information model may include a hierarchy of types of equipment, and / or instances of a piece of equipment, and / or designations for such instances in a plant. For example, a semantic information model may include classes of abstract types, such as "tank", "reactor", or "valve", which expand into subtypes of the corresponding classes, such as "storage tank", "mixing tank", "stirred reactor", "two-way valve", or "three-way valve". Each instance of a class may be linked to its corresponding class. A designation of an instance may be linked to its instance.

[0017] Semantically meaningful representations are placed in the embedding space so that when two or more representations are compared according to a given metric or criterion, the result of the comparison corresponds to the similarity and / or relationship between the corresponding entities according to the semantic information model. For example, different types of tanks, different types of reactors, and different types of valves may be relatively close together in the embedding space in the clusters "tanks", "reactors", and "valves". But these clusters may be far apart from each other in the embedding space. In particular, a metric with a suitable semantic concept (such as cosine similarity) can be used for the comparison.

[0018] Based on the results of a comparison between a representation of the record of the data on the one hand and a representation of a semantic entity on the other hand, a semantic meaning of the record of data regarding the structure, construction and / or function of the industrial plant is evaluated.

[0019] Research has found that if data records from multiple modalities are converted into one and the same embedding space, there is already a tendency to place semantically similar items close together in the embedding space. However, this is difficult to control and provide insight into exactly what will be considered similar; this can neither be inspected nor set intentionally. Instead, this behavior is essentially "black box".

[0020] As a simple example, consider mapping the attributes of a person to a representation in an embedding space. Here, people can be grouped according to a variety of possibilities. For example, there may be clusters of men, women, and different groups of people. People can be considered similar based on their job title, physique, or hobbies. However, the total set of a person's attributes forms a multi-dimensional vector, and if we simply transform this vector into an embedding space, it is difficult or impossible to control what will be considered similar. In particular, it is difficult to enforce that one specific attribute should dominate.

[0021] By introducing a semantic information model and mapping the entities of the semantic information model to the same embedding space, representations with defined semantic meaning are formed in the embedding space. If two representations of a person are close together in the embedding space, this does not immediately reveal which attribute this is based on. However, if two representations of a person are not only close to each other, but also close to the representation of "amateur radio operator", "blockchain user" or "traffic quarreler", it can be inferred that these people may belong to the corresponding group corresponding to the representation. In particular, when many data records of a person are converted into representations, it is possible to form clusters in the embedding space around the representations from the semantic information model.

[0022] The conversion to the embedding space need not be done all at once, starting with the records of data. Rather, the results of any conventional processing of the records of data, such as outputting a single object recognition from an image, can also be forward converted into a representation of the records of data. Furthermore, the representation of the ontological entities available in the embedding space can be obtained once at the beginning and then used (as a reference) to process and perform comparisons on all further incoming data.

[0023] In a particularly advantageous embodiment, comparing a representation of a record of data with a representation of a semantic entity (or even representations of other records of data) comprises computing the value of a given metric that maps a tuple of two or more representations to a scalar. Prime examples of this include distance metrics between two representations in a multidimensional embedding space, such as a cosine similarity metric, and directions of one representation relative to another. For example, such directions may be defined as one or more angles relative to a given reference direction.

[0024] In this case, most preferably, for converting records of the data into the embedding space, an encoder is selected that has been trained so that a measure of tuples of two or more representations depends on the similarity of the corresponding records of the data. For example, as described above, the encoder may have been trained so that the distance between representations in the embedding space corresponds to a similarity score according to a given similarity measure.

[0025] In another particularly advantageous embodiment, at least two records of data of different modalities are converted into corresponding representations in the embedding space. These representations are compared. The results of this comparison are also taken into account when evaluating the semantic meaning of one or more records of the data.

[0026] In particular, in this way, different modalities can help each other in situations where it is difficult to understand the semantic meaning of the recording of data. For example, if a piece of equipment has poor image quality, there may be ambiguity as to whether the equipment is class A or class B. However, when taking into account information from other modalities, it can be inferred that this further information excludes class A, for example. It can then be determined that the equipment belongs to class B.

[0027] For example, in an industrial plant, after identifying the equipment symbol in the P&ID diagram, it can be inferred from the textual control narrative how a particular valve is connected to a particular reactor, namely via pipe XYZ and using controller ABC. Using the associated I / O list, the tank from which pipe XYZ originates can be identified in the plant layout as tank K50.

[0028] Thus, different modalities can be processed into machine-readable and machine-understandable connections between different modalities. In particular, ambiguities that any single modality may have can be clarified using other modalities, and different modalities can improve on each other. Furthermore, by quantifying the extent to which representations generated from different modalities are consistent with each other and with specific representations of semantic entities, the (uncertainty) or credibility of semantic meaning derived from one modality can be quantified with the aid of automatic consideration of another modality. Due to the mutual improvement of having multiple modalities, uncertainty can be reduced and credibility can be increased.

[0029] The transformation to an embedding space converts the differences and relations between the modalities and between any modality and the semantic information model into metric relations, e.g. in the form of distances and / or angles, between the records of the newly input data on the one hand and the semantic entities from the semantic information model on the other hand, in a common embedding space.

[0030] In another particularly advantageous embodiment, the records of the data for each modality are transformed using an encoder that is specific to that modality. In this way, each encoder can specialize in how to process the data for one specific modality, but the representation of all records of the data will still be unified in a single embedding space.

[0031] Even if the different encoders are unaware of each other and the existence of the semantic information model, records of data from different modalities related to entities that exist as semantic entities in the semantic information model can still be expected to be converted into representations in the embedding space that are consistent with each other and with the representations of the semantic entities from the semantic information model. On the surface, this seems surprising. However, all modalities actually detect one and the same entity. Therefore, even independent training for different encoders should cause each encoder to capture the same basic concept of this entity and encode it into a representation in the embedding space. The same applies to the relationships between entities: the relationship between them should be the same for all modalities and should converge to one relationship indicated in the semantic information model.

[0032] Different encoders may still have some common processing. For example, text (such as a control narrative) can be processed into a representation through a large language model (LLM). At the same time, the image can be processed into a text description (such as "can") through a (Siamese) image recognition or classification neural network, which in turn can be fed into the same LLM. The end result is that both the image and the control narrative are converted into representations in a common embedding space (i.e., the LLM). Therefore, the encoder used on the control narrative is just the LLM, while the encoder used on the image is a combination of an image recognition neural network and the LLM.

[0033] The representation of all semantic entities, as well as all consistently possible combinations of semantic entities from an ontology or other semantic information model (e.g., concept-relation-concept, and chains of multiple such triplets) in the embedding space defines what is known from the semantic information model.

[0034] For example, a semantic information model may include the concept

[0035] Cans,

[0036] Valves,

[0037] Reactor,

[0038] ●Capacity,

[0039] ●Threshold, etc.

[0040] and the relationship between concepts

[0041] ●With connection,

[0042] ●With input,

[0043] ●With output,

[0044] ●Has parameters, etc.

[0045] Then possible triple combinations include:

[0046] The tank has a connected reactor,

[0047] The tank has an output valve,

[0048] The tank has parameters thresholds etc.

[0049] In particular, these triples can be represented in the embedding space in the same way as single concepts (tank, valve, reactor, etc.), for example by means of an LLM that inputs the triples as "sentences".

[0050] The semantic information model, in turn, nominally describes all entities present in the industrial plant. Therefore, each representation of a record of data should be associated with at least one representation of a semantic entity. If a representation of a record of data is not associated with any representation of a semantic entity, this indicates that the content that produced the data record does not fit the semantic information model. However, the semantic information model needs to be consistent with the record of data that represents what is planned to implement the industrial plant or what actually happens in the industrial plant. An inconsistency indicates that something that was not planned in the semantic information model is about to happen (or has actually happened) and is therefore wrong, or that the semantic information model is not up to date. In the latter case, the engineering of the industrial plant may be correct, the industrial plant may be operating correctly, but there is no proper detailed catalog of what is in the industrial plant and what is happening.

[0051] One possible reason for inconsistency between the representation of the record of data and the representation of the semantic entity is that the entity that produced the record of the data does not belong to the industrial plant. For example, many different pieces of equipment, such as valves or controllers, are of various different types. Based on the semantic information model, it may be intentional to use a particular type in the industrial plant. Human error may cause a piece of equipment of the wrong type to enter the design of the plant or even into the actual physical plant. For example, confusion may occur during maintenance across multiple plants that use different types of valves.

[0052] Another possible cause of inconsistency is that the records of the data and / or their processed representation in the embedding space are not good enough to determine which semantic entity the records of the data are related to. This can be resolved by obtaining improved records of the data and / or improved representations of the records of the data.

[0053] If the reason for the inconsistency between the recorded representation of the data and any representation of the semantic entity cannot be resolved, the help of an operator or human expert can be sought to associate the recorded representation of the data with the representation of the semantic entity. For example, a particular type of device may change its appearance because its manufacturer believes that this will help sales. The functionality of the device may be exactly the same as the previous version, but when analyzing the camera images of the new version, it may not be mapped to a representation in the embedding space that is completely consistent with the representation of the corresponding semantic entity from the semantic information model. The operator or human expert can then provide information such as: despite the change in appearance, the device is still of the same type and should be recognized as the same type.

[0054] Therefore, in another particularly advantageous embodiment, one and the same representation of the record of data is compared with multiple representations of the semantic entity. In response to determining that the comparison indicates or does not indicate an association of the representation of the record of data with any of the multiple representations of the semantic entity according to a given criterion, an action is taken to provide the association. The action includes one or more of the following:

[0055] ● obtaining an improved record of data relating to the same aspects and / or entities of the industrial plant as the original record of the data;

[0056] ● Improved presentation of records for acquired data;

[0057] ● Modify and / or enhance the semantic information model; and

[0058] ● Prompt an operator or human expert to associate a recorded representation of the data with a representation of a semantic entity.

[0059] In particular, improved records of captured data may include:

[0060] ● modifying operating parameters of the recording measuring instruments and / or data processing equipment used to obtain the data; and

[0061] - Obtaining improved records of data using measuring instruments and / or data processing equipment operating according to modified operating parameters.

[0062] In particular, measuring instruments can be used during the operational phase of the life cycle of an industrial plant. Data processing equipment can be used earlier in the life cycle, i.e. in the engineering phase of an industrial plant. For example, data processing equipment can generate a P&ID diagram or a control narrative for an industrial plant.

[0063] In a visual example, if an object cannot be clearly identified when viewed under a microscope, the focus and lighting can be changed, and the object can be moved around until an image is clear enough to allow identification of the object.

[0064] Advantageously, obtaining an improved representation of a record of data may include:

[0065] modifying parameters characterizing the behavior of the trained encoder; and

[0066] An improved representation is obtained by feeding records of data to a trained encoder operating according to modified parameters.

[0067] For example, if the appearance of a piece of equipment has changed, an encoder configured to map images of the equipment to a representation in the embedding space can be further trained to still recognize the new version as the same piece of equipment and convert its images to the same representation in the embedding space.

[0068] For example, an encoder may include an image recognition base model that is a pre-trained model (whose weights are frozen) and only the weights of the last layer or last n layers are retrained or fine-tuned, such as on a dataset for the specific context of P&ID symbols.

[0069] Image recognition models can be built on standard approaches used in zero-shot / one-shot / few-shot learning (e.g., based on Siamese networks) for P&ID recognition and symbol detection, and can also implement state-of-the-art enhancement mechanisms such as symbol knowledge bases, or be processed and trained using stretched / squeezed / rotated, enlarged / reduced, noisy symbols.

[0070] Furthermore, for P&ID symbols, the model can be trained not only on the symbols themselves, but also on categories and styles.

[0071] Furthermore, semantically informed model-driven enhancements of these few-shot learning techniques (such as engineering domain-aware / context-aware symbol space restrictions, e.g., targeting specific P&A engineering sub-domains) can also be used here.

[0072] In another particularly advantageous embodiment, the acquisition of improved records of data, the acquisition of improved representations, the modification and / or enhancement of the semantic information model, and / or the prompting of operators or human experts with the goal of optimizing the overall consistency of the representations of the records of data with the representations of semantic entities is performed multiple times for a given set of records of data. The overall degree of consistency measured after the optimization is completed is a more objective measure of the degree of correspondence of the planned or realized industrial plant with a given semantic information model.

[0073] In another particularly advantageous embodiment, in response to determining that one and the same representation of a record of data can be associated with multiple representations of a semantic entity, the candidate associations are weighted according to a predetermined method, algorithm and / or rule set. In this way, additional knowledge about the industrial plant can be put to use in the form of methods, algorithms and / or rule sets. For example, the rule set can be content represented in an ontology or other semantic information model. For example, a method can include mathematical methods for uncertainty quantification, which is more generally applicable here and in other fields.

[0074] In particular, such additional knowledge can be used to resolve ambiguities. For example, some symbols in a P&ID drawing may be clearly identified as "what they actually represent", while other symbols may be identified as "could be X or Y". If the relationships in the embedding space between representations of records of data on the one hand and representations of semantic entities on the other hand acknowledge that a record of data can be related to multiple semantic entities, rules can confirm that a record is indeed related to one of those semantic entities, or exclude certain entities. In particular, if leveraging multiple available modalities is not sufficient to resolve ambiguity, a ranking of a list of entities (e.g., content or type) can be established based on their corresponding credibility (or uncertainty), allowing uncertainty quantification and improvement / training, for example by requesting dedicated human expert feedback.

[0075] Furthermore, the ontology representation may be linked to and thus use / feed a knowledge graph and / or graph database that contains some historical information, e.g. about similar plant setups, so that probabilistic / statistical learning can be built on top of a multimodal setup.

[0076] For example, at least one rule in a rule set may specify:

[0077] ● if two or more representations indicate the presence of one and the same entity in an industrial plant, it is more likely that the entity actually exists; and / or

[0078] ● if a specific entity exists, this means that the industrial plant as a whole is likely to be of a specific type; and / or

[0079] The presence of two specific entities, and / or types of entities, in an industrial plant are mutually exclusive; and / or

[0080] ● the presence of certain entities, and / or types of entities, is possible or impossible, or mandatory or prohibited in certain types of industrial plants; and / or

[0081] ●In the vicinity of a specific entity and / or type of entity, the existence of another specific entity and / or type of entity is possible or impossible.

[0082] For example, the settings may incorporate / consider the symbol probability in the environmental context / surroundings of other symbols for identification / classification of the symbol to be determined, such as:

[0083] ●Always / never have a reactor symbol near, before or after the pressure sensor symbol, etc.; or

[0084] ● In chemical plants, there is never an XYZ symbol; or

[0085] ●If there is ABC symbol identified, it means we are dealing with an oil and gas platform.

[0086] Furthermore, the set of actually possible / allowed symbols can be further reduced (and thus the chances of identifying the correct symbol from a smaller set of symbols) by incorporating domain / expert knowledge (e.g., represented in a semantic information model), such as:

[0087] If, after the XYZ tank symbol is identified with 100% certainty (or high probability), the set of possible other symbols in the corresponding P&ID diagram can be restricted / constrained (based on the expert knowledge represented by the ontology that ABC valves do not exist in a plant containing an XYZ tank); or

[0088] ● If the abc symbol is identified, this means that we are dealing with an oil and gas platform, because the ontological concept of the abc symbol indicates that this symbol only exists in the oil and gas context. This in turn can exclude other symbols or features.

[0089] Even P&ID-based symbol-to-symbol connections can be identified and digitized based on joint multimodal + ontology embeddings. For example, when a fuzzy line from symbol A to X or Y is found, then checking symbol A in the joint multimodal + ontology embedding will yield the known concept A, and when in turn checking what ontological relationships are from concept A to other concepts, then what other concepts and therefore symbols may be linked to symbol A and how becomes clear. This can resolve ambiguities, for example, X is possible, but Y is not.

[0090] In addition, other modalities and the relationships they establish may also contribute to the connection. For example, if a fuzzy line from symbol A to M or N is found in the P&ID, and it is known from the multimodal + ontology embedding that the associated control narrative mentions component M but does not mention N anywhere, then M is more likely to represent another symbol and thus indicate connectivity.

[0091] Of course, confidence levels / indicators or uncertainty measures can also be displayed / presented to system users as a result of the processed information. This may also include reasoning about these measures / indicators, such as "the match between the two..." or "something was ruled out because..." or "something odd was inconsistent because..." etc.

[0092] It is unlikely to achieve 100% accurate recognition of all symbols in a P&ID diagram or all elements in a control narrative or other modality. But even 80% should be beneficial / valuable to the user to achieve. Another 20% can be marked as "uncertain" / "unknown" so that a human engineer can point to them, so that he / she can (with little time and effort) further complete the machine-recognized 80% towards the full 100%. That is, every single association between a record of data and a semantic entity from a semantic information model discovered according to the method proposed here is an improvement. This improvement is not an "on-off" effect that will only occur if the entire industrial plant can be automatically interpreted by a machine.

[0093] In particular, the semantic meaning of the records of data about the structure, construction and / or function of an industrial plant may specifically include:

[0094] ● an indication of what type of equipment in the industrial plant and / or which specific piece of equipment the recording of the data relates to; and / or

[0095] An indication of how a piece of equipment in an industrial plant (1) is related to and / or connected to at least one other piece of equipment; and / or

[0096] - An indication of the operating status of at least one piece of equipment in an industrial plant, and / or of the industrial plant as a whole.

[0097] This information is among the most commonly sought when engineering and / or operating industrial plants.

[0098] In particular, in another particularly advantageous embodiment, based at least in part on the semantic meaning of the records of data about the structure, construction, and / or function of the industrial plant, a correction for the layout, topology, settings or specifications of the industrial plant and / or for the operating mode of the industrial plant can be determined. The goal of the correction can be to improve the structure, construction and / or function of the industrial plant. According to the corresponding correction, the improvement is achieved by changing the layout, topology, settings or specifications of the industrial plant and / or the operating mode of the industrial plant.

[0099] In another particularly advantageous embodiment, the determination of the correction comprises:

[0100] • determining a value for at least one given performance indicator of the industrial plant based at least in part on the semantic meaning of the record of data relating to the structure, configuration, and / or function of the industrial plant; and

[0101] - Determining modifications that may improve the value of the performance indicator when applied to the layout, topology, setup or specification of the industrial plant, and / or when applied to the operating mode of the industrial plant.

[0102] In this way, the engineering and / or operation of an industrial plant can be adapted to any goal that can be captured by a performance metric.

[0103] Examples of performance indicators that are particularly relevant to the engineering of industrial plants include

[0104] ● the feasibility of physically realizing an industrial plant; and / or

[0105] ●Inconsistencies in the layout, topology, setup or specifications of industrial plants; and / or

[0106] ●Reuse of energy in industrial plants.

[0107] For example, the reuse of energy can occur through heat exchange. For example, if a reactor requires thermal energy, it can be located near other plant components that have and release "waste heat".

[0108] In particular, in a primarily manual engineering process based on disparate information sources, it is easy to overlook that making changes based on one such information source may result in missing constraints that are clearly coming from another information source, since at any one information source is primarily looked at at any one time. For example, when switching to a new type of pump or valve in order to increase the production capacity of a plant, it may be overlooked that the size of the new pump or valve is slightly larger than the size of the old one and does not fit into the physical space available in the plant. With the presently proposed method, errors that may prevent physical implementation can be automatically discovered.

[0109] The same applies to inconsistencies in the layout, topology, setup or specification of an industrial plant. In particular, inconsistencies may arise if two mutually exclusive requirements occur for the same property, a piece of equipment and / or an operating mode of an industrial plant. For example, for one and the same pipe diameter, there may be a first requirement of "at least 10 cm" and a second requirement of "at most 8 cm".

[0110] As mentioned above, the main improvement compared to previous approaches for evaluating multimodal data is achieved through representations of entities coming from a semantic information model. These representations introduce a new modality that is completely transparent and controllable, on top of all other modalities. In this way, what happens in the embedding space no longer happens in a "black box", as mentioned above. One way to create these representations of semantic entities in the embedding space is to feed these semantic entities to an encoder that has been specially trained.

[0111] Therefore, the present invention also provides a computer-implemented method for training an encoder to create representations of semantic entities of an industrial plant in an embedding space.

[0112] During the training method, at least one semantic information model of the industrial plant is provided. As mentioned above, in particular, the semantic information model can be an ontology. In the semantic information model, each semantic entity carries a semantic meaning about the structure, construction and / or function of the industrial plant, and the semantic information model includes relations between the semantic entities.

[0113] With the help of the encoder to be trained, the records corresponding to the entities of the semantic information model are converted into representations in the embedding space. The records can be in any form. The only requirement is that the similarity of the records in any space from which they were originally created is commensurate with the similarity and / or relationship of the corresponding semantic entities in the context of the industrial plant.

[0114] The representations generated by the encoders to be trained are compared. The results of the comparison are rated according to the semantic information model by a predetermined loss function, corresponding to the degree of similarity and / or goodness of the relationship between the corresponding entities. The parameters characterizing the behavior of the encoders are optimized towards the goal of improving the rating by the loss function.

[0115] Through this contrastive learning, the encoder is empowered to create representations with known semantics and relations in the embedding space.

[0116] Training can use one or more ontologies. This not only benefits from these ontologies, but also generalizes to unseen ontologies. In particular, if the semantic information model is revised in order to improve an industrial plant, the encoder can still be used in order to create new representations of semantic entities in the embedding space.

[0117] Because they are computer-implemented, the methods can be implemented in the form of software. Therefore, the present invention also relates to a computer program with machine-readable instructions, which, when executed by one or more computers and / or computing instances, causes one or more computers and / or computing instances to perform the above-mentioned method. Examples of computing instances include virtual machines, containers or serverless execution environments in the cloud. The present invention also relates to a machine-readable data carrier and / or a download product with a computer program. A download product is a digital product with a computer program, which can be sold, for example, in an online store for immediate implementation and download to one or more computers. The present invention also relates to one or more computing instances with a computer program and / or a machine-readable data carrier and / or a download product. BRIEF DESCRIPTION OF THE DRAWINGS

[0118] In the following, the present invention is described using the accompanying drawings, but is not intended to limit the scope of the present invention. The accompanying drawings show:

[0119] Figure 1 : An exemplary embodiment of a method 100 for evaluating at least one record of data;

[0120] Figure 2 : Schematic diagram of analyzing multimodal data 2a, 2b with the aid of representations 7 of semantic entities 6 of a semantic information model 8;

[0121] Figure 3 : An exemplary embodiment of a method 200 for training an encoder 11 to create a representation 7 in a common embedding space 5. DETAILED DESCRIPTION

[0122] Figure 1 is a schematic flow chart of an embodiment of a method 100 for evaluating at least one record 2a, 2b of data, and in particular measuring data. The one or more records 2a, 2b of data characterize at least one component of an industrial plant 1 and / or its operating state.

[0123] In step 110 , the trained encoder 3 a , 3 b converts at least one record 2 a , 2 b of data into a representation 4 a , 4 b in the embedding space 5 .

[0124] According to block 110, an encoder 3a, 3b may be selected which has been trained such that a metric for a tuple of two or more representations 4a, 4b depends on the similarity of corresponding records 2a, 2b of the data.

[0125] According to block 112 , at least two records 2a , 2b of data of different modalities may be converted into corresponding representations 4a , 4b in an embedding space 5 .

[0126] According to block 112a, the records 2a, 2b of data for each modality may be converted using an encoder 3a, 3b specific to that modality.

[0127] In step 120, one or more representations 4a, 4b are compared with representations 7 of semantic entities 6 from a semantic information model 8 in the same embedding space 5 as the representations 4a, 4b. Each semantic entity 6 carries a semantic meaning 6* about the structure, construction, and / or function of the industrial plant 1. The semantic information model 8 also includes relations between the semantic entities 6.

[0128] According to block 121 , the comparison comprises computing the value of a given metric that maps two or more tuples representing 4a, 4b, 7 to a scalar.

[0129] According to block 122, representations 4a, 4b of multiple modalities may be compared to one another.

[0130] Alternatively or in combination therewith, according to block 123 , one and the same representation 4 a , 4 b of the records 2 a , 2 b of the data is compared with a plurality of representations 7 of the semantic entity 6 .

[0131] In step 130 , the semantic meaning 2 a *, 2 b * of the record of the data sought regarding the structure, construction and / or function of the industrial plant is evaluated from the result 120 a of the comparison performed in step 120 .

[0132] According to block 131 , if representations of multiple modalities have been compared according to block 122 , the results of this comparison may also be taken into account when evaluating the semantic meaning 2a*, 2b* of one or more of the records 2a, 2b of data.

[0133] According to block 132, if one or more representations 4a, 4b of a record 2a, 2b have been compared with a plurality of representations 7 of a semantic entity 6, it may be checked whether the comparison indicates an association of the representation 4a, 4b of the record 2a, 2b of the data with any of the plurality of representations 7 of the semantic entity 6. If this is not the case (truth value 0), different remedies may be tried, either alone or in combination:

[0134] • According to block 133, obtaining an improved record 2a, 2b of data relating to the same aspect and / or entity of the industrial plant 1 as the original record 2a, 2b of data. This may in turn include:

[0135] According to block 133a, modifying operating parameters of a measuring instrument and / or a data processing device used to obtain the record 2a, 2b of data; and

[0136] According to block 133b, an improvement record 2a, 2b is obtained using measurement instruments and / or data processing equipment operating according to the modified operating parameters.

[0137] According to block 134, an improved representation 4a, 4b of the record 2a, 2b of data may be obtained. This may in turn include:

[0138] According to block 134a, modifying parameters characterizing the behavior of the trained encoder 3a, 3b; and

[0139] An improved representation 4a, 4b is obtained by feeding the records 2a, 2b of data to a trained encoder 3a, 3b operating according to modified parameters.

[0140] ● According to block 135 , the semantic information model 8 is modified and / or enhanced.

[0141] • According to block 136 , an operator or human expert may be prompted to associate the representations 4a , 4b of the records 2a , 2b of the data with the representations 7 of the semantic entities 6 .

[0142] According to block 137, it may be checked whether one and the same representation 4a, 4b of a data record 2a, 2b can be associated with multiple representations 7 of a semantic entity 6, judging by the proximity of the representations 4a, 4b to the representation 7 of the semantic entity 6 or any other suitable association or relevance measure. If this is the case (truth value 1), in order to resolve the ambiguity, according to block 138, these candidate associations may be weighted according to a predetermined method, algorithm and / or rule set 9. According to block 138a, the rule set 9 may specify:

[0143] If two or more representations 4a, 4b indicate the presence of one and the same entity in the industrial plant 1, it is more likely that the entity actually exists; and / or

[0144] ● if a specific entity exists, this means that the industrial plant 1 as a whole is likely to be of a specific type; and / or

[0145] The presence of two specific entities, and / or types of entities, in the industrial plant 1 are mutually exclusive; and / or

[0146] The presence of certain entities, and / or types of entities, is possible or impossible, or mandatory or prohibited in a certain type of industrial plant 1; and / or

[0147] ●In the vicinity of a specific entity and / or type of entity, the existence of another specific entity and / or type of entity is possible or impossible.

[0148] exist Figure 1In the example shown, in step 140, the semantic meanings 2a*, 2b* of the records 2a, 2b of the data on the structure and / or function of the industrial plant 1 determined as described above are further utilized. At least partly based on the semantic meanings 2a*, 2b*, a revision 10 for the layout, topology, settings or specifications of the industrial plant 1, and / or for the operating mode of the industrial plant 1 is determined. In particular, the goal of the revision 10 can be to improve the industrial plant 1 and / or its operating mode with respect to any given standard. In step 150, according to the corresponding revision 10, the layout, topology, settings or specifications of the industrial plant 1, and / or the operating mode of the industrial plant 1 are changed.

[0149] According to block 141, a value for at least one given performance indicator 1a of the industrial plant 1 is determined based at least in part on the semantic meaning 2a*, 2b* of the records 2a, 2b of data about the structure, construction, and / or function of the industrial plant 1. According to block 142, a correction 10 that may improve the value of the performance indicator 1a when applied to the layout, topology, setup or specification of the industrial plant 1 and / or when applied to an operating mode of the industrial plant 1 can then be determined.

[0150] According to block 141a, the performance indicators 1a include one or more of the following:

[0151] ● the feasibility of physically realizing the industrial plant 1; and / or

[0152] ● inconsistencies in the layout, topology, setup or specifications of the industrial plant 1; and / or

[0153] ●Reuse of energy in industrial plants 1.

[0154] Figure 2 The creation of a multimodality of representations 4a, 4b in an embedding space 5 and the use of representations 7 of semantic entities 6 from a semantic information model 8 in the same embedding space 5 is illustrated.

[0155] exist Figure 2 In the example shown, the semantic entity 6 has a semantic meaning 6*. Three such semantic meanings 6* are shown as examples, namely tank 61, reactor 62 and valve 63. The record 12 of the entity 6 corresponding to the semantic information model 8 is mapped by the trained encoder 11 to a representation 7 in the embedding space 5 to serve as an "anchor point" with a defined semantic meaning 6*.

[0156] The records 2a of the data of the first modality include text information (here: control statements). They are converted by the first encoder 3a into representations 4a in the embedding space 5. Here, one and the same record 2a (here: one control statement) mentions all tanks 61, reactors 62 and valves 63, so that three representations 4a are generated in the embedding space 5.

[0157] The record 2b of the data of the second modality includes images (here: process and instrumentation P&ID diagrams). They are converted by the second encoder 3b into representations 4b in the embedding space 5. Here, the record 2b contains the complete P&ID diagram with all tanks 61, reactors 62 and valves 63, so three representations 4b are generated in the embedding space 5.

[0158] like Figure 2 As shown on the right side of , the final result is that the embedding space 5 includes:

[0159] ● Representation 7 of semantic entities 6;

[0160] • due to proximity to a representation 7 annotated with a semantic meaning 6* of a tank 61, a reactor 62 or a valve 63, a representation 4a of a text record 2a may be assigned a corresponding semantic meaning 6*; and

[0161] • Due to proximity to a representation 7 annotated with a semantic meaning 6* of a tank 61 , a reactor 62 or a valve 63 , the representation 4 a of the image record 2 b can be assigned a corresponding semantic meaning 6*;

[0162] Therefore, the corresponding text record 2a and image record 2b can be assigned the semantic meaning 6* determined in this way as their semantic meaning 2a*, 2b*.

[0163] Figure 3 is a schematic flow chart of an embodiment of a method 200 for training an encoder 11 to create representations 7 of semantic entities 6 of an industrial plant 1 in an embedding space 5 .

[0164] In step 210, at least one semantic information model 8 of the industrial plant 1 is provided. In the semantic information model 8, each semantic entity 6 carries a semantic meaning 6* about the structure, construction, and / or function of the industrial plant 1. Furthermore, the semantic information model 8 includes relations between the semantic entities 6.

[0165] In step 220 , the record 12 corresponding to the entity 6 of the semantic information model 8 is converted into a representation 7 in the embedding space by the encoder 11 to be trained.

[0166] In step 230 , the representations 7 thus generated are compared.

[0167] In step 240, the comparison result 230a is rated according to the semantic information model 8 with the aid of a predetermined loss function 13 in terms of the degree to which it corresponds to the similarity and / or relationship between the corresponding entities 6. The rating is marked with reference symbol 13a.

[0168] In step 250, parameters 11a characterizing the behavior of the encoder 11 are optimized toward the goal of improving the rating 13a through the loss function 13. The optimization may terminate once any suitable termination criteria, such as convergence, test accuracy, or a number of iterations, are met. The final optimized state of the parameters 11a is marked with reference symbol 11a*. This state also characterizes the final training state 11* of the encoder 11.

[0169] List of Reference Numerals

[0170] 1Industrial Factory

[0171] 1a Performance indicators of industrial plant 1

[0172] 2a, 2b data recording

[0173] 2a*, 2b* record the semantic meaning of 2a, 2b

[0174] 3a, 3b Encoder for recording data 2a, 2b

[0175] 4a, 4b Representation of records 2a, 2b in embedding space 5

[0176] 5 Embedded Space

[0177] 6 Semantic entities from the semantic information model 8

[0178] 6*Semantic meaning of semantic entity 6

[0179] 61 as a can with semantic meaning 6*

[0180] 62 as a reactor of semantic meaning 6*

[0181] 63 as a valve for semantic meaning 6*

[0182] 7 Representation of semantic entities 6 in the embedding space 5

[0183] 8 Semantic information model of entities 6 of factory 1 and their relationships

[0184] 9 Methods, algorithms and / or rule sets for resolving ambiguities

[0185] 10 Amendments to industrial plant 1 and / or operating mode

[0186] 11 is used to encode record 12 into the encoder representing 7

[0187] 11*The trained state of encoder 11

[0188] 11a Parameters characterizing the behavior of encoder 11

[0189] 11a* The final optimized state of parameter 11a

[0190] 12 records corresponding to semantic entity 6

[0191] 13 Loss Function

[0192] 13 Rating output by loss function 13

[0193] 100 Method for evaluating records 2a, 2b 110 Converting records 2a, 2b into representations 4a, 4b

[0194] 111 Select an encoder trained to match metrics and similarities 112 Select records 2a, 2b from multiple modalities

[0195] 112a Compare representations 4a, 4b with representation 7 using encoders 3a, 3b of a specific mode 120

[0196] 121 Calculate the value of a scalar metric

[0197] 122 will represent 4a, 4b compared with another

[0198] 123 compares a representation 4a, 4b with a representation 7

[0199] 130 Determine the semantics 2a*, 2b* sought

[0200] 131 Consider the results of comparing representations 4a and 4b

[0201] 132 Check the association with representation 7

[0202] 133 Get improved records 2a, 2b

[0203] 133a Modify the parameters of the measuring instrument / data processing device 133b Obtain an improved record 2a, 2b with the modified parameters 134 Obtain an improved representation 4a, 4b

[0204] 134a Modify the parameters of the encoder 3a, 3b 134b Obtain the representation 4a, 4b with the modified parameters

[0205] 135 Modify and / or enhance semantic information model 8

[0206] 136 Prompt operator or human expert

[0207] 137 Check associations with multiple representations 7 138 Weight candidate associations according to method, algorithm, rule set 9 138a Select rule set 9 with specific rules

[0208] 140 Confirmed Correction 10

[0209] 141 Determine the value of performance indicator 1a

[0210] 141a Select a specific performance indicator 1a 142 Determine the correction 10

[0211] 150Apply modifier 10 to industrial plant 1

[0212] 200 Method for training encoder 11 210 Providing semantic information model 8

[0213] 220 Convert record 12 to represent 7

[0214] 230 Comparison Representation 7

[0215] 230a compare the results of 230 240 use the loss function 13 to rate the results 230a 250 optimize the parameters 11a

Claims

1. A computer-implemented method (100) for evaluating data, and in particular at least one record (2a, 2b) of measurement and / or plant specification data, the at least one record (2a, 2b) of data characterizing at least one component of an industrial plant (1), and / or an operating state of at least one component of the industrial plant (1), the method comprising the following steps: converting (110) the at least one record (2a, 2b) of the data into a representation (4a, 4b) in an embedding space (5) by means of a trained encoder (3a, 3b); comparing (120) the representations (4a, 4b) with representations (7) of semantic entities (6) from a semantic information model (8) in the same embedding space (5), wherein each semantic entity (6) carries a semantic meaning (6*) about the structure, construction, and / or function of the industrial plant (1), and the semantic information model (8) includes relations between the semantic entities (6); as well as The semantic meaning (2a*, 2b*) of the record (2a, 2b) of data about the structure, the construction, and / or the function of the industrial plant (1) is evaluated (130) from the result (120a) of the comparison.

2. The method (100) of claim 1, wherein the comparison comprises computing (121) a value of a given metric that maps a tuple of two or more representations (4a, 4b, 7) to a scalar.

3. A method (100) according to claim 2, wherein an encoder (3a, 3b) is selected (111) which has been trained so that the metric for a tuple of two or more representations (4a, 4b) depends on the similarity of corresponding records (2a, 2b) of the data.

4. The method (100) according to any one of claims 1 to 3, wherein At least two records (2a, 2b) of data of different modalities are converted (112) into corresponding representations (4a, 4b) in the embedding space (5); These representations are compared (122); and The result of this comparison is also taken into account (131) when evaluating the semantic meaning (2a*, 2b*) of one or more of the records (2a, 2b) of data.

5. A method (100) according to claim 4, wherein the records (2a, 2b) of data for each modality are converted (112a) using an encoder (3a, 3b), said encoder (3a, 3b) being specific to that modality.

6. The method (100) according to any one of claims 1 to 5, further comprising: comparing (123) one and the same representation (4a, 4b) of said record (2a, 2b) of data with a plurality of representations (7) of a semantic entity (6); and In response to determining (132) that the comparison indicates or does not indicate an association of the representation (4a, 4b) of the record (2a, 2b) of data with any of the plurality of representations (7) of a semantic entity (6) according to a given criterion, Acquiring (133) an improved record (2a, 2b) of data relating to the same aspects and / or entities of the industrial plant (1) as the original record (2a, 2b) of the data; and / or obtaining (134) an improved representation (4a, 4b) of said record (2a, 2b) of data; and / or modifying and / or enhancing (135) the semantic information model (8); and / or An operator or human expert is prompted (136) to associate said representation (4a, 4b) of said record (2a, 2b) of data with a representation (7) of a semantic entity (6).

7. The method (100) of claim 6, wherein obtaining an improved record of data (2a, 2b) comprises: modifying (133a) operating parameters of a measuring instrument and / or a data processing device used to obtain said record (2a, 2b) of data; and The improved record (2a, 2b) of data is obtained (133b) using the measuring instrument and / or the data processing equipment operating according to the modified operating parameters.

8. The method (100) according to any one of claims 6 to 7, wherein obtaining (134) an improved representation (4a, 4b) of the record (2a, 2b) of data comprises: modifying (134a) parameters characterizing the behavior of the trained encoder (3a, 3b); and The improved representation (4a, 4b) is obtained (134b) by feeding the record (2a, 2b) of data to the trained encoder (3a, 3b) operating according to the modified parameters.

9. A method (100) according to any one of claims 6 to 8, wherein the acquisition (133) of improved records (2a, 2b) of data, the acquisition (134) of improved representations (4a, 4b), the modification and / or enhancement (135) of the semantic information model (8), and / or the prompting (136) of an operator or human expert with the goal of optimizing the overall consistency of the representation (4a, 4b) of the records of data with the representation (7) of semantic entities (6) are performed multiple times for a given set of records (2a, 2b) of data.

10. A method (100) according to any one of claims 1 to 9, wherein in response to determining (137) that one and the same representation (4a, 4b) of the record (2a, 2b) of data is capable of being associated with multiple representations (7) of a semantic entity (6), these candidate associations are weighted (138) according to a predetermined method, algorithm and / or rule set (9).

11. The method (100) according to claim 10, wherein at least one rule in the predetermined rule set (9) provides (138a): If two or more representations (4a, 4b) indicate the presence of one and the same entity in said industrial plant (1), it is more likely that said entity actually exists; and / or If a specific entity exists, this means that the industrial plant (1) as a whole is likely to be of a specific type; and / or The presence of two specific entities, and / or types of entities, in said industrial plant (1) are mutually exclusive; and / or In a certain type of industrial plant (1), the presence of certain entities, and / or types of entities, is possible or impossible, or mandatory or prohibited; and / or In the vicinity of a particular entity and / or type of entity, the existence of another particular entity and / or type of entity is possible or impossible.

12. The method (100) according to any one of claims 1 to 11, wherein the semantic meaning (2a*, 2b*) of the record (2a, 2b) of data about the structure, the construction, and / or the function of the industrial plant (1) specifically comprises: an indication of which type of equipment and / or which specific piece of equipment in the industrial plant (1) the record (2a, 2b) of data relates to; and / or an indication of how a piece of equipment in the industrial plant (1) is related to and / or connected to at least one other piece of equipment; and / or An indication of the operating status of at least one piece of equipment in the industrial plant (1), and / or of the industrial plant (1) as a whole.

13. The method (100) according to any one of claims 1 to 12, further comprising: determining (140) a layout, topology, setup or specification for the industrial plant (1), and / or a modification (10) for an operating mode of the industrial plant (1), based at least in part on the semantic meaning (2a*, 2b*) of the records (2a, 2b) of data regarding the structure, the construction, and / or the function of the industrial plant (1); and The layout, the topology, the settings or the specifications of the industrial plant (1), and / or the operating mode of the industrial plant (1) are changed (150) according to the corresponding amendment (10).

14. The method (100) of claim 13, wherein the determining of modifying (10) comprises: determining (141) a value for at least one given performance indicator (1a) of the industrial plant (1) based at least in part on the semantic meaning (2a*, 2b*) of the records (2a, 2b) of data regarding the structure, the construction, and / or the functioning of the industrial plant (1); and Determine (142) a correction (10) that may improve the value of the performance indicator (1a) when applied to the layout, the topology, the settings or the specifications of the industrial plant (1) and / or when applied to the operating mode of the industrial plant (1).

15. The method (100) of claim 14, wherein the performance indicator (1a) comprises (141a) one or more of the following: the feasibility of physically realizing said industrial plant (1); and / or inconsistencies in the layout, the topology, the setup or the specifications of the industrial plant (1); and / or Recycling of energy in the industrial plant (1).

16. The method (100) according to any one of claims 1 to 15, wherein the one or more records (2a, 2b) of data have one or more of the following modalities: An image of pixels in a regular grid assigning values ​​of at least one measurement quantity; Assigning values ​​of at least one measured quantity to a point cloud of unconnected points in space; a time series of values ​​of at least one measured quantity, in particular an audio signal; a textual description, in particular a control description of said industrial plant (1); a list of input / output relationships of the industrial plant (1); and A process and instrumentation diagram or other component scheme of the industrial plant (1).

17. A computer-implemented method (200) for training an encoder (11) to create a representation (7) of a semantic entity (6) of an industrial plant (1) in an embedding space (5), the method comprising the following steps: Providing (210) at least one semantic information model (8) of the industrial plant (1), wherein in the semantic information model (8), each semantic entity (6) carries a semantic meaning (6*) about the structure, construction, and / or function of the industrial plant (1), and the semantic information model (8) includes relations between the semantic entities (6); Converting (220) a record (12) corresponding to an entity (6) of the semantic information model (8) into a representation (7) in the embedding space (5) by the encoder (11); comparing (230) the representation (7) thus generated; By means of a predetermined loss function (13), rating (240) the results (230a) of the comparison according to the semantic information model (8) in terms of the degree of similarity and / or relationship between the corresponding entities (6); and Parameters (11a) characterizing the behavior of the encoder (11) are optimized (250) towards the goal of improving the rating (13a) by means of the loss function (13).

18. A computer program comprising machine-readable instructions which, when executed by one or more computers and / or computing instances, cause the one or more computers and / or computing instances to perform the method (100, 200) according to any one of claims 1 to 17.

19. A non-transitory machine-readable data carrier and / or a download product having a computer program according to claim 18.

20. One or more computers and / or computing instances having a computer program according to claim 18 and / or having a machine-readable data carrier and / or a download product according to claim 19.

Images