ML-based and knowledge-enhanced topology parsing, transformation and formalization for brown topologies

Abstract

The invention relates to ML-based and knowledge-enhanced topology parsing, transformation and formalization for brown topologies. The invention relates to a method. The method comprises the following steps: inputting a topological image into an artificial intelligence / machine learning AI / ML model; identifying and / or indicating embedded representations of symbols in at least segments of the topological image; comparing the result of the identification with at least one of the predetermined symbols and / or with at least one of the predetermined embedded representations indicating the symbol, wherein the predetermined symbol is associated with the predetermined characteristic; determining, based on a result of the comparison, at least one feature associated with the identified result; and deriving a structured representation from image content of the topological image based on a result of the determination.

Description

Technical Field

[0001] The present invention relates to an apparatus, method and system for topology parsing, transformation and formalization based on machine learning and knowledge enhancement for brownfield topology. Background Art

[0002] In the field of process and automation engineering (PAEng), it is expected that brownfield projects will continue to be the main focus. For such brownfield projects, "images", such as P&ID images or other types of topology images / screenshots, may often be obtained as input data. Summary of the Invention

[0003] However, in view of the above, old topology images and / or screenshots are usually processed by human engineers to allow further processing with the help of existing tools. Therefore, such processing by human engineers is resource-intensive and can be very error-prone.

[0004] Therefore, it is valuable to be able to automatically process information like old topology images and / or screenshots, i.e., parse it and transform it into a structured format using artificial intelligence / machine learning (AI / ML), and then use (traditional and / or deterministic) engineering tools again on this basis.

[0005] Therefore, it would be advantageous to support or automate (where possible) the human engineering step of processing old topology images and / or screenshots. This support or automation can be achieved with the help of modern AI / ML algorithms. Therefore, ideally, if the old topology images are input into such AI / ML models, a formal representation and / or model can be output, such as an ontology and / or graph format.

[0006] Thus, AI / ML can be used to (further) automate the processing of brownfield topologies, including parsing, transformation, and formalization. For example, according to the method overview, pre-trained ("out-of-the-box") AI / ML models, such as generative AI models such as large language models (LLMs), large visual models (LVMs), or large multimodal (LMMs), can be used and applied to convert (old) topology images into, for example, textual descriptions. Continuously asking users and / or experts for feedback on such obtained automated transformation / conversion results can allow for improving the AI / ML models, and thus can allow for improving the obtained automated transformation / conversion results.

[0007] In view of the above, in order to address one or more of these problems, in a first aspect, a method is provided, comprising inputting a topological image into an artificial intelligence / machine learning AI / ML model; identifying symbols and / or embedded representations of indicative symbols in at least a segment of the topological image; comparing the identified result with at least one symbol in predetermined symbols and / or with at least one embedded representation of the indicative symbol in a predetermined embedded representation indicating the predetermined symbol, wherein the predetermined symbol is associated with a predetermined characteristic; based on a result of the comparison, determining at least one characteristic associated with the identified result; and based on the determined result, deriving a structured representation from the image content of the topological image.

[0008] In this way, the method can allow brownfield topology information to be quickly processed using AI / ML in order to thereafter allow deterministic further processing of the structured / formalized information. Thus, at least manual (human) parsing / transformation can be reduced, which ties up a lot of resources and is very error-prone. Thus, support (or even full automation) of the topology parsing and transformation steps is provided, which can significantly reduce costs and / or errors. Moreover, the method can also allow a more deterministic and streamlined process that is not affected by the individual work performance of human workers, i.e., there are no differences between people. Additionally, if the system (including the AI / ML model) is trained on various symbolic variations, the method can also allow application to third-party topologies.

[0009] In an example, identifying may include identifying a symbol and / or an embedded representation by using an AI / ML model; and / or identifying may include identifying an embedded representation indicating a symbol and recognizing the symbol from the identified embedded representation; and / or comparing may include comparing the identified symbol and / or the identified embedded representation by using at least one of: a nearest neighbor search, a similarity search, a lookup table storing predetermined symbols and associated characteristics, and a representation indicating a predetermined embedded representation; and / or determining may include determining at least one characteristic by using an AI / ML model; and / or deriving may include deriving a structured representation by using an AI / ML model.

[0010] Moreover, in yet another example, determining may include determining at least one of the following as at least one characteristic: an input and / or output associated with the identified symbol and / or the identified embedded representation, and a predetermined symbol and / or predetermined embedded representation associated with the identified symbol and / or the identified embedded representation.

[0011] In addition, in another example, identification may include identifying at least two symbols and / or at least two embedded representations; and wherein determining may include determining the proximity of the identified symbols and / or the identified embedded representations to each other as at least one characteristic, and determining the relationship between the identified symbols and / or the identified embedded representations based on the result of the comparison and the determined proximity.

[0012] Additionally, in yet another example, the exporting may include at least one of parsing, identifying, transforming, and formalizing the image content.

[0013] Moreover, in yet another example, the method may further include receiving feedback regarding the identification result and / or the comparison result and / or the determination result and / or the derived result; and training and / or retraining the AI / ML model based on the feedback.

[0014] Furthermore, in yet another example, the method may further include generating a code and / or a control graphic based on the structured representation; and processing the image content based on the code and / or the control graphic.

[0015] Additionally, in yet another example, the topology image may show a brownfield topology and / or may include brownfield topology information.

[0016] Also, in yet another example, the structured representation may be at least one of a text description indicating image content of the topological image, an ontology format indicating image content, and a graph format indicating image content.

[0017] Additionally, in yet another example, the AI / ML model can be based on further using at least one of: joint embedding and / or nearest neighbor search, image classification and / or segmentation and proximity-based relationship identification, transformer-based neural networks, and an analysis pipeline including lookup tables and similarity searches.

[0018] Additionally, in yet another example, the symbol and / or predetermined symbol can be associated with at least one of: a valve, a tank, a sensor, a reactor, a boiler, a mixer, a separator, a controller and an input / inlet / output / outlet, and / or at least one feature and / or predetermined feature can include at least one of: an input of the symbol and / or predetermined symbol, an output from the symbol and / or predetermined symbol, a relationship of the symbol and / or predetermined symbol to another symbol and / or predetermined symbol, a connection of the symbol and / or predetermined symbol to another symbol and / or predetermined symbol, an indication as to whether the relationship is unidirectional or bidirectional and an indication as to whether the relationship is marked with text indicating the type of relationship and / or giving other details and / or attributes.

[0019] According to a second aspect, there is provided a control device for a topological image processing control system, the control device being configured to perform the method of the first aspect outlined above.

[0020] According to a third aspect, there is provided a topological image processing control system configured to perform the method of the first aspect outlined above.

[0021] According to a fourth aspect, an industrial automation system is provided, comprising the control device of the second aspect and / or the topological image processing control system of the third aspect.

[0022] The method of the first aspect may be computer implemented. Optional features of the first aspect may form part of any one of the second to fourth aspects mutatis mutandis.

[0023] According to a fifth aspect, there is provided a computing system configured to perform the method of the first aspect.

[0024] According to a sixth aspect, there is provided a computer program (product) comprising instructions which, when executed by a computing system, enable the computing system to perform the method of the first aspect or cause the computing system to perform the method of the first aspect.

[0025] According to a seventh aspect, there is provided a computer-readable (storage) medium comprising instructions that, when executed by a computing system, enable the computing system to perform the method of the first aspect or cause the computing system to perform the method of the first aspect. The computer-readable medium may be transient or non-transient, volatile or non-volatile.

[0026] By using AI / ML to (further) automate the processing of brownfield topology, as outlined above and described in more detail below in this article, that is, using AI / ML to quickly process brownfield topology information so as to allow deterministic further processing of the structured / formal information using our existing tools. At least manual (human) parsing / transformation can be reduced, which is resource-intensive and very error-prone. Therefore, it is possible to support (or even fully automate) the topology parsing and transformation steps, which can allow for significant reductions in costs and / or errors. In addition, it can enable a more deterministic and streamlined process, e.g., no differences between people. Furthermore, if the system (including the AI / ML model) is trained on various symbolic variations, it can be applied to third-party topologies.

[0027] By “(process) automation system” is meant an industrial plant or production plant comprising one or more pipelines, production lines and / or assembly lines for transforming one or more educts into products and / or assembling one or more components into a final product.

[0028] As used herein, the term "obtaining" may include, for example, receiving from another system, device, or process; receiving via interaction with a user; loading or retrieving from storage or memory; measuring or capturing using a sensor or other data acquisition device.

[0029] As used herein, the term "determining" encompasses various actions and may include, for example, computing, calculating, processing, deriving, investigating, looking up (e.g., looking up in a table, a database, or another data structure), ascertaining, etc. Furthermore, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. Furthermore, "determining" may include resolving, selecting, choosing, establishing, etc.

[0030] The indefinite article "a" or "an" does not exclude a plurality. In addition, the articles "a" and "an" as used herein should generally be construed to mean "one or more" unless specified otherwise or clear from the context to be directed to a singular form.

[0031] Unless otherwise specified or obvious from the context, the phrases "one or more of A, B, and C," "at least one of A, B, and C," and "A, B, and / or C" as used herein are intended to mean all possible permutations of one or more of the listed items. That is, the phrase "A and / or B" means (A), (B), or (A and B), while the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C).

[0032] The term "comprising" does not exclude other elements or steps. In addition, the terms "including", "comprising", "having" and the like can be used interchangeably in this document.

[0033] The present invention may include one or more aspects, examples or features independently or in combination, regardless of whether they are specifically disclosed in combination or independently.Any optional feature or sub-aspect of one of the above aspects may be appropriately applied to any other aspect.

[0034] These and other aspects will become apparent and elucidated with reference to the detailed description provided hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] A detailed description will now be given, by way of example only, with reference to the accompanying drawings, in which:

[0036] Figure 1 illustrates an example overview of an AI / ML model-based system including a feedback loop that derives a structured representation from image content of a topological image;

[0037] Figure 2 The diagram illustrates that by using, for example, Figure 1An example of a system for converting a topological image into a topological graph; and

[0038] Figure 3 An example method for deriving a structured representation from image content of a topological image is illustrated. DETAILED DESCRIPTION

[0039] To address the shortcomings of the prior art outlined above, this document discloses the use of AI / ML models to derive formal and / or structured representations from image content (including parsing, identifying, transforming and / or formalizing image content), such as, for example, ontology representations and / or graph representations, which can be used by different engineering tools. As a supplement to the AI / ML model, a knowledge representation component can support and provide higher confidence by providing a lookup table including standard symbols and / or connections between standard symbols, which can be used for similarity searches, consistency checks and / or uncertainty reduction. Moreover, an integrated feedback component can further and / or continuously improve the system.

[0040] Reference Figure 1 , Figure 1 Illustrated is an example overview of an AI / ML model-based system including a feedback loop that derives a structured representation from image content of a topological image.

[0041] Thus, an AI / ML and knowledge-augmented topology parsing, topology identification, topology transformation, and topology formalization for topological images, including, for example, brownfield topology, is disclosed. For example, but not limited to, possible AI / ML approaches utilizing this approach are (1) joint embedding and nearest neighbor search, (2) image classification and / or segmentation and proximity-based relationship identification, (3) transformer-based neural networks, and (4) analysis pipelines using lookup tables and similarity search. The methods outlined under items (1), (2), and (3) can use pre-trained ("out-of-the-box") models, such as, for example, LLM-based embeddings, and such pre-trained models can be applied to convert topological images into structured representations, such as, for example, text descriptions. Continuously asking users (expert users) for feedback on the results obtained from such pre-trained models allows for further improvement of the AI / ML models. Thus, disclosed herein is the training of AI / ML models to convert topological images into structured representations, such as, for example, text descriptions.

[0042] Therefore, refer to Figure 1 A basic idea of the present disclosure is that it is possible to understand that a topological image (or screenshot) is input to an AI / ML model and obtain a formal model and / or formal representation of the input topological image as output. This formal model and / or formal representation can be an ontology or a graph, such as the DEXPI format. Thereafter, the formal model and / or formal representation can be transformed using existing (deterministic) engineering tools.

[0043] In the following, according to the reference Figure 1 The example outlines several processing steps.

[0044] In step 1, a topology image 110 (or 120) (reference numeral 110 refers to a first topology image, which shows the topology / system as an offline system (indicating a cross); reference numeral 120 refers to a second topology image, which shows the same topology / system as the first topology image, but as an online system (without indicating a cross)) is given as input to an AI / ML model (or AI / ML / analysis pipeline) 130 to derive a description in the form of, for example, text from the topology image 110 (or 120).

[0045] like Figure 2 , the topological image 110 (or 120) or the image content of the topological image 110 (or 120) shows an inlet 210 connected to a storage tank 230. The connection between the inlet 210 and the storage tank 230 includes a valve 220. Furthermore, water 240, crude oil 260, and gas 280 can be supplied to the storage tank 230 via corresponding control valves 250, 270, and 290.

[0046] In the topology image 110 (or 120), for example, the storage tank 230, the valve 220, and the control valves 250, 270, and 290 are illustrated by corresponding symbols. The AI / ML model 130 identifies these symbols from (the image content of) the topology image 110 (or 120). Figure 1 As shown, a similarity search can be performed on the identified symbol. More specifically, the identified symbol can be matched with predetermined and / or standard symbols stored in a lookup table 140 using a nearest neighbor search (the predetermined symbols can represent standard symbols). The lookup table 140 can store characteristics of the predetermined and / or standard symbols, such as, for example, corresponding inputs and / or outputs. Furthermore, the lookup table 140 can also store information about interconnections between certain predetermined and / or standard symbols, as other characteristics. Thus, after matching, the identified symbol is associated with characteristics of the matched predetermined and / or standard symbol.

[0047] It should be noted that such matching is not limited to one-to-one matching. Rather, in a one-to-many matching, one identified symbol may be matched with several predetermined and / or standard symbols, e.g., where the identified symbol may represent an old, obsolete, or outdated symbol, that symbol (its function) may now be represented by several predetermined and / or standard symbols. In a many-to-one matching, several old, obsolete, or outdated symbols (their functions) may now be represented by only one predetermined and / or standard symbol. In a similar manner, many-to-many matching may also be possible.

[0048] The matching identification symbol (and its characteristics) may also be given as input to the AI / ML model 130 .

[0049] In step 2, the AI / ML model 130 may be composed of, but not limited to, one of the following:

[0050] A pre-trained transformer-based neural network is used to train the neural network to convert the topological image into a formal model such as DEXPI or ontology. These models can directly convert the topological image 110 (or 120) (the image content of the topological image) into a structured representation, such as text / description text.

[0051] Use of lookup tables and / or embeddings, wherein the lookup table 140 and / or embeddings consist of predetermined and / or standard symbols for, for example, tanks and valves (see Figure 2 ). The topological image 110 (or 120) can be segmented to identify different symbols in the image (in the image content). Similarity searches can be used to match the identified symbols with existing predetermined and / or standard symbols (stored in the lookup table 140). The AI / ML model 130 can then derive the input and / or output of the predetermined and / or standard symbols (and therefore, the input and / or output of the corresponding matching identified symbols) from the information stored in the lookup table 140. This lookup table approach can improve the accuracy of verifying and / or matching the identified symbols. Moreover, the use of a lookup table can provide further useful information about certain symbol characteristics, such as, for example, which other element(s) (such as, for example, "water", "crude oil", "gas") and / or symbol(s) are (usually) connected, can be connected and / or must not be connected to a certain symbol.

[0052] Using (traditional) AI / ML, where image classification and / or segmentation can be used to identify different symbols in an image (in the image content). Based on the proximity of the identified symbols to each other, relationship(s) and / or link(s) (such as connection(s) or connection(s) rules) can be identified, for example, by verifying and / or matching the relationship(s) and / or link(s) with standard symbols. For example, the identified symbols and their links can be matched with standard text or ontology.

[0053] In step 3, the user 150 may then provide feedback to the generated structured representation (such as, for example, a text description), which may be incorporated into the AI / ML model 130 to train and / or retrain the AI / ML model 130 for further use.

[0054] In step 4, once a structured representation of the image (image content) (such as a textual description) is available, this structured representation can be used for various applications such as, but not limited to, code generation or control graphics generation. Examples of codes include IEC 61131-3 and IEC 61499.

[0055] Now refer to Figure 2 , Figure 2 The diagram illustrates that by using, for example, Figure 1 An example of a system that converts a topology image into a topology graph.

[0056] Topological image 120 is similar to the above reference Figure 1 For example, the topological image 120 shows a tank 230 connected to an inlet 210 via a valve 220. Then, as indicated in step S210, if the topological image 120 is obtained by, for example, using the above reference Figure 1 If the system is converted as outlined above, a topology diagram can be obtained. Such a topology diagram can include a first element reading "valve" (e.g., a first circle) and a second element reading "tank" (e.g., a second circle), wherein both elements are connected by an arrow pointing from "valve" to "tank", indicating that the tank is fed via the valve. That is, for example, a topology diagram can illustrate the flow from the flowmeter according to the flowmeter. Figure 1 How is the tank 230 of the topological image 110 (or 120) of 2 connected to the Figure 1 2. It should be noted that, for example, a textual description may be used instead of (or in addition to) the topological image 110 (or 120), wherein such a textual description may, for example, describe how the tank 230 is connected to the valve 220. Thus, in doing so, the topological image may be converted into a structured representation.

[0057] Now refer to Figure 3 , Figure 3 An example method for deriving a structured representation from image content of a topological image is illustrated.

[0058] In step S310 , the method includes inputting the topological image into an artificial intelligence / machine learning AI / ML model.

[0059] It should be noted that this input can represent the above reference Figure 1 At least a portion of this input outlined in step 1. Furthermore, this topological image may represent the above referenced Figure 1 and 2 Such a topological image 110 or 120 is outlined. Additionally, the AI / ML model may represent the Figure 1 and 2The AI / ML model 130 is described in detail. It should also be noted that the AI / ML model is not limited to any specific AI / ML model(s). Rather, it is suitable for use with reference to Figure 3 Any (present and / or future) AI / ML model in the outlined method is intended to be covered by such AI / ML model as mentioned with respect to step S310 .

[0060] In step S320 , the method further comprises identifying an embedded representation of the symbol and / or indicator in at least a segment of the topological image.

[0061] It should be noted that such identification may include reference to Figure 1 At least a portion of such identification as outlined in step 2. Furthermore, such symbol may represent the above reference Figure 1 and 2 At least one of the symbols outlined may thus be, for example, a tank, a valve or a control valve, without however being restricted thereto. On the contrary, the term symbol is to be understood as meaning and / or comprising any type and / or kind of element and / or component shown in the (topological) image. Such an element and / or component may have a certain (technical) function, may have certain inputs / outputs, may be connected to another such element and / or component and / or may have any relationship (e.g. locally, technically, functionally) with another such element or component. An embedded representation may be understood as meaning the embedding of one or several symbols in an arrangement (e.g. a structural arrangement) of one or several other symbols and / or in one or several connections / interconnections with, for example, one or several such other symbols, without being restricted thereto.

[0062] In step S330, the method further comprises comparing the identified result with at least one of predetermined symbols and / or with at least one embedded representation of an indicative symbol of predetermined embedded representations indicative of predetermined symbols, wherein the predetermined symbol is associated with the predetermined characteristic.

[0063] It should be noted that this comparison can be understood as matching and can include the above reference Figure 1 At least a portion of this matching (or verification) outlined in steps 1 and 2. The predetermined symbol may represent the above reference Figure 1 and 2 The predetermined embedded representation may be understood to mean, for example, a known and / or stored and / or described (e.g. by text) and / or standard embedded / embedded representation, but is not limited thereto. Furthermore, the predetermined feature may mean the above referenced Figure 1At least a portion of such features outlined in steps 1 and 2. Thus, for example, such predetermined features include, but are not limited to, inputs and / or outputs associated with a symbol (e.g., a tank), connections to other symbols (e.g., other tanks or valves), possible / allowed connections to other symbols, and prohibited connections to other symbols. The match may be a one-way match (matching the identified symbol to the predetermined symbol or matching the predetermined symbol to the identified symbol), or may be a two-way / bidirectional match (matching the identified symbol to the predetermined symbol and matching the predetermined symbol to the identified symbol). Furthermore, the result of the identification may include at least one identified symbol and / or an embedded representation of at least one identification.

[0064] In step S340 , the method further includes determining at least one characteristic associated with the identified result based on the result of the comparison.

[0065] It should be noted that such determination may include, for example, assigning characteristics of a comparison / matching standard tank (such as, for example, certain inputs and / or outputs of such a comparison / matching standard tank) to the representation symbol / associating with the identification symbol (i.e., being determined to be valid (at least within a certain amount) for the identification symbol) for example, as a tank.

[0066] In step S350 , the method further comprises deriving a structured representation from the image content of the topological image based on the determined result.

[0067] It should be noted that such exporting may represent at least a portion of such conversion of a topological image into a topological graph, as described above with reference to Figure 2 Furthermore, the structured representation may indicate the symbols shown in the topology image and / or the connections / interconnections of these symbols.

[0068] Moreover, according to an example, identification may include identifying a symbol and / or an embedded representation by using an AI / ML model; and / or identification may include identifying an embedded representation indicating a symbol and recognizing the symbol from the identified embedded representation; and / or comparison may include comparing the identified symbol and / or the identified embedded representation by using at least one of: a nearest neighbor search, a similarity search, a lookup table storing predetermined symbols and associated characteristics, and at least one of a representation indicating a predetermined embedded representation; and / or determination may include determining at least one characteristic by using an AI / ML model; and / or deriving may include deriving a structured representation by using an AI / ML model.

[0069] It should be noted that similarity search can be expressed as above with reference to Figure 1 This similarity search outlined in step 1. Further, the lookup table can represent the above reference Figure 1Such a lookup table 140 is outlined. Moreover, this representation may be understood to also represent a (same or different) lookup table, e.g., from which a predetermined embedded representation may be accessed. Such a predetermined embedded representation may be associated with a characteristic in that the predetermined embedded representation indicates a predetermined symbol associated with the characteristic.

[0070] Further, in yet another example, determining may include determining as at least one characteristic an input and / or output associated with the identified symbol and / or the identified embedded representation and at least one of a predetermined symbol and / or a predetermined embedded representation associated with the identified symbol and / or the identified embedded representation.

[0071] It should be noted that such a predetermined symbol associated with an identification symbol may represent a valve as a predetermined symbol, which is associated with the tank as the identification symbol by being connected to the tank. Alternatively, such a relationship may indicate a prohibited connection, i.e., the valve may not be connected to the tank. The relationship may be a unidirectional relationship (the relationship between the identification symbol and the predetermined symbol, or the relationship between the predetermined symbol and the identification symbol) or a bidirectional relationship (the relationship between the identification symbol and the predetermined symbol, or the relationship between the predetermined symbol and the identification symbol).

[0072] In addition, in another example, identification may include identifying at least two symbols and / or at least two embedded representations; and wherein determining may include determining as at least one characteristic the proximity of the identified symbols and / or the identified embedded representations to each other, and determining the relationship between the identified symbols based on a result of the comparison and the determined proximity.

[0073] For example, refer to Figure 2 , it can be determined that the identified inlet 210, the identified valve 220, and the identified tank 230 are in a certain proximity to each other. For example, but not limited to, such proximity can be determined as follows, since it can be determined how close the components (e.g., symbols) in the topological image are arranged, for example in centimeters or a number of pixels. Alternatively and / or additionally, it can be determined that the components and / or (a plurality of) individual components (e.g., with reference to Figure 2Whether such arrangement of the inlet 210, valve 220, tank 230) is connected by means of pipes / tubes (e.g., indicating a certain (multiple) material / energy / (multiple) signal flow), whether it is represented as a solid line / dashed line, and / or whether it is shown in different colors (e.g., indicating different groupings, relationships and / or interconnections). Therefore, the distance between the components (inlet 210, valve 220, tank 230) can be evaluated compared to the entire length of the topological image (e.g., the number of pixels or the length in centimeters when the topological image is generated on the screen). Therefore, if it can be determined that the components (inlet 210, valve 220, tank 230) are arranged in a segment of the topological image that forms, for example, a quarter or less of the topological image, the proximity can be considered relevant because a connection between at least some of these components (inlet 210, valve 220, tank 230) may be expected / possible.

[0074] Additionally, in yet another example, the exporting may include at least one of parsing, identifying, transforming, and formalizing the image content.

[0075] It should be noted that "parsing" may mean loading and analyzing, segmenting and structuring and understanding (from a computer's perspective) the image (i.e., pixels and their relationships to each other), "recognition" may include recognizing symbols / components in the image, and "transformation and formalization" may mean that it may first obtain an abstract representation (e.g., embedding) and thereafter a graph / structured representation (e.g., xml / owl output).

[0076] Moreover, in yet another example, the method may further include receiving feedback regarding the identification result and / or the comparison result and / or the determination result and / or the derived result; and training and / or retraining the AI / ML model based on the feedback.

[0077] It should be noted that this feedback loop can be represented by the above reference Figure 1 At least part of such a feedback loop outlined in step 3. Additionally, the matched, determined and / or derived result(s) may be evaluated by the AI / ML model with respect to confidence and / or accuracy (or any component for evaluating the confidence and / or accuracy of the result(s)) before being provided to, for example, a user for feedback reasons. The user may then give feedback on such result(s) if the result(s) may have a confidence value and / or accuracy value that is below a predetermined confidence threshold and / or below a predetermined accuracy threshold. Feedback on result(s) having a confidence value and / or accuracy value that is below the corresponding threshold may be fed back to the AI / ML model for training and / or retraining purposes. No feedback may be required if the confidence value and / or accuracy value of the result(s) is equal to or above the corresponding threshold.

[0078] Furthermore, in yet another example, the method may further include generating a code and / or a control graphic based on the structured representation; and processing the image content based on the code and / or the control graphic.

[0079] It should be noted that such code and / or graphics generation may represent the above referenced Figure 1 At least a portion of such code and / or graphics is generated as outlined in step 4. Examples of generated codes include, but are not limited to, IEC 61131-3 and IEC 61499.

[0080] Additionally, in yet another example, the topology image may show a brownfield topology and / or may include brownfield topology information.

[0081] It should be noted that this topological image can represent a reference Figure 1 and 2 A topological image 110 or 120 is shown.

[0082] Also, in yet another example, the structured representation may be at least one of a text description indicating image content of the topological image, an ontology format indicating image content, and a graph format indicating image content.

[0083] It should be noted that this diagram format can represent reference Figure 2 This topology is shown in the figure.

[0084] Additionally, in yet another example, the AI / ML model can be based on further using at least one of: joint embedding and / or nearest neighbor search, image classification and / or segmentation and proximity-based relationship identification, transformer-based neural networks, and an analysis pipeline including lookup tables and similarity searches.

[0085] Additionally, in another example, the symbol and / or predetermined symbol can be associated with at least one of the following, but not limited to: a valve, a tank, a sensor, a reactor, a boiler, a mixer, a separator, a controller and an input / inlet / output / outlet, and / or at least one feature and / or predetermined feature can include at least one of the following: an input of the symbol and / or predetermined symbol, an output from the symbol and / or predetermined symbol, a relationship of the symbol and / or predetermined symbol to another symbol and / or predetermined symbol, a connection of the symbol and / or predetermined symbol to another symbol and / or predetermined symbol, an indication as to whether the relationship is unidirectional or bidirectional and an indication as to whether the relationship is marked with text indicating the type of relationship and / or giving other details and / or attributes.

[0086] Furthermore, according to various examples, a control device for a topological image processing control system is disclosed, wherein the control device is configured to perform the above-mentioned Figures 1 to 3For example, the control device may include a processor and a memory for storing instructions that, when executed by the processor, may cause the control device to perform, for example, the method described above with reference to Figure 3 For this execution, the control device may include several functional parts, such as for executing the Figure 3 The input portion of the process of step S310 is used to execute the Figure 3 The identification part of the process of step S320 is used to perform the Figure 3 The comparison part of the process of step S330 is used to perform the Figure 3 The determination portion of the process of step S340 and the method for executing the Figure 3 Further, such a part can be understood as representing a component for performing a specific function or representing a part configured to perform a specific function.

[0087] Further, according to various examples, a topological image processing control system is disclosed, which is configured to perform the above-referenced Figures 1 to 3 Summarize the method and / or method steps.

[0088] Furthermore, according to various examples, an industrial automation system is disclosed that includes a control device and / or a topological image processing control system.

[0089] Further, according to various examples, a computer readable medium is disclosed that includes instructions that, when executed by a computing system, cause the computing system to perform the above-referenced Figures 1 to 3 Summarize the method and / or method steps.

[0090] Additionally, according to various examples, reference Figures 1 to 3 The outlined methods and / or method steps may be computer-implemented.

[0091] Further, according to various examples, a computing system is provided, which is configured to perform the above reference Figures 1 to 3 Summarize the method and / or method steps.

[0092] Furthermore, according to various examples, a computer program (product) is disclosed that includes instructions that, when executed by a computing system, enable the computing system to perform the above-referenced Figures 1 to 3 The methods and / or method steps outlined above or causing a computing system to perform the Figures 1 to 3 The method and / or method steps outlined herein may comprise a computer-readable medium comprising instructions of the computer program product.

[0093] Any unit, module, circuit system or method described herein can be implemented using hardware, software and / or firmware configured to perform any operation described herein. Hardware can include one or more processor cores, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs) and the like. Software can be implemented as software packages, codes, instructions, instruction sets and / or data recorded on at least one transient or non-transient computer-readable storage medium. Firmware can be implemented as code, instructions or instruction sets and / or data hard-coded in a memory device (e.g., a non-volatile memory device).

[0094] If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or sent via a computer-readable medium. Computer-readable media include computer-readable storage media. Computer-readable storage media can be any available storage medium that can be accessed by a computer. By way of example and not limitation, such computer-readable storage media can include flash memory storage media, RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer. As used herein, disks and optical disks include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks and Blu-ray disks (BDs), wherein disks typically copy data magnetically, while optical disks typically copy data optically using lasers. Further, propagation signals can be included within the scope of computer-readable storage media. Computer-readable media also include communication media, which include any media that facilitates transferring a computer program from one place to another. For example, a connection can be a communication medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies (such as infrared, radio, and microwave), then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (such as infrared, radio, and microwave) are included in the definition of communications media. Combinations of the above should also be included within the scope of computer-readable media.

[0095] Applicants hereby independently disclose each individual feature described herein, as well as any combination of two or more such features, as long as such feature or combination can be implemented in light of the common general knowledge of a person skilled in the art based on the present specification as a whole, regardless of whether such feature or combination of features solves any problem disclosed herein, and without limiting the scope of the claims. Applicants indicate that various aspects of the present invention may consist of any such individual feature or combination of features.

[0096] It should be noted that the embodiments of the present invention are described with reference to different categories. In particular, some examples are described with reference to methods, while other examples are described with reference to apparatuses. However, a person skilled in the art will infer from the description that, unless otherwise indicated, any combination of features belonging to one category, as well as any combination of features belonging to different categories, is also considered disclosed by this application. However, all features can be combined to provide synergistic effects that are not simply the sum of the features.

[0097] Although the present invention has been illustrated and described in detail in the drawings and the foregoing description, such illustration and description should be considered illustrative rather than restrictive. The present invention is not limited to the disclosed embodiments. Other variations of the disclosed embodiments may be understood and implemented by those skilled in the art by studying the drawings, the disclosure, and the appended claims.

[0098] The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0099] Any reference signs in the claims should not be construed as limiting the scope.

[0100] Reference numerals:

[0101] 110, 120: Topology image; 130: AI / ML model; 140: Lookup table; 150: User; 210: Inlet; 220: Valve; 230: Storage tank; 240: Water; 250: Control valve; 260: Crude oil; 270: Control valve; 280: Gas; 290: Control valve

Claims

1. A method for processing a topological image, comprising: Input the topological image into the artificial intelligence / machine learning AI / ML model; identifying symbols and / or indicating embedded representations of symbols in at least a segment of the topological image; comparing the result of the identification with at least one of predetermined symbols and / or with at least one embedded representation of an indicative symbol of predetermined embedded representations indicative of predetermined symbols, wherein the predetermined symbols are associated with predetermined characteristics; determining, based on a result of said comparing, at least one characteristic associated with said result of said identifying; and Based on a result of the determining, a structured representation is derived from the image content of the topological image.

2. The method according to claim 1, wherein The identifying comprises identifying the symbol and / or the embedded representation by using the AI / ML model; and / or The identifying comprises identifying an embedded representation indicative of the symbol and identifying the symbol from the identified embedded representation; and / or The comparing comprises comparing the identified symbols and / or the identified embedded representations by using at least one of: a nearest neighbor search, a similarity search, a lookup table storing the predetermined symbols and associated characteristics, and a representation indicative of the predetermined embedded representation; and / or The determining comprises determining the at least one characteristic by using the AI / ML model; and / or The deriving includes deriving the structured representation by using the AI / ML model.

3. The method according to claim 1 or 2, wherein the determining comprises determining at least one of the following as the at least one characteristic: inputs and / or outputs associated with the identified symbol and / or the identified embedded representation, and A predetermined symbol and / or predetermined embedded representation related to the identified symbol and / or the identified embedded representation.

4. The method according to any one of claims 1 to 3, wherein said identifying comprises identifying at least two symbols and / or two embedded representations; and wherein the determining comprises determining a proximity of the identified symbols and / or the identified embedded representations to each other as the at least one characteristic, and determining a relationship between the identified symbols and / or the identified embedded representations based on the result of the comparison and the determined proximity.

5. The method according to any one of claims 1 to 4, wherein the deriving comprises at least one of: parsing, identifying, transforming and formalizing the image content.

6. The method according to any one of claims 1 to 5, further comprising: receiving feedback regarding said result of said identifying and / or said result of said comparing and / or said result of said determining and / or said result of said deriving; as well as The AI / ML model is trained and / or retrained based on the feedback.

7. The method according to any one of claims 1 to 6, further comprising: generating code and / or control graphics based on the structured representation; as well as The image content is processed based on the code and / or the control graphic. 8 . The method according to claim 1 , wherein the topology image shows a brownfield topology and / or comprises brownfield topology information.

9. The method according to any one of claims 1 to 8, wherein the structured representation is at least one of: a textual description indicating the image content of the topological image, an ontological format indicating the image content, and Indicates the image format of the image content.

10. The method according to any one of claims 1 to 9, wherein the AI / ML model is based on further using at least one of: Joint embedding and / or nearest neighbor search, Image classification and / or segmentation and proximity-based relationship identification, Transformer-based neural networks, and An analysis pipeline comprising the lookup table and similarity search.

11. The method according to any one of claims 1 to 10, wherein the symbols and / or predetermined symbols are associated with at least one of: Valves, tanks, sensors, reactors, boilers, mixers, separators, controllers, and inputs / inlets / outputs / outlets; and / or The at least one characteristic and / or predetermined characteristic comprises at least one of the following: input of said symbols and / or predetermined symbols, output from said symbols and / or predetermined symbols, the relationship between the symbol and / or predetermined symbol and another symbol and / or predetermined symbol, the connection between the symbol and / or predetermined symbol and another symbol and / or predetermined symbol, an indication of whether the relationship is unidirectional or bidirectional, and An indication of whether the relationship is labeled with text indicating the type of relationship and / or giving other details and / or properties. 12 . A control device for a topological image processing control system, the device being configured to execute the method according to claim 1 .

13. A topological image processing control system, configured to execute the method according to any one of claims 1 to 11. 14 . An industrial automation system comprising the device according to claim 12 and / or the topological image processing and control system according to claim 13 .

15. A computer program product comprising instructions which, when executed by a computing system, enable the computing system to perform the method according to any one of claims 1 to 11 or cause the computing system to perform the method according to any one of claims 1 to 11.

Images