Confidence heat map for AI-based P&AEng document processing

Abstract

Embodiments of the present disclosure relate to an AI-based Pamp; the invention relates to a confidence coefficient heat map of AEng document processing. A method for aiding in evaluating a confidence estimate in a structured representation of information in an industrial plant environment, comprising: obtaining a target structured representation of information comprising one or more target information segments associated with a confidence estimate determined by an information model, at least a portion of the one or more target information fragments overlap with one or more information fragments included by the one or more different structured representations; generating a confidence heat map based on the target structured representation, the generated confidence heat map indicating a confidence of the one or more target pieces of information based on an associated confidence estimate; the generated confidence heat map is overlaid on at least a portion of one or more input documents input to the IM, and a target structured representation and one or more different structured representations are obtained by the IM based on the one or more input documents, the one or more input documents being associated with the same process plant.

Description

Technical Field

[0001] This invention relates to a method and data processing apparatus for probabilistic estimation in a structured representation of information in an industrial plant environment to assist in the evaluation. The invention also relates to a data processing system, a computer-readable medium, a computer program product, and the use of a probability heatmap generated according to the method. Background Technology

[0002] Due to limited support for the interpretation and processing of process design specification documents, the automation engineering of process automation systems in industrial plant environments still requires a significant amount of manual effort. Although standards for digital data exchange do exist between process engineering and automation engineering, these formats are rarely used, and therefore the enormous automation potential in automation engineering remains largely untapped.

[0003] Therefore, in automation engineering, there is still room for improvement and a need for improvement in the use of automation potential, especially in the use of artificial intelligence / machine learning (AI / ML) model confidence / uncertainty values ​​in process design specification documents. Summary of the Invention

[0004] In view of the above, the purpose of this disclosure is to overcome at least some of the deficiencies in the use of AI / ML model confidence / uncertainty values ​​in process design specification documents.

[0005] Therefore, to address one or more of these deficiencies, in a first aspect, a method is provided for assisting in the evaluation of confidence estimates in structured representations of information in an industrial plant environment. The method includes: acquiring a target structured representation of information, which includes one or more target information fragments associated with confidence estimates determined by an information model, wherein at least a portion of the one or more target information fragments overlaps with one or more information fragments included by one or more different structured representations. The method further includes: generating a confidence heatmap based on the target structured representation, wherein the generated confidence heatmap indicates confidence for the one or more target information fragments based on the associated confidence estimates. The method further includes: overlaying the generated confidence heatmap onto at least a portion of input documents input into one or more input documents in an IM, and acquiring the target structured representation and one or more different structured representations from the IM based on the one or more input documents, wherein the one or more input documents are associated with the same process plant.

[0006] It should be noted that the structured representation of information refers to the representation of information that can be obtained from unstructured representations (such as input images) in a structured manner, in a way that can be read and / or understood by computers or IM.

[0007] Regarding each information fragment, it refers to information provided in the structured representation. According to several examples, an information fragment can be an element identified in the input image, such as a symbol in a (topological) image. Such a symbol can represent any component of a process plant, such as a tank, reactor, or valve, but is not limited to this. Furthermore, according to several examples, an information fragment can also be textual information, such as words, phrases, sentences, or paragraphs. Such textual information can be associated with any component of a process plant, such as a tank, reactor, or valve, but is not limited to this. Additionally, according to several examples, an information fragment can also be numerical information, such as numbers or codes provided in a table. Such numerical information can be associated with any component of a process plant, such as a tank, reactor, or valve, but is not limited to this.

[0008] The information fragments included in the structured representation are obtained by IM processing from the information fragments included in one or more input documents. It should be noted that the structured representation of information can represent a structured representation of the information provided in the input document (which can be understood as representing an unstructured representation). Therefore, the information fragments included in the structured representation correspond to and / or depend on the information fragments included in the input document. In other words, for each information fragment included in the structured representation, the corresponding information fragment is included in the input document.

[0009] It should be noted that confidence estimates can be expressed as percentage values ​​within the range of zero to one, zero to ten, or zero to 100, but are not limited to these.

[0010] It should also be noted that the overlap of information segments may mean that different structured representations include the same information segment. For example, it can be assumed here that there is a first structured representation and a second structured representation. The first structured representation can be obtained from a first input document, for example, as an input image. The second structured representation can be obtained from a second input document, for example, as text information. The first input document may include a symbol representing a storage tank as an information segment. Therefore, the first structured representation may include a first information segment representing a storage tank. The second input document may include paragraphs defining the technical details of the storage tank as information segments. Therefore, the second structured representation may include a second information segment representing a storage tank. Therefore, the first information segment of the first structured representation and the second information segment of the second structured representation both represent or refer to the same storage tank. Therefore, the first and second information segments included by different structured representations should be understood to have some overlap, i.e., they overlap with each other. This also applies to three or more structured representations in addition to the first and second structured representations as outlined above.

[0011] Therefore, overlapping information segments can be understood as identical or related information segments. In other words, it can be understood that the content of the first information segment overlaps with the content of the second information segment (partially or entirely), or that the content of the first information segment is the same as the content of the second information segment (partially or entirely). This also applies to three or more information segments in addition to the first and second information segments as outlined above.

[0012] Regarding different structured representations, it can mean obtaining a first structured representation from a first input document and a second structured representation from a second input document, and that the first and second structured representations are distinct structured representations. However, regarding different structured representations, it can also mean obtaining a common or combined structured representation from (jointly or subsequently processed) the first and second input documents. This common or combined structured representation can then be understood as including a first part and a second part, the first part including information obtained from the first input document, and the second part including information obtained from the second input document. For simplicity, such a first part and second part can also be considered as distinct first and second structured representations. The same applies to three or more structured representations and / or three or more parts.

[0013] It should also be noted that the indicated confidence level can be, for example, a confidence value, a confidence level, a clustered confidence category (such as low confidence, medium confidence, and high confidence), or any combination thereof.

[0014] Regarding input documents, it can refer to engineering design specification documents associated with the same engineering project and / or the same process plant.

[0015] Regarding overlaying a generated confidence heatmap onto at least a portion of an input document, it can mean: overlaying a confidence heatmap indicating the confidence level for one or more target information segments onto at least a portion of the input document, which includes information segments corresponding to / representing each target information segment from a target structured representation. In other words, the confidence heatmap indicates the confidence level for one or more target information segments, where the target information segments may be, for example, a tank or a valve. The input document overlaid with the confidence heatmap may include (e.g., as image information or text information) one or more information segments, where the information segments may, for example, represent such a tank and / or a valve. Therefore, when a confidence heatmap is overlaid on an input document, the overlay is performed in such a way that the confidence value for a target information segment is associated with the corresponding information segment included in the input document. For example, it can be said that the confidence value of the confidence heatmap for a tank and / or a valve is overlaid on the tank and / or valve included in the input document. In the input document, a particular tank and / or valve may be represented, for example, by symbols, numbers, or (descriptive) text. Therefore, by overlaying a confidence heatmap onto at least a portion of the input document, such symbols, numbers, or (descriptive) text can be associated with the corresponding confidence levels indicated in the confidence heatmap.

[0016] The advantage of the approach based on the first aspect is that it enables human engineers in industrial environments to more easily and quickly understand or perceive uncertainty. Consequently, the advantage lies in that human engineers can more easily, quickly, and comprehensively understand the reliability (or unreliability) or confidence (or insecurity) of the AI's work, and how the AI ​​arrives at the results it provides.

[0017] According to several examples of this disclosure, the confidence estimates associated with each target information fragment overlapping with each information fragment of different structured representations can be joint confidence estimates. The method may further include: determining a joint confidence estimate for the target information fragment overlapping with one or more information fragments of different structured representations by calculating a weighted sum of the corresponding confidence estimates associated with the overlapping target information fragments and one or more information fragments of different structured representations.

[0018] It should be noted that the joint confidence estimate can be understood as, for example, the average confidence estimate determined on the corresponding confidence estimates associated with the overlapping information fragments.

[0019] Therefore, the potential uncertainty provided by the first structured representation can be further evaluated by considering other structured representations. Thus, the potential uncertainty can be identified or resolved. In any case, the confidence level of human engineers in the confidence estimate will increase.

[0020] According to several examples of this disclosure, calculating a weighted sum may include assigning a higher weight to the confidence estimate associated with the first certainty than to the confidence estimate associated with a second certainty that is lower than the first certainty.

[0021] Therefore, estimates with higher certainty receive greater attention compared to those with lower certainty, thus improving overall certainty.

[0022] According to several examples of this disclosure, obtaining a target structured representation may include: based on inputting one or more input documents into an IM, obtaining a structured representation including information of a first structured representation, wherein each of the obtained structured representations includes one or more information fragments associated with a confidence estimate determined by the IM. The obtaining may further include: determining a joint confidence estimate for each information fragment of different structured representations in the obtained structured representations that overlap with each other, based on the confidence estimate. The obtaining may further include: associating one or more first information fragments of a first structured representation that overlap with one or more information fragments of different structured representations in the obtained structured representations with the determined corresponding joint confidence estimate. The obtaining may further include: based on the association, replacing at least a portion of the first confidence estimate with the determined corresponding joint confidence estimate for one or more first information fragments associated with the first confidence estimate determined by the IM. The obtaining may further include: obtaining a target structured representation as a result from the replacement.

[0023] Therefore, it is possible to use any structured representation of the acquired structured representation as the basis for obtaining the target structured representation. Thus, it is possible to generate several target structured representations.

[0024] According to several examples of this disclosure, the method may further include: displaying an input document and a generated confidence heatmap overlaid on at least a portion of the input document.

[0025] It should be noted that the superimposed confidence heatmap can be made transparent.

[0026] Therefore, human engineers can identify potential uncertainties more easily, quickly, and reliably, thereby identifying potential risks.

[0027] According to several examples of this disclosure, the display may also include: displaying a notification message if the confidence heatmap includes confidence values ​​that violate a predetermined confidence threshold.

[0028] Therefore, human engineers can identify potential alerts more easily, quickly, and reliably.

[0029] According to several examples of this disclosure, the information fragments included in the target structured representation may be obtained from information fragments included in one or more input documents.

[0030] Therefore, it is possible to overlay the generated confidence heatmap onto at least a portion of the input document.

[0031] According to several examples of this disclosure, generating a confidence heatmap may include indicating different confidence levels included by the confidence heatmap using different colors and / or different patterns.

[0032] Therefore, it further improves the manageability of human engineers.

[0033] According to several examples of this disclosure, the method may further include: modifying the target structured representation by determining a joint confidence estimate based on the uncertainty indicated by the confidence levels included in the confidence heatmap and by using an additional structured representation obtained from another input document associated with the same process plant.

[0034] It should be noted that, generally, the more input documents used to obtain the target structured representation, the more reliable the target structured representation will be. In this way, a single information fragment associated with high uncertainty has a smaller impact on the target structured representation, because more overlapping fragments with low uncertainty can be used to obtain a joint confidence estimate.

[0035] Therefore, human engineers will have even greater confidence in the results obtained from AI (or IM).

[0036] According to several examples of this disclosure, the method may further include acquiring multiple input documents associated with the same process plant. Acquiring a target structured representation may include acquiring a corresponding target structured representation for each of the multiple input documents. Generating a confidence heatmap based on the target structured representation may include generating multiple confidence heatmaps based on the multiple target structured representations. The method may further include overlaying each of the generated multiple confidence heatmaps onto its corresponding input document among the multiple input documents.

[0037] Therefore, it makes it easier, faster, and more reliable to assess the determinism of various parts of the entire process plant.

[0038] According to several examples of this disclosure, the method may further include: acquiring multiple input documents associated with the same process plant. The method may further include: determining a joint confidence heatmap for the multiple input documents based on combining multiple confidence heatmaps generated for the multiple input documents. The method may further include: overlaying the joint confidence heatmap onto the multiple input documents representing at least a portion of the same process plant.

[0039] Therefore, it makes it easier, faster and more reliable to assess the determinism of an overview representation of at least a portion of the entire process plant.

[0040] According to a second aspect, a data processing apparatus is provided for assisting in the evaluation of probability estimates in a structured representation of information in an industrial plant environment. The data processing apparatus includes a processor configured to perform the method of the first aspect.

[0041] The advantage of the data processing device according to the second aspect is that it can participate in enabling human engineers in industrial environments to more easily and quickly understand or perceive uncertainty. As a result, its advantage is that human engineers can more easily, quickly, and comprehensively understand the reliability (or unreliability) or confidence (or disbelief) of the work of artificial intelligence, and how the artificial intelligence arrives at the results it provides.

[0042] According to a third aspect, a data processing system is provided for assisting in the evaluation of probabilistic estimations in a structured representation of information in an industrial plant environment. The data processing system includes the data processing apparatus of the second aspect. Additionally or alternatively, the data processing system includes components for performing the method of the first aspect.

[0043] The advantage of the data processing system according to the third aspect is that it can help human engineers in industrial environments to understand or perceive uncertainty more easily and quickly. As a result, its advantage is that human engineers can more easily, quickly, and comprehensively understand the reliability (or unreliability) or confidence (or disbelief) of the work of artificial intelligence, and how the artificial intelligence arrives at the results it provides.

[0044] According to a fourth aspect, the present invention provides an industrial plant comprising the data processing apparatus of the second aspect and / or the data processing system of the third aspect.

[0045] According to several examples, "industrial plant" can mean an industrial factory, an autonomous industrial plant, or an industrial production plant, including one or more pipelines, production lines, and / or assembly lines for transforming one or more effluents into products and / or for assembling one or more components into a final product. According to several examples, it can mean an industrial plant in the oil, gas, mining, chemical, wind and electricity, or food and beverage industries.

[0046] The advantage of industrial plants, according to the fourth aspect, lies in their ability to enable human engineers in industrial environments to more easily and quickly understand or perceive uncertainty. Consequently, the advantage is that human engineers can more easily, quickly, and comprehensively understand the reliability (or unreliability) or confidence (or disbelief) of AI's work, and how AI arrives at the results it provides.

[0047] According to a fifth aspect, a computer-readable medium including instructions that, when executed by a computing system, cause the computing system to perform the method of the first aspect. The computer-readable medium may be transient or non-transient, volatile or non-volatile.

[0048] According to the fifth aspect, the advantage of this computer-readable medium is that it enables human engineers in industrial environments to more easily and quickly understand or perceive uncertainty. As a result, its advantage lies in allowing human engineers to more easily, quickly, and comprehensively understand the reliability (or unreliability) or confidence (or lack thereof) of the work of artificial intelligence, and how the AI ​​arrives at the results it provides.

[0049] According to a sixth aspect, a computer program product including instructions is provided that, when executed by a computing system, cause or enable the computing system to perform the method of the first aspect. The computer program product may include a computer-readable medium comprising the instructions of the computer program product.

[0050] According to the sixth aspect, the advantage of this computer program product lies in its ability to enable human engineers in industrial environments to more easily and quickly understand or perceive uncertainty. As a result, its advantage is that human engineers can more easily, quickly, and comprehensively understand the reliability (or unreliability) or confidence (or insecurity) of the AI's work, and how the AI ​​arrives at the results it provides.

[0051] According to the seventh aspect, there is use for at least one of the probability heatmap of the first aspect, the data processing device of the second aspect, the data processing system of the third aspect, the industrial plant of the fourth aspect, the computer-readable medium of the fifth aspect, and the computer program product of the sixth aspect.

[0052] The advantage of the application according to the seventh aspect is that it can help human engineers in industrial environments to understand or perceive uncertainty more easily and quickly. As a result, its advantage is that human engineers can more easily, quickly, and comprehensively understand the reliability (or unreliability) or confidence (or disbelief) of the work of artificial intelligence, and how the artificial intelligence arrives at the results it provides.

[0053] The optional features of the first aspect can form part of any of the second to seventh aspects, but necessary modifications are required.

[0054] The first approach can be implemented, at least in part, by a computer.

[0055] The fifth aspect is a computer-readable medium on which the computer program product of the sixth aspect can be stored.

[0056] As used herein, the term “acquisition” can include, for example, receiving from another system, apparatus, or process; receiving via interaction with a user; loading or retrieving from a storage device or memory; or measuring or capturing using a sensor or other data acquisition device.

[0057] The indefinite article “a” or “one” does not exclude the plural. Furthermore, the article “a” or “one” as used in this text should generally be interpreted as “one or more” unless otherwise specified or clearly indicated from the context to be in the singular form.

[0058] Unless otherwise specified or clearly stated from the context, the phrases “one or more of A, B, and C,” “at least one of A, B, and C,” and “A, B, and / or C” as used herein are intended to refer to all possible permutations of one or more of the listed items. That is, the phrase “A and / or B” means (A), (B), or (A and B), while the phrase “A, B, and / or C” means (A), (B), (C), (A and B), (A and C), (B and C), or (A and B and C).

[0059] The term "comprising" does not exclude other elements or steps. Furthermore, the terms "comprising," "including," and "having," etc., are used interchangeably herein.

[0060] This invention may include one or more aspects, examples, or features, either individually or in combination, whether specifically disclosed in the combination or individually. Any optional feature or sub-aspect of one of the foregoing aspects is applicable to any other aspect.

[0061] The above aspects will become apparent from the detailed description provided below, and will be clarified by reference. Attached Figure Description

[0062] A detailed description will now be given with reference to the accompanying drawings, by way of example only, in which:

[0063] - Figure 1 The illustrations depict several examples of engineering data funnels, based on this disclosure, for processing engineering design specification documents into a comprehensive structured representation, and artificial intelligence (AI).

[0064] - Figure 2 The illustration shows example input images with superimposed confidence heatmaps according to several examples of this disclosure;

[0065] - Figure 3 The illustration shows, as shown in the diagram. Figure 2 The indicated representation of the entire process plant, which is divided into several parts;

[0066] - Figure 4 The illustration schematically depicts, according to several examples of the present disclosure, a superimposed confidence heatmap covering a factory area. Figure 3 The entire process of the factory is represented;

[0067] - Figure 5 The illustrations depict flowcharts indicating methods according to several examples of this disclosure; and

[0068] - Figure 6 A block diagram schematically illustrating a data processing apparatus according to several examples of the present disclosure is shown. Detailed Implementation

[0069] See now Figure 1 Through the "Engineering Data Funnel" (EDF) schematically illustrated therein, this disclosure provides, based on several examples, an AI-based approach and prototype for structuring and formalizing multimodal unstructured process design information, such as in PDF, Excel, and / or Word formats, using agent and hybrid expert systems, such as those based on ontology-enhanced large language models (LLM), and making it available to engineering tools that have long been referred to as "automation of automation."

[0070] However, since AI-based data processing also has an inherent probabilistic nature in industrial factory environments, there is a problem or drawback: there is always a certain degree of uncertainty in the data processing and therefore in the processing results.

[0071] In view of this, based on several examples of this disclosure, this disclosure is intended to enable engineering expert users in industrial environments to understand or actually experience this uncertainty, for example by providing a “probability heatmap” or also known as a “confidence heatmap”, which enables the labeling of portions, intervals and paragraphs in original and / or further processed documents, data or (generally) structured representations based on estimated uncertainty and AI-based data processing confidence.

[0072] As a result, human engineers can more easily, quickly, and comprehensively understand the reliability (or unreliability) or confidence (or lack thereof) of AI's work, and how AI arrives at the results it provides.

[0073] See now Figure 1 , Figure 1 The illustration shows several examples of EDF 140 according to this disclosure, which can be understood as artificial intelligence (AI) or information model (IM) for processing input data (such as engineering design specification documents) into a comprehensive structured representation.

[0074] More specifically, according to Figure 1 The EDF system 100 illustrates three different types of examples for engineering design specification documents: the first specification document 110 includes text information; the second specification document 120 includes images or topological diagrams of several connection symbols; and the third specification document 130 includes tables. Specification documents 110, 120, and 130 can be understood as input documents that are fed into EDF 140. Specification documents 110, 120, and 130 can be in data formats; for example, the first specification document 100 can be in Word format, the second specification document 120 can be in PDF format, and the third specification document 130 can be in Excel format. However, specification documents are not limited to these formats. Generally, specification documents 110, 120, and 130 can represent unstructured data, such as data obtained from EPC.

[0075] Each of the specification documents 110, 120, and 130 includes one or more information segments. For example, the first specification document 110 includes information segments related to the storage tank, reactor, and valve, i.e., text information. Furthermore, the second specification document 120 includes information segments related to the storage tank, reactor, and valve, i.e., connection symbols. Additionally, the third specification document 130 includes information segments related to the storage tank, reactor, and valve, i.e., the contents of rows, columns, or cells in a table.

[0076] Specification documents 110, 120, and 130 are entered (e.g.) Figure 1 As indicated in steps S110, S120, and S130, the data is processed by EDF 140. As a result of the processing, EDF 140 outputs a structured representation in S140.

[0077] During processing in EDF 140, the information segments, namely tanks, reactors, and valves, are identified in specification documents 110, 120, and 130. It should be noted that how these information segments are specifically identified in specification documents 110, 120, and 130 does not form part of this disclosure. However, several well-known LLM-based and / or image processing-based solutions can be used to identify these (and / or other) information segments in specification documents 110, 120, and 130.

[0078] The structured representation obtained from EDF 140 can be understood as a representation of the joint information available from specification documents 110, 120, and 130. Alternatively, the structured representation obtained from EDF 140 can be understood as a representation of the corresponding structured representation in each of specification documents 110, 120, and 130. As an example of improving understandability, EDF 140 may have identified from the second specification document 120, based on image processing, that a tank may be connected to a valve, and that the valve may also be connected to a reactor. However, because the quality of the second specification document 120 may be low—for example, the image may be pixelated—the certainty that the tank is connected to the reactor via a valve may be low. For example, from the first specification document 110, EDF 140 can know with greater certainty that the tank is indeed connected to the reactor via a valve; while from the third specification document 130, EDF 140 can know with some certainty the possible types of the tank and the possible types of the reactor. Therefore, for example, a structured representation can include with relatively high certainty a tank connected to a reactor via a valve, and the tank and reactor may belong to a specific type.

[0079] However, as already derived, the structured representation thus obtained still includes uncertainty, particularly the fragments of information associated with certain uncertain (or deterministic) values. Identifying, understanding, and processing this uncertainty is a complex, time-consuming, and error-prone task for human engineers. Based on several examples in this disclosure, at least some of these drawbacks are mitigated.

[0080] Generally, EDF 140 can output one or more structured representations in different ways or formats, such as in the form of structured visualizations 150 or structured text information 160 (such as JSON files).

[0081] More specifically, EDF 140 or EDF system 100 may include, among other things, a visual model trained to recognize parts and / or symbols and connections in an input document (e.g., P&ID documents like first and second specification documents 110 and 120, such as PDF files, image files). Additionally or alternatively, EDF 140 or EDF system 100 may include a language model trained to process text in a controlling narrative document (e.g., input documents like specification documents 110, 120, and 130, such as PDF files, text, text files, or tables, etc.).

[0082] However, the EDF system 100 may never be perfect and may struggle to handle certain symbols and / or connections (e.g., as shown in the second specification document 120), or certain text paragraphs (e.g., as shown in the first specification document 110), for example, when the symbols are low resolution, rotated, or different from the training data, or, for example, when the text / sentence is complex and has multiple ambiguous relationships, which places excessive demands on the transformer's attention mechanism, etc.

[0083] In these cases, to address this limitation, according to several examples of this disclosure, for example, a system and method are provided that, given one or more input engineering design specification documents (such as specification documents 110, 120, and 130), provides a confidence heatmap overlaid on the original documents (e.g., specification documents 110, 120, and 130), which can show an estimated probability of a machine learning (ML) model capable of detecting or understanding the correct things (e.g., symbols and / or connections or textual meanings and / or relationships in a given pixel location / region and / or text sentence / paragraph).

[0084] Therefore, using the proposed system and method, it is now possible not only to show what parts and pipelines or attributes / features are detected in P&ID and CN, but also to provide users with a "confidence heatmap" that can show the estimated probability of the ML model that can (correctly) detect parts (as well as connections and attributes, etc.) at a given location in an image or in a part / paragraph of text.

[0085] For example, a set of low-resolution symbols, blurry or intersecting lines, or complex long sentences or paragraphs can have higher estimation uncertainty than, for example, a simple chain of frequently occurring high-resolution symbols or a set of simple SPO sentences with a univariate linear relationship.

[0086] It should be noted that the systems and methods disclosed herein can be modified in any way when combined with one or more engineering design specification documents:

[0087] • It can generate and provide confidence heatmaps (or probability heatmaps, hereinafter also referred to as probabilistic heatmaps) for single expert model outputs: that is, a processed engineering design specification document (e.g., P&ID) will produce an output, such as a list of detected parts / symbols and connections, and a heatmap that can be overlaid.

[0088] However, confidence heatmaps can also be composed of combinations of several single-expert model outputs. For example, two documents (P&ID and control narrative) from the same project or process plant will naturally complement each other: given document 1 (P&ID, such as second specification document 120) and document 2 (CN, such as first specification document 110), from which outputs and corresponding probability distributions have already been obtained, for document 2, the information obtained from document 1 is now used to compute a *conditional* (i.e., combined) probability distribution P(X_document1|Y_document2), where X and Y are symbols or components in different documents 1 and 2. This may result in P(X_document1) being enhanced or altered, now becoming P(X_document1|Y_document2) (where Y can be anything in document 2, including X in document 2).

[0089] Visualization can also be achieved by overlaying confidence heatmaps.

[0090] Furthermore, this layering approach can also be used to intentionally process (if available) additional documents from the same project to reduce uncertainty in a given area: for example, if an area in a P&ID might have high uncertainty / low confidence defects, and this can be shown to a human engineer or user, then he / she can try to provide more information to EDF 140, i.e., additional design specification documents about the corresponding area, or simply (in the case of a large P&ID) provide a high-resolution image of the corresponding section, as an example. Figure 2 As indicated in the document.

[0091] It should be noted that the overlay method with confidence heatmaps can also be used to reconfirm confidence intervals by overlaying confidence heatmaps generated from other modalities. That is, if a “specific” confidence heatmap interval that may be generated from a P&ID image (e.g. from the second specification document 120) can remain “specific” after being overlaid with a confidence heatmap generated from a CN text (e.g. from the first specification document 110), then this can reconfirm the potentially correct processing.

[0092] In general, based on several examples of this disclosure, the confidence heatmap can be an optional “transparent overlay” that can be mixed to show / hide which parts the AI ​​is confident in and which parts it is not.

[0093] This functionality is very beneficial to engineering experts or users, as it can help them discover potential errors or contradictory / incomplete structural representations more quickly, i.e., contradictory / incomplete outputs from EDF 140.

[0094] To allow for the transferability and wider use of confidence information, according to several examples of this disclosure, probability, uncertainty, or confidence information is included in an XML format along with detected content (such as symbols, pipes, text, components, relationships, etc.). Therefore, standard engineering tools (such as those available in the applicant's product / solution portfolio) can use it for visualization in common or well-known tools.

[0095] See now Figure 2 , Figure 2 The illustration shows example input images with superimposed confidence heatmaps according to several examples of this disclosure.

[0096] For example, according to Figure 2 Input image 200 can represent according to Figure 1 The "Assessed" second specification document 120.

[0097] As an example, Figure 2 Three different patterns are shown, which can represent three different levels of certainty (uncertainty). For example, low-level uncertainty is illustrated by a pattern where lines extend from the bottom left to the top right. Medium-level uncertainty is illustrated by a pattern where lines extend from the top left to the bottom right. High-level uncertainty is illustrated by a pattern where lines also extend from the bottom left to the top right, but the lines are closer together than in the low-level uncertainty pattern. It is possible to indicate (fewer or more than three) levels of uncertainty, for example, (only one or two) four or more levels.

[0098] Additionally or alternatively, different shades of gray or different colors can be used to represent different levels of certainty (uncertainty) or values ​​of certainty (uncertainty). For example, refer to Figure 2 Low-level uncertainty can be represented in green, medium-level uncertainty in yellow, and high-level uncertainty in red. However, different and / or more colors can be used in addition to "green," "yellow," and "red." For example, for certainty (uncertainty) values ​​in the range of 1 to 100, uncertainty values ​​close to 1 can be represented in light gray, and the closer the uncertainty value is to 100, the darker the light gray becomes, until the uncertainty value reaches its darkest gray at 100. It should be noted that, as an example, in... Figure 4 The diagram illustrates such representations of different gray levels. As another example, in addition to a continuous change from light gray to dark gray, a continuous change from a first color (“starting color”) to a second color (“ending color”) can also be illustrated. The change from the starting color to the ending color can occur via one or more intermediate colors, such as from “green” through “yellow” to “red”.

[0099] To be more detailed, Figure 2The diagram illustrates the symbol for storage tank 230. Input connections 210a and 210b are shown, providing input flow to storage tank 230. Output connections 220a, 220b, and 220c are shown, providing output flow from storage tank 230. Storage tank 230 and its corresponding input connections 210a, 210b and output connections 220a, 220b, 220c are visually indicated as being associated with second-level determinism, such as intermediate determinism (i.e., intermediate uncertainty). Figure 2 Flow control entities 240a and 240b for controlling flow are further shown, and they are visually indicated to be associated with third-level determinism, such as low-level determinism (i.e., high-level uncertainty). Furthermore, Figure 2 Flow control entities 250a, 250b, 250c, and 250d for controlling flow are also shown, and they are visually indicated to be associated with first-level determinism, such as high-level determinism (i.e., low-level uncertainty). Furthermore, Figure 2 Different types of valves 260 and 270a to 270h are also shown, which are visually indicated to be associated with first-level determinism, such as high-level determinism (i.e., low-level uncertainty).

[0100] Next, we will consider two scenarios.

[0101] Based on the first scenario, it should be noted that... (refer to...) Figure 1 The second specification document 120 can be processed by EDF 140 alone (i.e., without other specification documents), and the result of such processing can be based on... Figure 2 Image 200 or (topological image). Therefore, the second specification document 120 in its original version may include components 210a to 270h as input images, such as Figure 2 The diagram is shown in the figure and does not include any associated deterministic assessments. However, after processing in EDF 140, determinism, such as deterministic level or deterministic value, can be associated with each of parts 210a to 270h, which can be illustrated by the corresponding confidence heatmap, thus obtaining... Figure 2 The image shown. Therefore, these associated levels of certainty or values ​​of certainty can be included in the confidence or probability heatmap, and such a confidence heatmap can be overlaid in its original version onto the second specification document 120. This can lead to, for example... Figure 2 The image shown. As... Figure 2 The image shown can be further processed, for example, similar to what is indicated below with reference to the second case.

[0102] However, according to the second case, it should be noted that, according to several examples of this disclosure, the second specification document 120 can be processed by EDF 140 alone (i.e., without other specification documents) at the level, and the result of such processing may be that several of components 210a to 270h are associated with low deterministic levels or low deterministic values; for example, tank 230 may be associated with a low deterministic level or low deterministic value. Based on this, EDF 140 can perform additional processing on one or more other specification documents (e.g., first and third specification documents 110 and 130, where specification documents 110, 120, and 130 correspond to the same engineering project or the same process plant). Thus, for example, as referenced above... Figure 1 As already indicated, tank 230 can be identified with higher certainty, and the result of such additional processing on the first and / or third specification documents 110 and 130 may be that 230 in the second specification document 120 can be identified with higher certainty. Therefore, by overlaying the confidence heatmap of the second specification document 120 in its original version, tank 230 can be indicated as associated with a medium-certainty or moderate-certainty value. In other words, according to the second case, according to... Figure 2 The image shown can represent the result of such additional processing. For example... Figure 2 The image shown can be further processed, for example, the processing according to the second case can be repeated and / or continued.

[0103] Therefore, considering the first and second scenarios described above, it should be noted that, more generally, an "evaluated" specification document, i.e., a specification document with an associated level of certainty or a certainty value in its original version (e.g., obtained after processing by EDF 140), can be understood as a specification document covered with a confidence heatmap. Such an associated level of certainty or a certainty value can be obtained by individually evaluating the specification document in its original version (i.e., the first scenario), or by subsequently and / or simultaneously evaluating two or more specification documents in the original version and / or the "evaluated" version (i.e., the second scenario).

[0104] See now Figure 3 The diagram illustrates several specification documents 200 and 301 to 312. These specification documents 200 and 301 to 312 correspond to the same engineering project or the same process plant. Specification documents 200 and 301 to 312 can represent, as per... Figure 2 The illustrated “evaluated” specification document, and / or may represent the specification document in its original version, as shown in the figure. Figure 1 The standard documents are 110, 120 and 130.

[0105] Each of the specification documents 200 and 301 to 312 includes corresponding input and output connections. For example, referring to specification document 200, reference symbols 210a and 210b denote input connections (according to...). Figure 2 Guided storage tank 230), reference symbols 220a, 220b, 220c and 220d indicate output connections (according to...). Figure 2 (Export tank 230). For example, output connection 220b is connected to input connection 301a in specification document 301. In specification document 301, the output of tank 230 can be further processed and / or used in subsequent processing steps. More connection details can be found from... Figure 3 Export from [source].

[0106] See now Figure 4 , Figure 4 The illustration schematically depicts several examples based on this disclosure. Figure 3 The representation of a process plant with a superimposed confidence heatmap over the plant area. According to Figure 4 The confidence heatmap coverage is based on Figure 3 The process plant is represented as follows. As indicated above, lighter shades of gray represent lower uncertainty values, while darker shades represent higher uncertainty values. Based on this, when the output of tank 230 reaches the subsequent processing system via input connection 301a and is further used or processed therein, a human engineer or user can easily and quickly identify potential uncertainties (e.g., potential misidentifications) in the processing of the process plant. Similarly, a human engineer or user can easily and quickly identify potential uncertainties (e.g., potential misidentifications) in the processing of the process plants associated with specification documents 309, 310, and 311.

[0107] The following text should be noted. Based on several examples of this disclosure, in accordance with... Figure 1 In cases where the first specification document 110 can be used as the basis for a confidence heatmap, such a confidence heatmap can associate a level of certainty or a value of certainty with individual words, phrases, sentences, or paragraphs included in the textual information of the first specification document 110.

[0108] Therefore, based on several examples of this disclosure, a system and method are provided for visualizing confidence / uncertainty values ​​of AI / ML models in process design specification documents. For example, a system and method are provided for visualization using confidence heatmaps, wherein the visualization of the confidence heatmaps can be for single or multiple / combined documents. This system and method can be based on a method for obtaining probability / uncertainty (confidence values) for AI-based engineering data or document processing. Furthermore, a system and method are provided for obtaining an XML representation of the confidence heatmap, for example, to present confidence information as an (optional and composable) transparent overlay in conventional engineering tools.

[0109] Based on several examples in this disclosure, more detailed examples are provided for ease of understanding.

[0110] Taking P&ID (image file, such as second specification document 120) and EDF 140 with image processing expert model (visual model) as examples:

[0111] Explain in detail how ML models / neural networks (especially visual models) detect symbols in images (e.g., P&ID) and determine the most probable symbol (from a set of possible target symbols to be identified, e.g., from a certain criterion) and the probability associated with each symbol:

[0112] 1. Neural Network Structure

[0113] Visual models, such as convolutional neural networks (CNNs), or similar transformer-based models, are commonly used for image recognition tasks. A brief overview of their possible structures (in the case of CNNs) is as follows:

[0114] - Input layer: Takes the image as input. The image is typically represented as a multidimensional array of pixel values.

[0115] - Convolutional layers: Convolutional filters are applied to the input image to detect features such as edges, textures, and simple shapes. These layers help the model learn the spatial hierarchy of features.

[0116] - Pooling layers: reduce the dimensionality of feature maps, making computation more efficient and helping to achieve invariance to small translations.

[0117] - Fully connected layer: Flatten the pooled feature map and pass it through the fully connected layer to combine features and make predictions.

[0118] - Output layer: Produces the final output, typically a probability distribution over possible target symbols.

[0119] 2. Detection Symbol

[0120] The process of detecting symbols in an image involves several steps:

[0121] a. Image preprocessing: Input images are typically preprocessed (e.g., resized, normalized) to fit the model's input requirements.

[0122] b. Feature Extraction: Convolutional layers use various filters to scan the image to extract features. Shallower layers may detect simple patterns, while deeper layers detect more complex structures, such as the parts of a symbol.

[0123] c. Flattening and Classification: The extracted features are flattened into vectors and passed through fully connected layers. These layers combine the features to make the final prediction.

[0124] 3. Find the most likely symbol

[0125] The output layer of a network typically uses the softmax activation function, especially in classification tasks. (However, note that other activation functions also exist for several examples in this disclosure.) The softmax function converts the raw output scores (logit) into probabilities, ensuring that their sum is 100%.

[0126] The softmax function is defined as follows:

[0127]

[0128] in, It is the raw score (logit) for category i, and P(y) i ) is the probability of target category i.

[0129] The output of the softmax function is the probability distribution over all possible target symbols.

[0130] For example, if the possible symbols are {A, B, C}, where A is the tank symbol, B is the reactor symbol, and C is the distillation column symbol, the output might be as follows:

[0131] P(A) = 0.7

[0132] P(B) = 0.2

[0133] P(C) = 0.1

[0134] 4. Determine the most likely symbol

[0135] The model selects the symbol with the highest probability as the most likely prediction. In the example above, the model predicts A as the most likely symbol because it has the highest probability (0.7).

[0136] 5. Output Results

[0137] The model typically outputs two main information segments:

[0138] - Predicted symbol: The symbol with the highest probability.

[0139] - Probability distribution: the probability of all possible symbols.

[0140] In practical applications, this might be represented as:

[0141] -Prediction: Symbol A

[0142] - Probability: {A: 0.7, B: 0.2, C: 0.1}

[0143] The resulting example workflow might be interpreted as follows:

[0144] 1. Input: A P&ID image with the symbol to be identified.

[0145] 2. Preprocessing: Optionally, the image size is resized to 28x28 pixels and the pixel values ​​are normalized.

[0146] 3. Feature extraction: Convolutional layers detect character edges and curves.

[0147] 4. Classification: Fully connected layers combine features to predict characters.

[0148] 5. Softmax activation: Converts logits into probabilities (regardless of the actual ML model architecture, i.e., whether it is a CNN or a transformer NN, etc.).

[0149] 6. Output: The most likely symbol (e.g., "A") and its probability (e.g., 0.7), as well as the probabilities of other symbols.

[0150] This process / workflow allows ML models to efficiently identify and classify symbols in images with quantifiable confidence.

[0151] Please note that this example describes an image processing scenario (detecting symbols in an image), but equivalent procedures and settings also apply to other ML models. That is, the output layers of these models also have a softmax activation function and the resulting probability distribution, and will output both the prediction and the corresponding probability distribution.

[0152] Explained probability

[0153] - Most likely symbol: The symbol with the highest probability (e.g., A with a probability of 0.7).

[0154] - Unlikely symbols: Symbols with low probability (e.g., B with a probability of 0.2, C with a probability of 0.1).

[0155] These probabilities give the model's predictions for them. Confidence The higher the probability of the predicted symbol (i.e., the lower the uncertainty), the higher the confidence level.

[0156] Now, it no longer shows probability values ​​to human engineers or users, but instead... Confidence heatmap These probabilities are represented by a heatmap that can be overlaid onto the original engineering design specification document (e.g., P&ID (Second Specification Document 120) or CN (First Specification Document 110)).

[0157] For example, such heatmaps can have color codes from red to green, where red represents high uncertainty of the model and green represents low uncertainty / high confidence of the model.

[0158] Therefore, high confidence / certainty (low uncertainty) means that the model is very certain about its predictions. This means that the probability of predicting a particular class is high. Similarly, low confidence / certainty (high uncertainty) means that the model is less certain about its predictions. This means that the probability of predicting a particular class is low, while the probability distribution for other classes is more uniform.

[0159] Based on several examples of this disclosure, it should be noted that, in addition to the steps described above, creating a confidence heatmap that overlays the original input image and uses color codes to represent model confidence or uncertainty may optionally involve steps such as classification activation mapping (CAM) or Grad-CAM or other CAM variants:

[0160] CAM can also be used to evaluate the confidence of a model across different regions of an image (e.g., different pieces of information) (rather than just a single / independent object / symbol) (and heatmaps based on this). The CAM method highlights image regions that are important for model decisions. Its work is as follows:

[0161] 1. Forward pass: Before applying the softmax function, the input image is passed through the network to obtain class scores.

[0162] 2. Feature map extraction: Extract the feature map from the last convolutional layer.

[0163] 3. Weight Extraction: Extract the weights of the output layer (the fully connected layer immediately following the last convolutional layer).

[0164] 4. Weighted sum: The feature maps are calculated using weights corresponding to the predicted categories.

[0165] The resulting heatmap will highlight the areas that contribute the most to the prediction.

[0166] The CAM can then be normalized to the range [0,1] and mapped to a color scale (e.g., from green to red) to represent different levels of confidence / determinism.

[0167] In the resulting heatmap:

[0168] - Green areas can indicate intervals with high model confidence (high activation).

[0169] - The red area can indicate the interval where the model has low confidence (low activation).

[0170] By visualizing this heatmap on the original image, one can see which parts of the image the model relies on most in its predictions and which parts have uncertainty.

[0171] Therefore, this approach provides a more advanced visual representation of model confidence, which helps to better understand and interpret the model's predictions.

[0172] See now Figure 5 , Figure 5 The diagram illustrates a flowchart illustrating a method according to several examples of this disclosure. This method is used to assist in evaluating confidence estimates in structured representations of information in an industrial plant environment.

[0173] Based on several examples of this disclosure, according to Figure 1 The EDF 140 or EDF system 100 can be configured to perform according to Figure 5 The method is illustrated in the figure.

[0174] This method starts with S500.

[0175] In S510, the method includes acquiring a target structured representation of information 150, 160, the target structured representation of information including one or more target information fragments 160a, 160b associated with a confidence or uncertainty (deterministic) estimate determined by IM 140, wherein at least a portion of the one or more target information fragments overlaps with one or more information fragments included by one or more different structured representations.

[0176] In S520, the method includes generating a confidence heatmap based on a target structured representation, wherein the generated confidence heatmap indicates the confidence level for one or more target information fragments based on associated confidence estimates.

[0177] In S530, the method includes overlaying the generated confidence heatmap onto at least a portion of the input documents in one or more input documents 110, 120, 130 input to the IM, and obtaining a target structured representation and one or more different structured representations from the IM based on the one or more input documents, wherein the one or more input documents are associated with the same process plant 300.

[0178] This method ends at S540.

[0179] Based on several examples of this application, specific examples may be as follows.

[0180] In the first step, a first input document or a first specification document can be input into the IM, and a first structured representation of the information can be obtained from the IM based on the input first input document. In the second step, a second input document or a second specification document can be input into the IM, and a second structured representation of the information can be obtained from the IM based on the input second input document. The first structured representation may include a first information fragment associated with a first uncertainty (deterministic) estimate determined by the IM, and the second structured representation may include a second information fragment associated with a second uncertainty (deterministic) estimate determined by the IM, wherein the content of the first information fragment overlaps with the content of the second information fragment.

[0181] Additionally or alternatively, in the first step, an "evaluated" first input document may be obtained from the IM, i.e., a first input document may be obtained from the IM, and at least one information fragment (e.g., a symbol or text element) included in the first input document may be associated with an uncertainty level or uncertainty value determined by the IM. Additionally or alternatively, in the second step, an "evaluated" second input document may be obtained from the IM, i.e., a second input document may be obtained from the IM, and at least one information fragment (e.g., a symbol or text element) included in the second input document may be associated with an uncertainty level or uncertainty value determined by the IM.

[0182] In the third step, based on the first and second uncertainty estimates, a third uncertainty estimate regarding the overlap between the content of the first information segment and the content of the second information segment can be determined, for example, by the IM. It should be noted that the first and second uncertainty estimates can be obtained from the first and second structured representations and / or from the "evaluated" first and second input documents. Furthermore, according to various examples of this disclosure, and generally not limited to this particular example, it should be noted that the first and second structured representations (or any other structured representations) can be understood as representing first and second portions of the same structured representation.

[0183] In the fourth step, at least one of the first information fragment in the first structured representation and the second information fragment in the second structured representation may be associated with the calculated third uncertainty estimate. Additionally or alternatively, in the fourth step, at least one of the first information fragment in the "evaluated" first input document and the second information fragment in the "evaluated" second input document may be associated with the calculated third uncertainty estimate.

[0184] In a possible fifth step, the “evaluated” first or second input document associated with the calculated third uncertainty estimate and the “evaluated” third input document can be used as inputs to the IM, and the first to fourth process steps outlined above can be repeated using such “evaluated” first or second input documents and such third input documents (these steps are outlined as examples with reference to two input documents or two structured representations).

[0185] In doing so, the use of two input documents or two structured representations means using several more input documents or several more structured representations. This is because, according to several examples of this disclosure (not limited to this particular example), an input document can also be understood as an "evaluated" input document resulting from IM having processed two or more ("evaluated" and / or "unevaluated") input documents. Furthermore, according to several examples of this disclosure (not limited to this particular example), a structured representation can also be understood as a structured representation resulting from IM having processed two or more input documents or structured representations.

[0186] Therefore, refer to Figure 5 In step S510, in addition to obtaining the target structured representation or as an alternative, the method may also include obtaining the target "evaluated" input document. Therefore, one or more different structured representations can be understood as one or more different "evaluated" input documents. Similarly, S520 can also be understood as generating a confidence heatmap based on the target "evaluated" input document. Therefore, S530 can also be understood as covering at least a portion of the target "evaluated" input document with the generated confidence heatmap.

[0187] Therefore, based on several examples of this disclosure, an “evaluated” input document can also be understood as a structured representation of information.

[0188] See now Figure 6 , Figure 6 A block diagram schematically illustrating a data processing apparatus 600 according to several examples of the present disclosure is shown. Specifically, according to several examples of the present disclosure, a data processing apparatus 600 is provided for assisting in the evaluation of probability estimates in a structured representation of information in an industrial plant environment. The data processing apparatus 600 includes components configured to perform... Figure 5 The processor 601 of the method.

[0189] According to several examples of this disclosure, the data processing apparatus 600 may include components used as referenced above. Figure 1 Overview of the components of this type of EDF 140.

[0190] More specifically, based on various examples, it is configured to execute Figure 5 The data processing apparatus 600 of the method may include a processing circuitry, processing functions, processing components, processing units, or a processor 601, which enables the data processing apparatus 600 to participate in probabilistic estimations in a structured representation of information in an industrial plant environment to assist in the evaluation. The processor 601 may include one or more processing portions or functions, wherein the processing portions or functions may be provided as one or more physical or virtual entities. The data processing apparatus 600 may include one or more communication interfaces 602. The data processing apparatus 600 may also include a memory or memory unit 603 for storing data, programs, and / or instructions to be executed by the processor. The memory 603 may be internal to the data processing apparatus 600 or external to the data processing apparatus 600, such as at a cloud server. The processor 601 may include one or more portions that enable the data processing apparatus 600 to perform, for example... Figure 5 The method. According to several examples of this disclosure, the acquisition portion 610 can be configured to perform according to... Figure 5 The S510's acquisition of this type, the generation part 620 can be configured to perform according to Figure 5 Such generation of the S520, and the coverage portion 630, can be configured to perform according to Figure 5 Such coverage of the S530.

[0191] According to several examples of this disclosure, corresponding portions of the data processing apparatus 600 may also be understood as components for performing specific functions.

[0192] According to several examples of this disclosure, a data processing system is provided for assisting in the evaluation of probabilistic estimations in structured representations of information in an industrial plant environment. The data processing system includes... Figure 6 Data processing device 600 and / or for performing according to Figure 5 The method is a component. This data processing system can be represented as described in the reference above. Figure 1 Overview of EDF system 100.

[0193] According to several examples of this disclosure, an industrial plant is provided, comprising, according to Figure 6 The data processing device 600 and / or the data processing system as described above.

[0194] According to several examples of this disclosure, a computer-readable medium including instructions that, when executed by a computing system, cause the computing system to perform actions according to... Figure 5 The method. The computer-readable medium may be temporary or non-temporary, volatile or non-volatile.

[0195] According to several examples of this disclosure, a computer program product including instructions is provided that, when executed by a computing system, cause the computing system to perform actions according to... Figure 5 The method. The computer program product may include a computer-readable medium that includes the instructions of the computer program product. The computer-readable medium as mentioned above may store the computer program product thereon.

[0196] According to several examples of this disclosure, there is a use for a data processing apparatus 600, a data processing system as outlined above, an industrial plant as outlined above, a computer-readable medium as outlined above, and / or a computer program product as outlined above.

[0197] according to Figure 5 The method can be implemented, at least partially, by a computer.

[0198] according to Figure 5 The optional features of the method may form part of the data processing apparatus 600, the data processing system, the industrial plant, the computer-readable medium, the computer program product and its uses, but necessary modifications are required.

[0199] Any unit, module, circuit system, or method described herein may be implemented using hardware, software, and / or firmware configured to perform any of the operations described herein. Hardware may include one or more processor cores, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), etc. Software may be implemented as software packages, code, instructions, instruction sets, and / or data recorded on at least one transient or non-transitory computer-readable storage medium. Firmware may be implemented as code, instructions, or instruction sets and / or data hard-coded in a memory device (e.g., a non-volatile memory device).

[0200] If implemented in software, the functionality can be stored on or transmitted via a computer-readable medium as one or more instructions or code. Computer-readable media includes computer-readable storage media. A computer-readable storage medium can be any available storage medium accessible to a computer. For example, and not limitingly, such computer-readable storage media can include FLASH storage media, RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, disk storage devices or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and is accessible to a computer. As used herein, “disk” and “optical disc” include compact optical discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs (BDs), where disks typically copy data magnetically and optical discs typically copy data optically using lasers. Furthermore, the propagation of signals can also be included within the scope of computer-readable storage media. Computer-readable media also includes communication media, which includes any medium that facilitates the transfer of a computer program from one place to another. For example, a connection can be a communication medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are all included in the definition of communication media. Combinations of the above should also be included within the scope of computer-readable media.

[0201] The applicant hereby individually discloses each individual feature described herein, as well as any combination of two or more such features, provided that such features or combinations can be implemented based on the entire specification and in view of common general knowledge of those skilled in the art, regardless of whether such features or combinations of features solve any problem disclosed herein, and without limiting the scope of the claims. The applicant notes that aspects of the invention can consist of any such individual features or combinations of features.

[0202] It should be noted that embodiments of the present invention are described with reference to different categories. Specifically, some examples are described with reference to methods, while others are described with reference to apparatus. However, those skilled in the art will understand from the specification that, unless otherwise stated, any combination of features related to different categories, in addition to any combination of features belonging to the same category, is also considered to be disclosed in this application. However, all features can be combined to provide a synergistic effect greater than the simple sum of the features.

[0203] While the invention has been detailed and described in the accompanying drawings and foregoing description, these illustrations and descriptions should be considered exemplary rather than limiting. The invention is not limited to the disclosed embodiments. Those skilled in the art will understand and implement other variations of the disclosed embodiments by studying the drawings, the disclosure, and the appended claims.

[0204] The fact that certain measures are stated in mutually different dependent claims does not indicate that...

[0205] The combination of these measures cannot be used advantageously.

[0206] Any reference symbols in the claims should not be construed as limiting the scope.

Claims

1. A method for assisting in assessing confidence estimates in structured representations of information in an industrial plant environment, the method comprising: acquiring (S510) a target structured representation (150, 160) of information, the target structured representation of information comprising one or more target information segments (160a, 160b) associated with confidence estimates determined by an information model, IM (140), wherein at least a portion of the one or more target information segments overlap one or more information segments comprised by one or more different structured representations; generating (S520) a confidence heat map based on the target structured representation, wherein the generated confidence heat map indicates confidence for the one or more target information segments based on the associated confidence estimates; and overlaying (S530) the generated confidence heat map over at least a portion of an input document of one or more input documents (110, 120, 130) input (S110, S120, S130) to the IM, and the target structured representation and the one or more different structured representations are acquired by the IM based on the one or more input documents, wherein the one or more input documents are associated with a same process plant (300).

2. The method of claim 1, wherein the confidence estimates associated with multiple target information segments overlapping multiple information segments of different structured representations are joint confidence estimates; wherein the method further comprises: determining a joint confidence estimate for a target information segment overlapping one or more information segments of the different structured representations based on computing a weighted sum of respective confidence estimates associated with the target information segment and the one or more information segments of the different structured representations overlapping each other.

3. The method of claim 2, wherein computing the weighted sum comprises: assigning a higher weight to a confidence estimate associated with a first certainty than to a confidence estimate associated with a second certainty, the second certainty being lower than the first certainty.

4. The method of any one of claims 1 to 3, wherein acquiring the target structured representation comprises: acquiring structured representations of information comprising a first structured representation based on inputting the one or more input documents (110, 120, 130) to the IM, wherein each of the acquired structured representations comprises one or more information segments associated with confidence estimates determined by the IM; determining joint confidence estimates for multiple information segments of different structured representations of the acquired structured representations overlapping each other as a function of the confidence estimates; associating one or more first information segments of the first structured representation overlapping one or more information segments of different structured representations of the acquired structured representations with the determined corresponding joint confidence estimates; based on the association, for the one or more first pieces of information associated with the first confidence estimate determined by the IM, replacing at least part of the first confidence estimate with the determined corresponding joint confidence estimate; and obtaining (S140) the target structured representation as a result of the replacing.

5. The method of any one of claims 1 to 4, further comprising: displaying the input document and the generated confidence heat map overlaid on the at least part of the input document.

6. The method of claim 5, wherein the displaying further comprises: if the confidence heat map comprises a confidence value that violates a predetermined confidence threshold, displaying notification information.

7. The method of any one of claims 1 to 6, wherein the plurality of pieces of information (160a, 160b) comprised by the target structured representation is obtained from a plurality of pieces of information (210a, 210b, 220a-220c, 230, 240a, 240b, 250a-250d, 260, 270a-270h) comprised by the one or more input documents (110, 120, 130).

8. The method of any one of claims 1-7, wherein generating the confidence heat map comprises: different confidences comprised by the confidence heat map are indicated in different colors and / or by different patterns.

9. The method of any one of claims 1 to 8, further comprising: based on an uncertainty indicated by a confidence comprised by the confidence heat map, based on determining the joint confidence estimate using a further structured representation obtained from a further input document associated with the same process plant (300), modifying the target structured representation (150, 160).

10. The method of any one of claims 1 to 9, further comprising: obtaining a plurality of input documents (110, 120, 130, 200, 301-312) associated with the same process plant (300); obtaining the target structured representation comprises obtaining a respective target structured representation for each of the plurality of input documents; generating the confidence heat map based on the target structured representation comprises generating a plurality of confidence heat maps based on the plurality of target structured representations; and overlaying each of the generated plurality of confidence heat maps on a respectively corresponding input document of the plurality of input documents.

11. The method of any one of claims 1 to 10, further comprising: obtaining the plurality of input documents (110, 120, 130, 200, 301-312) associated with the same process plant (300); determining a joint confidence heat map for the plurality of input documents based on combining the plurality of confidence heat maps generated for the plurality of input documents; and overlaying the joint confidence heat map on the plurality of input documents representing at least part of the same process plant.

12. A data processing apparatus (600) for assisting in evaluating probability estimates in structured representations of information in an industrial plant environment, the data processing apparatus comprising a processor configured to perform the method of any one of claims 1 to 11.

13. A data processing system (100) for assisting in the evaluation of a probability estimate in a structured representation of information in an industrial plant environment, the data processing system (100) comprising a data processing apparatus (600) according to claim 12.

14. A computer readable medium comprising instructions, which when executed by a computing system, cause the computing system to perform the method according to any one of claims 1 to 11.

15. A computer program product comprising instructions, which when executed by a computing system, enable the computing system to perform and / or cause the computing system to perform the method according to any one of claims 1 to 11.

Images