Data processing brain ontology enhanced autonomous agent and hybrid expert system

Abstract

The embodiment of the invention relates to a data processing brain ontology enhanced autonomous agent and hybrid expert system. A method for integrated engineering data processing in an industrial plant is disclosed. The method includes providing access to a domain knowledge representation associated with an engineering project to a large language model-based (LLM-based) autonomous agent. The method includes applying, based on and / or in alignment with a domain knowledge representation, an LLM-based autonomous agent, the LLM-based autonomous agent providing access to orchestrate, by the LLM-based autonomous agent, at least the following: obtaining first data indicative of engineering data associated with an engineering project; selecting one or more processing tools for processing the first data based on one or more types of information provided in the engineering data and / or based on the type of the engineering project; applying the selected one or more processing tools to the first data; second data is obtained based on the application.

Description

Technical Field

[0001] This invention relates to the brain of EPC data processing. Specifically, this invention relates to an ontology-enhanced autonomous agent and hybrid expert system for data processing (orchestration) in P&A engineering. Background Technology

[0002] To date, human engineering experts have had to manually process data from EPC clients, such as piping & instrumentation diagrams (P&ID), control narratives (CN), input / output lists (IO lists) or label lists, single-line diagrams, functional descriptions, logic diagrams, motor and consumer lists, and instrument lists, in order to derive insights and representations applicable to further steps in the process and automation engineering (P&AEng) workflow, and these insights and representations are available using existing P&AEng tools.

[0003] If this is automated, it can not only save a lot of manpower, but also allow for the reduction of errors when processing data one by one and thus overlooking other parts of the data.

[0004] For automating the processing of unstructured data, AI / ML models are likely the best choice. Therefore, the current state of technology is strong for specific expert models used for data processing, such as object detection models for P&ID and text extraction models for controlling narratives. Many of these are now based on transformer architectures.

[0005] However, processing individual project data one by one does not provide a complete picture, and if the data is not processed as a whole, a great deal of information will be lost or ignored, such as the project's P&ID data as well as the control narrative (CN) and IO list for the same project.

[0006] Therefore, there are several drawbacks in processing data for individual engineering projects. Summary of the Invention

[0007] In view of the above, the purpose of this disclosure is to overcome at least some of the shortcomings that may exist in the processing of data for individual engineering projects.

[0008] Therefore, to address one or more of these shortcomings, in a first aspect, a method for comprehensive engineering data processing in an industrial plant is provided. The method includes providing access to a domain knowledge representation associated with an engineering project to an autonomous agent based on a Large Language Model (LLM). The method also includes applying the LLM-based autonomous agent, based on and / or aligned with the domain knowledge representation, the LLM-based autonomous agent providing access to orchestrate at least the following: acquiring first data indicating engineering data associated with an engineering project; selecting one or more processing tools for processing the first data based on one or more types of information provided in the engineering data and / or based on the type of an engineering project; applying the selected one or more processing tools to the first data; based on the application, acquiring second data indicating one or more intermediate results obtained from processing the first data by the selected one or more processing tools; performing a comprehensive analysis of the second data based on examining inconsistencies and / or optimization potential within the second data; and based on the results of the comprehensive analysis, acquiring third data indicating a structured representation of the first data.

[0009] The first approach can be implemented using a computer.

[0010] A comprehensive analysis of the second data may include a comprehensive analysis of the first and second data based on examining inconsistencies and / or optimization potential within the second data.

[0011] It should be noted that the term "domain knowledge representation" can refer to a structured, formalized representation of a domain of interest (e.g., the P&A Eng domain), defining relevant concepts / things, their attributes / features, their (mutual) relationships or dependencies, and related management rules or axioms. Such domain knowledge representation can be as simple as a vocabulary (i.e., a list of concepts / things in the domain), a taxonomy, or even a more sophisticated ontology, or other types of domain information models (applicable to the domain). Furthermore, the term "orchestration" can refer to coordinating a series of subsequent and / or parallel actions (performed by software). Therefore, orchestration can mean that subsequent and / or parallel actions (performed by software) are coordinated. Acquiring first data can also include acquiring engineering data. Acquiring second data can also include acquiring intermediate results. For example, comprehensive analysis can also be understood as joint analysis. Acquiring third data can also include acquiring structured representations.

[0012] The advantage of the approach based on the first aspect is that it can participate in processing the entire dataset from the corresponding engineering project in order to obtain a comprehensive and consistent output containing all the information, just as a human engineering expert would. Therefore, this method considers the outputs from different expert models, such as the output from an expert model for images like P&ID and the output from an expert model for text like CN, and combines them into a larger context while accessing and maintaining consistency with the appropriate underlying expert domain knowledge representation.

[0013] According to several examples of this disclosure, the engineering data may be first engineering data provided in a first data file. The first data may also indicate second engineering data provided in a second data file. The second engineering data may be associated with the first engineering data in the same engineering project. Selection may include choosing one or more processing tools for processing the first data based on at least one of the following:

[0014] -The data type of the first data file,

[0015] -The data type of the second data file,

[0016] -One or more types of information provided in the first engineering data

[0017] -One or more types of information provided in the second engineering data, and

[0018] - The same type of engineering project.

[0019] Therefore, comprehensive processing can be achieved. Consequently, more complete, reliable, and accurate intermediate results and / or structured representations can be obtained.

[0020] According to several examples in this disclosure, the type of an engineering project may include at least one of the following: mining, oil and gas, food and beverage, and chemical and refining. One or more types of information include at least one of the following: text, tables, images, and time-series data. Data types may include at least one of the following: proprietary formats, user-specific formats, PDF files, XML files, JSON files, and JPG files.

[0021] Therefore, comprehensive processing can be further achieved. Consequently, more complete, reliable, and accurate intermediate results and / or structured representations can be obtained.

[0022] According to several examples of this disclosure, selecting one or more processing tools may include choosing one or more processing tools from a pre-selected pool of processing tools. Additionally or alternatively, selecting one or more processing tools may be further based on obtaining application and compatibility information regarding the pre-selected processing tools. Additionally or alternatively, selecting one or more processing tools may also include configuring the selected processing tools based on one or more types of information provided in the first and / or second engineering data and / or based on the type of an engineering project.

[0023] Therefore, it enables more comprehensive, specific, and personalized processing. Consequently, it allows for the acquisition of more comprehensive, reliable, and accurate intermediate results and / or structured representations.

[0024] According to several examples of this disclosure, applying one or more selected processing tools to first data may include applying at least one of the selected one or more processing tools to at least one of the first and second engineering data. The application may be based on at least one of the following

[0025] -The type of information provided in the first engineering data,

[0026] -The data type of the first data file,

[0027] -The type of information provided in the second engineering data,

[0028] - The data type of the second data file, and

[0029] - A type of engineering project.

[0030] Therefore, it is possible to achieve more comprehensive, specific, and personalized processing. Consequently, more comprehensive, reliable, and accurate intermediate results and / or structured representations can be obtained.

[0031] According to several examples of this disclosure, acquiring second data may include acquiring one or more first intermediate results from processing first engineering data using at least one of one or more selected processing tools, acquiring a first confidence value for the acquired one or more first intermediate results based on the processing, and assigning the acquired first confidence value to the acquired one or more first intermediate results. Additionally or alternatively, acquiring second data may include acquiring one or more second intermediate results from processing second engineering data using at least one of one or more selected processing tools, acquiring a second confidence value for the acquired one or more second intermediate results based on the processing, and assigning the acquired second confidence value to the acquired one or more second intermediate results.

[0032] Therefore, based on the confidence score, transparency is improved, user understandability is enhanced, and user trust in the results increases. Additionally, a low confidence score can prompt the user that further manual verification is needed for the relevant aspects or content.

[0033] According to several examples of this disclosure, the comprehensive analysis can further examine inconsistencies and / or optimization potential based on one or more first intermediate results and one or more second intermediate results and / or across one or more first intermediate results and one or more second intermediate results. Examining inconsistencies and / or optimization potential includes at least one of the following:

[0034] - Compare at least portions of one or more first intermediate results with at least portions of one or more second intermediate results, and

[0035] - Compare at least a portion of the first confidence value and at least a portion of the second confidence value with each other.

[0036] It should be noted that examining contradictions and / or optimization potential can be understood as including the separate analysis of one or more first intermediate results and one or more second intermediate results, and examining contradictions and / or optimization potential can be further understood as including the combined analysis of one or more first intermediate results and one or more second intermediate results, i.e., the combined analysis across one or more first intermediate results and one or more second intermediate results.

[0037] Therefore, the alignment, consistency, reliability, and accuracy of intermediate results, outcomes, and structured representations are further improved by comparing and adding relevant insights.

[0038] According to several examples of this disclosure, the comprehensive analysis may also include, if one or more contradictions and / or optimization potentials are identified in the second data and / or in one or more first intermediate results and one or more second intermediate results, repeatedly applying one or more selected processing tools to the first data, taking into account the identified one or more contradictions and / or one or more optimization potentials.

[0039] It should be noted that the phrase "considering one or more identified contradictions and / or one or more optimization potentials" can mean that one or more identified contradictions and / or one or more optimization potentials can be used as input for repetition, so that the LLM-based autonomous agent is aware of the one or more identified contradictions and / or one or more optimization potentials when performing repetition.

[0040] It should be noted that, for reasons of understandability, the application of one or more processing tools selected is not hard-coded, but orchestrated by an LLM-based autonomous agent, for example, based on the LLM-based autonomous agent's understanding of "contradictions" (or "consistencies").

[0041] Therefore, due to repetition, contradictions can be reduced at least, and optimization potential can be utilized more effectively.

[0042] According to several examples of this disclosure, the method may further include repeated application, acquisition, and comprehensive analysis until predetermined analysis criteria are met. The predetermined analysis criteria may be at least one of the following:

[0043] - The comprehensive analysis results in no contradictions and / or no longer include optimization potential.

[0044] - Reach the predetermined maximum number of iterations

[0045] - The contradiction of achieving the predetermined minimum quantity, and

[0046] - For example, reaching a predetermined minimum optimization potential threshold, thus achieving a predetermined minimum amount of optimization potential.

[0047] Therefore, this standard enables efficient comprehensive analysis.

[0048] According to several examples of this disclosure, obtaining third data may include, if the result of a comprehensive analysis is obtained based on meeting predetermined analysis criteria, combining the second data into a consistent context based on a predetermined alignment mechanism; and generating a structured representation from the consistent context.

[0049] It should be noted that the so-called consistent context can mean, for example, that the processed / extracted content from the first data (e.g., P&ID) is not contradictory (i.e., consistent) with the processed or extracted content from the second data (e.g., CN). Specifically, this might mean that the information extracted from the first data can be extended by the content extracted from the second data, but this should not lead to inconsistencies (e.g., P&ID seems to show that tank X is *directly* connected to valve A at the input, but CN describes an additional component (e.g., a controller) between valve A and tank X).

[0050] Therefore, a more reliable and accurate structured representation can be obtained due to the consistent context.

[0051] According to several examples of this disclosure, the method may also include providing the user with at least one of the following information:

[0052] -One or more intermediate results for the user

[0053] - Assign one or more intermediate results with confidence values ​​to the user.

[0054] -The process of obtaining the user's first data.

[0055] -The process of obtaining second data for users

[0056] -The process of obtaining third-party data for users.

[0057] - The process of selecting one or more processing tools

[0058] - The process of applying one or more selected processing tools, and

[0059] -The process of comprehensive analysis.

[0060] Additionally or alternatively, logging capabilities may be provided for workflows and / or data processing of an LLM-based autonomous agent used to provide information to users.

[0061] It should be noted that the information provided to users can be understood as information that improves transparency or traceability, which can improve the user's transparency and / or traceability in the orchestration and / or control and / or processing of LLM-based autonomous agents.

[0062] As a result, transparency and traceability are improved, and users' trust in the data and / or results obtained is further enhanced.

[0063] According to several examples of this disclosure, the method may also include receiving user feedback regarding the provided information; and based on the received user feedback, performing and / or repeating at least one of the following:

[0064] - Fine-tuning the autonomous agent based on a large language model

[0065] - Fine-tune one or more of the selected processing tools

[0066] - Obtain the first data,

[0067] - Select one or more processing tools,

[0068] - Apply one or more processing tools of your choice.

[0069] - Obtain the second data,

[0070] -Comprehensive analysis, and

[0071] - Retrieve third-party data.

[0072] Therefore, due to user feedback, the quality of the acquired data and / or results is further improved and may be more suitable for specific situations, such as specific types of engineering projects.

[0073] According to several examples of this disclosure, the method may further include applying an LLM-based autonomous agent, which provides access to further orchestrate the following based on domain knowledge representation: the achievement of a planning objective, which indicates the acquisition of a structured representation from data obtained from the LLM-based autonomous agent; and planning, execution, and / or repetition of at least one of the following:

[0074] - Select one or more processing tools,

[0075] - Apply one or more processing tools of your choice.

[0076] - Obtain the second data,

[0077] -Comprehensive analysis, and

[0078] - Retrieve third-party data.

[0079] It should be noted that planning can include an LLM-based autonomous agent autonomously planning (based on its language-based analysis and understanding of first data / (mid-term) / outcomes / second data / (mid-term) / outcomes and domain knowledge representations, as well as available tools) several processing steps to be performed in order to potentially achieve a predetermined goal. The planning of an LLM-based autonomous agent can be improved through training or fine-tuning (e.g., training is not necessarily required in the case of using LLM).

[0080] According to several examples in this disclosure, in order to train an LLM-based autonomous agent (an LLM-based autonomous agent system), the LLM (which is to act as an LLM-based autonomous agent) can be trained or fine-tuned or prompted, for example, with a few hints, to specify its plan for solving problems or generating results by means of an output string.

[0081] Therefore, automation has been further improved, and users are provided with further and / or more suitable and goal-oriented support. Thus, specific tool uses for specific fields (such as, for example, oil & gas and mining) can be indicated.

[0082] According to several examples of this disclosure, orchestration via an LLM-based autonomous agent may also include reflecting on and / or self-reflecting on the acquisition of second data and / or the acquisition of third data based on the objectives to be achieved. Additionally or alternatively, orchestration via an LLM-based autonomous agent may also include further processing and / or providing the second and / or third data based on the results of the reflection and / or self-reflection.

[0083] It should be noted that reflection and / or self-reflection may include the ability of an LLM-based autonomous agent to compare its executed processes with the processes planned prior to the processes. If an LLM-based autonomous agent can determine that a process has deviated from its plan by more than a predetermined amount, for example, due to the non-use or non-as planned use of one or more tools, the LLM-based autonomous agent may adapt its processes based on the results of the reflection or self-reflection, for example, by using the unused tools.

[0084] Therefore, due to self-reflection and / or introspection, the comprehensiveness, consistency, consistency, accuracy, and reliability of the acquired data and the acquired structured representation are further improved. According to a second aspect, a data processing apparatus for comprehensive engineering data processing in an industrial plant is provided. The data processing apparatus includes a processor configured to perform the method of the first aspect.

[0085] The advantage of the data processing device according to the second aspect is that it can participate in processing the entire dataset from the corresponding engineering project in order to obtain a comprehensive and consistent output containing all the information, just as a human engineering expert would. Therefore, the data processing device considers the outputs from different expert models, such as the output from an expert model for images like P&ID and the output from an expert model for text like CN, and combines them into a larger context while accessing and maintaining consistency with the appropriate underlying expert domain knowledge representation.

[0086] According to a third aspect, a data processing system for integrated engineering data processing in an industrial plant is provided. The data processing system includes the data processing apparatus of the second aspect, and / or the data processing system includes components for performing the method of the first aspect.

[0087] The advantage of the data processing system according to the third aspect is that it can participate in processing the entire dataset from the corresponding engineering project in order to obtain a comprehensive and consistent output containing all the information, just as a human engineering expert would. Therefore, the data processing system considers the outputs from different expert models, such as the output from an expert model for images like P&ID and the output from an expert model for text like CN, and combines them into a larger context while accessing and maintaining consistency with the appropriate underlying expert domain knowledge representation.

[0088] According to the fourth aspect, an industrial plant is provided, which includes the data processing apparatus of the second aspect and / or the data processing system of the third aspect.

[0089] Based on several examples, the term "industrial plant" may refer to an industrial plant or industrial production plant that includes one or more pipelines, production lines, and / or assembly lines for converting one or more segregants into products and / or for assembling one or more components into a final product. Based on several examples, this may refer to industrial plants in the oil, gas, mining, chemical, wind and power, or food and beverage industries.

[0090] The advantage of the industrial factory according to the fourth aspect is that it can participate in processing the entire dataset from the corresponding engineering project in order to obtain a comprehensive and consistent output containing all the information, just as a human engineering expert would. Therefore, the industrial factory considers the outputs from different expert models, such as the output from an expert model for images like P&ID and the output from an expert model for text like CN, and combines them into a larger context while accessing and maintaining consistency with the appropriate underlying expert domain knowledge representation.

[0091] According to a fifth aspect, a computer-readable medium is provided, comprising instructions that, when executed by a computing system, cause the computing system to perform the method of the first aspect. The computer-readable medium may be temporary or non-temporary, volatile or non-volatile.

[0092] The advantage of the computer-readable medium according to the fifth aspect is that it can participate in the processing of the entire dataset from the corresponding engineering project in order to obtain a comprehensive and consistent output containing all the information, just as a human engineering expert would. Therefore, the computer-readable medium takes into account the outputs from different expert models, such as the output from an expert model for images like P&ID and the output from an expert model for text like CN, and combines them into a larger context while accessing and maintaining consistency with the appropriate underlying expert domain knowledge representation.

[0093] According to a sixth aspect, a computer program product is provided, comprising instructions that, when executed by a computing system, enable the computing system to perform the method of the first aspect or cause the computing system to perform the method of the first aspect. The computer program product may include a computer-readable medium that includes the instructions of the computer program product.

[0094] The advantage of the computer program product according to the sixth aspect is that it can participate in processing the entire dataset from the corresponding engineering project in order to obtain a comprehensive and consistent output containing all the information, just as a human engineering expert would. Therefore, the computer program product considers the outputs from different expert models, such as the output from an expert model for images like P&ID and the output from an expert model for text like CN, and combines them into a larger context while accessing and maintaining consistency with the appropriate underlying expert domain knowledge representation.

[0095] According to the seventh aspect, there is provided a data processing apparatus of the second aspect, and / or a data processing system of the third aspect, and / or the use of an industrial plant of the fourth aspect.

[0096] The advantage of the application according to the seventh aspect is that it can participate in the processing of the entire dataset from the corresponding engineering project in order to obtain a comprehensive and consistent output containing all the information, just as human engineering experts would do. Therefore, this application takes into account the outputs from different expert models, such as the output from an expert model for images like P&ID and the output from an expert model for text like CN, and combines them into a larger context while accessing and maintaining consistency with the appropriate underlying expert domain knowledge representation.

[0097] The optional features of the first aspect can be modified as necessary in detail to form any part of the second to seventh aspects.

[0098] As used herein, the term “acquisition” can include, for example, receiving from another system, device, or process; receiving via interaction with a user; loading or retrieving from a storage device or memory; or measuring or capturing using a sensor or other data acquisition device.

[0099] As used herein, the term "determine" encompasses a wide variety of actions and may include, for example, calculation, estimation, processing, deduction, investigation, lookup (e.g., searching in a table, database, or other data structure), ascertainment, etc. Furthermore, "determine" may include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Additionally, "determine" may include resolving, selecting, picking, establishing, etc.

[0100] The indefinite articles “a” or “a kind” do not preclude plural forms. Furthermore, unless otherwise specified or the context clearly indicates a singular form, the articles “a” and “a kind” should generally be interpreted as “one or more”, as used herein.

[0101] Unless otherwise specified or the context clearly indicates otherwise, as used herein, the phrases “one or more of A, B, and C,” “at least one of A, B, and C,” and “A, C, and / or B” are intended to refer to all possible permutations of one or more of the listed items. That is, the phrase “A and / or B” means (A), (B), or (A and B), while the phrase “A, B, and / or C” means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C).

[0102] The term "comprising" does not exclude other elements or steps. Furthermore, the terms "comprising," "including," and "having" are used interchangeably herein.

[0103] This invention may include one or more aspects, examples, or features, either individually or in combination, whether specifically disclosed in the combination or individually. Any optional feature or sub-aspect of one of the foregoing aspects may suitably apply to any other aspect.

[0104] The above aspects will become clear and clarified by referring to the detailed description provided below. Attached Figure Description

[0105] A detailed description will now be given by way of example only, with reference to the accompanying drawings, in which:

[0106] - Figure 1 The agent system is shown;

[0107] - Figure 2 Several example systems according to this disclosure are shown;

[0108] - Figure 3 Flowcharts indicating methods according to several examples of this disclosure are shown; and

[0109] - Figure 4 A block diagram schematically illustrating several examples of a data processing apparatus according to this disclosure is shown. Detailed Implementation

[0110] In the context of an engineering data processing funnel (for illustrative purposes, this includes, for example, processing several input data points to obtain one output data point, i.e., feeding several input data points into a "funnel" that outputs one output data point), different types of engineering design specification data from various engineering projects (such as current greenfields and subsequent brownfields) will be processed, such as engineering design, procurement, and construction (EPC) or end-customer data. These different types of data can include P&IDs, control statements, and IO / tag lists, to name just a few. For greenfields, data can be newly entered; for brownfields, data can be, for example, as a P&ID input highlighting changes.

[0111] The goal is to extract a structured representation of the content contained in such unstructured input data. For example, the structured representation to be obtained could be in a graph format, such as OWL or RDF, or any other kind of structured, hierarchical format, such as XML or JSON-based. To automatically process unstructured input data to obtain a structured representation, large-scale artificial intelligence / machine learning (AI / ML) models are likely the best choice, many of which are likely based on transformer architectures.

[0112] However, processing individual project data piecemeal does not provide a complete picture, and if the data is not processed as a whole but rather in such a step-by-step manner, a significant amount of information will be lost or overlooked. In cases where several data points (such as different data files) are available and associated with the same project, step-by-step processing can be understood as processing each file in these different data files sequentially and independently of each other. Therefore, each data point, data block, or data file is processed individually or specifically. Consequently, there is a risk of significant information loss or oversight. For example, insights cannot be gained from cross-analysis between different data files. Additionally or alternatively, in cases where data is available, associated with the same project, and includes several different types of information, such as images with text (i.e., image-based information and text-based information), step-by-step processing can be understood as processing each of these different types of information sequentially and independently of each other. Therefore, each type of information is processed individually or specifically. Consequently, there is a risk of significant information loss or oversight. For example, insights cannot be gained from cross-analysis between different types of information.

[0113] Therefore, processing the entire dataset from the corresponding engineering project would be meaningful and advantageous (compared to process-by-process) in order to obtain a comprehensive and consistent output containing all the information, just as a human engineering expert would do for the corresponding engineering project. For this purpose, a (more sophisticated) data processing system is needed that can consider the outputs from different expert models (such as an expert model for images like P&ID and an expert model for text like CN), and that combines these outputs into a larger context while having access to appropriate underlying expert domain knowledge representations. While having access, the data processing system can maintain consistency with appropriate underlying expert domain knowledge representations. Based on several examples of this disclosure, a suitable setup for such a data processing system is described.

[0114] Based on several examples of this disclosure, a system and method for comprehensive engineering data processing are provided, wherein the system (and method) are described in more detail below.

[0115] This system can be based on an autonomous agent or autonomous agent system that manages the overall data processing and, as part of it, autonomously routes input data (such as, for example, project data) to appropriate further processing expert models and / or expert algorithms, as detailed below. These "experts" can then be executed sequentially for the actual input data and information processing and / or information transformation, for example, to route image input files toward an image processing expert model.

[0116] The system can be based on a hybrid of the aforementioned "experts," namely, a "hybrid expert" (MoE) that incorporates expert ML / AI models and / or expert data processing algorithms. Examples include a visual transformer model for processing images like P&ID, a large language model (LLM) model for processing text like CN, and a hard-coded if-else-tree algorithm for processing tables like engineering I / O / tag lists.

[0117] The system can take into account (or the system can converge) the acquired embedding representations and / or outputs from different expert models (within the MoE).

[0118] This system can combine the acquired embedding representations and / or outputs into a larger or consistent context, i.e., the embedding representation, to generate structured or summarized outputs. The consistency of such structured or summarized outputs can be guaranteed by an underlying alignment mechanism. This underlying alignment mechanism can either be performed on separately processed data (handled by the respective experts) or properly addressed during the training of multimodal embeddings, for example, by using contrastive learning methods or similar approaches.

[0119] When performing the data processing described above, the system can access (and remain consistent with) underlying expert domain knowledge representations, such as domain information models or domain ontology, to control, protect, and / or guide data processing and output generation.

[0120] In addition, the system can allow for feedback mechanisms to record traceability and transparency, and can enable I / O visualization to improve user understanding and user trust.

[0121] By using a proxy wrapper—that is, an autonomous proxy that wraps around the aforementioned tools, underlying domain knowledge, and MoE—the system gains the ability to plan, use tools, collaborate across multiple tools, and reflect. In doing so, the system can derive optimal results in a comprehensive, structured representation.

[0122] Based on several examples of this disclosure, the generated output can be used to populate the underlying knowledge representation ontology with instances, i.e., specifically, ontology concepts, thereby allowing the retrieval of a comprehensive and consistent representation or representation instantiation from the entire EPC data or EPC information, i.e., from different sets of data and data modalities. Conversely, it should be noted that, for example, if the context is unclear, each individual data or data file itself may only allow the extraction of partial information, or may even allow the extraction of uncertain, indeterminate, and / or erroneous information.

[0123] According to several examples of this disclosure, an output convergence expert model can be set at the end of the processing pipeline. This output convergence expert model may be an LLM, or "output guardian LLM," to converge the output from the "expert" before the final result of the structured information is output from the engineering data funnel to the user, and, for example, to self-reflect and / or maintain consistency.

[0124] According to several examples of this disclosure, the system may further have logging capabilities for both workflow and data processing, providing users (who may primarily be human engineering experts) with updates on what the system is currently doing, what actions it is taking and why it is taking those actions, and what intermediate results or intermediate data processing results it has obtained. Therefore, users can follow the reasoning and step-by-step data processing of the system and expert models and / or expert algorithms.

[0125] According to several examples of this disclosure, the system may further have output visualization capabilities, such as in a user interface (UI), which may additionally enable the reception and processing of feedback (e.g., user feedback) for continuous improvement of the underlying system, expert models and / or expert algorithms, or embedded representations or knowledge representations (i.e., structured representations). Thus, for example, by using techniques such as PEFT LoRA, a set of initial models (and model checkpoints, e.g., their weights) and / or algorithms can be further or continuously fine-tuned, allowing the exchange or updating of active model configurations via additional low-rank adapters (wrappers).

[0126] Based on several examples in this disclosure, systems and methods for processing comprehensive engineering data will be outlined in more detail below.

[0127] This system (and method) is based on an autonomous agent system, the core of which may contain an LLM (Limited Management Model). The autonomous agent's task may be to manage data processing and, as part of this, autonomously route input data (such as, for example, project data) to expert models and / or expert algorithms for further processing. Therefore, these "experts" can then be executed sequentially for the input data and information processing, for example, to route image input files towards an image processing expert model. In this regard, it should be noted that the term "autonomous agent" refers to software typically based on one or more LLMs that can take high-level guidance, autonomously determine complex sequences or combinations of actions, and execute them to deliver results.

[0128] For example, such autonomous agents can be implemented based on and using the functions, agents, and tools of "langchain"; for example, they can be provided open source through the "langchain" Python library.

[0129] According to several examples of this disclosure, an autonomous agent or autonomous agent workflow may include: receiving instructions to process input data; selecting the most effective tool from a toolset to achieve a predetermined objective; describing at least some of the tools in the set using FAIR data principles for representation, so that the autonomous agent can “understand” the purpose and function of the tools; executing the selected tool along with the input data line, possibly even more than once; interpreting and motivating the intermediate steps of the execution, and displaying and collecting the intermediate results of the execution; putting such intermediate results together, for example, by joint analysis or combination, and determining the next processing step based on the intermediate results and / or based on the results put together (i.e., based on the results of joint analysis); and displaying and interpreting the final result obtained from the put together as the final result obtained from the execution of (the series of) steps and / or tools. Figure 1 An example provided by the "Transformer Agent" is shown schematically (this example is unrelated to P&AEng), where Figure 1 This is the documentation for the "Transformer" Python library from Huggingface, and is intended to provide a very simple example and overview of what a proxy system is, how it works, and what it can be used for. For example, (1) instructions are translated into (2) prompts, which are executed by (3) a proxy that has access to (4) tools that can be executed, for example, by means of a Python engine (5) (this is just an example and does not provide any limitations or constraints to this disclosure), in order to finally obtain (6) the desired output.

[0130] Based on several examples in this disclosure, autonomous agents can assume and require the use of formal descriptive methods to describe tools and "experts," which can be of any maturity level, for example, via tagged text descriptions, from simple natural language, or up to mature ontology representations.

[0131] Based on several examples of this disclosure, in addition to the P&A Eng domain, such as for P&A Eng components, services, IO, interrelationships, connections, and tools, models, applications, and their functions, parameters, and IO, the autonomous agent may preferably assume and utilize the same formal descriptions. Such similar formal descriptions may include FAIR data representations. Therefore, the autonomous agent may be able to utilize MoE, for example, for miscellaneous modal data processing, and for automating, supporting, and / or enhancing P&A Eng data processing and analysis.

[0132] Based on several examples in this disclosure, autonomous agents can access a “simple” set of data processing and storage tools, such as, for example, state-of-the-art document loaders, parsers, tokenizers, chunking methods, embedding methods and tools, vector storage components, vector indexing tools, etc., which are provided open source through the “langchain” library or similar libraries.

[0133] According to several examples of this disclosure, an autonomous agent can access a domain knowledge representation database containing, for example, a domain information model consistent with an ebase database for the mining or energy industry, or a domain ontology related to the same mining or energy industry and / or the MTP standard / VDI 2658, or the ebase format, or other proprietary vendor formats.

[0134] Based on several examples in this disclosure, in addition to the tools mentioned above, autonomous agents can access repositories or databases of ontology, information models, or Excel adapters / parsers, for example, for different industries such as mining, oil & gas, food & beverage, or chemical and refining, to extract concepts, such as acting as classification targets, output pattern targets, protection, etc.

[0135] Based on several examples in this disclosure, autonomous agents can access state-of-the-art retrieval augmented generation (RAG) capabilities, not only for documents, but also specifically for knowledge graphs or ontology (KG-RAG) and other data representations (X-RAG).

[0136] According to several examples in this disclosure, autonomous agents can be based on a MoE with expert ML / AI models and / or expert data processing algorithms, such as a model for processing images like P&ID (e.g., a visual transformer), a model for processing text like CN (e.g., an LLM), and an algorithm for processing tables like IO lists (e.g., a hard-coded if-otherwise tree). It should be noted that the experts can be ML / AI models or hard-coded conventional data processing algorithms. It should also be noted that each of these experts can be guided by domain knowledge represented in a knowledge representation database (mentioned above).

[0137] According to several examples of this disclosure, image processing tools or object recognition and classification tools can classify detected objects into a set of classes derived from a set of concepts in a domain ontology. Such detected objects can be understood as representing "classification targets".

[0138] According to several examples of this disclosure, when two or more detected symbols (such as, for example, cans and valves) in P&ID may (or may not) be connected, image processing tools or object recognition and classification tools can utilize the interrelationships or connections, as represented in the domain ontology, to determine whether these symbols can / may actually be connected, if relationships may exist in the ontology. Therefore, the ontology can be used to reaffirm or enhance the confidence of object detection and connection identification mechanisms, or to refute impossible connections.

[0139] According to several examples in this disclosure, the same applies to language or text transformers, such as those that control the text in a narrative, like talking about two or three “things,” “elements,” or “entities” in a sentence or context, but according to the ontology, only two of them can be related. Therefore, for example, the third “thing” must refer to the preceding or following sentence.

[0140] According to several examples in this disclosure, a database or repository for low-rank adapters (LoRA adapters) for ML / AI models can provide a set of industry-specific LoRA adapters that can be loaded and attached to a general expert model to, for example, act as wrappers around the expert model. After being acquired, for example, through specialized fine-tuning, the industry-specific LoRA adapters can represent specific characteristics or attributes of data from a particular industry, such as mining, oil & gas, food & beverage, or chemical refining.

[0141] Based on several examples in this disclosure, MoE can be equipped with several models for the same purpose, used to process the same data in parallel, and thus have greater confidence in the generated output. This can be understood as a confidence enhancement feature.

[0142] According to several examples of this disclosure, expert models with the same purpose can, for example, collaborate again via the routing function of an autonomous agent, and the result of such collaboration can be a weighted sum, such as an average, of the outputs of different expert models with the same purpose.

[0143] Based on several examples in this disclosure, the expert model set in the MoE may also include one or more commercial models, such as the future GPT5-Vision-Turbo, if any of these commercial models may perform better than the customer-specified model on certain specific data or modalities (e.g., on P&ID images).

[0144] Based on several examples in this disclosure, an autonomous agent can consider or converge acquired embedding representations and / or outputs from different expert models. The autonomous agent can combine these representations and / or outputs into a larger context or a consistent context (i.e., composing an embedding representation) to generate structured or aggregated outputs, where consistency can be guaranteed by an underlying alignment mechanism.

[0145] According to several examples of this disclosure, such alignment mechanisms can properly handle the alignment of different modalities, such as images and text, and / or underlying ontology data processing. Therefore, according to several examples of this disclosure, it can be used with Siamese neural networks (Siamese NNs) with appropriate loss functions, or it can utilize cross-attention mechanisms to directly assist in multimodal data processing.

[0146] Alternatively or additionally, according to several examples of this disclosure, contrastive learning can be used to appropriately tune or train the (global) transformer network and embeddings. For a given dataset of “similar” symbol pairs, such as P&ID symbols, and / or their corresponding words or names in a control narrative, and / or their corresponding concept names in an ontology or information model, the neural network can be tuned to move similar “thing embeddings” together and separate dissimilar “thing embeddings”.

[0147] Based on several examples of this disclosure, as mentioned above, the domain information models and / or ontology available in the knowledge representation database can be directly used for class name extraction or derivation in the corresponding expert models for multi-class classification.

[0148] According to several examples of this disclosure, it is also possible to access (and maintain consistency with) a suitable underlying expert domain knowledge representation, such as a domain information model or domain ontology, to guide or control, protect, and / or direct data processing and output generation.

[0149] According to several examples of this disclosure, by guiding the content extraction and output representation generation toward concepts containing information models or ontology representations, the content of engineering data files can be extracted and directed toward the generation of output in a Module Type Package (MTP) (VDI2658) or similar format.

[0150] Based on several examples of this disclosure, the generated output can be used to populate the underlying knowledge representation ontology with instances, i.e., specifically, ontology concepts, thereby allowing for the deriving of a comprehensive and consistent representation or instantiation from the entire EPC data or information, i.e., from a set of different data and / or data modalities. Conversely, for example, if the context is unclear, each individual data or data file may only allow the extraction of partial information, or even information that may be uncertain or unclear.

[0151] Based on several examples in this disclosure, another example of a self-governing agent workflow with self-governing agent settings might look like this:

[0152] - Receive instructions to process input data or the first data indicating the input data, such as P&ID and CN.

[0153] -Then, the autonomous agent, or the LLM used by the autonomous agent, will propose (possibly in multiple steps) a plan to achieve the goal, such as extracting all or at least part of the content or information from the first data or from the input data into a structured output format, which the LLM then executes.

[0154] Therefore, a selection is made from a set of tools, which are described so that the autonomous agent / LLM can “understand” the purpose and function of the tools, requiring one or more tools and / or the most effective one or more tools. For example, starting with document loaders, parsers, word segmenters, and embedders for image and text files or PDF files, and then using image processing expert models and text processing expert models.

[0155] - Autonomous agents / LLMs trigger to execute one or more selected tools along with initial or input data, possibly more than once, and possibly in parallel. In doing so, multi-tool collaboration, for example, with several tools, can be possible, working with and / or experts to break down large tasks into subtasks and steps, and / or plan the achievement of goals. For example, this yields structured output, such as in JSON or XML format (via another tool, such as an output parser), listing the objects identified in the P&ID, what I / O they have, how they are connected, and from the CN, what components are described, and what their specifications are.

[0156] - Explain and motivate intermediate steps, and display and / or collect secondary data indicating intermediate results. For example, collect two separate JSON / XML outputs as indicated above.

[0157] - Combine the intermediate results and, based on these intermediate results, such as the results of joint processing of second data or intermediate results, determine one or more next steps. For example, as indicated above, collect two separate JSON / XML outputs and have an autonomous agent (e.g., another LLM) run another model (e.g., another LL) to analyze how these outputs can be combined.

[0158] - Display and interpret the final result or indicate the final result, such as third data obtained from the execution of (a series of) steps and / or tools.

[0159] -Optionally, as part of the autonomous agent system setup, an output convergence expert model, possibly an LLM, such as an "output guardian LLM," can be set at the end of the processing pipeline to converge the output from the expert before feeding the final results of the engineering data funnel back to the user, and to reflect on or reiterate the documentation or expert model to maintain, for example, consistency between outputs.

[0160] Therefore, based on several examples of this disclosure, by wrapping around tools, underlying domain knowledge, and autonomous agents of MoE, the system has the ability to plan, use tools, collaborate with multiple tools, and reflect in order to ultimately arrive at the best possible outcome.

[0161] According to several examples of this disclosure, the system may further have suitable logging capabilities for both workflow and data processing in order to provide users (e.g., human engineering experts) with updates on the following:

[0162] What is the system currently doing?

[0163] -What actions is the system taking and why?

[0164] -What intermediate data processing results does the system produce?

[0165] Therefore, users can follow the main reasoning and step-by-step data processing of the system and expert models and / or expert algorithms.

[0166] Such information updates allow for improved traceability and transparency, and therefore, increased trust in the system or its outputs. Furthermore, these updates make it easier for system users or developers to identify where the system (i.e., the autonomous agent) might have gone wrong. Moreover, the system and its internal workings, i.e., its step-by-step internal processes, are also detectable in this way.

[0167] According to several examples of this disclosure, the system may further have output visualization capabilities, such as in a user interface (UI), which can also enable the reception and processing of user feedback for use in improving or continuously improving the underlying overall system, expert model, embedded representation, or knowledge representation. Thus, for example, by using techniques such as PEFT LoRA, a set of initial models or model checkpoints, and their weights, can be further or continuously fine-tuned, allowing the corresponding active model(s) configuration to be exchanged or updated via additional low-rank adapters (wrappers).

[0168] As shown in the drawing. Figure 2 This is drawn on two sheets of paper, namely sheets 2 / 4 and 3 / 4. In doing so, the arrows indicated by A1* and A2* on sheet 2 / 4 continue as A1** and A2** on sheet 3 / 4, respectively. Furthermore, the square arrows A3*, A4*, A5*, and A6* on sheet 2 / 4 correspond to the square arrows A3**, A4**, A5**, and A6** on sheet 3 / 4, respectively. Additionally, the arrows indicated by A7*, A8*, and A9* on sheet 2 / 4 continue as A7**, A8**, and A9** on sheet 3 / 4, respectively. The box indicating the engineering data funnel 220 extends from sheet 2 / 4 to sheet 3 / 4.

[0169] Figure 2 The entire system 200 is illustrated schematically. For example, in Figure 2 The left-hand portion schematically illustrates EPC data 210 for an engineering project. EPC data 210 comprises three different files or three different types of data: a first file (e.g., an image file) 211 displaying diagrams (or P&IDs), a second file (e.g., a text file) 212 including text (e.g., CN), and a third file (e.g., a table file) 213 including tables. It should be noted that, for visibility reasons, [the remaining text is incomplete and requires further context]. Figure 2 In the second document 212 shown, the term "[text]" indicates that text is included. In each of these three documents, based on symbols in the images shown in the first document 211, or based on text in the plain text included in the second document 212, or based on entries in the tables in the third document 213, including reactors, tanks, and valves. EPC data 210, which can be understood as representing the first data, is input into the engineering data funnel 220, specifically into the autonomous agent 221. The autonomous agent 221 can access several tools ( Figure 2 Artificial intelligence (AI) preparation tools 231 and several databases ( Figure 2The document, ontology, information model, and database are 232. The autonomous agent 221 selects one or more tools based on access to the tools 231 and database 232, which are suitable for application to first data or input data including a first document 211, a second document 212, and a third document 213. Therefore, the autonomous agent 221 acquires a hybrid expert (MoE) 222, which includes, for example, a visual transformer model (visual™) 222b for analyzing the first document 211, a text™ 222a for analyzing the second document 212, and a table parser algorithm 222c for analyzing the third document 213. The MoE 222 can be selected or set with further consideration of the LoRA adapter 240. Therefore, the models and / or algorithms in the MoE 222 may be suitable and / or preferred for a specific type of industry to which an engineering project belongs (e.g., mining or oil & gas industry), and / or the models and / or algorithms in the MoE 222 may be pre-determined and / or pre-adjusted for a specific type of industry to which an engineering project belongs. For example, the weights used in a model or algorithm can be predetermined based on a specific industry. When MoE 222 analyzes files 211 to 213, it acquires second data, which indicates, for example, intermediate results from the analysis. Such intermediate results can be results obtained by the autonomous agent 221 from a model or algorithm during the application of that model or algorithm. For example, when applying text TM 222a, text TM 222a can analyze the second file 212 only once or several times, and can output the results of each analysis. One or more such results acquired at the MoE level can be understood as representing one or more intermediate results. However, it should be noted that text TM 222a is not limited to being applied only to the second file 212. Rather, generally, a model or algorithm in MoE 222 can be applied to one or more data or data files from EPC data 210. Similarly, a data or data file from EPC data 210 can be analyzed by one or more models or algorithms in MoE 222. For example, text TM 222a can be used to analyze the text in the first file 211 (where possible, if text is available). The autonomous agent 221 can then generate third data 223 indicating a structured representation based on the second data. For example, the autonomous agent 221 combines and places sparse information obtained from the MoE 222 within a context. More specifically, the autonomous agent 221 can synthesize or jointly analyze the second data, i.e., the autonomous agent 221 can synthesize and analyze intermediate results from the MoE 222. In doing so, inconsistencies can be identified and further removed. For example, the vision TM 222b applied to the first document 211 can indicate that the tank can be directly connected to the reactor as an intermediate result.However, intermediate results can be obtained from the text TM 222a applied to the second document 212, which outlines that the tank must not be directly connected to the reactor, but rather that a valve must be connected between the tank and the reactor. For example, similar intermediate results regarding the indirect connection between the tank and the reactor can be obtained from analyzing the third document 213. Therefore, the autonomous agent 221 knows that the intermediate results obtained from the visual TM 222b applied to the first document 211 may be contradictory. Therefore, utilizing additional knowledge or insights, such as these, as boundary conditions, the autonomous agent 221 can reapply the visual TM 222b to the first document 211 and / or can accordingly correct the intermediate results obtained from the visual TM 222a. Thus, in doing so, the autonomous agent 221 can guarantee the consistency of data composition and / or data contextualization based on predetermined or available consistency criteria. The third data 223 can indicate or include a joint multimodal embedding representation of the original input data content (i.e., the first data content). The third data 223 can then be input into the LLM (...). Figure 2 The ultimate guardian (LLM) 224 in the system allows the autonomous agent 221 to generate a structured representation 250 from third-party data, and this structured representation 250 can be output to a user, such as a human engineering expert 260. Therefore, the autonomous agent 221 can be understood as an LLM-based autonomous agent 221. The user can further provide feedback, for example, regarding the structured representation and / or one or more of the intermediate results (i.e., regarding the processing and / or analysis of MoE 222). For example, as... Figure 2 As indicated in the documentation, user feedback can be integrated into the feedback loop for fine-tuning the model in MoE 222 (or the adapter and for joint embedding calibration).

[0170] It's important to note that in machine learning, embeddings are low-dimensional vector representations of high-dimensional data points. This is a method of representing data in a space where similar items are grouped closer together and dissimilar items are grouped further apart. This process involves mapping each data point to a point in a continuous vector space. Embeddings are commonly used in natural language processing (NLP), computer vision, and other fields where the data is inherently high-dimensional.

[0171] For example, in natural language processing, words can be represented as dense vectors in an embedding space where words with similar meanings are close to each other. This allows machine learning models to capture semantic relationships between words and generalize better across different tasks.

[0172] (Joint) multimodal embedding is a representation that combines information from multiple modalities (such as text, images, audio, or any other type of data). It is a method of unifying different types of information into a single vector space, in which the relationships between data points from different modalities can be captured.

[0173] For example, in a multimodal embedding space, if both the image of a cat and its corresponding text description are about cats, then both the image of a cat and its corresponding text description can be mapped to nearby points. This allows machine learning models to understand and infer relationships between data from different modalities.

[0174] Based on several examples of this disclosure, the use of the described system 200 can be anticipated using the following two methods:

[0175] Access the Engineering Data Funnel 220 via API / menu from, for example, a Web App UI, or via a RESTful interface, or from one of the established automation engineering tools, such as Orchestration Designer, Control Generator M, Automation Studio, Freelance Engineering, Process Graphics Editor, eBase, or Café Tools. Note that it can also connect to other tools, or the results can be provided to, for example, tools for electrical data, simulation tools, or testing.

[0176] - Access can be made remotely or locally via a Web API through function calls to the "engineering data funnel" backend on the server, with the aid of secure data processing or encryption, such as using a two-key system.

[0177] In addition to the actual EPC data (or EPC engineering data) 210 (i.e., one or more documents), users can also add additional information to the provided EPC data 210 via the UI, such as which industry to focus on, such as the processing industry (PI) or the energy industry (EN), or even which business, such as mining, O&G, chemicals and refining. With this knowledge, the system 200 can directly resolve or direct data processing requests to the appropriate experts (or to tools and appropriate industry / business-specific LoRA wrappers around the multi-purpose model), or fine-tune the experts for future use.

[0178] Based on several examples in this disclosure, the overall requirements for autonomous agent-based systems 200 to be promoted are given below. For example, the labeled description and formal representation of PAEng domains and tools (factories, components, services, functions, I / O, parameters, interrelationships, key performance indicators (KPIs), tool libraries, use cases, I / O, etc.) to enable analysis or enhanced analysis by means of autonomous agents or AI agents.

[0179] Based on several examples in this disclosure, examples of proxy activities and corresponding record outputs are given below, including planning, routing, accessing tools and information (e.g., method calls with method signatures of args / kwargs (parameters and keyword arguments), collaboration (using tools), self-reflection, self-criticism, maintaining consistency with the underlying ontology / information model, iteration, drawing conclusions, displaying interim and final results, etc.

[0180] Based on several examples of this disclosure, information regarding the processing of autonomous agents by the user of system 200 is outlined using several non-limiting examples. Autonomous agent 221 can communicate with the user as follows:

[0181] Hello, I am your helpful engineering data funnel assistant agent!

[0182] I will process the data you provide as input and explain to you step by step what I am doing, for example,

[0183] What are my plans for trying to process your files, and what tools would I use?

[0184] - I have already obtained what the (mid-game) output is, and

[0185] -How will I proceed with them, etc.

[0186] -I will provide you with the final result once I obtain it.

[0187] I will continue to monitor data processing and reflect on whether it still aligns with the initial plan or objectives and the given constraints (e.g., domain information model).

[0188] If unforeseen circumstances arise, or if things no longer align, I will inform you of an update to my plan to best mitigate the situation and further efforts to complete the task.

[0189] For all data processing, I have access to the following toolset (and their corresponding formal function descriptions):

[0190] - "Unstructured Data-Structured Tools": ...

[0191] -“Image-Object-Recognition-Tools”:…

[0192] - "Image-ocr-tools": ...

[0193] - "Text-Processing-Tools": ...

[0194] - "Table-Extraction-Tools": ...

[0195] - "Ontology - Extraction - Tools": ...

[0196] -“RAG-Tools”:…

[0197] - "JSON-output-parser-tools": ...

[0198] Specifically, for (self-)reflection, I can access the following tools:

[0199] - "Criticism-Tool": ...

[0200] Based on several examples of this disclosure, sample inputs to the engineering data funnel 220 are shown as examples:

[0201] -P&ID image / pdf files, and

[0202] -Control Narrative (CN) PDF file

[0203] For a newly established engineering project.

[0204] Based on several examples in this disclosure, the LLM-based autonomous agent 221 can respond to user-provided prompts (tasks):

[0205] I want you to process the provided unstructured engineering data and extract the content from this unstructured engineering data into a structured list (JSON object), listing all mentioned components, their unique identifiers / tags, their attributes, and how they might be linked (referring to other unique identifiers).

[0206] In response, autonomous agent 221 can communicate the following to the user:

[0207] As part of the larger topology, I've provided you with sample JSON output for a single sample component: <sample_JSON_output(extract; for each "1 object" detected / identified)>

[0208] Based on several examples in this disclosure, the LLM-based autonomous agent 221 can respond to the following prompts (tasks) given by the user:

[0209] Your job is to come up with a plan, solve the task through a series of simple Python command steps, and execute the tools and process the data.

[0210] Then, execute the plan, that is, carry out the steps one by one.

[0211] Here, after each step, reflect on what you have accomplished and whether it still aligns with your original plan, or whether you need to rethink your approach to achieve your goals.

[0212] To perform the task, you can access the following tools listed below, along with brief descriptions of what each tool can do, what I / O it consumes / generates, and what optional parameters can be set:

[0213] <Tools List>

[0214] To engage in reflection, please use the criticism tool:

[0215] <Criticism - Tools>

[0216] Please let me know every step you take (using brief printouts from the console and log files):

[0217] For example (users can choose at least one of the following):

[0218] - Tell me your initial plans.

[0219] - Tell me the corresponding steps you will take (what Python command you will execute, which tool you will use, and explain why you think it makes sense to execute the command with this tool).

[0220] - If it makes sense to do this, print out the intermediate results.

[0221] - Reflect on whether the (intermediate) results look correct and are consistent with other results obtained so far, and let me know how you proceeded in any situation.

[0222] - Please remember to provide deterministic / confidence information for all outputs (as indicated in the sample JSON output format) and always reason about the processed data relative to the domain information model / ontology.

[0223] Based on several examples in this disclosure, the LLM-based autonomous agent 221 can respond to user prompts (tasks).

[0224] Below, you will find some sample tasks and the steps to achieve the overall goal of obtaining structured output from an input file.

[0225] For example, when only CN is given, all components and named entities, along with their tags and attributes or characteristics, are extracted and then stored in a structured / hierarchical JSON file.

[0226] For example, when only P&ID is given, all symbols and their labels / IDs are extracted, along with their connections to other symbols (which also have labels / IDs), and then all of these are saved in a structured / hierarchical JSON file.

[0227] For example, given CN and P&ID, extract not only all named entities and their tags and attributes or features from CN, but also all symbols, their tags, and their connections from P&ID. For both subtasks, remember to maintain consistency with the domain ontology. Store both results in a structured, hierarchical JSON file. Furthermore, merge the two separately extracted contents and check for overall consistency. If you find any inconsistencies, annotate them with the corresponding parts of the results and iterate further to eliminate the inconsistencies, resolving any contradictions you find and resolve.

[0228] Generally, for each task, provide deterministic / confidence information for all the components, entities, and features you have identified, and store them in the corresponding fields in the structured JSON output.

[0229] Generally, for each task, combine sparse information from all tools (expert models) and place it in context, and try to maintain overall consistency by reasoning your results relative to the provided domain information model or ontology!

[0230] Please refer to the sample execution and related console / recording output below.

[0231] According to several examples in this disclosure, the output from autonomous agent 221, such as the <sample JSON output (extracted; each detected / identified "object")> indicated above, may include the following information:

[0232] <json>

[0233] Component

[0234] -ID / Tag: RE001

[0235] -Type: High pressure reactor

[0236] -Confidence:

[0237] -Individual expert confidence: Float value (%)

[0238] -Individual expert confidence (ontology enhanced): Float value (%)

[0239] -Multiple expert average confidence: Float value (%) (average of individual expert values)

[0240] --Contradictions / inconsistencies:

[0241] -Type: Low pressure reactor

[0242] -Properties:

[0243] -Capacity: 150 L

[0244] -Fill level:

[0245] -Min: 20 L

[0246] -Max: 120 L

[0247] -…

[0248] -Connected to:

[0249] -Input:

[0250] -ID / Tag: LC004 (Level controller)

[0251] -ID / Tag: Pump 002

[0252] -Output:

[0253] -ID / Tag: PC009 (Pressure controller)

[0254] -ID / Tag: Tank 037

[0255] -Type: Low pressure reactor

[0256] -Confidence: …

[0257] -Properties: …

[0258] -Connected to: …

[0259] -ID / Tag: LC004

[0260] -Type: Level controller

[0261] -…

[0262] -…

[0263] According to several examples of the present disclosure, a tool list (<tool_list>) can include a set of tools and their respective formalized (simple / atomic) functional description, i.e., a description of what can be done with the respective tool, what I / O the tool consumes or produces, and what optional parameters can be set for the tool. The tool list can include one or more of the following examples:

[0264] -“unstructured-data-structured-tool”: This tool extracts data (in miscellaneous file formats) and converts it into clean consistent JSON that is ready for chunking, embedding, and integration, e.g., into a vector database. Thus, it extracts and normalizes documents and enriches the extracted information with metadata. In addition to this, it uses techniques for document image analysis, including layout detection and visual transformers, to extract and understand the content of PDFs, images, and tables.

[0265] -I / O can be any file / / JSON.

[0266] -“image-object-recognition-tool”: This tool is for image object recognition. It provides a list of objects and their respective bounding boxes, in particular for images with objects in the P&a Eng domain, e.g., P&IDs, SCDs, etc. If an ontology is additionally provided (with a set of classes & instances), it will utilize a YOLO-X (or similar) algorithm for multi-class classification and also detect connections between identified objects (e.g., edges in P&IDs).

[0267] This tool can also optionally utilize an “image-ocr-tool” in order to additionally identify IDs / labels and detected symbols.

[0268] -I / O can be image files / / detected_object_list, bounding_box_list, label_list, connection_list.

[0269] -“image-ocr-tool”: This tool is for optical character recognition. It can identify characters, words, or numbers in an image file (such as a document scan or P&ID) and convert these into text or identifiers that can be further processed.

[0270] -I / O can be image, detected_object_list, bounding_box_list, connection_list / / label_list.

[0271] - "Text-Processing-Tool": This tool is used to process (semi-clean) text data. It can embed textual information into abstract embedding representations or extract, for example, named entities. If different identifiers (such as ontology concepts) are provided, it can extract information about this predefined concept from the text, such as how the concept relates to other things mentioned in the text, or what properties the concept has based on the text.

[0272] - "Table Extraction Tool": This tool is used to process (semi-clean) tabular data (such as real tables or tables with markup styles). It can embed tabular data into an abstract embedded representation. If a different identifier (such as an ontology concept) is provided, it can extract information about this predefined concept from the tabular data, such as what additional characteristics the concept has (e.g., in rows / columns or cross-references).

[0273] - "Ontology Extraction Tool": This tool is used to mine ontological data, such as ontology files (e.g., in OWL format) or knowledge graphs (e.g., in RDB or Graph DB). For example, it can be used to extract classes from an ontology for later multi-class classification (e.g., with the help of the "Image Object Recognition Tool").

[0274] - "RAG-Tool": This tool facilitates the retrieval of augmented generation. It can facilitate standard RAG, KG-RAG (e.g., working with ontology information or KG), etc.

[0275] - "JSON-Output-Parser-Tool": This tool ensures that the output is correct JSON; not only in terms of format, but also optionally according to a predefined JSON pattern (which it populates with parameters).

[0276] Based on several examples of this disclosure, regarding the reflection or self-reflection of autonomous agent 221, users can, for example, give autonomous agent 221 the following tasks through a "criticism-tool".

[0277] For example, when you have two tools that do the same thing, use both tools to get results, and then compare the results. Reflect on whether they are different, interconnected, or contradictory. And if they contradict each other, explain why, which piece of information is likely correct, and which is incorrect. Where you can provide confidence information (e.g., ontology confirmation).

[0278] For example, when you have two tools that process data from different modalities but about the same "thing" (e.g., using unique labels / IDs), you then compare the results, reflect on whether they are different, contradictory, or extended, and if they are different or contradictory, explain why, which one is correct, or provide confidence information.

[0279] For each tool, if a domain information model (e.g., an ontology) is provided, the extracted information is reasoned and proven based on the ontology.

[0280] Based on several examples of this disclosure, the results obtained may be as follows.

[0281] If the Image-Object-Recognition tool finds 10 objects in P&ID, and, for example, one is identified as a high-pressure reactor (85% certainty / confidence) or a low-pressure reactor (15% certainty or confidence), then it can be checked in the ontology to determine, for example, whether two or only one of these reactors is actually qualified in the area of ​​interest (e.g., oil & gas and mining), or what I / O connections these reactors will have (based on the information provided in the ontology that allows for certain I / O connections). Then, for example, if the detected symbol has two input connections and one output connection, and the IM indicates that only the low-pressure reactor also has such a connection (while the high-pressure reactor has only one input and one output), then, even though the high-pressure reactor has a higher probability (85% vs. 15%), it can be excluded from the ontology due to the enhancement.

[0282] According to several examples of this disclosure, the autonomous agent 221 may add such a deterministic or confidence value (or uncertainty quantification) to the information of the corresponding component in the structured output. In the examples above, this means that the "single expert confidence" will be 85% for the high-pressure reactor in the simple case and 0% in the ontologically enhanced case, and 15% for the low-pressure reactor in the simple case and 100% in the ontologically enhanced case. Therefore, reflection or self-reflection on the output (i.e., the result or intermediate result obtained by the autonomous agent 221) may lead to repeated execution of the probabilistic tool, for example, after some additional information has been obtained from another tool and after it is found to be inconsistent with the previous output.

[0283] Based on several examples in this disclosure, exemplary execution and console output / logging information are provided below.

[0284] Proxy = Initialize_Proxy(Chain = Data_Process_to_JSON)

[0285] Agent.Combined_tools(tools_list = [tool1, tool2, ..., toolN]) #See the list above!

[0286] Result = Agent.Run(

[0287] The command = """Processes a given input document into a structured list of things in JSON format, taking into account the given context.""", #See the more detailed tips above!

[0288] Document_list = [document1, document2, ..., documentN],

[0289] Context = [paeng_body],

[0290] Lengthy = True )

[0292] It should be noted that processing at autonomous agent 221 can be initiated using the commands outlined above.

[0293] ==Explanation from the agent==

[0294] You provided me with two input files, document 1 and document 2.

[0295] See document 1. You provided me with a jpg file, which is an image file.

[0296] I'm assuming this is an image file related to an engineering project (like a P&ID file).

[0297] I will use the following tool: `image_object_detection`, to identify the objects contained in the image and how they are linked.

[0298] If I find text, I will also use the 'ocr_ tool' to extract the text or labels associated with objects in the image.

[0299] It should be noted that the autonomous agent 221 provides users with feedback on the input fields and how the autonomous agent 221 plans to process these input files.

[0300] ==Code generated by the proxy==

[0301] Temporary Result 1 = Image Object Detection (

[0302] Image = Document1_Image,

[0303] Context = paeng_ontology,

[0304] Iteration_plan = None )

[0306] ==Ask a Question to the User==

[0307] Do you want me to execute this code in the provided saved sandbox Python environment? Yes / No

[0308] >>>Yes

[0309] ==Output from the proxy==

[0310] ...implement...

[0311]

[0312] success.

[0313] ==Result==

[0314] Using the tool "Image Object Detection", I obtained the following interim results:

[0315] '"temporary_result1.json"'

[0316] JSON

[0317] Components

[0318] -ID / Tag: RE001

[0319] Type: High-pressure reactor

[0320] - Confidence level:

[0321] - Individual expert confidence level: 85%

[0322] - Individual expert confidence (ontology augmentation): 0%

[0323] - Multi-expert average confidence level: 0%

[0324] -Contradictory / Inconsistent:

[0325] Type: Low-pressure reactor

[0326] -property:

[0327] - Capacity: 150L

[0328] - Filling level:

[0329] Minimum: 20L

[0330] -Maximum: 120L

[0331] -…

[0332] -connect:

[0333] -enter:

[0334] -ID / Tag: LC004 (Level Controller)

[0335] -ID / Tag: Pump 002

[0336] Output:

[0337] -ID / Tag: PC009 (Pressure Controller)

[0338] -ID / Tag: Jar 037

[0339] Type: Low-pressure reactor

[0340] - Confidence level:

[0341] - Individual expert confidence level: 15%

[0342] - Individual expert confidence (ontology augmentation): 100%

[0343] - Average confidence level from multiple experts: 100%

[0344] -property:...

[0345] -connect:...

[0346] -ID / Tag: LC004

[0347] -ID / Tag: ...

[0348] -...

[0349] It should be noted that the autonomous agent 221 outputs intermediate results showing potentially identified objects. For example, based solely on the image-object detection tool, the autonomous agent 221 identifies an object as a high-pressure (HP) reactor with 85% certainty, or as a low-pressure (LP) reactor with 15% certainty. Figure 2 As shown, this may be the result obtained by the autonomous agent 221 specifically analyzing the first file 211. However, when considering ontology, such as intermediate results obtained by different (selected) models or algorithms applied to the same or different input files, it may be necessary to adapt the results from object identification. For example, based on ontology enhancement analysis, the certainty that the identified object is a high-pressure reactor decreases from 85% to 0%, and the certainty that the identified object is a low-pressure reactor increases from 15% to 100%. For example, the autonomous agent 221 may have already derived the results from the above references. Figure 2 As shown in the second document 212 and / or the third document 213, a high-pressure reactor is not permitted for a given project.

[0350] ==Ask a Question to the User==

[0351] Do you want me to continue? Yes / No

[0352] >>>Yes

[0353] ==Explanation from the agent==

[0354] After processing document 1, I will move on to document 2.

[0355] See document 2; you provided me with a PDF file.

[0356] PDF files can contain various types of content, such as text, tables, and images.

[0357] I will first use the following tool: `unstructured-data-structured`, to extract the contained data and convert it into clean, consistent JSON, which is then ready for chunking, embedding, and integration, such as integration into a vector database.

[0358] ==Code generated by the proxy==

[0359] JSON-formatted document2 = unstructured data - structured data (file = document2)

[0360] ==Ask a Question to the User==

[0361] Do you want me to execute this code in the provided saved sandbox Python environment? Yes / No

[0362] >>>Yes

[0363] ==Output from the proxy==

[0364] ...implement...

[0365]

[0366] success.

[0367] ==Result==

[0368] Using the tool "unstructured_data_structured", I obtained the following interim results: [...]

[0370] ==Ask a Question to the User==

[0371] Do you want me to continue? Yes / No

[0372] >>>Yes

[0373] ==Explanation from the agent==

[0374] I have discovered that the JSON-formatted content contains both text and tables.

[0375] Therefore, I will first proceed with the text section, using the following tool `Text-Processing-Tool` to process the JSON content, and for the tables, I will use the following tool `Table-Processing-Tool` later.

[0376] It should be noted that Autonomous Agent 221 applies two tools to a single input file.

[0377] ==Code generated by the proxy==

[0378] Temporary Result 2 = Text Processing (

[0379] Input = JSON-formatted_document2,

[0380] Context = paeng_ontology,

[0381] Iteration_plan = None )

[0383] ==Ask a Question to the User==

[0384] Do you want me to execute this code in the provided saved sandbox Python environment? Yes / No

[0385] >>>Yes

[0386] ==Output from the proxy==

[0387] ...implement... [...] [...] [...]

[0391] ==Explanation from the agent==

[0392] I have now processed all the input files you provided in one go.

[0393] I will now synthesize and analyze the obtained output and reflect on its content.

[0394] For example, I will check the overall consistency with the provided domain information model (context = paeng_ontology).

[0395] For example, I will examine potential inconsistencies between interim results from different tools.

[0396] etc.

[0397] It should be noted that the autonomous agent 221 performs comprehensive analysis.

[0398] ==Code generated by the proxy==

[0399] Iteration_Plan = Self_Criticism_Of One's Own_Results (

[0400] raw_input = [document1, document2],

[0401] The generated result = [temporary result 1, temporary result 2, ...],

[0402] Context = paeng_ontology,

[0403] The prompt is: "Criticize the generated results. Examine their consistency with the provided ontology information model and suggest how and where to improve them. Examine potential inconsistencies between interim results from different tools and suggest how and where to correct them." )

[0405] ==Ask a Question to the User==

[0406] Do you want me to execute this code in the provided saved sandbox Python environment? Yes / No

[0407] >>>Yes

[0408] ==Output from the proxy==

[0409] ...implement... [...]

[0411] ==Result==

[0412] I have discovered several inconsistencies with the domain information model.

[0413] I have identified several potential contradictions between interim results from different tools.

[0414] ==Explanation from the agent==

[0415] First, I recommend reusing the tool 'Image Object Detection', along with the information I've already obtained from the tool 'Text Processing' regarding the components and their associated uncertainties. [...]

[0417] ==Code generated by the proxy==

[0418] Iteration_Plan = ["Criticize the provided results, check their consistency with the provided ontology information model or other interim results, and make suggestions on how and where to improve them."]

[0419] Interim Results 1 = Image Object Detection (

[0420] Image = Document1_Image,

[0421] Context = paeng_ontology,

[0422] Iteration_plan = Iteration_plan[0], )

[0424] ==Ask a Question to the User==

[0425] Do you want me to continue? Yes / No

[0426] >>>Yes

[0427] ==Output from the proxy==

[0428] ...implement... [...]

[0430] ==Result== [...] [...]

[0433] Repeated WHILE iterations - plan! = None

[0434] It should be noted that here, autonomous agent 221 can perform comprehensive analysis through several iterations.

[0435] According to several examples in this disclosure, in order to train an LLM-based autonomous agent system or system 200, the LLM (which is to act as an autonomous agent 221) can be trained or fine-tuned or prompted with a small amount of cueing to specify its plan for solving problems or generating results by outputting strings such as the following:

[0436] "””

[0437] {Tool: Image-Object-Detection, Input: image.jpg, Output: [image.jpg, list of detected objects.txt, list of bounding boxes.txt]}

[0438] {Tool: Image OCR tool, Input: [image.jpg, bounding_box_list.txt], Output: detected_label_list.txt}

[0439] """

[0440] Optionally, it may be prompted to output a corresponding code snippet, or it may be triggered to execute that corresponding code snippet, such as:

[0441] "””

[0442] {Tool: Image-Object-Detection, Input: image.jpg, Output: [image.jpg, list of detected objects.txt, list of bounding boxes.txt], Context = paeng_body}

[0443] {Tool: Image OCR tool, Input: [image.jpg, bounding box list.txt], Output: detected label list.txt}

[0444] ==Explanation from the agent==

[0445] I will use the following tool: `image_object_detection` to detect objects in the image.

[0446] ==Code generated by the proxy==

[0447] Image, list of detected objects, list of bounding boxes = image_object_detection(image = ...)

[0448] Input: _document_1, context = paeng_ontology)

[0449] Detected_Label_List = Image_OCR_Tool(Image = Input_Document_1, bb_List = Bounding_Box_List, Context = Paeng_Ontology)

[0450] Now for reference Figure 3 , Figure 3 A flowchart illustrating methods according to several examples of this disclosure is shown. These methods, according to several examples of this disclosure, are for integrated engineering data processing in an industrial plant.

[0451] according to Figure 3 The method can be applied to, for example, reference Figure 2 The example shown is an autonomous agent 221.

[0452] This method begins in S300.

[0453] In S310, the method includes providing access to a domain knowledge representation 231 associated with an engineering project to an autonomous agent 221 based on a large language model (LLM).

[0454] In S320, the method includes applying an LLM-based autonomous agent 221 based on and / or aligned with a domain knowledge representation, which provides access for orchestration via the LLM-based autonomous agent 221.

[0455] In S330, the method includes acquiring first data indicating engineering data 211, 212, 213 associated with an engineering project.

[0456] In S340, the method includes selecting one or more processing tools 222a, 222b, 222c to process the first data based on one or more types of information provided in engineering data 211, 212, 213 and / or based on the type of an engineering project.

[0457] In S350, the method includes applying one or more selected processing tools 222a, 222b, 222c to the first data.

[0458] In S360, the method includes, based on the application, acquiring second data indicating one or more intermediate results obtained by processing the first data by one or more selected processing tools 222a, 222b, 222c.

[0459] In S370, the method includes a comprehensive analysis of the second data based on examining inconsistencies and / or optimization potential within the second data.

[0460] In S380, the method includes obtaining third data based on the results of a comprehensive analysis, which indicates a structured representation 250 of the first data.

[0461] The method ends in S390.

[0462] The comprehensive analysis of the second data can include the comprehensive analysis of the first data and the second data.

[0463] Now for reference Figure 4 , Figure 4 A block diagram schematically illustrating a data processing apparatus 400 according to several examples of the present disclosure is shown. Specifically, according to several examples of the present disclosure, a data processing apparatus 400 for integrated engineering data processing in an industrial plant is provided. The data processing apparatus 400 includes a processor 401 configured to perform... Figure 3 The method.

[0464] More specifically, based on various examples, it is configured to execute Figure 3 The data processing apparatus 400 of the method may include a processing circuit, processing functions, processing components, processing units, or a processor 401, enabling the data processing apparatus 400 to participate in integrated engineering data processing in an industrial plant. The processor 401 may include one or more processing portions or functions, wherein the processing portions or functions may be provided as one or more physical or virtual entities. The data processing apparatus 400 may include one or more communication interfaces 402. The data processing apparatus 400 may also include a memory or storage unit 403 for storing data, programs, and / or instructions executed by the processor. The memory 403 may be internal to the data processing apparatus 400 or external to the data processing apparatus 400, such as at a cloud server. The processor 401 may include one or more portions, for example, that enable the data processing apparatus 400 to perform... Figure 3 The method. According to several examples of this disclosure, provision 410 can be configured to... Figure 3 The S310 performs such provisioning, and the application section 420 can be configured to... Figure 3 The S320 executes such an application, and the 430 can be configured to obtain part according to Figure 3 The S330 performs this acquisition, and the selection portion 440 can be configured to... Figure 3 The S340 performs such a selection, and the application section 450 can be configured according to... Figure 3 The S350 executes such an application, and the 460 portion can be configured to... Figure 3 The S360 performs this acquisition, and the comprehensive analysis section 470 can be configured to perform it according to... Figure 3 The S370 performs such a comprehensive analysis, and the 480 can be configured to perform this analysis based on... Figure 3 The S380 performs this acquisition.

[0465] Based on several examples of this disclosure, a data processing system for integrated engineering data processing in an industrial plant is provided. The system includes components as referenced above. Figure 4 The data processing apparatus 400 described herein. Additionally or alternatively, the system includes components for performing... Figure 3 The components of the method.

[0466] Data processing systems can be as described in the reference above. Figure 2 The system described herein is 200.

[0467] Based on several examples of this disclosure, an industrial plant is provided, which includes components as described in the references above. Figure 4 The data processing apparatus 400 is described above. Additionally or alternatively, the industrial plant includes, as referenced above... Figure 2 The system 200 is described in the overview.

[0468] According to several examples of this disclosure, a computer-readable medium is provided that includes instructions that, when executed by a computing system, cause the computing system to perform... Figure 3 The method. Computer-readable media can be temporary or non-temporary, volatile or non-volatile.

[0469] According to several examples of this disclosure, a computer program product is provided that includes instructions, which, when executed by a computing system, cause the computing system to perform... Figure 3 The method or to make the computing system perform Figure 3 The method. A computer program product may include a computer-readable medium that includes the instructions of the computer program product.

[0470] Based on several examples of this disclosure, a method is provided as described above. Figure 4 The data processing apparatus 400 described above and / or as referenced above Figure 2 The system 200 is described in the overview and / or the industrial plant use as described above.

[0471] according to Figure 3 The method can be implemented by a computer.

[0472] according to Figure 3 The optional features of the method can be incorporated into any data processing device 400, data processing system, industrial plant, computer-readable medium, computer program product, and application, and modified as necessary.

[0473] Any unit, module, circuit arrangement, or method described herein may be implemented using hardware, software, and / or firmware configured to perform any of the operations described herein. Hardware may include one or more processor cores, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), etc. Software may be embodied as software packages, code, instructions, instruction sets, and / or data recorded on at least one transient or non-transitory computer-readable storage medium. Firmware may be embodied as code, instructions, or instruction sets and / or data hard-coded in a memory device (e.g., a non-volatile memory device).

[0474] If implemented in software, these functions can be stored or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media includes computer-readable storage media. A computer-readable storage medium can be any available storage medium that can be accessed by a computer. By way of example, and not limitation, such computer-readable storage media can include flash memory storage media, RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, disk storage devices or other magnetic storage devices, or any other medium that can be used to carry or store required program code in the form of instructions or data structures and that can be accessed by a computer. As used herein, disks and optical discs include optical discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs (BDs), wherein disks typically reproduce data magnetically, while optical discs typically reproduce data optically using lasers. Furthermore, transmitted signals can be included within the scope of computer-readable storage media. Computer-readable media also include communication media, which includes any medium that facilitates the transfer of computer programs from one place to another. For example, a connection can be a communication medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, or microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, or microwave are included in the definition of communication media. Combinations of the above should also be included within the scope of computer-readable media.

[0475] The applicant hereby discloses individually described features and any combination of two or more such features, provided that such features or combinations can be implemented based on this specification as a whole, in accordance with common general knowledge of those skilled in the art, regardless of whether such features or combinations of features solve any problem disclosed herein, and not to limit the scope of the claims. The applicant notes that aspects of the invention can consist of any such individual features or combinations of features.

[0476] It should be noted that embodiments of the present invention are described with reference to different categories. Specifically, some examples are described with reference to methods, while others are described with reference to apparatus. However, those skilled in the art will understand from the specification that, unless otherwise notified, any combination of features related to different categories, in addition to any combination of features belonging to one category, is also considered to be disclosed in this application. However, all features can be combined to provide synergistic effects that go beyond the simple addition of features.

[0477] While the invention has been shown and described in the accompanying drawings and the foregoing description, such showing and description are to be considered exemplary and not limiting. The invention is not limited to the disclosed embodiments. Other variations of the disclosed embodiments will be understood and implemented by those skilled in the art through a study of the drawings, the disclosure, and the appended claims.

[0478] The fact that certain measures are referenced in mutually different dependent claims does not indicate that combinations of these measures cannot be used advantageously.

[0479] No reference numerals in the claims should be construed as limiting the scope.< / json>

Claims

1. A method for comprehensive engineering data processing in an industrial plant, the method comprising: Provide (S310) an autonomous agent (221) based on a large language model (LLM) to access (231) a domain knowledge representation (231) associated with an engineering project; as well as Based on the domain knowledge representation (231) and / or aligned with the domain knowledge representation (231), the LLM-based autonomous agent (221) is applied (S320), and the LLM-based autonomous agent (221) is provided with the access to orchestrate at least the following by the LLM-based autonomous agent (221): Obtain (S330) first data, the first data indicating engineering data (211, 212, 213) associated with the engineering project; Based on one or more types of information provided in the engineering data (211, 212, 213) and / or based on the type of the engineering project, select (S340) one or more processing tools (222a, 222b, 222c) for processing the first data; Apply the selected one or more processing tools (222a, 222b, 222c) to the first data (S350); Based on the application, (S360) second data is obtained, the second data indicating one or more intermediate results obtained from processing the first data by the selected one or more processing tools (222a, 222b, 222c); Based on the examination of contradictions and / or optimization potential in the second data, a comprehensive analysis (S370) is performed on the second data; as well as Based on the results of the comprehensive analysis, (S380) third data is obtained, which indicates the structured representation of the first data (250).

2. The method according to claim 1, The engineering data mentioned above is the first engineering data provided in the first data file. The first data further indicates second engineering data provided in the second data file, wherein the second engineering data and the first engineering data are associated with the same engineering project, and The selection includes choosing one or more processing tools for processing the first data based on at least one of the following: The data type of the first data file, The data type of the second data file, One or more types of information provided in the first engineering data One or more types of information provided in the second engineering data, and The same type of engineering project.

3. The method according to claim 1 or 2, The type of a project includes at least one of the following: mining, Oil and natural gas, Food and beverages, and Chemicals and oil refining; The information of one or more types includes at least one of the following: text, sheet, Images, and Time series data; The data type mentioned above includes at least one of the following: Proprietary format, User-specific format, PDF file, XML file, JSON file, and JPG file.

4. The method according to any one of claims 1 to 3, The selection of one or more processing tools includes: Select one or more of the pre-selected processing tools; and / or The selection of one or more processing tools is also based on obtaining application and compatibility information about the pre-selected processing tools; and / or Selecting one or more processing tools further includes configuring the selected processing tool based on one or more types of information provided in the first engineering data and / or the second engineering data and / or based on the type of the engineering project.

5. The method according to any one of claims 1 to 4, wherein applying the selected one or more processing tools to the first data comprises: Apply at least one of the selected processing tools to at least one of the first and second engineering data. The application is based on at least one of the following: The type of information provided in the first engineering data, The data type of the first data file, The type of information provided in the second engineering data, The data type of the second data file, and The type of an engineering project.

6. The method according to any one of claims 1 to 5, wherein obtaining the second data comprises: Obtain one or more first intermediate results from the processing of the first engineering data by at least one of the selected processing tools; obtain a first confidence value for the obtained one or more first intermediate results based on the processing; and assign the obtained first confidence value to the obtained one or more first intermediate results; and / or One or more second intermediate results are obtained from the processing of the second engineering data by at least one of the selected processing tools, a second confidence value is obtained for the obtained one or more second intermediate results based on the processing, and the obtained second confidence value is assigned to the obtained one or more second intermediate results.

7. The method according to any one of claims 1 to 6, The comprehensive analysis described therein also examines inconsistencies and / or optimization potential based on the one or more first intermediate results and the one or more second intermediate results, and / or across the one or more first intermediate results and the one or more second intermediate results. The examination of contradictions and / or optimization potential includes at least one of the following: Compare at least portions of the one or more first intermediate results with at least portions of the one or more second intermediate results, and At least a portion of the first confidence value and at least a portion of the second confidence value are compared with each other.

8. The method according to any one of claims 1 to 7, wherein the comprehensive analysis further comprises, If one or more contradictions and / or optimization potentials are identified in the second data and / or in one or more first intermediate results and one or more second intermediate results. Considering the identified one or more contradictions and / or one or more optimization potentials, the selected one or more processing tools are repeatedly applied to the first data.

9. The method according to claim 8, further comprising: Repeat the application, acquisition, and comprehensive analysis until a predetermined analysis criterion is met, which is at least one of the following: The results of the comprehensive analysis are consistent and / or no longer include optimization potential. The predetermined maximum number of iterations has been reached. The predetermined minimum amount of contradiction was achieved, and The predetermined minimum optimization potential has been achieved.

10. The method according to any one of claims 1 to 9, wherein the acquisition of the third data includes, If the predetermined analysis criteria are met and the results of the comprehensive analysis are obtained, Based on a predetermined alignment mechanism, the second data is combined into a consistent context; and The structured representation is generated from the consistent context.

11. The method according to any one of claims 1 to 10, further comprising providing the user with at least one of the following information: The one or more intermediate results given to the user The one or more intermediate results assigned to the user with confidence values. The process of obtaining the first data given to the user. The process of obtaining the second data for the user. The process of obtaining the third-party data provided to the user. The process of selecting one or more of the processing tools. The process of applying the selected one or more processing tools, and The process of comprehensive analysis; And / or the provision of recording functions for the workflow and / or data processing of the LLM-based autonomous agent, based on the provision of the information to the user.

12. The method of claim 11, further comprising: Receive user feedback regarding the information provided; as well as Based on the received user feedback, perform and / or repeat at least one of the following: Fine-tuning the autonomous agent based on the aforementioned large language model, Fine-tune the selected one or more processing tools. Obtain the first data. Select one or more of the processing tools. Apply the one or more processing tools selected. Obtain the second data. The comprehensive analysis, and Obtain the third data.

13. The method according to any one of claims 1 to 12, further comprising applying the LLM-based autonomous agent provided with said access to further orchestrate the following items based on the domain knowledge representation through the LLM-based autonomous agent: The achievement of the planning objective, wherein the objective indicates the acquisition of a structured representation from the data obtained from the LLM-based autonomous agent, and Based on the plan, perform and / or repeat at least one of the following: Select one or more of the processing tools. Apply the one or more processing tools selected. Obtain the second data. The comprehensive analysis, and Obtain the third data.

14. The method according to any one of claims 1 to 13, wherein the orchestration by the LLM-based autonomous agent further comprises: Based on the stated objectives to be achieved, reflect on and / or self-reflect on the acquisition of the second data and / or the acquisition of the third data; as well as Based on the results of the reflection and / or the self-reflection, the second data and / or the third data may be further processed and / or provided.

15. A data processing apparatus (221) for integrated engineering data processing in an industrial plant, the data processing apparatus comprising a processor configured to perform the method according to any one of claims 1 to 14.

16. A computer-readable medium comprising instructions that, when executed by a computing system, cause the computing system to perform the method according to any one of claims 1 to 14.

17. A computer program product comprising instructions that, when executed by a computing system, enable the computing system to and / or cause the computing system to perform the method according to any one of claims 1 to 14.

Images