Method and system for information extraction from structured and / or unstructured data

Abstract

The invention relates to a computer-implemented method for information extraction from structured and / or unstructured data, the method comprising: - receiving, by a data agent (3), one or more input data documents and a task, - processing, by the data agent (3), the information contained within the one or more input data documents and generating an updated document from the information contained within the one or more input data documents, - generating, by a logging agent (4), a structured document from the updated document and storing (S4) said structured document in a memory, wherein the information in the updated document is clustered into several attributes, each attribute relating to an information element within the structured document, - extracting, by an extraction agent (5), information from the structured document, wherein the extraction agent (5) comprises at least two sub extraction agents (6) each of which comprises an LLM and is configured to extract, by utilizing its respective LLM, information from one specific attribute, wherein the extraction agent (5) selects (S5) and instructs one or more of the sub extraction agents (6) to each extract (S7) information that is relevant to the task from one attribute and to provide said extracted information as output to the extraction agent (5), - generating (S8), by the extraction agent (5), a final output (9) based on the outputs of the sub extraction agents (6).

Description

[0001] METHOD AND SYSTEM FOR INFORMATION EXTRACTION FROM STRUCTURED AND / OR UNSTRUCTURED DATA

[0002] The present invention relates to a method for information extraction from structured and / or unstructured data.

[0003] The present invention further relates to an LLM-integrated system.

[0004] Moreover, the present invention relates to a computer program.

[0005] In recent years data processing has become more advanced and widespread. The large volume and complexity of data make it impractical for humans to process it efficiently and accurately. This necessitates the automation of data processing tasks. It has been known to utilize simple algorithms to analyze structured data. However, the data information may be erroneous, incomplete or hard to process. Additionally, the data may be unstructured, which cannot be analyzed with simple algorithms. This hinders widespread application of human machine interaction.

[0006] In existing NLP (Natural Language Processing) Information Extraction Technology, common Information Extraction models such as transformer-based Information Extraction model and NLE Milie are designed to extract all potential <subject, predicate, object> triples from a given sentence, their extraction focuses on the text structure (i.e. which part of the sentence is subject, object or predicate). Therefore, this method fails to deliver specific user intended information. For instance, given a task of extract “country restriction”, this system would analyze and report back a summary of an input text, instead of extracting only countries from the text and reporting those back in a structured way.

[0007] In existing NLP Name Entity Recognition Technology (NER), standard NER models such as BERT and NLE NSP are able to identify specific type of information from the text, for instance, if the task is to identify “person” in the text, NER model can detect “Maria” is a person. Combining NER models with certain indexing technology, it is possible to extract the “Maria” word from the text to fulfill the “extraction” task. However, all existing standard NER models are trained in a supervised manner, i.e. if a user wants a BERT model to identify “country restriction” in text, the user has to prepare a “country restriction” dataset, first, in which all “countries” in each sentence are clearly labeled out and use this dataset to train the model and normally this dataset should be large enough to train a deep neural network model (at least X thousand examples). However, in realistic application, a document contains a bunch of entities / attributes to identify, such as “partial election” and “minimum multiple”. Those entities are highly domain-specific hence there are no pre-trained NER models to directly use. To use NER technology, the user would have to train a specific model to detect each entity, and the dataset is a big bottleneck.

[0008] It has been known to implement Large Language Models (LLM) to analyze structured and unstructured data. However, simple machine learning approaches such as zero-shot prompts for LLMs, are not sufficient to address those tasks as they struggle with the complexity.

[0009] Specifically, existing LLM models such as ChatGPT are able to extract required information from text and to output the result in certain structured format such as in json format with only few examples (training is not needed). For instance, providing ChatGPT a few examples which contains input “MT Messages” and corresponding structured output (few-shot prompting), it is able to roughly generate a result in similar format. However, it still has several bottlenecks and therefore cannot really fulfill the task request:

[0010] 1 ) Due to existing LLMs being generic models trained on common knowledge, they usually extract extra information which may be not wrong in common sense but not required by the task and this results in the system user needing to invest extra human effort to carefully check and filter the output. Therefore, this reduces the effectiveness of the original intention of “automation”;

[0011] 2) If the task requires a complex structured format, e.g. it constraints the data type depending on information value and need to form the extractions in a customized XML format, existing LLM cannot fulfill this kind of complex format request with zero / few-shot prompting.

[0012] It is therefore an object of the present invention to improve and further develop a method, a system and a computer program of the initially described type for information extraction from structured and / or unstructured data, that is able to efficiently process large amounts of data regardless of whether the data is in a structured or unstructured format.

[0013] In accordance with the invention, the aforementioned object is accomplished by a computer-implemented method for information extraction from structured and / or unstructured data, the method comprising:

[0014] - receiving, by a data agent, one or more input data documents and a task,

[0015] - processing, by the data agent, the information contained within the one or more input data documents and generating an updated document from the information contained within the one or more input data documents,

[0016] - generating, by a logging agent, a structured document from the updated document and storing said structured document in a memory, wherein the information in the updated document is clustered into several attributes, each attribute relating to an information element within the structured document,

[0017] - extracting, by an extraction agent, information from the structured document, wherein the extraction agent comprises at least two sub extraction agents each of which comprises an LLM and is configured to extract, by utilizing its respective LLM, information from one specific attribute, wherein the extraction agent selects and instructs one or more of the sub extraction agents to each extract information that is relevant to the task from one attribute and to provide said extracted information as output to the extraction agent,

[0018] - generating, by the extraction agent, a final output based on the outputs of the sub extraction agents.

[0019] In accordance with the invention, the aforementioned object is further accomplished by an LLM-integrated system, comprising:

[0020] - a data agent configured to receive one or more input data documents and a task and to process the information contained within the one or more input data documents and to generate an updated document from the information contained within the one or more input data document,

[0021] - a logging agent, configured to generate a structured document from the updated document and to store said structured document in a memory, wherein the information in the updated document is clustered into several attributes, each attribute relating to an information element within the structured document,

[0022] - an extraction agent, configured to extract information from the structured document, wherein the extraction agent comprises at least two sub extraction agents each of which comprises an LLM and is configured to extract, by utilizing its respective LLM, information from one specific attribute, wherein the extraction agent is configured to select and instruct one or more of the sub extraction agents to each extract information that is relevant to the task from one attribute and to provide said extracted information as output to the extraction agent, wherein the extraction agent is configured to generate a final output based on the outputs of the sub extraction agents.

[0023] In accordance with the invention, the aforementioned object is moreover accomplished by a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the proposed method.

[0024] According to the invention it has first been recognized that structured and unstructured data can be better analyzed, by breaking the data down into individual chunks which can be analyzed separately. This allows the individual sub extraction agents to utilize individual data services to better analyze their respective data chunks. To this end, the data is first parsed by a data agent, who aggregates information from or more input documents into an updated document. Then, the logging agent analyzes said updated document and stores the information in a structured way, such that individual information elements can be quickly found and addressed, thereby creating a structured document. Afterwards, the extraction agent analyzes said structured document and extracts the individual information elements. To this end, the extraction agent utilizes several sub extraction agents, each designed to extract information from one kind of attribute. Moreover, because the information of the structured document is broken down into several attributes, which are given to individual sub extraction agents, the individual prompts for the LLMs of the sub extraction agents can be shorter and more concise. This increases the quality of the output of the LLMs. On top of that, one advantage of the invention is that the method enables to precisely extract highly domain-specific information and automatically form it into required structural format by strengthening the capability of LLMs on understanding generic human-text with the domain-awarded agentic system.

[0025] The data agent may receive input data from a user and / or from another system / agent. The term, “input data documents” is to be understood in its broadest term and relates in the description, preferably in the claims, to a document containing relevant information. The document may comprise text, picture, drawings, videos and / or audio samples. Preferably the document comprises text. The information may be structured - e.g. in form of keywords for different topics - or unstructured - e.g. in the form of free text. The agent may receive more than one document, all relating to a common task, and the documents may be of different types.

[0026] The term, “attribute” is to be understood in its broadest term and relates in the description, preferably in the claims, to a data structure within the structured document, that has an identifier and a text. Optionally, the attribute may comprise further information, such as a type of the data structure (Boolean, String, Integer, Float, etc.). An attribute relates to individual information elements within the document. An attribute may comprise only little information, such as “Customer Name”: “John”, “Balance”: “250.00”, “Date of Birth”: “01.01.2000”, wherein “Customer Name”, “Balance” and “Date of Birth” are the identifiers and “John”, “250.00” and “01.01.2000” are the values. An attribute may also comprise more information such as a free text with regard to the “intended use” of a banking transaction. An attribute may be broken down into several sub-attributes, which can be analyzed separately, especially if the attribute value contains a high amount of information.

[0027] The term, “structured document” is to be understood in its broadest term and relates in the description, preferably in the claims, to a document, where the information is ordered and the individual information elements are addressable. The structured document may be in the form of a nested list, wherein the top level comprises metadata and the information of the document is ordered into individual attributes, that can be referenced directly by the extraction agent. The structuring of the document is only done, if necessary.

[0028] It is possible that the provided document by the end user is already in a suitably structured form. In this case, the logging agent can provide said document directly to the extraction agent.

[0029] The term, “updated document” is to be understood in its broadest term and relates in the description, preferably in the claims, to an aggregate of information from one or more input data documents. The updated document may be the same as the input data document, if the input data document already contains all the necessary information.

[0030] The term, “data service” is to be understood in its broadest term and relates in the description, preferably in the claims, to an external entity, that can provide additional information and / or transform data given to said external entity. For example, the data service may be a search engine, a translator, a public data base and / or a corporate server. Specifically, the data service may provide public knowledge to the sub extraction agents to help them better understand the information input. In particular, a data service can be regarded as a tool that a sub extraction agent can utilize to gain knowledge required to extract information from their specific attribute.

[0031] The term, “task” is to be understood in its broadest term and relates in the description, preferably in the claims, to an implicit or explicit assignment of an end user how to handle the input data and what the output result form or structure should be. The task may be to provide a structured receipt containing information for a bank transaction, an overview of a corporate action, such as dividends, mergers or stock splits, or a structured summary of the input data documents.

[0032] The terms “extraction agent” and “sub extraction agent” are to be understood in their broadest term and relate in the description, preferably in the claims to an entity that is capable of interpreting and / or analyzing data in a document. The extraction agent and / or the sub extraction agents may be initialized in a preparation stage, if they have not been initialized yet. To this end, the extraction agent and / or a plurality of two or more sub extraction agents are created, wherein the extraction agent and the plurality of sub extraction agent each are initialized with a LLM. Then each sub extraction agent is specialized to a specific attribute, such that it is trained to extract information from that specific attribute and / or to extract a certain type of information. A sub extraction agent can be regarded as an attribute agent, as they are designed to extract information only from one specific attribute. Each individual sub extraction agent can be its own independent unit. Each individual sub extraction agent can be granted access to at least one data service to help them facilitate their information extraction. I.e. a sub extraction agent specialized to read the purpose of a banking transaction may be connected to a translation service, such that it can understand declarations of purpose in different languages. After the sub extraction agents have analyzed their respective attributes, the extraction agent aggregates their respective outputs and constructs a final output to the end user. This final output may be in the form of a structured document.

[0033] The form of the structured document, which is output by the extraction agent can be determined by the end user. For instance, it could be an XML file. The output of the system can be used to guide a human-computer interaction process.

[0034] According to an embodiment, the step of processing the one or more input data documents comprises performing, by the data agent, a quality inspection by performing one or more of the steps:

[0035] - a) validating the quality and / or integrity of input data documents,

[0036] - b) correcting errors in the input data documents,

[0037] - c) verifying that the input data documents contain all required data points.

[0038] The data agent is responsible to pre-process the input data. To this end, the data agent can perform three actions, whereby the data agent may perform none, only one, a combination of any two or all three actions. The data agent may provide message quality inspection and / or correction. The quality of an input message is assessed based on reference data, schemas and / or definitions. This can include the verification of mandatory information (e.g., are those present) and validity. To this end, the data agent can comprise an LLM-assistant, preferably an LLM- assistant that is trained on ISO-principles, to analyze and / or correct, if possible, information of the input data documents. According to an embodiment, the step of processing the one or more input data documents comprises combining, by the data agent, data from multiple documents into the updated document, wherein the multiple input data documents relate to the same topic and / or choosing, by the data agent, an input data document with the highest reliability from a plurality of input data documents, wherein the input data corresponds to the same information from different sources, wherein the reliability is determined by a user input and / or by a majority vote and / or by an LLM-assistant. The data agent may provide multi-message handling. A single task might require considering and / or processing several documents or even handling a version history of the same document. At the same time, it is necessary to keep all involved documents and document types as they may contain exclusive information. The data agent combines the information of these several documents into one updated - or “Master” - document. Preferably, the several documents are all of the same type and / or relate to the same topic. “Several” in this context means two or more documents and / or two or more versions of on individual document. Specifically, the data agent may differentiate between recent versus outdated information or those which extend already existing information. Therefore, the data agent may combine multiple documents that each describe part of the necessary information and / or that describe the changes of information in chronological order. An advantage of this is that the extraction agent may better extract information, as an LLM alone might struggle to analyze information that differs in chronological order. It is possible that the same information is provided by different sources, e.g., different documents describing the same scenario, event, and / or situation. Specifically, it might be possible that different documents contradict each other. In this case, the data agent may only use the document with the highest reliability. The reliability can be defined by the end-user (e.g., a ranked list) and / or determined automatically by the data agent. In the latter case, a possible implementation would be a majority vote and in case of conflict to influence an LLM to select the most reasonable one. An advantage of this is that the extraction agent does need to differentiate between valid and invalid data.

[0039] According to an embodiment the sub extraction agents are configured to request additional input for the respective information extraction from one or more data services that provide a data acquisition and / or a data transmutation service. This means that each sub extraction agent is able to communicate with none or one or more data services. Due to the fact that the sub extraction agents can individually query external data services, they can efficiently generate all the knowledge they need to process their individual attributes, without using a lot of bandwidth. The additional input might be translation service or general information regarding their attribute. An advantage of this is that the sub extraction agents can be specifically tailored to best analyze their respective attribute.

[0040] According to an embodiment, the step of generating, by a logging agent, a structured document from the updated documents comprises adding a link to external data to the attributes of the structured document and / or adding structured data pointers containing information about the content and type of individual data sections to the attributes, preferably wherein the relevant data from the external data and / or from the data service is saved in a cache memory, preferably based on the link to external data of the structured document. Due to the structure of the structured document, the extraction agent can directly reference individual data points within the document. However, the attributes of the document, i.e. individual data elements, may reference further data from external sources. For example, an attribute might reference an internet article, which is only known by its URL. The logging agent may add said reference to the attribute, such that said reference is directly addressable from addressing said attribute. In this case an attribute may have the form “Identifier” - “Value” - “link to external data”. The link may point to a URL, an API-Endpoint, another function and / or another document. An advantage of this is that external data can be easily and efficiently integrated into the LLM model.

[0041] Specifically, the external data can be stored in a cache memory. A cache memory may be a memory with fast addressing speed, such that data can be faster accessed than data stored on an external source. The cache memory may hold information about the linked external sources described above. Specifically, the contents of the linked external source may be stored in said cache memory. This is especially useful, if the linked external source is linked to more than one attribute. I.e. if multiple information elements reference a common external source, said external source can be stored in cache to avoid high bandwidth usage. Additionally or alternatively, said cache can be used to store data retrieved from the data services, specifically data that is useful for more than one attribute. An advantage of this is that redundant data requests to external data sources are minimized.

[0042] According to an embodiment, the method further comprises mapping, by the extraction agent, preferably by a mapper of the extraction agent, the sub extraction agents to the at least one data service and / or mapping, by the extraction agent the sub extraction agents to general attribute types. The extraction agent itself may utilize an LLM and each of the sub extraction agents may also utilize an LLM. The extraction agent may use its LLM to analyze and optimize the mapping of sub extraction agents to individual data services, such that the sub extraction agents only communicate with the data services they need. This mapping can also be learned by a separate Al based mapper, that utilizes a machine learning model, for example an LLM, to learn a suitable mapping. Additionally or alternatively, the extraction agent may analyze and optimize the behavior of the sub extraction agent and / or their mapping to the data services, in order to find sub extraction agent best suited for specific types of attributes. For example, the extraction agent may determine that a specific sub extraction agent is suited for analyzing free text attributes that relate to banking transactions and utilize this knowledge in further enquiries. Thereby, the extraction agent may generate a mapping, which sub extraction agent is best suited for a given attribute type.

[0043] According to an embodiment, the mapping is learned and / or updated, preferably by the mapper, based on the outputs of the sub extraction agents, an information exchange between the extraction agent and / or the sub extraction agents. This means that the extraction agent and / or the mapper can continuously improve the mapping, in order to increase the quality of the outputs of the sub extraction agents. For example, the mapper could analyze the output of the sub extraction agents with regard to their attribute und if the mapper determines that the output is insufficient, he can update the mapping. Thereby, the sub extraction agent can request input from additional data services which he needs to fulfill its task.

[0044] The sub extraction agents are part of the extraction agent. Therefore, any functionality described above or below with respect to the extraction agent or the sub extraction agents also applies to the reverse entity. I.e. any function performed by the sub extraction agents may also be performed by the extraction agent and vice versa.

[0045] According to an embodiment, the extraction agent chooses the sub extraction agents to be used based on a selection of attributes, wherein the set of attributes is selected from the provided document type, based on schemata, guidelines and / or rules or they are provided by an end user and / or the extraction agent instantiates a new sub extraction agent when no sub extraction agent is available and / or exists, yet, for one of the selected attributes. After the logging agent provides the structured document, the extraction agent can analyze the structured document and identify the required attributes of the document. The required attributes may be derived from the type of document or may be provided by the user. For example, if the document is a “dividend event” a required attribute may be a “partial election” and hence, the extraction agent would check for such an attribute and if found sends it to the sub extraction agent best suited for said attribute. It is also feasible that the required attributes may be input by the user, specifically in a prompt to the extraction agent. I.e. “analyze this document for actions that relate to payment of fees”. An advantage of this is that the most relevant attributes are always accounted for. If the extraction agent has already instantiated a specific sub extraction agent for a given attribute, he can instantly choose said sub extraction agent. Otherwise, he can instantiate a new attribute agent.

[0046] According to an embodiment, the sub extraction agents generate a prompt to query their LLMs, wherein the prompt is transmuted by injecting information about the attribute and domain knowledge of the field, wherein the domain knowledge is provided to the sub extraction agents in the form of guidance provided by domain experts, patterns learned from historical datasets and / or is acquired through the data services. The sub extraction agents utilize LLMs to generate their output. As the sub extraction agents are designed to interpret specific attributes, they are highly specialized. To further increase their knowledge in their respective fields or attributes respectively, they can utilize public domain knowledge to include into their prompts, such that the output of the LLMs is of higher quality, as the prompt is more specific. This injection of domain knowledge may be preprogrammed depending on the specific attribute and / or the sub extraction agents may generate the required knowledge, themselves. This can be done through analyzing historical data and / or querying their respective data providers for input. An advantage of this may be, that the sub extraction agents may be more knowledgeable in their respective field.

[0047] According to an embodiment, the extraction agent and / or the sub extraction agents generate explanations for their outputs, wherein the explanation contains information about the source of their output, the sub extraction agents’ interpretation of texts and / or proofs of non- hallucination output. The explanations can be utilized to verify that an output of the extraction agent or a single sub extraction agent is valid and well founded. Moreover, it can be used to find hallucinations in the output, where an LLM made assumptions without basis. Moreover, the explanations might give information regarding how an LLM of the extraction agent or of a sub extraction agent has interpreted a text, thereby allowing a third party to verify that the respective LLM has correctly analyzed the text.

[0048] According to an embodiment, the extraction agent plans an order sequence, wherein the order sequence defines a sequence in which the sub extraction agents generate their responses, wherein the order sequence is based on instructions by an end user and / or interdependencies between the attributes. The multitude of sub extraction agents are autonomous and may query their individual data services independently from each other and may also generate their outputs independently from each other. However, depending on the information in the structured document, there might be interdependencies between the individual attributes of the structured document. Thus, a first sub extraction agent may utilize the output of another sub extraction agent to correctly generate an output. The extraction agent may take this into account by creating an order sequence, which states which sub extraction agent has to finish their prompt before another may start. This sequence may also be provided by the end user. It is feasible that the sub extraction agents utilize the data services to plan the order sequence. An advantage of this is that the sub extraction agent has a way of communication that improves the overall quality of outputs. According to an embodiment, the extraction agent updates the attributes, updates a mapping of sub extraction agents to attributes and / or changes the order sequence based on the explanations and / or the sub extraction agents may update their prompts based on the output of their respective LLM. The extraction agent and / or the sub extraction agents can iteratively improve themselves by comparing their output against user requirements. Specifically, the sub extraction agents may analyze their outputs to check if they fulfill certain formal requirements and if not, they adapt their own prompt to the LLM in order to rectify a determined issue. Additionally or alternatively, the extraction agent may analyze the explanations given by the sub extraction agents to determine if the chosen sub extraction agents were in fact suitable for a given attribute. If the extraction agent determines that a sub extraction agent did not give a satisfying output, the extraction agent may invoke a suitable remedy. The sub extraction agent, may change the order sequence to allow a certain sub extraction agent access to the prompts of another sub extraction agent, may give a sub extraction agent additional access to another data service and / or may choose another sub extraction agent for a given attribute, thereby changing the mapping of said sub extraction agent to said attribute. An advantage of this is that the system of extraction agent and sub extraction agents can be iteratively improved, based on their real time performance.

[0049] According to an embodiment, the method further comprises checking, by a fact checking agent, the validity of the output of the sub extraction agents by comparing it against the attribute, the input data and / or the explanations and / or checking, by the extraction agent and / or the fact checking agent, the validity of the output of the extraction agent by comparing it against end user provided data and / or ground truth data. There are multiple ways to fact check the output of the LLMs. For example, a fact checking agent may compare the output of the sub extraction agents and their given explanations for their output to verify that their explanations are logical with respect to their output and their respective attribute. Moreover, the fact checking agent may verify that the output of the LLMs of the sub extraction agents does not contain hallucinated information. Additionally or alternatively, the extraction agent may verify that its output is valid by comparing it against feedback in the form of ground truth data. It is also feasible that the end user provides feedback in the form of user feedback to the extraction agent. In any case, the extraction agent and / or the sub extraction agents may continuously improve and refine their output by providing an output, checking it against the feedback, updating their prompts based on the feedback, recalculating their outputs and providing the recalculated output. Additionally or alternatively, it is possible that the extraction agent uses the feedback to give hints to the end user, as to how to correctly use the system.

[0050] There are several ways to design and further develop the teaching of the present invention in an advantageous way. To this end, it is to be referred to the dependent claims on the one hand and to the following explanation of preferred embodiments of the invention by way of example, illustrated by the figure on the other hand. In connection with the explanation of the preferred embodiments of the invention by the aid of the figure, generally preferred embodiments and further developments of the teaching will be explained. In the drawing

[0051] Fig. 1 is an overview of an LLM-integrated system according to an embodiment of the invention.

[0052] Fig. 2a is an overview of a data agent according to an embodiment of the invention.

[0053] Fig. 2b is a detailed view of a data agent according to an embodiment of the invention.

[0054] Fig. 3 is a detailed view of a logging agent according to an embodiment of the invention.

[0055] Fig. 4a is an overview of an extraction agent according to an embodiment of the invention.

[0056] Fig. 4b is a detailed view of a sub extraction agent according to an embodiment of the invention.

[0057] Fig. 4c is a refinement process performed by an extraction agent according to an embodiment of the invention. Fig. 5 is an overview of a method for information extraction according to an embodiment of the invention.

[0058] Fig. 1 shows an overview of an LLM-integrated system according to an embodiment of the invention.

[0059] The invention is now explained with respect to two specific use cases - Corporate Finance (CF) and Healthcare Data (HD). The invention will be described in greater detail with regard to corporate actions. Corporate actions refer to events initiated by a corporation that affect its securities, such as dividends, mergers, or stock splits, requiring communication and sometimes action from shareholders. However, the invention is not limited to those use cases.

[0060] The system 1 comprises several agents. First, a data agent 3 of the system 1 receives an input 2 from an external system or a human end user 8. The input 2 is in the form of one or more documents regarding a task. A task can hence comprise multiple MT (Message Type) messages where the task represents a corporate action event, such as dividend issuing or security purchasing. Considering the exemplary use cases mentioned above, the input 2 could be, in case of CF, incoming SWIFT Messages, specifically MT messages, or, in case of HD, electronic health records. MT messages are standardized formats used in the SWIFT (Society for Worldwide Interbank Financial Telecommunications) network for securely transmitting financial information between banks and financial institutions, covering various transaction types such as payments, securities, and trade finance. MT Messages are text files maintained in globally standardized formats used in the SWIFT network for securely transmitting financial information between banks and financial institutions. There are different message types used for different communication purposes. However, independent of the type, a MT message always includes both structured and unstructured information. While those messages have a standardized format, they can differ from one another. The quality of those messages varies across institutions, i.e., mandatory fields are missing, or information is incorrect (e.g., typos). Further, these messages have several free text attributes leading to different ways of presenting certain information (e.g., different technical language, links to other resources, etc.).

[0061] Specifically, the input data of input 2 could consist of multiple MT564 messages and multiple MT568 messages.

[0062] According to an embodiment, the data agent 3 analyses the input 2 by performing three steps. These steps are shown in Figs. 2a and 2b.

[0063] First, the data agent 3 performs message quality inspection and correction, as shown at S1 in Figs. 2a and 2b. This comprises validating the quality, correcting errors and verifying that all required data points are available. Specifically, the data agent 3 verifies that the input 2 comprises all required information.

[0064] For example, a SWIFT Message follows the principles of International Standard ISO 15022, which clearly sets the optional and mandatory information required in different SWIFT Message types. For instance, in any given message, “if field :22F::CAEV / / OTHR is present, then sequence F must be present, and field :70E::ADTX must be present in the message (CF). In the case of Healthcare (HD), some healthcare data might need to be transmitted in redacted or anonymous form. The data agent 3 hence verifies if all those required fields are present and validly filled. If possible, the data agent 3 may perform data correction. For example, the data agent 3 might convert a transaction date from the notation MMDDYYYY into a standard notation YYYYMMDD.

[0065] Second, the data agent 3 performs multi-message handling, as shown at S2 in Figs. 2a and 2b. This means that the data agent 3 takes several documents, i.e. several SWIFT-Messages (CF) or several electronic health records (HD) and combines them into one updated document. The updated document comprises the information of the several documents, wherein information that is present in several documents is only saved once in the updated document. For example, the data agent 3 may combine three SWIFT-Messages that belong to one transaction and combine them into one (CF), or the data agent 3 may combine all healthcare records that belong to one specific patient (HD). This is a common use case as one task could consist of, e.g., ten MT564 messages and five MT568 messages. Specifically, the data agent 3 may be configured to combine data in such a way that the chronological order of the data is taken into account.

[0066] Third, the data agent 3 performs multi-source handling, as shown at S3 in Figs. 2a and 2b. This means that the data agent 3 takes several documents and combines them into one updated document, although the several documents contain conflicting information and / or documents from different sources. For example, two SWIFT Messages with the same declared purpose and the same amount of transferred money might indicate, that one of them was send in error (CF) or the data agent 3 may have gotten healthcare records for a patient, wherein one records state the patient is female and one states that the patient is male. It is also feasible that a single order is requested by multiple custodians, i.e., several MT564 documents from multi sources are available for the same order The data agent 3 can autonomously decide which document to discard and which document to use or may prompt the user for input.

[0067] Referring back to Fig. 1 , the data agent 3 creates the updated document and provides it to the logging agent 4. The logging agent 4 is further described in connection with Fig. 3.

[0068] Fig. 3 is a detailed view of a logging agent according to an embodiment of the invention.

[0069] The logging agent 4 takes as input the updated document from the data agent 3 and transforms it into a structured document 15a, 15b. This is done by splitting the information in the updated document into several attributes 14a, 14b. In Fig. 3, two structured documents 15a, 15b are shown, each consisting of meta data and several attributes 14a, 14b, wherein only one attribute 14a, 14b from each structured document 15a, 15b is shown. The structured documents 15a, 15b are stored in memory 16. As part of the data structure, an attribute 14a, 14b is an array of objects representing individual data points within the document. Each attribute object has one or more of the following properties:

[0070] • Name: Unique identifier for the field within the document (e.g., "customer_name").

[0071] • Value: The actual data stored in the field (string, number, boolean, etc.).

[0072] • Type: Optional property specifying the data type (useful for validation).

[0073] • Link: Optional object containing information for fetching data from an external source: o Type: Specifies the type of link (e.g., "document", "API", "function"). o Target: Identifier or URL pointing to the external data source. o Cache: Boolean flag indicating if fetched data should be cached within the document file.

[0074] In Fig. 3, the first structured document 15a comprises only the properties Name, Value and Type, wherein the structured document 15b comprises all four listed properties.

[0075] Moreover, the structured document is provided with metadata, which allows other entities to quickly reference specific attributes within the document.

[0076] The attributes 14a, 14b may contain links to external data, which is accessible via an External Endpoint 11. For example, the healthcare records may point to X-Ray- Images stored on another server (HD) or the SWIFT-Messages may point to URL of an issuing bank. If multiple attributes 14a, 14b reference the same link, the contents of said link can be stored in a cache 12 for faster lookup.

[0077] Referring back to Fig. 1 , the structured document is then provided to an extraction agent 5.

[0078] The extraction agent 5 is further described in connection with Figs. 4a to 4c. Fig. 4a is an overview of an extraction agent according to an embodiment of the invention.

[0079] The extraction agent 5 takes as input the structured document form the logging agent 4. Then the extraction agent 5 performs several steps to extract the required information from the structured document and provide the end user with a suitable output.

[0080] First, the extraction agent 5 performs an attribute selection, as shown at S5.1. The required attributes for any given document can be given by a user to may be derived from the provided document type. For example, the structured document may be a SWIFT-Document and the extraction agent 5 may know from that, that the required attributes are “name”, “date”, “transaction type”, “receiving account”, “sending account” and “money value” (CT) or the structured document may be a healthcare document and the user inputs the task to find all patients with a specific disease (HD), in which case the attributes would be “name” and “disease”. Not every attribute might be required with regard to every transaction, e.g. “partial election” is an attribute required by a “dividend” event, but it is not required by “security repurchase” event.

[0081] Next, the extraction agent 5 instantiates (see step S5.2) sub extraction agents 6. This step may be optional, if the extraction agent 5 has already instantiated the sub extraction agents 6 in a prior task. The extraction agent 5 instantiates a new sub extraction agent 6 for each attribute that was identified in the first step which has not been instantiated, yet. An attribute can be any type of information, i.e. , structured and unstructured information. The accuracy of the individual attribute agents significantly influences the overall quality of the extraction agent 5.

[0082] Additionally, the extraction agent 5 might plan (see step S6) the sequence of execution of the sub extraction agent 6 during the “Execution Planning” step, to take into account dependencies between the different attributes. The sequence is influenced by two parameters: First, any instructions (e.g., existing schema) by the end-user. Second, any by the extraction agent 5 or its LLM detected dependency between the attributes. This also includes the consideration of reducing network load in respect when querying required information by the attribute agents.

[0083] Before explaining the further steps conducted by the extraction agent 5, the sub extraction agents 6 will be further described.

[0084] The sub extraction agents are further described in connection with Fig. 4b.

[0085] Each sub extraction agent 6 is its own independent unit. A sub extraction agent 6 has the task to extract a certain type or piece of information. A sub extraction agent 6 embeds a (specialized) LLM model, that is enriched with data services 7 (e.g., Knowledge Base API) through an Al-based mapper 10. The Al-based mapper 10 is another machine learning model which learns in an end-to-end fashion which data services 7 are most suitable for a sub extraction agent 6. The Al-based mapper might comprise an LLM. Thereby, a mapping between the sub extraction agents 6 and the data services 7 is created. Most suited means that the performance of the sub extraction agent 6 improves. Because a sub extraction agent 6 is highly specialized, they are each designed to analyze specific attributes. Hence, they need access to different kind of data services 7.

[0086] The Al-based Mapper can update and improve the mapping over time, giving the sub extraction agents 6 access to new data services 7 and / or denying access to old data services 7 based on the requirements of the sub extraction agents 6. Specifically, the outputs of the sub extraction agents 6 can be analyzed by the Al- based mapper 10, in order to update the mapping.

[0087] The task of each sub extraction agent 6 is to generate an output of the extracted information of their specific attribute. This information extraction step S7 can be divided into four sub steps.

[0088] The first step “Extraction Planning” S7.1 covers the ability to design prompts and to plan the prompts order considering dependencies among intermediate results. This step can be supported by data services. For example, a sub extraction agent 6 designed to analyze a purpose for a transaction might utilize a translation data service to translate the purpose into English first (CT), or a sub extraction agent 6 designed to list the symptoms of living patient might wait for another sub extraction agent’s 6 output, whether that patient is currently alive.

[0089] The next step “Knowledge Injecting” S7.2 comprises injecting domain knowledge into the LLM prompts to generate most accurate results. The domain knowledge could be guidance provided by domain experts, patterns learned from historical dataset, or acquired through the data services. E.g. a sub extraction agent 6 that is designed to list the symptoms of a patient might consult a dictionary, if the sub extraction agent 6 does not know the meaning of a word (HD) or a sub extraction agent 6 might look up different norms of SWIFT-Messages.

[0090] In the third step ’’Generation Controlling” S7.3, the output of the LLM is refined, e.g., to fulfil certain requirements in respect of the format (e.g., monetary value as float value or country name as ISO code). The refinement happens by automatically adapting the prompts and related parts.

[0091] Finally, in the step “Explaining” S7.4, the sub extraction agent 6 and / or its LLM generates corresponding explanations, such as highlighting the source information in input data or interpret the result from a long text given in input data. For example, the sub extraction agent 6 might link to a website that gives a definition of a medical term (HD) or the sub extraction agent 6 might give an explanation, why the sub extraction agent 6 ignored a key information element.

[0092] Afterwards, the sub extraction agents 6 will provide their output to the extraction agent 5. The subsequent handling of the output of the sub extraction agents 6 will be further explained with regard to Fig. 4a.

[0093] Referring back to Fig. 4a, the extraction agent 5 takes as input the output from the several sub extraction agents 6 and generates a complete output 9 during the step “Message Generation” S8. Potential sub-attributes are assembled to derive the required attribute. In the simplest case, this document follows a key-value structure. The message may be a “BB Message”, that refers to the CF example and stands for “Banking Book Message”. It might also be an ordered list of healthcare information, which are extracted from multiple healthcare document (HD).

[0094] In an optional step, “Fact Checking” S9, the output of the extraction agent 5 might be verified. Thus, a fact checking agent (not shown) checks if the output contains a wrong information, e.g., it does not exist in input data (hallucination) or if the explanations of the sub extraction agents 6 do not match with their output, attribute or input. The extraction agent 5 also provides a functionality to consider feedback be it derived from ground truth data or by a human-end user. In this context, the “Feedback Analyzing” stage S10 conducts error analysis by comparing the generated BB Message with pre-defined ground truth data. This allows the system to analyze continuously errors occurred in generation process and to learn from the feedback to automatically refine the generation process Therefore, during a selfimprovement step S11 , the system might be adapting prompts or extending the hints provided along the prompt to increase the quality of the output. Fig. 4c describes this refinement process.

[0095] Fig. 4c describes a refinement process.

[0096] The extraction agent 5 gets feedback of any kind, from the user or external sources. First, the extraction agent 5 analyzes the type of feedback and where the extraction agent 5 or one of the sub extraction agents 6 can use the feedback in the prompt S10. For example, a user might tell the extraction agent 5, that the extraction agent 5 declared proration in its output, however, this was not the case. The extraction agent 5 reforms its prompt and / or orders one of its sub extraction agents 6 to reform their prompt by adding the condition that proration will only be mentioned to the user, if it is explicitly mentioned. Fig. 4c illustrates the latter case where the prompt of the sub extraction agent 6 is reformed. The sub extraction agent 6 hence learns from the feedback and updates it prompt to reflect said feedback. Based on this updated prompt, a new output is generated and presented to the user. This refinement process can be iteratively repeated, until the output is of sufficient quality, or a threshold of iterations is reached. Thus, a fact checked and final output of the extraction agent 5 is generated.

[0097] Referring back to Fig. 1 this final output is presented to the end user.

[0098] Fig. 5 shows an overview of the method for information extraction.

[0099] The workflow to extract information from a document is as follows:

[0100] First the data agent 3 gets as input 2 one or more documents. The data agent 3 analyzes and inspects these documents for quality and errors, as shown at S1. If the input comprises several documents with different topics, the data agent 3 integrates them into one document, as shown at S2. If the input comprises several documents from different entities with conflicting information, the data agent incorporates only the relevant data, as shown at S3. Thus, the data agent 3 creates an updated document.

[0101] Second, a logging agent 4 takes the updated document and structures it by attributes. Thus, the logging agent 4 creates a structured document which is stored for further processing (see step S4 in Fig. 5).

[0102] Next, the extraction agent 5 takes the structured document as input and selects (shown at S5) and if necessary, instantiates, several sub extraction agents 6 and plans (shown at S6) the execution process for the sub extraction agents 6 for each attribute. The sub extraction agents 6 perform information extraction on their specific attribute (see step S7). Then, the extraction agent 5 constructs a combined output from the output of all the sub extraction agents 6 (see step S8).

[0103] The combined output can be fact checked by a fact checking agent, as shown at S9. If the combined output is of sufficient quality, the combined output is sent to the user as final output. Otherwise, the errors in the combined output are analyzed (see S10.1 ) and if possible, the extraction agent 5 learns potential remedies from the feedback (see S10.2). Afterwards, the prompts for the LLMs of the extraction agent 5 or the sub extraction agents 6 are corrected (see S11 ). Additionally, the order sequence of the sub extraction agents 6 can be updated. Then, a new combined output of the sub extraction agents 6 and / or of the extraction agent is generated S7, S8 and again fact checked S9. This process repeats until the output is of sufficient quality, or a threshold is reached.

[0104] Two use cases for the aforementioned workflow are now described.

[0105] 1. Information Processing: Robust Multi-Source Information Extraction with Contextual Understanding and Error Mitigation

[0106] Use Case: Investment companies receiving a flood of MT messages about stock splits, mergers, and other corporate actions, often manually created by humans. While those MT messages can also already contain incomplete or wrong information, human case handlers, processing these MT messages, can introduce additional errors. Thus, the process is prone to errors due to complex language and potential translation mistakes. This manual approach is also time-consuming and expensive, hindering efficient corporate action processing.

[0107] The LLM integrated system can sort through those documents in form of MT messages, deciphering the key attributes like which company is involved and what option was selected. This information might then be translated into a format that the company's systems can understand, streamlining and automating the corporate action process.

[0108] Input: The data source for this Al system would be the incoming MT messages (SWIFT messages) themselves. These messages may originate from various financial institutions, custodian banks, or other entities involved in conveying corporate action information.

[0109] The Method: The LLM integrated system can automate corporate action processing by understanding incoming MT messages. It can use natural language processing to extract key attributes and convert free text into a structured format. This allows for faster, more accurate processing of corporate actions, reducing manual effort and improving efficiency. Output: The output of the LLM integrated system is a structured document containing relevant information. This automation can streamline the entire corporate action process, saving time and reducing errors. Therefore, it may reduce the risk that investors miss the best reaction time (e.g. message generated too slow by human, that company A proposes its security holders the option to convert existing securities into different ones with an overall amount limitation. Slow message could cause the reaction delay and result in financial loss). Also, it may reduce the risk that investors react on faulty information.

[0110] 2. Information Processing: Robust Multi-Source Information Extraction with Contextual Understanding and Error Mitigation (Health Care)

[0111] Use Case: Healthcare institutions generate vast volumes of patient data, including clinical notes, lab results, imaging reports, and administrative information. These data are often scattered across various systems and formats, making it challenging to access, analyze, and leverage for improved patient care. Manual data entry, transcription errors, and inconsistent data formats introduce inaccuracies and inefficiencies into the EHR system. This fragmented data landscape may hinder clinical decision-making, research, and population health management.

[0112] The LLM integrated system can solve this issue by centralizing, standardizing, and enriching EHR data. The system can extract and structure critical information from unstructured data sources, improving data quality and completeness. Additionally, the LLM integrated system can uncover hidden patterns and insights, enabling data- driven decision-making and personalized patient care.

[0113] Input: The data source encompasses a wide range of electronic health record (EHR) components. These include structured data elements such as patient demographics, medical history, diagnoses, medications, lab results, and imaging reports. Additionally, the system processes unstructured data from clinical notes, discharge summaries, and other free-text documents.

[0114] The Method: The LLM integrated system automates EHR data management by extracting and structuring critical information from both structured and unstructured data sources. It converts free-text clinical notes, discharge summaries, and other documents into a standardized format. This enables faster, more accurate data analysis, leading to improved clinical decision-making, enhanced patient care, and streamlined operations.

[0115] Output: The LLM integrated system can provide a structured overview of patient data from EHRs. By eliminating the need for manual data entry and transcription, the LLM integrated system may reduce errors and inconsistencies in patient records. The system can transform complex medical information into structured tasks and overviews. This automation can accelerate clinical workflows, improves data quality, and supports evidence-based decision-making. By providing timely and accurate information, the LLM integrated system can help clinicians make informed diagnoses, prescribe appropriate treatments, and monitor patient outcomes effectively.

[0116] Many modifications and other embodiments of the invention set forth herein will come to mind to the one skilled in the art to which the invention pertains having the benefit of the teachings presented in the foregoing description and the associated drawings. Therefore, it is to be understood that the invention is not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

[0117] L ist of refe re n ce s i g ns

[0118] 1 LLM-integrated system

[0119] 2 Input

[0120] 3 Data Agent

[0121] 4 Logging Agent

[0122] 5 Extraction Agent

[0123] 6 Sub Extraction Agent

[0124] 7 Data Service

[0125] 8 External System or End User

[0126] 9 Complete Output

[0127] 10 Al Based Mapper

[0128] 11 External Endpoint

[0129] 12 Cache

[0130] 13 Fact Checking Agent

[0131] 14 Attribute

[0132] 15 Structured Document

[0133] 16 Memory

[0134] 51 Message Quality Inspection

[0135] 52 Multi-Message Handling

[0136] 53 Multi-Source Handling

[0137] 54 Store and log processed message

[0138] 55 Select Sub Extraction Agents

[0139] 55.1 Attribute Selection

[0140] 55.2 Instantiate Attribute Agents

[0141] 56 Execution Planning

[0142] 57 Information Extraction

[0143] 57.1 Extraction Planning

[0144] 57.2 Knowledge Injecting

[0145] 57.3 Generation Controlling

[0146] 57.4 Explaining Message Generation Fact Checking Feedback Analyzing Analyze Errors Learn from Errors Self-Improving

Claims

- 29 -C l a i m s1. A computer-implemented method for information extraction from structured and / or unstructured data, the method comprising:- receiving, by a data agent (3), one or more input data documents and a task,- processing, by the data agent (3), the information contained within the one or more input data documents and generating an updated document from the information contained within the one or more input data documents,- generating, by a logging agent (4), a structured document from the updated document and storing (S4) said structured document in a memory, wherein the information in the updated document is clustered into several attributes, each attribute relating to an information element within the structured document,- extracting, by an extraction agent (5), information from the structured document, wherein the extraction agent (5) comprises at least two sub extraction agents (6) each of which comprises an LLM and is configured to extract, by utilizing its respective LLM, information from one specific attribute, wherein the extraction agent (5) selects (S5) and instructs one or more of the sub extraction agents (6) to each extract (S7) information that is relevant to the task from one attribute and to provide said extracted information as output to the extraction agent (5),- generating (S8), by the extraction agent (5), a final output (9) based on the outputs of the sub extraction agents (6).

2. The method according to claim 1 , wherein the step of processing the one or more input data documents comprises performing, by the data agent (3), a quality inspection (S1 ) by performing one or more of the steps:- a) validating the quality and / or integrity of input data documents,- b) correcting errors in the input data documents,- c) verifying that the input data documents contain all required data points.

3. The method according to claim 1 or 2, wherein the step of processing the one or more input data documents comprises combining, by the data agent (3), data from multiple documents into the updated document, wherein the multiple input data- 30 - documents relate to the same topic and / or choosing, by the data agent (3), an input data document with the highest reliability from a plurality of input data documents, wherein the input data corresponds to the same information from different sources, wherein the reliability is determined by a user (8) input and / or by a majority vote and / or by an LLM-assistant.

4. The method according to any of claims 1 to 3, wherein the sub extraction agents (6) are configured to request additional input for the respective information extraction from one or more data services (7) that provide a data acquisition and / or a data transmutation service.

5. The method according to any of claims 1 to 4, wherein the step of generating, by a logging agent (3), a structured document from the updated documents comprises adding a link to external data to the attributes of the structured document and / or adding structured data pointers containing information about the content and type of individual data sections to the attributes, preferably wherein the relevant data from the external data and / or from the data service (7) is saved in a cache memory, preferably based on the link to external data of the structured document.

6. The method according to any of claims 1 to 5, further comprising mapping, by the extraction agent (5), preferably by a mapper (10) of the extraction agent (5), the sub extraction agents (6) to the at least one data service (7), and / or mapping, by the extraction agent (5), the sub extraction agents (6) to general attribute types.

7. The method according to claim 6, wherein the mapping is learned and / or updated, preferably by the mapper (10), based on the outputs of the sub extraction agents (6), an information exchange between the extraction agent (5) and / or the sub extraction agents (6).

8. The method according to any of claims 1 to 7, wherein the extraction agent (5) chooses the sub extraction agents (6) to be used based on a selection (S5.1 ) of attributes, wherein the set of attributes is selected from the provided document type,based on schemata, guidelines and / or rules or they are provided by an end-user (8) and / or the extraction agent instantiates (S5.2) a new sub extraction agent (6) when no sub extraction agent (6) is available and / or exists, yet, for one of the selected attributes .

9. The method according to claim 8, wherein the sub extraction agents (6) generate a prompt to query their LLMs (S7.1 ), wherein the prompt is transmuted by injecting information about the attribute and domain knowledge of the field (S7.2), wherein the domain knowledge is provided to the sub extraction agents (6) in the form of guidance provided by domain experts, patterns learned from historical datasets and / or is acquired through the data services (7).

10. The method according to any of claims 1 to 9, wherein extraction agent (5) and / or the sub extraction agents (6) generate explanations for their outputs (S7.4), wherein the explanation contains information about the source of their output, the sub extraction agents’ (6) interpretation of texts and / or proofs of non-hallucination output.11 . The method according to any of claims 1 to 10, wherein the extraction agent (5) plans an order sequence (S6), wherein the order sequence defines a sequence in which the sub extraction agents (6) generate their responses, wherein the order sequence is based on instructions by an end user (8) and / or interdependencies between the attributes.

12. The method according to any of claims 1 to 11 , wherein the extraction agent (5) updates the attributes, updates a mapping of sub extraction agents (6) to attributes and / or changes the order sequence (S11 ) based on the explanations and / or the sub extraction agents (6) may update their prompts based on the output of their respective LLM (S10.2).

13. The method according to 1 to 12, further comprising checking, by a fact checking agent (13), the validity of the output of the sub extraction agents (6) by comparing it against the attribute (S9), the input data and / or the explanations, and / orchecking, by the extraction agent (5) and / or the fact checking agent (13), the validity of the output (9) of the extraction agent (5) by comparing it against end user (8) provided data and / or ground truth data.

14. An LLM-integrated system, preferably configured to perform the method according to one of the claims 1 to 13, the system comprising:- a data agent (3) configured to receive one or more input data documents and a task and to process the information contained within the one or more input data documents and to generate an updated document from the information contained within the one or more input data document,- a logging agent (3), configured to generate a structured document from the updated document and to store said structured document in a memory, wherein the information in the updated document is clustered into several attributes, each attribute relating to an information element within the structured document,- an extraction agent (5), configured to extract information from the structured document, wherein the extraction agent (5) comprises at least two sub extraction agents (6) each of which comprises an LLM and is configured to extract, by utilizing its respective LLM, information from one specific attribute, wherein the extraction agent (5) is configured to select (S5) and instruct one or more of the sub extraction agents (6) to each extract (S7) information that is relevant to the task from one attribute and to provide said extracted information as output to the extraction agent (5), wherein the extraction agent (5) is configured to generate (S8) a final output (9) based on the outputs of the sub extraction agents (6).

15. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of one of the claims 1 to 13.

Images