Information system for supervising data transportation

Abstract

The invention relates to a system for analysing and supervising data transportation traces between applications of an information system. The innovation concerns the analysis of the traces in terms of 3 aspects: their description at execution, the mapping of the information system and their temporality.

Description

Information system for monitoring data transport Technical field of the invention

[0001] The present invention relates to an information system for monitoring data transport. It applies, in particular, to the overall architecture of the information system, including hardware and software components, servers, communication devices, databases, and user interfaces.

[0002] The present invention covers methods and systems for monitoring and supervising data transports, including event detection, error handling, transfer traceability, and report generation.

[0003] The domain may include aspects related to data security, such as encryption, authentication, protection against computer threats and attacks, and access management.

[0004] The field may encompass techniques and algorithms aimed at optimizing the efficiency of data transfers, including compression, latency reduction, bandwidth management, and data prioritization.

[0005] The domain can also address methods to track and record information from each data transfer, thus enabling complete traceability and auditability for compliance and performance management. Thus, the domain can include aspects related to seamless integration with different applications and systems, thus ensuring compatibility and interoperability.

[0006] Finally, the domain may encompass aspects related to the user interface of the system, including real-time monitoring features, configuration, reporting, and user interaction.

[0007] Some monitoring solutions are directly integrated with intermediation tools, such as Enterprise Service Bus (ESB), Extract-Transform-Load (ETL), and API Management offered by renowned vendors such as Oracle, IBM, Google, and Microsoft, a registered trademark. These monitoring consoles provide information on running programs, deployed programs, and the status of unit processes.

[0008] Other external solutions focus on collecting logs and metrics from operating systems. These solutions read text files or performance data and are used to monitor the overall health of systems. It is important to note that the content of log files is generally free and does not necessarily have to be structured in any specific way.

[0009] To track technical transactions, monitoring tools are available, both from commercial vendors and in open source versions. Using agents that automatically add "markers" to application behaviors is a common method for linking technical calls with protocol markers. These solutions are grouped under the term "Application Performance Management" (APM).

[0010] It is important to note that these solutions focus primarily on performance and technical dependencies and do not provide functional information or in-depth analysis.

[0011] The many existing solutions are aimed at technology experts and administrators of these tools. Strong IT skills are a prerequisite for using these solutions.

[0012] Document US2002087704 is known, which describes systems and methods for routing messages to communication devices over a communication network. However, this document has drawbacks.

[0013] The main trace storage base operates on a group of servers on the same network.

[0014] The document imposes user declaration, which requires having a specific account without providing a solution. If each specific user uses messages, it is obvious to have its origin and destination. One of the goals of the present invention is to have a reliable system without knowing the user.

[0015] The system described in the document requires careful and accurate time management to ensure reliability and consistency of operations. This makes it difficult to monitor without rigorous protocols in place.

[0016] Document US10033611 is known, which describes a transaction management platform configured to perform end-to-end tracking of transactions including messages. However, this document has drawbacks.

[0017] The platform described in the document requires traceability of information. However, one of the goals of the present invention is to have a reliable system without knowing the user. Presentation of the invention

[0018] The present invention aims to overcome these drawbacks with a completely innovative approach whose main objective is the reconstruction of the functional and conceptual vision of data transport between a source application and a set of target applications from technical data.

[0019] The essence of this technology lies in its ability to reconstruct and infer the initial purpose of data transport as well as its current state, based on the individual transport traces collected.

[0020] In other words, this system, hereinafter referred to as the invention, aims to establish a complete understanding of the data transport process by reconstructing the initial vision of the operation, including the underlying intentions and objectives. It carefully analyzes the information collected during the transport, the processing carried out along the way and the interactions with the different target applications to develop a representation

[0021] These objectives, as well as others which will appear subsequently, are achieved, using an information system for supervising data transport, comprising at least two servers linked together by a communication network, computer client applications connecting to said servers, remarkable in that it comprises: - transport traces having at least the information coming from said following computer client applications from a server; said transport traces comprise: - a unique identifier corresponding to a sequence of characters configured to associate a unique identifier with each transport trace; - a message carrying at least one temporal information corresponding to a date, a time with milliseconds and a time slot; - a transport route corresponding to a character string defining the processing in charge of the transport;- a current status corresponding to one of the following values: information, success, timeout, alert, and error; - a list of references of the transported data corresponding to a type and an identifier; - a database of transport traces configured to organize the information relating to the transport traces, a database of the exchange mapping of the supervised information system configured to reference the exchanges with characteristics of the source applications, target applications and its time limit, and a database for the exchanges calculated from the traces configured to define the type of each transport trace. ; - means using at least one microprocessor to process transport traces and exchanges in the database of transport traces and the database of calculated exchanges, called calculated exchange;said means for processing transport traces and exchanges modifying the status of the calculated exchange according to a server if the transport trace has not arrived at the corresponding server in the exchange database then the current status is modified and one of the following values ​​is modified: information, success, out of time, alert, and error; - means for viewing transport traces and calculated exchanges; said system references the exchanges with the characteristics of the source applications, the target applications and its allocated time in a mapping database;said system assigns the transport traces with elements called "Mediation", defined by at least one name of one of the elements called "Mediation", and a given time limit of one of the elements called "Mediation", and represented in the mapping database as well as in the database of calculated exchanges as one of the elements called "Mediation" calculated, using in priority the transport route configured to process the transport load then the transported data.;

[0022] The invention is based on a fundamental technical architecture that encompasses several key elements, each of which is of crucial importance for achieving effective monitoring of data transport.

[0023] Connecting servers to each other through a communications network is fundamental to establishing a robust communications infrastructure. This network allows servers to exchange data, coordinate operations, and ensure the consistency of transmitted information. It also provides the possibility of two-way communication between servers and client applications.

[0024] Computer client applications play a vital role in the system's use by end users. These applications allow users to connect to servers and interact with the system to monitor and manage data transport. The presence of these computer client applications responds to functional services and ensures a user-friendly and intuitive interface for system operators.

[0025] The invention thus offers a new perspective on data transport by allowing users to better understand information flows, identify possible deviations from the initial objective and make more informed decisions regarding the management and optimization of this transport.

[0026] The invention represents a significant advance in the field of monitoring and analysis of data transfers, opening up new possibilities for improving the efficiency, quality and security of communication operations between applications.

[0027] The main objective of the invention is the monitoring of data transport. The combination of servers, the communication network and client applications allows real-time monitoring of data flows, identification of potential problems, generation of alerts and facilitation of decision-making. This monitoring function is crucial to guarantee the performance, security and reliability of data transfers.

[0028] The database of transport traces received by the invention allows all information relating to transport traces to be stored in an organized and structured manner, while the exchange database defines the type of each transport trace. This organization guarantees complete traceability and allows precise management of the transported data.

[0029] Transport trace and exchange processing facilities offer an automated processing function, allowing the status of transport traces to be tracked and appropriate action to be taken based on their progress. Automating this process improves efficiency and reduces the possibility of human error.

[0030] The processing facilities enable real-time monitoring of data transport. If a transport trace does not reach the corresponding server in the exchange database, this may indicate a problem or a deviation from the planned route. Changing the current status, for example, from "success" to "error," allows potential problems to be quickly identified and corrective actions initiated.

[0031] The current status deduction feature is innovative. It deduces the status of functional exchanges, “success,” “error,” “alert,” “information,” or “out of time,” based on the status of transport traces received by the servers. This feature helps improve quality of service by categorizing anomalies and data transfer incidents. This allows for rapid response to minimize negative impacts on the system.

[0032] The feature of automatically changing the current status to “out of time” when a transport trace does not reach the corresponding server within the given time frame helps improve the quality of service by quickly reporting missing and absent processing within the monitored system. This allows for a rapid response to minimize negative impacts on the system.

[0033] The invention finds application in various complex computing environments, including those involving sensitive data transfers, distributed systems, or large-scale networks. The ability to adapt the current status of the exchange based on the statuses of the numerous transport traces is a technical feature that enhances the robustness and usefulness of the system in various scenarios.

[0034] The invention is advantageously implemented according to the embodiments and variants set out below, which are to be considered individually or in any technically effective combination.

[0035] In one embodiment, the system assigns the elements called "Mediations" with elements called "Flows", defined by at least one name of one of the elements called "Flows", and a given time limit of one of the elements called "Flows", represented in the mapping database, as well as in the database of calculated exchanges as one of the elements called "Flows" calculated, according to the dependencies defined in the mapping database, and the given time limits of one of the elements called "Mediation".

[0036] In one embodiment, the system assigns the calculated Flows with calculated exchanges according to the dependencies defined in the mapping database, and the given deadlines of one of the elements called "Flows".

[0037] In one embodiment, the status of one of the elements called "Mediation" calculated is determined according to the status of the oldest transport trace affected, said traces ordered on a time axis relative to a clock of a microcontroller used to generate the transport trace. Brief description of the figures

[0038] Other advantages, aims and characteristics of the present invention emerge from the following description given, for explanatory and non-limiting purposes, with reference to the appended drawings, in which:

[0039] It represents a basic diagram of an information system;

[0040] Larepresents an example of transporting data from XY to B and C;

[0041] Represents the operation to emit traces;

[0042] It represents a basic diagram of how databases work.

[0043] Lamontre shows a schematic diagram of an information system composed of 3 applications: A, B and C.

[0044] This system allows the efficient and secure movement of data generated, xy, by a source application, A, to 2 target applications: B and C.

[0045] The following terms are defined by giving the definition:

[0046] - Transport trace database: database storing traces in their entirety.

[0047] - Exchange mapping database: physical, application and business views of the supervised information system with exchanges, flows and mediations.

[0048] - Database for calculated exchanges: database grouping the reconstruction of the route, of the data within the referenced elements.

[0049] - Mediation: a program dedicated to transporting data. It reads data in a specific format, such as XML or JSON, and following a communication protocol, such as File or HTTP.

[0050] - Flow: a transport of data between two applications storing the data in a proprietary format. The movement is achieved by a succession of mediations

[0051] The data transport process involves several essential steps, including the generation of content by the source application in the form of a file or message. This data is then routed to dedicated intermediary applications, which play a key role in managing this information.

[0052] Intermediary applications are responsible for reading, adapting, transforming, and distributing content between other intermediary applications, as well as to the final destination applications. This mediation process ensures that data is presented in a manner appropriate for use by target applications, whether similar to or different from the source application. These applications are represented in the.

[0053] There are technical products that are exchanged on a site. Application A then transfers data xy to application B and C with ids such as 0001, 0002, 0003, etc.

[0054] The expected result is a list of customer and product data exchanges between applications A, B and C; as shown in.

[0055] List of executed trades:

[0056] 20230731 10:34: A -> B: data: xy, id:0001 successful

[0057] 20230731 10:35: A -> B: data: xy, id:0002 in error

[0058] 20230731 10:35: A -> C: data: xy, id:0002 successful

[0059] Naturally, the invention is described in the foregoing by way of example. It is understood that those skilled in the art are able to carry out different variant embodiments of the invention without departing from the scope of the invention.

[0060] Shows an example of a design for transporting XY data from application A to applications B and C.

[0061] The transport is organized in a so-called “Y” flow. That is to say in 3 distinct stages, 1 for the extraction of data from its source format, from A. Then 2 for the propagation of data to its target format, towards B and C.

[0062] Each of these steps is supported by technological foundations, which can be different. In this example: an ETL (acronym for: Extract-Transform-Load in English for Extract the transformation and load), then an ESB (acronym for: Enterprise Services Bus in English for Enterprise Services Bus), then an API Management (acronym for: Application Programming Interface Management in English for Application Programming Interface Management).

[0063] Shows an example implementation of transporting XY data from application A to applications B and C.

[0064] This figure allows to transcend the technical and often arduous nature of data transport and consequently of its tracking by transport traces. The invention translates them into a more accessible concept: an exchange of data between different applications, such as in. This simplification process is essential to allow users, whether experts or not, to quickly and easily grasp the meaning of transport traces and to integrate them into their overall understanding of the functioning of computer systems.

[0065] Four individual processes or functions are implemented to transport “xy”:

[0066] f1: generation of the file, by an ETL platform.

[0067] f2: reading the file then separating it into 1 message per line, using an ETL platform.

[0068] f3: output n° s1 to B: reading of the message, by an ESB, then call of an HTTP service, acronym of Hypertext Transfer Protocol for standard communication protocol, exposed by an API Management

[0069] f4: output n° s2 to C: reading of the message, by an ESB, then generation of a file.

[0070] Shows how to generate transport traces.

[0071] The format of the data ingested by this trace management system, commonly referred to as "transport traces", represents the first essential element of the system. These traces are generated by various intermediation applications, illustrated here by applications f1 to f4, covering the families commonly encountered in the transport field. At the output of the applications there can be fi files, or mes or m messages. The objective is to have one or more t traces at the output of the application.

[0072] The transport trace format is structured into five individual blocks, each playing a specific role in managing and representing this data.

[0073] These five blocks are as follows:

[0074] Generality Block: This block contains crucial information about the trace, including the transport status, which can be classified as success, error, alert, or information. Additionally, it can contain information about the environment where the trace was produced, such as "Production" or "Receipt."

[0075] Message block: In this block, we store information such as the date of receipt, including the date, time, milliseconds, and time slot. In addition, technical headers can be included, as can the message body. Each message is also associated with a so-called correlation identifier, which is optional but recommended, and a unique identifier, a free but definitely unique string for each message.

[0076] Route block: This block specifies the details of the route taken by the message. It can include the process name, its version, and its unique identifier, a free but unique string for each transport process, whether it is a mediation or a function.

[0077] "Business" block: For each piece of data transported, this block records its type and identifier. This allows for a conceptual but precise definition of the data handled during the transport process.

[0078] "Infrastructure" block: This block contains information related to the infrastructure, such as the ID of the server where the processing took place and the ID of the instance where the processing took place.

[0079] "Error" block: Finally, the last block is the error block, which records the code associated with the error that occurred and an explanatory message. When an error occurs during processing, this block is used to describe the error encountered in detail. It includes a unique code, the type of the class involved, the error details, and possibly the code's call stack.

[0080] This five-block structure ensures a complete and consistent representation of transport traces, making them easier to manage and analyze within the system.

[0081] Here are examples of “transport trace”: these examples carry mandatory and optional data.

[0082] Transport trace successful:

[0083] in the General block:● transport status: SUCCESS● environment: UAT

[0084] in the Message block: a unique identifier: AAA-BBBB-CCCC-001 the creation date: 2024-09-14'T'14:12:45.0234 the correlation identifier: 123-456-789 body of the message: <element> 123 < / element>

[0085] in the Route block:● an identifier of the route that transports it: SAP-IDOC-0345● a name: “Consumption of IDOC 0345 from SAP”

[0086] in the Business block:● the data transported:● ABC invoice● DEF customer

[0087] Transport trace failed:

[0088] in the General block: ● transport status: ERROR ● environment: UAT

[0089] in the Message block: a unique identifier: AAA-BBBB-CCCC-002 creation date: 2024-09-14'T'14:13:45.0234

[0090] in the Route block: ● an identifier of the route that transports it: SAP-IDOC-0345

[0091] in the Business block:● the data transported:● company XYZ● customer DEF

[0092] in the Error block:● a code: ERROR-0001● exception class: ClassCastException● detail: “This is a Casting error”

[0093] in the Infrastructure block: the server identifier: SRV-10234

[0094] Transport trace on alert:

[0095] in the General block:● transport status: WARNING● environment: PROD

[0096] in the Message block: a unique identifier: AAA-BBBB-CCCC-003 creation date: 2024-09-15'T'09:13:45.0234

[0097] in the Route block:

[0098] ● an identifier of the route that transports it: EDI-FIC-123● the body with the contents of the file: abcfdev.

[0099] in the Business block:● the data transported:● ABC file● DEF client

[0100] In the example above, the character string identifying the road is eleven characters long. In another example, there are at least eleven characters.

[0101] Within the system of the invention, four matching processes are implemented to result in the discovery of an exchange:

[0102] Processing by transported business data,

[0103] Processing by mapping the information system concerned,

[0104] A treatment by the temporal concordance of the traces between them,

[0105] A treatment by managing a correlation identifier.

[0106] Lamontre shows a schematic diagram of how databases work.

[0107] We find the treatments f1 to f4, of the, which generate traces t to tn.

[0108] These traces, t1 to tn, are then ingested by a database of traces bdd tn to be analyzed by one or more reconciliations R to constitute a database bdd E of exchanges E.

[0109] Rapprochement through mapping:

[0110] Information system mapping provides a set of crucial elements that play a determining role in the organization and effective management of the information system. These elements include:

[0111] Application Repository: This repository lists the different applications involved in the system, thus identifying the software, platforms, and application modules necessary for its operation.

[0112] Flow Repository: This repository reveals the complete mapping of data and information flows within the system, thus allowing an accurate visualization of the paths taken by data between different parts of the system.

[0113] Mediation Reference: This reference lists intermediation processing, namely the IT functions executed on servers or dedicated platforms, which play a central role in the transport and transformation of data within the system.

[0114] Data Repository: This repository centralizes all information on the data handled by the system, including their structure, format, and semantics, thus facilitating their management and use.

[0115] Service Contract Reference: This reference defines the service contracts that govern the interactions between the system components, thus ensuring smooth and reliable coordination.

[0116] What most strongly characterizes the organization of this information system are the close and complex dependencies between the different key elements.

[0117] These dependencies include:

[0118] Intermediation Processing and Flows: Intermediation processing is intrinsically linked to data flows, as it is responsible for managing and transforming these flows.

[0119] Flows and Applications: Data flows are closely linked to applications, as they transport information between the different applications in the system.

[0120] Streams and Data: Data streams are also dependent on the data they carry, as this data determines their content and structure.

[0121] Intermediation Processing and Service Contracts: Intermediation processing interacts directly with service contracts to ensure execution in accordance with the agreed agreements.

[0122] In order to better organize the elements of representation of a map, we have introduced three fundamental notions:

[0123] Exchange: This is the logical transport of one or more pieces of information between several systems. This concept offers a simplified view of the process.

[0124] Flow: The flow represents the complete technical transport from one application module to another. It encompasses all technological aspects of this transmission such as formatting, transformation, routing, and protocol adaptation. In the literature, it is commonly referred to as the half-flow concept.

[0125] Mediation: Mediation is a computer function or process executed on a server or platform provided. Each mediation exposes and uses interfaces and protocols to consume and produce data. It is the smallest identifiable element in the transport chain.

[0126] These notions are interconnected by logical dependencies: exchange -> flow -> mediation.

[0127] For example, there is also data matching. Here is an example:

[0128] Traces● 20230731 10:35: route: m1001, business: xy, status: info● 20230731 10:35: route: m2001, business: xy, status: success● 20230731 10:36: route: m3001, business: xy, status: success● 20230731 10:36: route: m4001, business: xy, status: success

[0129] Related streams:● 20230731 10:35: f1: data: xy, successful● 20230731 10:36: f2: data: xy, successful● 20230731 10:36: f3: data: xy, successful

[0130] Exchange deducted:

[0131] 20230731 10:34: A -> B: data: xy, id:0001 successful

[0132] A key aspect of the present invention is the reconciliation by transported data, which answers the crucial question of what is transported. However, it is important to note that the types of transported data vary depending on the source and target applications. This is why we have introduced a more abstract notion called Information, necessary for exchanges. The repository ensures the correspondence between Data Types and Information Types.

[0133] Correspondence between Mediation and Exchange

[0134] The transport traces communicated by the treatments in charge of part of the transport, the mediations, which manipulate the corresponding data are selected to be part of the same exchange, thus guaranteeing efficient management of the transported data.

[0135] For example, there is also the correlation matching. Here is an example of matching by correlation ID

[0136] Traces:● 20230731 10:41: route: m1001, correlation: abc-def, status: info● 20230731 10:41: route: m2001, correlation: abc-def, status: success● 20230731 10:42: route: m3001, correlation: abc-def, status: error● 20230731 10:44: route: m4001, correlation: abc-def, status: success

[0137] Related flows:● 20230731 10:41: f1: correlation: abc-def, successful● 20230731 10:42: f2: correlation: abc-def, in error● 20230731 10:44: f3: correlation: abc-def, successful

[0138] Exchange deducted:

[0139] 20230731 10:41: A -> B: data: xy, in error

[0140] A key mechanism for establishing a direct link between mediations is the correlation identifier. An identifier is added to the headers of transported messages, allowing distinct calls to be linked together, which facilitates the management of intermediation processes.

[0141] For example, there is also the temporal reconciliation. Here is an example of temporal reconciliation

[0142] Traces● 20230731 10:35: route: m1001, business: xy, status: info● 20230731 10:35: route: m2001, business: xy, status: success● 20230731 10:36: route: m3001, business: xy, status: success● 20230731 10:36: route: m4001, business: xy, status: success● 20230731 10:45: route: m1001, business: xy, status: info● 20230731 10:45: route: m2001, business: xy, status: success● 20230731 10:46: route: m3001, business: xy, status: success● 20230731 10:46: route: m4001, business: xy, status: success

[0143] Related streams:● 20230731 10:35: f1: data: xy, successful● 20230731 10:36: f2: data: xy, successful● 20230731 10:36: f3: data: xy, successful● 20230731 10:45: f1: data: xy, successful● 20230731 10:46: f2: data: xy, successful● 20230731 10:46: f3: data: xy, successful

[0144] Deduced exchange: 20230731 10:34: A -> B: data: xy, successful 20230731 10:45: A -> B: data: xy, successful

[0145] Time reconciliations are essential but complex to determine. It is imperative to define a maximum execution time for each concept: exchange, flow, and mediation. Based on their dependencies and these maximum deadlines, the traces are reconciled into sets that execute within an acceptable gap, thus ensuring optimized time management.

[0146] Then there are the dependencies of the Rapprochements

[0147] Following successive reconciliations in accordance with the steps described, the exchanges are discovered and made available. Each exchange contains crucial information, including its status, its source and target applications, as well as the data transported.

[0148] It is emphasized that all features, as they emerge for a person skilled in the art from this description, the drawings and the attached claims, even if they have been specifically described only in relation to other specific features, both individually and in any combinations, may be combined with other features or groups of features disclosed herein, provided that this has not been expressly excluded or that technical circumstances make such combinations impossible or meaningless.

Claims

Information system for monitoring data transport, comprising at least two servers connected to each other by a communication network, computer client applications connecting to said servers, characterized in that it comprises:- transport traces having at least the information coming from said following computer client applications from a server; said transport traces comprise:- a unique identifier corresponding to a sequence of characters configured to associate a unique identifier with each transport trace;- a message carrying at least one temporal information corresponding to a date, a time with milliseconds and a time slot;- a transport route corresponding to a character string defining the processing in charge of the transport;- a current status corresponding to one of the following values: information, success, out of time, alert, and error;- a list of references of the transported data corresponding to a type and an identifier; - a database of transport traces configured to organize the information relating to the transport traces, a database of the exchange mapping of the supervised information system configured to reference the exchanges with characteristics of the source applications, target applications and its time limit, and a database for the exchanges calculated from the traces configured to define the type of each transport trace. ; - means using at least one microprocessor to process transport traces and exchanges in the database of transport traces and the database of calculated exchanges, called calculated exchange;said means for processing transport traces and exchanges modifying the status of the calculated exchange according to a server if the transport trace has not arrived at the corresponding server in the exchange database then the current status is modified and one of the following values ​​is modified: information, success, out of time, alert, and error; - means for viewing transport traces and calculated exchanges; said system references the exchanges with the characteristics of the source applications, the target applications and its allocated time in a mapping database;said system assigns the transport traces with elements called "Mediation", defined by at least one name of one of the elements called "Mediation", and a given time limit of one of the elements called "Mediation", and represented in the mapping database as well as in the database of calculated exchanges as one of the elements called "Mediation" calculated, using in priority the transport route configured to process the transport load then the transported data.; The system of claim 1, wherein the system assigns the elements called "Mediations" with elements called "Flows", defined by at least one name of one of the elements called "Flows", and a given time limit of one of the elements called "Flows", represented in the mapping database, as well as in the database of calculated exchanges as one of the elements called "Flows" calculated, according to the dependencies defined in the mapping database, and the given time limits of one of the elements called "Mediation". The system of claim 2, wherein the system assigns the calculated Flows with calculated exchanges according to the dependencies defined in the mapping database, and the time limits of one of the elements called "Flows". The system of claim 3, wherein the status of one of the elements called "Mediation" calculated is determined according to the status of the oldest transport trace affected, said traces classified on a time axis relative to a clock of a microcontroller used to generate the transport trace.

Images