Lightweight software infrastructure for integrating decision nodes into automated data mining processing

Abstract

Data processing pipeline, comprising a plurality of microservices (MicServ) sharing the same structure and configured to exchange data with each other via an enterprise service bus (ESB), each microservice being implemented by means of a basic block (100) comprising a logical core (Lgc) encapsulated by a software structure (FWK) limited to five interface blocks, each communicating with the logical core (Lgc): a first interface block (ASync) for asynchronous messaging; a second interface block (Sync) for direct bidirectional communication synchronously with the bus (ESB); a third interface block (Journ.Intrf) for recording the activity of the five interface blocks and the logical core (Lgc) in a log file (Journ); a fourth interface block (Strg.Intrf) for reading and writing to persistent storage (Strg); a fifth interface block (Config.Intrf) for configuration loading.Figure to be published with the abbreviation: Fig. 2.

Description

Title of the invention: Lightweight software infrastructure for integrating decision nodes into automated data mining processing. TECHNICAL FIELD OF THE INVENTION

[0001] The invention relates to a lightweight software infrastructure facilitating the integration of decision nodes in an automated data drilling and classification processing activity using artificial intelligence modules and elementary business rules. TECHNOLOGICAL BACKGROUND

[0002] Data mining refers to the analysis of raw data using various complementary methods, with the aim of transforming this raw data into useful information by establishing relationships between the data, for example by sorting the data. Such analysis relies on potentially complex algorithms that implement different types of analysis within a decision tree containing nodes called decision nodes.

[0003] Among the types of analysis implemented, we can mention the search for associations between data or events, the search for sequences of events, the search for new classifications relevant to the classification of data, data partitioning, which consists of dividing a dataset into homogeneous groups, or predictive analysis, which consists of analyzing present facts to derive predictive hypotheses about future events. A complete analysis may require the combination of each of these types of analysis, each of which can be broken down into relatively simple basic tasks taken individually, but whose assembly can prove difficult.

[0004] Indeed, routing data through different decision nodes, forming a complex set of data processing and inference mechanisms known as a "data processing pipeline," is all the more difficult when the desired analysis is exhaustive and the amount of available data is massive, continuously renewed, and originates from heterogeneous data formats. Numerous design principles exist for implementing such a pipeline, such as microservices or hexagonal architecture, but there is no concrete framework or operational guide for developing it and making it easy to integrate quickly, monitor, and maintain.

[0005] Thus, a satisfactory solution from a practical point of view has yet to be implemented. Description of the invention

[0006] One object of the invention is to provide a lightweight software infrastructure with application programming interfaces (APIs) and standardized logical encapsulation that accelerates and harmonizes the creation of microservices communicating with each other via the APIs, thus forming an integrated processing chain. Using this infrastructure, the microservices can exchange data bidirectionally and transparently via asynchronous or synchronous methods and implement a complex sequence of decision processes, for example, composed of artificial intelligence decision nodes and enterprise consolidation rules.

[0007] Each node, integrated into the processing chain, uses the software infrastructure according to the invention, designed to provide the essential functions for any algorithm (parameter loading, data persistence, generation of traces on internal activity, synchronous and asynchronous message exchanges) while (1) limiting the technologies used and (2) standardizing coding practices. This simplifies the monitoring and maintenance process by connecting each node of the processing chain consistently with a single method.

[0008] To achieve this goal, one aspect of the invention is a data processing pipeline, comprising a plurality of microservices sharing the same structure and configured to exchange data with each other via an enterprise service bus, in which each microservice is implemented by means of a basic block comprising a logical core encapsulated by a software structure limited to five interface blocks each communicating with the logical core, in which the five interface blocks are: a first interface block comprising asynchronous messaging; a second interface block configured to allow direct bidirectional communication synchronously with the enterprise service bus; a third interface block configured to record in a log file the activity of the five interface blocks and the logical core;a fourth interface block configured to allow reading and writing data to or from permanent data storage; a fifth interface block is configured to load the configuration of the logical core and the four other interface blocks mentioned above from a configuration file.

[0009] A data processing pipeline according to the invention is particularly simple and easy to set up, maintain and modify, based on a lightweight and extremely modular architecture based on blocks having the same structure and manageable through the same system of interfaces.

[0010] According to additional, non-limiting features of the first aspect of the invention, considered individually or in any technically feasible combination:

[0011] - each of the interface blocks can be configured to exchange data with the other interface blocks;

[0012] - each of the interface blocks can be configured to exchange data with the other interface blocks without going through the logical core;

[0013] - each microservice can be configured to exchange data with the others data pipeline microservices using the enterprise service bus (ESB) via the first interface blocks;

[0014] - Each microservice can record its internal activity in the log file according to a standardized format common to the plurality of microservices;

[0015] - a record in the log file of an event of an internal activity of Each microservice can include at least a date, a criticality indicator, an identification of the microservice concerned, an identification of the event, and a description of the event;

[0016] - each microservice can use the same timestamp to record activity of the five interface blocks and the logical core in the log file the software structure is unique and shared between each of the microservices;

[0017] - the second interface block can be configured to direct an external call to one of the four other interface blocks or to the logical core, based on an identifier of the call type carried by the message;

[0018] - a first of the microservices may include a logical core configured for record an identity for each of the microservices connected to the data processing pipeline; and

[0019] - a second of the microservices may include a logical core configured for consult the identities of each of the microservices recorded by the logical core of the first of the microservices. BRIEF DESCRIPTION OF THE FIGURES

[0020] Other features and advantages of the invention will become apparent from the detailed description of the invention which follows with reference to the accompanying figures in which:

[0021] [Fig. 1] Fig. 1 represents a basic block of the software infrastructure according to the invention; and

[0022] [Fig.2] Figure [Fig.2] represents a data processing pipeline according to the invention, incorporating a plurality of basic blocks illustrated by [Fig.1]. DETAILED DESCRIPTION OF THE INVENTION

[0023] Figures 1 and 2 illustrate the software infrastructure according to the invention.

[0024] Figure 1 illustrates a basic block 100 of the software architecture according to the invention. This block constitutes a MicServ microservice. The logical core of the block, Lgc in Figure 1, consists of computer logic dedicated to a single task and translating an application algorithm, for example a business rule or a component of a data mining application.

[0025] Figure 2 illustrates the integration of five MicServ microservices within the software infrastructure according to the invention, forming, for example, a DataPip data processing pipeline connected to an App application by means of a SuperV monitoring module controlled by a user Usr via a Usr.Interf user interface, for example, a graphical interface. An Enterprise Service Bus (ESB) is configured to communicate with each of the MicServ microservices as well as with a Cal module of the App application, dedicated to data processing. Each microservice comprises its own logical core, performing a specific function within the data processing pipeline.

[0026] A software structure FWK encapsulates the logical core Lgc of each of the MicServ microservices in the data processing pipeline and isolates them from external communications. This software structure FWK comprises five interface blocks that allow the microservice's logical core to operate optimally by offloading it from all tasks not directly related to the single task to which it is dedicated. Furthermore, the software interface FWK is configured to provide direct connections between each of the interface blocks, enabling them to communicate directly with each other and preventing the Lgc logical core's computations from being disrupted by operations not directly related to its specific function.Of course, other modules or external services besides those mentioned in this document can communicate with the microservices, either via the ESB enterprise service bus or directly via the second Sync interface block.

[0027] A first ASync interface block is configured to allow indirect bidirectional communication via asynchronous messaging with another external component, such as another microservice, using a publish-subscribe method. The logical core can thus request to route a message on the Enterprise Service Bus (ESB). Other microservices subscribed to this route can receive and interpret this message. No response is expected from the message sender unless the logical core is configured to listen for and receive messages on a specific route, for example, a response channel defined in a configuration file.

[0028] A second Sync interface block is configured to allow direct bidirectional communication in a synchronous manner according to a ReST (Representational State Transfer) approach.

[0029] Certain external calls received by this second block can be processed directly at the interface block level, for example, to provide information about the status or version of the microservice. The second Sync interface block is configured to generate a message for one of the four other interface blocks depending on the type of call received, without going through the Lgc logical core.

[0030] Alternatively, an external call can be transferred to the logic core, provided that the processing of this call is linked to the single task of the logic core Lgc, such as the transmission of the last result calculated by the core, such as the prediction of a relevance coefficient or the result of a sorting operation.

[0031] An external call is a message issued by an element external to a given microservice. The external element can be another microservice, the application App, or any other component connected directly or indirectly to the microservice in question. This external call carries metadata including a call type identifier. The second interface block is configured to direct each external call to one of the four other interface blocks or to the logical core, depending on the call type, based on the call type identifier carried by the message and the identifier of its sender. The call type can indicate the degree of urgency of handling the call, the interface block to which it should be transmitted, or the fact that the call is related to the single task of the logical core of the given microservice.

[0032] In a microservice, the processing or calculations performed by the Lgc logical core and the two Sync and Async blocks are parallelized using multithreading so that they can act independently of each other. This principle is particularly useful when, for example, the logical core is performing a calculation: the logical core can make the progress of this calculation available to the Sync or Async block, which, in the event of an external query, can consult and transmit this progress to an element external to the microservice without stopping or disrupting the ongoing work of its logical core.

[0033] A third interface block, Journ.Intrf, is configured to record the internal activity of the elements comprising the MicroServ microservice (the five interface blocks and the logical core) in a single Jour log file, using a standardized format. This means that, within the data processing pipeline, the events recorded in the log file are identified according to a nomenclature common to all the relevant microservices in the pipeline.

[0034] The function of the log file is to record all events related to the use of the pipeline. It allows, for example, retrieval for a set of microservices exchanging messages containing data, the order in which the messages were sent and processed is recorded. The nomenclature is defined to associate each recorded event with an occurrence date, a criticality level, an identification of the sending element (one of the interface blocks or the logical core of an identified microservice), a descriptive code of the event type (start, stop, receive, send, etc.), and a detailed description. Thanks to the internal interconnection of the FWK software structure of the interface blocks, it is possible, for example, to reconfigure the nomenclature based on the information stored in the configuration file.

[0035] With regard to the specific case of the occurrence date, a common timestamp is used by the microservices in the pipeline. This can involve using a common clock via a protocol such as a network time protocol, or NTP (Network Time Protocol). For example, if all the microservers are hosted on the same server, the server's clock can serve as the common clock. Alternatively, it can involve multiple clocks from different servers when the pipeline microservices are hosted on these different servers; the important point is that these different clocks are synchronized. In all cases, the occurrence dates of the recorded events are immediately comparable to one another so that the order of occurrence of the events distributed in the log files of each microservice can be immediately reconstructed.

[0036] The advantage of this standardized centralisation is to allow easy aggregation of recorded data, enabling their temporal and event-based consolidation in an extremely simple way, and their subsequent processing by a log analyzer to understand globally the activity of a given microservice or as part of scaling up the entire DataPip data processing pipeline.

[0037] A fourth Strg.Intrf interface block is configured to allow reading and writing data into or to a permanent Strg data storage such as a database, for example to store the results of data processing by the Lgc logical core, without the latter taking charge of the details of writing the data but on the contrary delegating this task to the fourth interface block.

[0038] A fifth interface block, Config.Intrf, is configured to load the respective configurations of the logical core and the four other interface blocks mentioned above from a Config configuration file, and to transmit their respective configurations to its elements. This could involve, for example, loading the log configuration, the microservice version, or the specific configuration of the Lgc logical core.

[0039] Each of the interface blocks is configured to exchange data (configuration, log, data to be stored permanently, etc.) with the other interface blocks without going through the logical core.

[0040] Integrated within a data processing pipeline, each MicServ microservice is configured to exchange data with the other microservices in the data processing pipeline via the Enterprise Service Bus (ESB) through the first ASync interface blocks, each consisting of asynchronous messaging. This mode of operation optimizes the operation of the Lgc logical core by allowing it to complete ongoing processing without interruption.

[0041] In the data processing pipeline illustrated by [Fig.2], two of the five microservices have a supervisory role over the other microservices.

[0042] One of these two microservices constitutes a register: its logical core, designated as Reg in [Fig. 2], is configured to store in its Strg memory the identity and unique task associated with each of the MicServ microservices connected to the data processing pipeline. This microservice thus enables the centralized discovery of services offered by the pipeline: an application connecting to the pipeline would first consult the register to determine if a particular application exists, such as a classifier enabling binary prediction in a specific domain or any other function sought by the application and potentially offered by the pipeline.

[0043] The other of these two microservices plays a monitoring role, that is, a watchdog role, for the microservices in the data processing pipeline: its logical core, indicated as WDog in [Fig. 2], is configured to consult the register formed by the Reg logical core microservice, check the status of each of the microservices registered therein, for example by reading and copying the latest events recorded in their respective log files, and make the latest recorded events thus retrieved available to an external application through the Sync interface block. All these operations are performed without needing to access the logical cores of the monitored microservices, the interface blocks being sufficient for this task and thus offloading the core from it.

[0044] The data processing pipeline is configured so that a user (Usr) can, for example via an interface, exchange data with the second interface block (Sync). As described above, this block is configured to minimize external call transmissions to the logical core. However, it allows for good system responsiveness to the user.

[0045] Here, the WDog logical core microservice acts as an intermediary between the user-controlled SuperV monitoring module and the other microservices, playing a role of a hub preventing the monitoring module from having to query all the other microservices.

[0046] The FWK software structure can be duplicated so that each microservice has its own copy, or it can be shared by these microservices. In the latter case, pipeline maintenance is facilitated because an update of the software structure is implicitly applied to each of the microservices.

[0047] A software architecture such as the one described above, with basic blocks containing a logical core Lgc encapsulated by a software structure FWK limited to five interface blocks providing the essential services for the operation of an IT service, ensures excellent compatibility between microservices, each implemented using one of these basic blocks. Indeed, limiting the type of interfaces leads to a limitation on the type of data or calls to be managed, and thus facilitates the integration, monitoring, and maintenance of a data pipeline based on this software architecture.

[0048] Of course the invention is not limited to the described embodiment and alternative embodiments can be made without departing from the scope of the invention as defined by the claims.

Claims

Demands

1. A data processing pipeline, comprising a plurality of microservices (MicServ) configured to exchange data with each other, characterized in that the microservices are configured to exchange data with each other via an Enterprise Service Bus (ESB), in which each microservice is implemented by means of a basic block (100) comprising a logical core (Lgc) encapsulated by a software framework (FWK) which is the same for each of the microservices and which is limited to five interface blocks each in communication with the logical core (Lgc), in which the five interface blocks are: - a first interface block (ASync) comprising asynchronous messaging; - a second interface block (Sync) configured to allow direct bidirectional synchronous communication with the Enterprise Service Bus (ESB); - a third interface block (Joum.Intrf) configured to record in a log file (Joum) the activity of the five interface blocks and the logical core (Lgc); - a fourth interface block (Strg.Intrf) configured to allow reading and writing data into or to a permanent data storage (Strg); - a fifth interface block (Config.Intrf) is configured to load the configuration of the logical core and the four other interface blocks mentioned above from a configuration file (Config).

2. Data processing pipeline according to claim 1, wherein each of the interface blocks is configured to exchange data with the other interface blocks.

3. Data processing pipeline according to any one of claims 1 to 2, wherein each of the interface blocks is configured to exchange data with the other interface blocks without going through the logical core.

4. A data processing pipeline according to any one of claims 1 to 3, wherein each microservice (MicServ) is configured to exchange data with the other microservices of the data pipeline using the Enterprise Service Bus (ESB) via the first interface blocks (ASync).

5. Data processing pipeline according to any one of claims 1 to 4, wherein each microservice (MicServ) records its internal activity in the log file in a standardized format common to the plurality of microservices.

6. Data processing pipeline according to claim 5, wherein a log file record of an event of an internal activity of each microservice includes at least a date, an indication of criticality, an identification of the microservice concerned, an identification of the event, and a description of the event.

7. Data processing pipeline according to any one of claims 1 to 6, wherein each microservice (MicServ) uses the same timestamp to record the activity of the five interface blocks and the logical core (Lgc) in the log file.

8. Data processing pipeline according to any one of claims 1 to 7, wherein the software structure (FWK) is unique and shared between each of the microservices.

9. Data processing pipeline according to any one of claims 1 to 8, wherein the second interface block is configured to direct an external call to one of the other four interface blocks or to the logical core, based on an identifier of the call type carried by the message.

10. Data processing pipeline according to any one of claims 1 to 9, wherein a first of the microservices includes a logical core (Reg) configured to register an identity of each of the microservices (MicServ) connected to the data processing pipeline (DataPip).

11. Data processing pipeline according to claim 10, wherein a second of the microservices includes a logical core (WDog) configured to look up the identities of each of the microservices registered by the logical core (Reg) of the first of the microservices.

Images