Intelligent data processing system with multi-interface front-end and back-end

Abstract

The method includes identifying (402) via a user interface (310) a sequence of transformations to be performed on an input dataset (308). The method also includes identifying (404) a first context associated with the input dataset. The method further includes selecting (406) a first execution engine of a plurality of execution engines (332) to be used to perform the sequence of transformations on the input dataset based on the first context. The method further includes providing (416) first code (334) implementing the sequence of transformations to the first execution engine, and executing (418) the first code with the first execution engine to perform the sequence of transformations on the input dataset.

Description

[Technical field]

[0001] The present disclosure is directed generally to data processing systems, and more particularly, to an intelligent data processing system with a multi-interface front end and back end. [Background technology]

[0002] Various data processing systems, such as those used to implement machine learning or artificial intelligence (ML / AI) algorithms or other data processing algorithms, can support a wide range of techniques and tools used to acquire and process data. The design of a specific data processing system is often based on a set of assumptions that are first made about characteristics such as the size or amount of data to be processed by the data processing system, the type of data to be processed by the data processing system, and the required or desired performance of the data processing system. These assumptions heavily influence design decisions for the data processing system at both the interface (front-end) level and the implementation (back-end) level. Design decisions at the interface or front-end level typically include determining the design of a user interface or other interface that is used to interact with a user or application that accesses the data processing system. Design decisions at the implementation or back-end level typically include compiling code that is executed to perform the desired data processing operations. Summary of the Invention

[0003] The present disclosure relates to an intelligent data processing system with a multi-interface front end and back end.

[0004] In a first embodiment, a method includes identifying, via a user interface, a sequence of transformations to be performed on an input data set. The method also includes identifying a first context associated with the input data set. The method further includes selecting a first execution engine of a plurality of execution engines to be used to perform the sequence of transformations on the input data set based on the first context. The method further includes providing first code implementing the sequence of transformations to the first execution engine, and executing the first code with the first execution engine to perform the sequence of transformations on the input data set.

[0005] In a second embodiment, an apparatus includes at least one processing device configured to identify, via a user interface, a sequence of transformations to be performed on an input data set. The at least one processing device is further configured to identify a first context associated with the input data set. The at least one processing device is further configured to select a first execution engine of a plurality of execution engines to be used to perform the sequence of transformations on the input data set based on the first context. The at least one processing device is further configured to execute, with the first execution engine, first code implementing the sequence of transformations to perform the sequence of transformations on the input data set.

[0006] In a third embodiment, a non-transitory computer readable medium stores computer readable program code that, when executed, causes one or more processors to identify, via a user interface, a sequence of transformations to be performed on an input dataset. The medium also stores computer readable program code that, when executed, causes the one or more processors to identify a first context associated with the input dataset. The medium further stores computer readable program code that, when executed, causes the one or more processors to select, based on the first context, a first execution engine of a plurality of execution engines to be used to perform the sequence of transformations on the input dataset. The medium further stores computer readable program code that, when executed, causes the one or more processors to execute, with the first execution engine, first code implementing the sequence of transformations to perform the sequence of transformations on the input dataset.

[0007] Other technical features will be readily apparent to one skilled in the art from the following figures, descriptions, and claims. [Brief description of the drawings]

[0008] For a more complete understanding of the present disclosure, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which:

[0009] [Figure 1] 1 illustrates an example system supporting intelligent data processing with a multi-interface front-end and back-end in accordance with the present disclosure. [Diagram 2] 1 illustrates an example of a device that supports intelligent data processing with a multi-interface front end and back end in accordance with the present disclosure. [Diagram 3] 1 illustrates an example architecture of an intelligent data processing system with a multi-interface front end and back end in accordance with the present disclosure. [Figure 4]1 illustrates an example method for intelligent data processing with a multi-interface front-end and back-end in accordance with the present disclosure. [Diagram 5] 1 illustrates an example process for generating execution engine specific code for execution by a selected backend in accordance with the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] 1-5 described below, and the various embodiments used herein to describe the principles of the present disclosure, are for illustrative purposes only and should not be construed as limiting the scope of the present disclosure in any way. Those skilled in the art will appreciate that the principles of the present disclosure may be implemented in any type of suitably arranged device or system.

[0011] As mentioned above, various data processing systems, such as those used to implement machine learning or artificial intelligence (ML / AI) algorithms or other data processing algorithms, can support a wide range of techniques and tools used to acquire and process data. The design of a specific data processing system is often based on a set of assumptions that are first made about characteristics such as the size or amount of data to be processed by the data processing system, the type of data to be processed by the data processing system, and the required or desired performance of the data processing system. These assumptions heavily influence design decisions for the data processing system at both the interface (front-end) level and the implementation (back-end) level. Design decisions at the interface or front-end level typically include determining the design of a user interface or other interface that is used to interact with a user or application that accesses the data processing system. Design decisions at the implementation or back-end level typically include compiling code that is executed to perform the desired data processing operations.

[0012] Unfortunately, an initial set of assumptions for the design of a data processing system often results in data processing system criteria that are suboptimal, especially for use cases that are outside of that initial set of assumptions. Furthermore, there is often a clash between what should and need not be common between different data processing system designs. For example, commonality across front-end designs may be desirable because it allows users to become familiar with the interfaces of the various data processing system designs, improving the ease of use of the data processing system. In contrast, commonality across back-end designs may be undesirable because optimizing the implementation of different data processing system designs usually depends on the specific use case of the data processing system. Attempting to utilize the same data processing system design across different use cases can result in various disadvantages that usually outweigh any advantages. For example, an interface that is useful for streaming data may not be the best interface to use for batched data (and vice versa). As another example, a data exploration tool that is useful for processing small amounts of data at any given time may become less useful or non-functional when attempting to process large amounts of data. Even though multiple implementations are provided for use, users themselves often need to select which implementation is to be used at a given time for particular data.

[0013] The present disclosure provides an intelligent data processing system with a multi-interface front end and back end. As described in more detail below, the data processing system may support at least one front end (interface) that may be used by one or more users to identify data and a transformation (code) to be applied to the data. Depending on the implementation, the data processing system may support the use of a single front end or multiple front ends. The data processing system may capture the transformation and store information identifying the transformation, such as in a database. The data processing system may also use that information and a context associated with the data to perform a data processing operation. For example, the data processing system may select a particular execution engine from among multiple execution engines (representing multiple back ends) depending on the context, and code implementing the transformation to be performed may be generated or otherwise obtained for that particular execution engine. The particular execution engine may execute the code to perform the data processing operation on the data. The particular context may vary based on a number of factors, examples of which are provided below. The particular execution engine selected for use may also vary, such as when different users or applications are associated with different contexts, or when the context associated with the same user or application changes over time. Various codes for performing data transformations may be generated or otherwise obtained based on the stored information, and the various codes may be executed by different execution engines, such that the same sequence of transformations may be subsequently requested one or more times and executed using one or more execution engines.

[0014] In this manner, the data processing system can dynamically select an execution engine that best fits the context in which at least one user or application is attempting to execute a data transformation (code). The execution engines selected for different users or applications can vary based on their different contexts, and the execution engine selected for a single user or application can vary over time based on changing contexts. As a result, at least one user need only hand-write or otherwise provide code that defines a data transformation once, and the data transformation can be executed using different execution engines without requiring the user to rewrite the code for each execution engine. In essence, the data processing system can decouple the selection of the interface (front-end) from the selection of the implementation (back-end), allowing an appropriate back-end to be selected and used regardless of which front-end may be used. Overall, this allows the data processing system to switch between different execution engines depending on the nature of the data processing task being performed. This can be extremely useful in a number of scenarios, including scenarios in which use cases change over time or applications evolve over time. The data processing system can thereby provide a reliable data processing tool that is interactive, replicable, redundant, and secure. Below are various examples of specific benefits or advantages that may be obtained from using a data processing system designed in accordance with the present disclosure.

[0015] FIG. 1 illustrates an exemplary system 100 supporting intelligent data processing with a multi-interface front-end and back-end in accordance with the present disclosure. For example, the system 100 illustrated herein may be used to support a data processing system having a functional architecture providing one or more interfaces at the front-end and execution engine selection at the back-end, as described in more detail below. As illustrated in FIG. 1, the system 100 includes user devices 102a-102d, one or more networks 104, one or more application servers 106, and one or more database servers 108 associated with one or more databases 110. Each of the user devices 102a-102d communicates over the network 104, e.g., via a wired or wireless connection. Each of the user devices 102a-102d represents any suitable device or system used by at least one user to provide or receive information, such as a desktop computer, a laptop computer, a smartphone, and a tablet computer. However, any other or additional types of user devices may be used in the system 100.

[0016] Network 104 facilitates communication between various components of system 100. For example, network 104 may communicate Internet Protocol (IP) packets, Frame Relay frames, Asynchronous Transfer Mode (ATM) calls, or other suitable information between network addresses. Network 104 may include all or a portion of one or more local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), a global network such as the Internet, or any other communication system or multiple communication systems in one or more locations. In some cases, network 104 may include at least one network within a company or other organization that performs data processing operations.

[0017] The application server 106 is coupled to the network 104 and is coupled to or otherwise in communication with the database server 108. In some embodiments, the application server 106 supports a data processing capability architecture that provides one or more interfaces at the front end and execution engine selection at the back end, described in more detail below. For example, the application server 106 may execute one or more applications 112 that may enable a user to identify data and transformations to be applied to the data via at least one interface. Information associated with the identified data and the identified application is sent from the application server 106 to the database server 108 for storage in the database 110. The one or more applications 112 may also enable a user or application to initiate data processing operations that may cause the application server 106 to retrieve information from the database 110 via the database server 108. The one or more applications 112 may use the retrieved information to generate or otherwise obtain code for execution by the selected execution engine or engines. Each execution engine selected for a particular user or application may be determined based on a particular context associated with that user or application. It should be noted that database server 108 may be used within application server 106 to store information, in which case application server 106 itself may store information used to support data processing operations.

[0018] The database server 108 operates to store and facilitate retrieval of various information used, generated, or collected by the application server 106 and the user devices 102a-102d in a database 110. For example, the database server 108 may store various information regarding data transformations to be applied to data as defined by one or more users. The database server 108 may also store information regarding various execution engines, such as information specifying how a particular execution engine is selected based on different characteristics of the context for various users or applications.

[0019] Although Figure 1 illustrates an example of a system 100 supporting intelligent data processing with multiple interface front-ends and back-ends, various modifications may be made to Figure 1. For example, system 100 may include any number of user devices 102a-102d, a network 104, an application server 106, a database server 108, and a database 110. Furthermore, these components may be located in any suitable location and may be distributed over a wide area. Furthermore, while Figure 1 illustrates an example of an operating environment in which intelligent data processing with multiple interface front-ends and back-ends may be used, this functionality may be used in any other suitable system.

[0020] 2 illustrates an example device 200 supporting intelligent data processing with multiple interface front-ends and back-ends in accordance with the present disclosure. One or more instances of device 200 may be used to, for example, at least partially implement the functionality of application server 106 of FIG. 1. However, the functionality of application server 106 may be implemented in any other suitable manner. In some embodiments, device 200 illustrated in FIG. 2 may form at least a portion of user device 102a-102d, application server 106, or database server 108 of FIG. 1. However, each of these components may be implemented in any other suitable manner.

[0021] 2, device 200 represents a computing device or system that includes at least one processing device 202, at least one storage device 204, at least one communication unit 206, and at least one input / output (I / O) unit 208. Processing device 202 may execute instructions that may be loaded into memory 210. Processing device 202 may include any suitable number and type of processors or other processing devices in any suitable arrangement. Exemplary types of processing device 202 include one or more microprocessors, microcontrollers, data processing units (DPUs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or discrete circuits.

[0022] Memory 210 and persistent storage 212 are examples of storage device 204, which represent any structure capable of storing and facilitating retrieval of information (e.g., data, program code, and / or other suitable information, either temporary or persistent). Memory 210 may represent random access memory or any other suitable volatile or non-volatile storage device. Persistent storage 212 may include one or more components or devices that support longer-term storage of data, such as read-only memory, a hard drive, a flash memory, or an optical disk.

[0023] The communications unit 206 supports communication with other systems or devices. For example, the communications unit 206 may include a network interface card or a wireless transceiver to facilitate communication over a wired or wireless network, such as the network 104. The communications unit 206 may support communication over any suitable physical or wireless communication link.

[0024] I / O unit 208 allows for the input and output of data. For example, I / O unit 208 may provide a connection for user input through a keyboard, mouse, keypad, touch screen, or other suitable input device. I / O unit 208 may also send output to a display, printer, or other suitable output device. It should be noted that I / O unit 208 may be omitted if device 200 does not require local I / O, such as when device 200 represents a server or other device that can be accessed remotely.

[0025] Although Figure 2 illustrates one example of a device 200 supporting intelligent data processing with multiple interface front ends and back ends, various modifications may be made to Figure 2. For example, computing and communication devices and systems come in a wide variety of configurations, and Figure 2 does not limit the disclosure to any particular computing or communication device or system.

[0026] Figure 3 illustrates an example architecture 300 of an intelligent data processing system with a multi-interface front end and back end in accordance with the present disclosure. For ease of explanation, the architecture 300 illustrated in Figure 3 is described as being implemented in or supported by the application server 106 of the system 100 illustrated in Figure 1, where the application server 106 may be implemented using one or more instances of the device 200 illustrated in Figure 2. However, the architecture 300 illustrated in Figure 3 may be used in any other suitable system with any other suitable device.

[0027] 3, architecture 300 is generally divided into functionality associated with an authoring environment 302 and functionality associated with an execution environment 304. Authoring environment 302 generally includes functionality that allows data and data transformations (code) to be defined, and information associated with data transformations may be stored in database 110 and other suitable storage locations. Execution environment 304 generally includes functionality that allows code to be transformed (if necessary) and executed to perform data transformations upon request.

[0028] In this example embodiment, the authoring environment 302 may be used by one or more users 306 to define at least one set of data transformations to be applied to at least one input data set 308. For example, the authoring environment 302 may include one or more interfaces 310 representing one or more front ends available for use. Each interface 310 may enable at least one user 306 to load a data set 308 and define operations to be performed on the data set 308. It should be noted that a single interface 310 or multiple interfaces 310 may be provided in the architecture 300 depending on the implementation. Each interface 310 includes any suitable logic, such as an application programming interface (API), configured to receive data and information defining transformations to be applied to the data. Examples of specific types of interfaces 310 that may be used herein include a Pandas API and a Structured Query Language (SQL) API.

[0029] The tracking module 312 is used in conjunction with the interface 310 to track each sequence of transformations that have been applied to each dataset 308 by one or more users 306 using the interface 310. In other words, the tracking module 312 can identify how each user modifies the dataset 308 using the interface 310, and the tracking module 312 can store this information in the database 110. The tracking module 312 may generate information identifying the sequences of transformations and store information associated with each sequence of transformations in the database 110 in any suitable manner. For example, in some cases, the tracking module 312 may generate a Directed Acyclic Graph (DAG) that identifies each sequence of transformations that have been applied to the dataset 308.

[0030] The context module 314 is also used within the authoring environment 302 to identify information about each context in which the user 306 is performing a sequence of transformations, which information can also be stored in the database 110 or provided to the execution environment 304. Examples of the types of information that can be identified or generated by the context module 314 are described below. In this example, the tracking module 312 and the context module 314 are shown as collectively storing data 316 associated with the sequence of transformations and optionally the context as part of a "save_transformations" action. In some cases, the data 316 stored for each sequence of transformations may be associated with a user-defined identifier (ID) or other type of identification. Also, in some cases, the context module 314 may provide context information to the tracking module 312 for storage in the database 110, the tracking module 312 may provide information about the sequence of transformations to the context module 314 for storage in the database 110, or the context module 314 may provide the context information directly to the execution environment 304.

[0031] It should be noted that the user 306 herein is free to use any of the supported interfaces 310 when defining the sequence of transformations to be applied to the data set 308. In some embodiments, for example, the authoring environment 302 may include one or more predefined or pre-installed interfaces 310, and the user 306 may be limited to using those particular interfaces 310. In other embodiments, one or more interfaces 310 may be installed in the authoring environment 302 as needed or desired (with or without the one or more predefined or pre-installed interfaces 310 used in the authoring environment 302). In general, the authoring environment 302 may include any suitable number and type of interfaces 310, regardless of when and how the one or more interfaces 310 are made available to the user 306. In this manner, this approach may enable different users 306 to use various interfaces 310 as needed or desired to define the sequence of transformations and other information to be performed on the input data set 308.

[0032] Once one or more sequences of transformations are defined in the authoring environment 302 using at least one interface 310, each sequence of transformations is applied sequentially in the execution environment 304. In this example embodiment, the execution environment 304 may be used to execute the sequence of transformations to process the input data set 308 defined by the user 306 (during the first execution of each defined sequence of transformations). One or more users 318 and / or one or more applications 320 may also request that the same sequence of transformations be executed on their input data 322 (during subsequent executions of the sequence of transformations). The one or more users 318 represent users who wish to apply previously defined data transformations to the input data 322, and the one or more applications 320 represent logic (executed by one or more computing devices or other devices) that requests the application of previously defined data transformations to the input data 322. It should be noted that the one or more users 318 may or may not be the same as the one or more users 306. In this example, the application of the previously defined data transformations may be requested using a “run_transformations” request 324. In some cases, a previously defined data transformation may be requested by including in request 324 a user-defined identifier or other type of identifier associated with the previously defined data transformation.

[0033] The execution module 326 generally operates to control the execution of sequences of transformations on users 306 and their input data sets 308, and on users 318 / applications 320 and their input data 322. As each sequence of transformations is executed, the execution module 326 may receive information identifying a current context 328 associated with the data being processed. Each current context 328 is identified using the context module 314. For example, the context module 314 may be used to identify information about the context for each user 306, as described above, and the context module 314 may be used to identify information about the context in which the user 318 or application 320 is requesting the execution of a sequence of transformations for each request 324.

[0034] The context module 314 may be used to identify any suitable characteristics associated with the context in which the data transformation is taking place. For example, the context module 314 may determine values ​​of different context dimensions associated with each sequence of data transformations. In some embodiments, examples of different context dimensions that may be used by the context module 314 may include any one or any combination of the following context dimensions: The interactivity context dimension may represent an amount or level of user or application interaction during the processing of data in the sequence of transformations. The data size context dimension may approximate a size or amount of data processed during the sequence of transformations. The data type context dimension may represent a type of data processed during the sequence of transformations, such as real-time, batch, or streaming data. The data shape context dimension may represent whether the data processed during the sequence of transformations is structured or unstructured, and if unstructured, the particular type of unstructured data being processed (e.g., image data, video data, audio data, etc.). The security profile context dimension may represent a level of security required while processing data during the sequence of transformations. The resource availability context dimension may represent the amount or level of processing, memory, or other resources that may be required to process the data during the sequence of transformations. The personal identifiable information (PII) context dimension may represent whether the data processed during the sequence of transformations contains personally identifiable information, and if so, how that data needs to be treated. The retention policy context dimension may represent how long the data processed during the sequence of transformations, or the results thereof, needs to be retained.The computation comprehensiveness context dimension can represent the overall amount or level of data included in the computations in a sequence of transformations. Depending on the implementation, the contexts may be defined along with one or more of these contexts using discrete values, or values ​​that are continuous within a given value range.

[0035] For each sequence of transformations performed (either for user 306, user 318, or application 320), execution module 326 can receive information identifying a current context 328 associated with the data being processed. Execution module 326 can also retrieve information regarding the particular sequence of transformations to be performed from database 110, for example, by reading information defining the sequence of transformations associated with an identifier included in request 324 or a sequence of transformations defined by user 306. In some cases, for example, database 110 can be queried using a “retrieve_transformations” request 330, which can be used to retrieve information (e.g., a directed cyclic graph) associated with the sequence of transformations to be performed.

[0036] For each sequence of transformations to be performed, the execution module 326 can use at least a portion of the obtained information to select one of a plurality of execution engines 332 to be used to execute the sequence of transformations. The execution engines 332 represent different backends that can use different tools or techniques to execute the sequence of changes being requested. Any suitable execution engines 332 can be used here to execute the data transformations, for example backends that support various types of machine learning or artificial intelligence (ML / AI) algorithms or other data processing algorithms. In some cases, the execution module 326 can select the particular execution engine 332 to be used for each sequence of transformations to be performed based on the current context 328 of the data being processed. As a specific example, the execution module 326 can use an in-memory data structure or other mechanism that maps different combinations of values ​​of the context dimensions to different ones of the execution engines 332. Thus, the execution module 326 can take the values ​​of the current context 328 and select the execution engine 332 that is mapped to those values.

[0037] For each sequence of transformations to be performed, the execution module 326 may further generate execution-engine-specific code 334 to be executed by the selected execution engine 332. For example, the execution module 326 may traverse a directed cyclic graph associated with the sequence of transformations to be performed and modify the directed cyclic graph in an appropriate manner to enable code to be generated for the selected execution engine 332. Examples of the types of modifications that may be performed on the directed cyclic graph may include removing one or more nodes from the directed cyclic graph, replacing one or more nodes in the directed cyclic graph with one or more other nodes, and / or shuffling the position or order of one or more nodes in the directed cyclic graph. Each node in the directed cyclic graph may generally represent a data operation to be performed as part of a sequence of transformations. The specific manner in which the nodes of the directed cyclic graph are modified may vary based on a number of factors, such as the particular execution engine 332 on which the code 334 is to be executed, and in some cases, no modifications may be necessary to the directed cyclic graph. The modified or unmodified directed cyclic graph may be used by a compiler to generate code 334 for a selected execution engine 332. The compiler may also be used to generate execution engine specific artifacts based on more general artifacts received in response to the request 330. However the code 334 is generated, the code 334 can be executed by the selected execution engine 332 to perform the desired sequence of transformations.

[0038] It should be noted that although the generation of the code 334 for the execution engines 332 is described as being performed during run-time (e.g., after a user 306, 318 or application 320 requests execution of a sequence of transformations), other approaches may be used to generate the code 334. For example, the code 334 for each execution engine 332 may be generated after the data 316 associated with each sequence of transformations is retrieved and stored in the database 110. In some cases, the code 334 for each execution engine 332 may also be stored in the database 110. Once a particular execution engine 332 is identified (e.g., in response to a particular request 324), the previously generated code 334 for the appropriate sequence of transformations may be retrieved and provided (without further compilation) to that particular execution engine 332 for execution. In general, the present disclosure is not limited to any particular order of receiving a request and generating code.

[0039] As will now be appreciated, this approach allows the authoring environment 302 to be used to define any desired sequence of data transformations, where information defining the sequence of data transformations may then be stored in the database 110 or elsewhere for later use. This may be accomplished using any suitable interface 310 within the authoring environment 302. This approach also allows the execution environment 304 to be used to execute the sequence of transformations that are required, based (at least in part) on information retrieved from the database or elsewhere. In particular, the execution engine 332 for each sequence of transformations to be executed may be dynamically selected based on the associated context 328.

[0040] As such, the architecture 300 can provide various benefits or advantages depending on the implementation. The following are non-limiting examples of various types of benefits or advantages that may be obtained using the architecture 300. For example, the architecture 300 can enable code to be developed once and then reused between different use cases (different contexts). For example, a user 306 can write code to perform a sequence of data transformations using a data set 308 in the authoring environment 302, where the sequence of transformations is associated with one context and used to generate code executed by one execution engine 332. That code can later be transformed by the execution module 326 for execution by any number of other execution engines 332 and associated with different contexts. The particular execution engine 332 used for the user's 306's original use and the particular execution engine 332 selected for use by a subsequent request 324 can differ based on the particular context 328 associated with those operations. This allows the context, and therefore the execution engine 332 used to execute the same sequence of transformations, to differ based on the changing needs of the user 306, 318 or application 320 (among other things).

[0041] As another example, the architecture 300 may be used to provide flexibility when selecting between a system-driven backend and a user-driven backend. For example, in some embodiments, the execution module 326 may allow the user 306, 318 and / or application 320 to select the execution engine 332 that will be used to process their data. In particular, this may allow the user 306, 318 or application 320 to select a particular execution engine 332 to be used with particular data, which is useful when a particular execution engine 332 is known to produce superior results when performing a particular type of data processing task. However, the execution module 326 may also dynamically select the execution engine 332 to be used, for example, when the user 306, 318 or application 320 does not specifically identify the execution engine 332 or when the user 306, 318 or application 320 specifically requests dynamic selection of the execution engine 332.

[0042] As yet another example, the architecture 300 may be used to accelerate the time-to-value for executing data processing tasks. For example, data-intensive workloads often have back-end needs that change over time, typically necessitating time-consuming changes to the execution engines used for the data-intensive workloads. With the architecture 300, the complexity of authoring a sequence of data transformations may be reduced, and the same sequence of data transformations may be leveraged and used to execute code 334 on any number of execution engines 332 (including execution engines 332 selected by the user 306, 318 / application 320 or dynamically selected by the architecture 300). In some cases, the user 306 may define the transformations using a single interface 310, and the particular back-end used to execute the transformations may vary depending on the nature of the data processing task being performed (which may change dynamically). This may significantly reduce the time required to execute the data processing task.

[0043] As yet another example, the architecture 300 may be used to make server-aware or client-aware decisions when selecting which execution engine 332 to use. That is, customers (users 306, 318 and / or applications 320) may implement logic with devices acting as clients or servers depending on their particular installation. As a specific example of this, some installations support declarative programming while other installations support imperative programming. Thus, different execution engines 332 may be customized for different types of customer installations. In some embodiments, the architecture 300 uses self-declarative programming, which can leverage the benefits of both declarative and imperative programming.

[0044] It should be noted that the architecture 300 can still achieve a high level of performance when executing code with a dynamically selected execution engine 332. Notably, this is because the architecture 300 provides resource efficiency, time efficiency, and reliability to the users 306, 318 and / or applications 320 when the architecture 300 selects (and switches as needed) a different execution engine 332 that best suits the needs of the users 306, 318 and / or applications 320. For example, time efficiency can be obtained by providing an appropriate trade-off between interactive and non-interactive contexts. In some cases, for example, this may allow for faster output with lower reliability or slower output with higher reliability. Furthermore, the architecture 300 can be used to perform data transformations on a wide range of data types, including real-time and streaming data, even when minimal resources are available to process the data. Also, security-related aspects of the context for the data processing task are identified and respected within the architecture 300.

[0045] Results generated by the execution of the code 334 by the selected execution engine 332 for each sequence of transformations may be used in any suitable manner. For example, the results generated by each execution engine 332 may be provided, via a suitable interface 310 or other mechanism, to the user 306, 318 or application 320 that initiated the execution of the sequence of transformations. Because both the data processing tasks and the data being processed by the data processing tasks can vary widely, the results generated by the execution engines 332 may be used for any suitable purpose.

[0046] It should be noted that the functionality illustrated or described with respect to Figure 3 may be implemented in an electronic device, such as a computing device, in any suitable manner. For example, in some embodiments, at least a portion of the functionality illustrated or described with respect to Figure 3 may be implemented or supported using one or more software applications or other software instructions executed by one or more processing devices of application server 106, device 200, or other devices. In other embodiments, at least a portion of the functionality illustrated or described with respect to Figure 3 may be implemented or supported using dedicated hardware components. In general, the functionality illustrated or described with respect to Figure 3 may be performed using any suitable hardware or any suitable combination of hardware and software / firmware instructions.

[0047] Although Figure 3 illustrates an example of an intelligent data processing system architecture 300 with a multi-interface front end and back end, various changes may be made to Figure 3. For example, functions and components may be added, deleted, combined, subdivided, duplicated, or arranged in any other suitable configuration in architecture 300 according to particular needs. As a specific example, architecture 300 may include any suitable number of interfaces 310 and any suitable number of execution engines 332.

[0048] Figure 4 illustrates an example method 400 for intelligent data processing with a multi-interface front-end and back-end in accordance with the present disclosure. For ease of explanation, the method 400 illustrated in Figure 4 is described as being implemented in or supported by the application server 106 in the system 100 illustrated in Figure 1, which may be implemented using one or more instances of the device 200 illustrated in Figure 2. However, the method 400 illustrated in Figure 4 may be performed using any other suitable device and in any other suitable system.

[0049] As shown in Fig. 4, in step 401, an interface is provided to enable a user to load an input data set and define a sequence of transformations to be applied to the input data set. This may include, for example, the processing device 202 of the application server 106 presenting at least one interface 310 to the user 306. The interface 310 may enable the user 306 to load a data set 308 and identify a sequence of transformations to be performed in relation to the data set 308. A context associated with the input data set is identified in step 404. This may include, for example, the processing device 202 of the application server 106 using a context module 314 to determine one or more values ​​of one or more context dimensions.

[0050] Information defining the transformations in the sequence of transformations is obtained at step 406 and used to generate nodes in the graph at step 408. This may include, for example, the processing device 202 of the application server 106 using the tracking module 312 to obtain information associated with the transformations defined by the user 306 using the interface 310. This may also include the processing device 202 of the application server 106 generating nodes in the directed cyclic graph that identify the transformations and optionally signing the nodes at step 410. As long as the user continues to define transformations, the process may repeat steps 406-408 to gather information about the transformations and generate nodes in the graph. Once the terminal transformations are provided by the user, the completed graph is stored in storage at step 412. This may include, for example, the processing device 202 of the application server 106 storing the directed cyclic graph in the database 110. A directed cyclic graph can represent a sequence of backend-agnostic transformations, or the directed cyclic graph may be used to generate a sequence of backend-agnostic transformations.

[0051] An execution engine to be used to execute the sequence of transformations is identified based on the identified context at step 414. This may include, for example, the processing device 202 of the application server 106 using the execution module 326 to determine how to map values ​​of the context dimensions associated with the input data set 308 to a particular one of the execution engines 332. At step 416, code is generated to be executed by the selected execution engine. This may include, for example, the processing device 202 of the application server 106 using the execution module 326 to modify the directed cyclic graph (if necessary) based on the selected execution engine 332. This may also include a compiler compiling the modified or unmodified directed cyclic graph or other information to generate execution engine specific code 334. At step 418, the code is executed with the selected execution engine to execute the sequence of transformations on the data. This may include, for example, the processing device 202 of the application server 106 using the execution engine 332 to execute the execution engine specific code 334 to execute the sequence of transformations on the input data set 308. The results of the code execution are provided to at least one appropriate destination, step 420. This may include, for example, the selected execution engine 332 providing the data processing results to the user 306 via the interface 310.

[0052] At this point, it is possible that the same sequence of transformations may be requested by the same user 306, another user 318, or an application 320. As a result, in step 422, it is determined whether a request has been received for a sequence of transformations. This may include, for example, the processing device 202 of the application server 106 determining whether a request 323 has been received from a user 318 or application 320 identifying a sequence of transformations. In some cases, the request 324 may identify the sequence of transformations to be applied and the data 322 to which the sequence of transformations should be applied. The process may then perform steps 414-420 again, but this time a current context 328 is determined for the new request 324 and used to select an appropriate execution engine 332 (which may or may not be the same execution engine 332 originally used for the user 306). Note that steps 414-420 here may be repeated any number of times based on any number of requests 324, with each request 324 being associated with its own unique context 328.

[0053] While Figure 4 illustrates one example of a method 400 for intelligent data processing with a multi-interface front end and back end, various modifications may be made to Figure 4. For example, although shown as a series of steps, the various steps in Figure 4 may overlap, occur simultaneously, occur in a different order, or occur any number of times. As a specific example, as discussed above, execution engine specific code 334 may be generated for one or more of execution engines 332 prior to receipt of request 324.

[0054] FIG 5 illustrates an example process 500 for generating execution-engine specific code for execution by a selected backend according to the present disclosure. The process 500 illustrated in FIG 5 can represent, for example, a process for collecting user input via the interface 310, where the user input defines a sequence of data transformations to be performed. The process 500 illustrated in FIG 5 also represents how the tracking module 312 can generate information regarding the sequence of data transformations and how the execution module 326 can use this information to generate execution-engine specific code 334 suitable for execution by a selected execution engine 332.

[0055] 5, one or more data sources 502 represent one or more interfaces (e.g., one or more interfaces 310) that may be used to obtain data from one or more users 306. In some embodiments, each data source 502 may be associated with a different programming language and a different data storage mechanism. Depending on the implementation, there may be a single data source 502 or multiple data sources 502 provided for use herein.

[0056] Each user 306 can use one or more tools 504 to write code that defines sequences of data transformations in one or more specific programming languages. In some embodiments, the tools 504 accessed and used by each user 306 can vary depending on which data source 502 is used by the user 306, such that different data sources 502 can be associated with different tools 504. In some cases, for example, a tool 504 can represent a JUPYTER LAB tool that uses the Pandas programming language. Of course, different tools 504 can be provided to support different programming languages, if desired.

[0057] Each tool 504 here can be used to define a sequence of operations to be performed, where the sequence of operations represents a sequence of data transformations defined by the user 306. Each operation in the sequence can be represented by a cell 506, where each cell 506 can identify at least one operation defined by the user 306. In this example, the first cell 506 represents a request to load one or more source data files. The second cell 506 represents at least one request to perform one or more data integration operations, which may include any suitable operations to prepare source data for processing. The third cell 506 represents at least one request related to feature engineering, which may include any suitable operations required to identify features of the source data that should be further processed. The fourth cell 506 represents at least one request related to feeding the processed data (e.g., identified features) into a machine learning pipeline or other data processing architecture. The fifth cell 506 represents a request to update a package that encapsulates the sequence of operations. Of course, the operations represented by a cell 506 will vary based on the particular sequence of data transformations defined by a user 306. In some cases, a cell 506 may be digitally signed (e.g., by using a user's digital key) to associate a particular user 306 with a particular cell 506 and / or to prevent unauthorized modification of the cell.

[0058] Each user-defined sequence of operations generated by the user 306 using the tool 504 is converted into system-generated code 508, which (if executed) may be used to execute the user-defined sequence of operations. In some cases, the system-generated code 508 may be defined using a directed cyclic graph, where the directed cyclic graph includes a number of nodes 510. Each node 510 in the directed cyclic graph represents one or more operations to be executed, and each node 510 may correspond to one of the cells 506 generated using the tool 504. For example, each node 510 may include one or more data specifications ("data specs") that identify the particular operations to be executed in the corresponding cell 506. The information defined in the nodes 510 may identify the operations that correspond to each line of code generated by the user 306 using the tool 504. Thus, the nodes 510 of the directed cyclic graph may represent the logic required to implement the user-defined sequence of operations. However, the directed cyclic graph may be generic in that it is not tied to any particular execution engine 332. As can be seen, the nodes 510 form a directed cyclic graph because the nodes 510 are sequentially attached in a particular order defined by the arrows to / from / between the nodes 510. In some cases, the directed cyclic graph is at least one of the items that the tracking module 312 may generate and store in the database 110 when the user 306 defines a sequence of transformations using the interface 310.

[0059] After the user 306 issues a terminal transformation (i.e., the final transformation defined by the user 306), the directed cyclic graph may be stored in the database 110. A particular execution engine 332 may also be selected as described above based on a current context associated with the user 306. To execute the sequence of transformations defined by the directed cyclic graph on the selected execution engine 332, the directed cyclic graph may be translated into code 512 suitable for execution by the selected execution engine 332. Because the directed cyclic graph generically defines the operations to be performed, the directed cyclic graph may be easily translated into code 512 that is specific to the execution engine on which the code 512 is to be executed. The code 512 may then be compiled or otherwise prepared and sent to the selected execution engine 332 for execution. If the same sequence of transformations is requested again (e.g., via request 324), but a different execution engine 332 is selected, the directed cyclic graph may be retrieved again and used to generate code 512 for execution by the selected execution engine 332. It should be noted that after code 512 is generated for a particular execution engine 332, the code 512 may be stored (e.g., in database 110) such that the code 512 may be executed again later if required, without delay.

[0060] It should be noted that the particular operations performed within the code 512 will typically vary depending on (among other things) the execution engine 332 used to execute the code 512. Because the directed cyclic graph contains a data spec or other information that generally defines the operations to be performed as part of a sequence of user-defined operations, knowledge of the particular execution engine 332 can be used to generate logic that enables the execution module 326 to translate the data spec into code specific to the execution engine. By defining the appropriate logic for all available execution engines 332, the execution module 326 can translate the directed cyclic graph into the appropriate code 512 each time execution of a sequence of user-defined operations is requested.

[0061] Although Figure 5 illustrates one example of a process 500 for generating execution engine specific code for execution by a selected backend, various modifications may be made to Figure 5. For example, the user-defined code may include any suitable number of operations, which may be translated into any suitable operations to be executed by the selected execution engine 332. Also, the specific code 512 illustrated in Figure 5 is related to a particular execution engine 332, and any other suitable code 512 may be generated depending on the execution engine 332.

[0062] The following describes an exemplary embodiment of the present disclosure implementing an intelligent data processing system with a multi-interface front end and back end, although other embodiments may be used in accordance with the teachings of the present disclosure.

[0063] In a first embodiment, a method includes identifying, via a user interface, a sequence of transformations to be performed on an input data set. The method also includes identifying a first context associated with the input data set. The method further includes selecting a first execution engine of a plurality of execution engines to be used to perform the sequence of transformations on the input data set based on the first context. The method also includes providing first code implementing the sequence of transformations to the first execution engine, and executing the first code with the first execution engine to perform the sequence of transformations on the input data set.

[0064] In a second embodiment, an apparatus includes at least one processing device configured to identify, via a user interface, a sequence of transformations to be performed on an input data set. The at least one processing device is further configured to identify a first context associated with the input data set. The at least one processing device is further configured to select a first execution engine of a plurality of execution engines to be used to perform the sequence of transformations on the input data set based on the first context. The at least one processing device is further configured to execute, with the first execution engine, first code implementing the sequence of transformations to perform the sequence of transformations on the input data set.

[0065] In a third embodiment, a non-transitory computer readable medium stores computer readable program code that, when executed, causes one or more processors to identify, via a user interface, a sequence of transformations to be performed on an input dataset. The medium also stores computer readable program code that, when executed, causes the one or more processors to identify a first context associated with the input dataset. The medium further stores computer readable program code that, when executed, causes the one or more processors to select, based on the first context, a first execution engine of a plurality of execution engines to be used to perform the sequence of transformations on the input dataset. The medium further stores computer readable program code that, when executed, causes the one or more processors to execute, with the first execution engine, first code implementing the sequence of transformations to perform the sequence of transformations on the input dataset.

[0066] Any one of the following features or any suitable combination may be used in the first, second or third embodiment: The user interface may include one of a plurality of user interfaces, each user interface may be configured to enable one or more users to define a sequence of transformations to be performed on the input data set. A graph associated with the sequence of transformations may be stored, and the first code may be generated based on the graph. The first context may be based on at least one of an interactivity context dimension representing an amount or level of user or application interaction during the sequence of transformations, a data size context dimension representing a size or amount of data processed during the sequence of transformations, a data type context dimension representing a type of data processed during the sequence of transformations, a data shape context dimension representing whether the data processed during the sequence of transformations is structured or unstructured, a security profile context dimension representing a level of security required while processing data during the sequence of transformations, a resource availability context dimension representing an amount or level of resources required to process data during the sequence of transformations, a personally identifiable information (PII) context dimension representing whether the data processed during the sequence of transformations contains personally identifiable information, a retention policy context dimension representing how long the data processed during the sequence of transformations or its results should be retained, and a computation comprehensiveness context dimension representing an overall amount or level of data included in computations during the sequence of transformations.A request to perform a sequence of transformations on a specified data may be received, a second context associated with the request may be identified (wherein the second context may be different from the first context), a second one of the execution engines used to perform the sequence of transformations on the specified data may be selected based on the second context (wherein the second execution engine may be different from the first execution engine), and a second code implementing the sequence of transformations may be executed using the second execution engine to perform the sequence of transformations on the specified data. Information identifying the sequence of transformations to be performed on the input data set may be stored in the data storage together with an identifier, and the request may include the identifier. The sequence of transformations may be identified by identifying, via a user interface, a number of transformations to be performed on the input data set, and for each transformation, generating a node in a directed acyclic graph that defines the transformation and signing the node.

[0067] In some embodiments, various functions described herein are implemented or supported by a computer program formed from computer readable program code and embodied in a computer readable medium. The phrase "computer readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer readable medium" includes any type of medium that can be accessed by a computer, such as read only memory (ROM), random access memory (RAM), hard disk drive (HDD), compact disk (CD), digital video disk (DVD), or any other type of memory. "Non-transitory" computer readable medium excludes wired, wireless, optical, or other communication links that carry transient electrical or other signals. Non-transitory computer readable medium includes media on which data can be stored permanently, and media on which data can be stored and later overwritten, such as rewritable optical disks or erasable storage devices.

[0068] It may be advantageous to explain definitions of certain words or phrases used throughout this specification. The terms "application" and "program" refer to one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, associated data, or portions thereof that are adapted to be implemented in suitable computer code (including source code, object code, or executable code). The term "communicate" and its derivatives encompass both direct and indirect communication. The terms "including" and "having" and their derivatives are intended to include without limitation. The terms "or", "or" and "alternative" are inclusive, meaning "and / or". The phrase "related to" and its derivatives may mean "including," "contained in," "interconnected with," "containing," "contained in," "connected to," "coupled with," "communicated with," "cooperate with," "interleaved," "juxtaposed," "close to," "associated with," "having," "having the quality of," "related to," and the like. The phrase "at least one of," when used in conjunction with a list of items, means that different combinations of one or more of the items in the list may be used and that only one item in the list may be required. For example, "at least one of A, B, and C" includes any of the following combinations: A, B, C, A and B, A and C, B and C, and A, B, and C.

[0069] The description in this application should not be read as implying that any particular element, step, or function is essential or critical to inclusion in the scope of the claims. The scope of patentable subject matter is defined solely by the scope of the allowed claims. Moreover, none of the claims invoke 35 U.S.C. 112(f) with respect to any of the appended claims or claims, unless the precise terms "means for" or "step for" are expressly used in a particular claim following a participial phrase identifying the function. The use of terms such as, but not limited to, "mechanism," "module," "device," "unit," "component," "element," "member," "apparatus," "machine," "system," "processor," or "component" in the claims is understood and intended to refer to structures known to those skilled in the relevant art, as further modified or expanded by the features of the claims themselves, and is not intended to invoke 35 U.S.C. 112(f).

[0070] While the present disclosure has been described with respect to specific embodiments and generally related aspects, alterations and permutations of these embodiments and methods will be apparent to those skilled in the art. Thus, the above description of exemplary embodiments does not define or constrain the present disclosure. Other modifications, substitutions, and alterations are possible without departing from the spirit and scope of the present disclosure, as defined by the following claims.

Claims

1. Identifying, via a user interface, a sequence of transformations to be performed on the input data set; identifying a first context associated with the input data set; selecting a first execution engine of a plurality of execution engines to be used to execute the sequence of transformations on the input data set based on the first context; providing a first code implementing the sequence of transformations to the first execution engine; executing the first code using the first execution engine to perform the sequence of transformations on the input data set; The method according to claim 1,

2. The user interface comprises one of a plurality of user interfaces; each user interface is configured to enable one or more users to define a sequence of transformations to be performed on an input data set; The method of claim 1.

3. storing a graph associated with said sequence of transformations; generating the first code based on the graph; The method of claim 1 further comprising:

4. The first context is interactivity context dimensions that represent the amount or level of user or application interaction during said sequence of transformations; a data size context dimension representing the size or amount of data being processed during said sequence of transformations; a data type context dimension representing the type of data being processed during said sequence of transformations; a data shape context dimension that represents whether the data being processed during said sequence of transformations is structured or unstructured; a security profile context dimension that represents the level of security required while processing data during said sequence of transformations; a resource availability context dimension representing the amount or level of resources required to process data during said sequence of transformations; a personally identifiable information (PII) context dimension that indicates whether the data processed during the sequence of transformations contains personally identifiable information; A retention policy context dimension that indicates how long data processed during said sequence of transformations or their results should be retained; a computation comprehensiveness context dimension that represents the overall amount or level of data included in the computations in the sequence of transformations; Based on at least one of the following: The method of claim 1.

5. receiving a request to perform said sequence of transformations on specified data; identifying a second context associated with the request, the second context being different from the first context; selecting a second one of the execution engines to be used to perform the sequence of transformations on the specified data based on the second context, the second execution engine being different from the first execution engine; providing second code implementing the sequence of transformations to the second execution engine; executing the second code with the second execution engine to perform the sequence of transformations on the specified data; The method of claim 1 further comprising:

6. storing information identifying the sequence of transformations performed on the input data set together with an identifier in a data storage; The request includes the identifier. The method according to claim 5.

7. Identifying the sequence of transformations includes: Identifying, via the user interface, a number of transformations to be performed on the input data set; for each transformation, creating a node in a directed acyclic graph that defines that transformation and signing that node; having The method of claim 1.

8. 8. Apparatus comprising at least one processing device configured to perform the method according to any one of claims 1 to 7.

9. A non-transitory computer readable medium storing computer readable program code which, when executed, causes one or more processors to perform the method of any one of claims 1 to 7.

Images