Automatic two-way generation and synchronization of notebook and pipeline

Abstract

To provide systems and methods for automatic two-way generation and synchronization of notebook and pipeline.SOLUTION: A method includes: identifying a difference between pipeline and notebook representations of application logic; synchronizing the pipeline representation and notebook representation by, when the difference is a notebook paragraph without pipeline node, automatically generating a new pipeline node in the pipeline representation, and when the difference is a pipeline node that does not have a corresponding notebook paragraph, automatically generating a new notebook paragraph in the notebook representation; and updating either a pipeline user interface to show the new pipeline node or a notebook user interface to show the new notebook paragraph.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This disclosure claims the benefit of commonly assigned Indian Provisional Patent Application Serial No. 202121053246, filed November 19, 2021 by inventors Rajaram N. Vadapandeshwara, Tara Kant, and Farsana K, entitled "Automatic Two-Way Generation and Synchronization of Notebook and Pipeline," which is incorporated herein by reference in its entirety. [Background technology]

[0002] background Notebook interfaces or computational notebooks, such as Oracle® Data Studio, Jupyter, Zeppelin, Google® Notebook Service, and SageMaker Notebook, enable data scientists and modelers to script statistical and machine learning (ML) models that are useful for predictive use cases. Notebook interfaces are also increasingly being used to script deterministic computations. Summary of the Invention [Problem to be solved by the invention]

[0003] With the proliferation of notebooks, business domain users are becoming disconnected from the underlying business logic scripted within the notebook. Using notebooks for development results in a loss of auditability, explainability, and ease of regulatory audit. This is why statistical and ML analysis, while becoming mainstream, remains within the realm of regulatory approval.

[0004] BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are incorporated herein by reference and constitute a part of this specification, illustrate various systems, methods, and other embodiments of the present disclosure. It will be appreciated that element boundaries (e.g., boxes, groups of boxes, or other shapes) shown in the figures represent one embodiment of a boundary. In some embodiments, one element may be realized as multiple elements, or multiple elements may be realized as one element. In some embodiments, an element shown as an internal component of another element may be realized as an external component, and vice versa. Additionally, some elements may not be drawn to scale. [Brief explanation of the drawings]

[0005] [Figure 1] FIG. 1 illustrates one embodiment of a system related to automatic bidirectional creation and synchronization of notebooks and pipelines. [Figure 2] FIG. 1 is a conceptual diagram illustrating one embodiment of a notebook interface and a pipeline interface for application logic related to automatic bidirectional creation and synchronization of notebooks and pipelines. [Figure 3] FIG. 1 illustrates one embodiment of system components for automatic bidirectional creation and synchronization of notebooks and pipelines. [Figure 4] FIG. 1 illustrates one embodiment of a method related to automatic bidirectional creation and synchronization of notebooks and pipelines. [Figure 5] FIG. 1 illustrates another embodiment of a method related to automatic bidirectional creation and synchronization of notebooks and pipelines. [Figure 6] FIG. 1 illustrates one embodiment of a method for linking new pipeline nodes together in relation to automatic two-way creation and synchronization of notebooks and pipelines. [Figure 7]FIG. 1 illustrates one embodiment of a pipeline graphical user interface associated with automatic bidirectional creation and synchronization of notebooks and pipelines. [Figure 8] FIG. 1 illustrates one embodiment of a computing system configured with the disclosed exemplary systems and / or methods. DETAILED DESCRIPTION OF THE INVENTION

[0006] Detailed Description The challenges associated with the use of computer modeling (both statistical and ML) are numerous. Regulators do not readily accept results, decisions, numbers, or filings derived using modeling (including ML modeling) or data science algorithms. This is due, at least in part, to the lack of transparency of notebooks to business / domain users. Business / domain users typically have no way to validate, verify, and own the application logic, including the model or data science drivers. The decoupling of the business / domain user's preferred interface (i.e., pipeline) from the data scientist user's preferred interface (i.e., notebook) effectively precludes a collaborative "composition" approach between business / domain users and data scientists / modelers to augment rule-based computational solutions with model-based logic. Compatible paradigms supporting both visual design processes (as in a pipeline interface) and scripting processes (as in a notebook interface) do not exist.

[0007] Systems, methods, and other embodiments for automatic, bidirectional generation and synchronization of notebooks and pipelines described herein overcome these and other challenges. Described herein are systems, methods, and other embodiments that provide automatic generation and synchronization of notebooks with pipelines and automatic generation and synchronization of pipelines with notebooks. In particular, the systems, methods, and other embodiments enable (i) automatic generation or modification of business flow pipelines in response to user input changes to notebook code, and (ii) automatic generation or modification of notebook code in response to user input changes to the pipeline. Statistical and ML operations developed in notebooks can be presented to business domain users as objects in a pipeline view that is easily understood by business users, and changes to the pipeline result in automatic extension (code generation) of the notebook code along with user edits to the pipeline. The systems, methods, and other embodiments also enable business users and domain specialists to declaratively design business logic, with notebook code automatically generated behind the scenes. This enables data scientists to work collaboratively with domain analysts to thoroughly fine-tune the business logic / process flow. Any changes made by a business / domain analyst in the pipeline view or a modeler / data scientist in the notebook view are immediately visible to others using the other views. Thus, the pipeline flow and the notebook do not get out of sync with each other. Thus, in one embodiment, the automatic bidirectional creation and synchronization of notebooks and pipelines as described herein provides a low-code / no-code data science platform.

[0008] Initial testing of one embodiment of automatic bidirectional notebook and pipeline generation and synchronization has proven the generation and synchronization capabilities to be highly effective. The test system was able to convert 10,000 lines of notebook code into a pipeline flow, with every paragraph mapped as a named object to a corresponding task in the pipeline, and the notebook paragraph code mapped as embedded business logic in the tasks, callable as REST APIs. This transformed what was once a large amount of code in the notebook that was difficult to decipher into a visually auditable pipeline, making the end-to-end process debuggable, bidirectionally observable, and accountable for regulatory audits.

[0009] None of the acts or functions described or claimed herein are performed by the mind of a human being, and any interpretation that any act or function can be performed by the mind of a human being is inconsistent with and contrary to this disclosure.

[0010] -Example environment- 1 illustrates one embodiment of a system 100 relating to the automatic, bidirectional creation and synchronization of notebooks and pipelines. In one embodiment, system 100 includes a cloud infrastructure system 105, such as Oracle® Cloud Infrastructure, connected to an enterprise network 115 by the Internet 110 (or another suitable communications network or combination of networks).

[0011] In one embodiment, cloud infrastructure system 105 includes various systems and components, including data integration and modeling services 120 , other system components 125 , data stores 130 , and web interface servers 135 .

[0012] In one embodiment, data integration and modeling services 120 may include components of Oracle Financial Services' Model Management and Governance (MMG) tool for developing, deploying, and managing statistical, machine learning, computational, and simulation models. In one embodiment, other system components 125 may include cloud services that may be combined by data integration and modeling services 120 to build and run specialized computing applications, such as statistical, machine learning, computational, and simulation models. In one embodiment, other system components 125 may further include a user management module for managing user access to cloud infrastructure system 105.

[0013] Each of the components of cloud infrastructure system 105 is configured with logic for performing the functions that each component is described as performing. In one embodiment, the components of cloud infrastructure system 105 may be implemented as a set of one or more software modules executed by one or more computing devices specially configured for such execution. In one embodiment, the components of cloud infrastructure system 105 are implemented on one or more hardware computing devices or hosts interconnected by a data network. For example, the components of cloud infrastructure system 105 may be executed by networked computing devices in one or more computational hardware forms, such as a central processing unit (CPU) or general-purpose form, a dense input / output (I / O) form, a graphics processing unit (GPU) form, and a high-performance computing (HPC) form. In one embodiment, the components of cloud infrastructure system 105 are implemented by dedicated computing devices. In one embodiment, the components of cloud infrastructure system 105, although depicted as separate units in FIG. 1 , are implemented by a common (or shared) computing device. In one embodiment, the cloud infrastructure system 105 may be hosted by a dedicated third party, for example, in an infrastructure-as-a-service (IAAS), platform-as-a-service (PAAS), or software-as-a-service (SAAS) architecture.

[0014] In one embodiment, components of system 100 communicate with each other through electronic messages or signals. These electronic messages or signals may be configured as calls to functions or procedures, such as application programming interface (API) calls, to access features or data of the components. In one embodiment, these electronic messages or signals are transmitted between hosts in a format compatible with Transmission Control Protocol / Internet Protocol (TCP / IP) or other computer networking protocols. Each component of system 100 may (i) generate or configure electronic messages or signals to issue commands or requests to another component, (ii) transmit such messages or signals to other components of computing system 100, (iii) analyze the content of received electronic messages or signals to identify commands or requests that the component can execute, and (iv) automatically execute or implement such commands or requests in response to identifying the commands or requests. Electronic messages or signals may include queries against a database. The queries may be configured and executed in a query language compatible with the database and may be executed in a runtime environment compatible with the query language.

[0015] In one embodiment, a remote computing system (such as a remote computing system in the enterprise network 115) may access information or applications provided by the cloud infrastructure system 105 via the web interface server 135. In one embodiment, the remote computing system may send requests to and receive responses from the web interface server 135. In one example, access to the information or applications may be achieved by using a web browser on a personal computer 145, a remote user computer 155, or a mobile device 160. For example, these computing devices 145, 155, 160 in the enterprise network 115 may access a notebook graphical user interface (GUI) (also referred to as a data studio) or a pipeline GUI (also referred to as a canvas) for developing application logic. In one example, communications may be exchanged between the web interface server 135 and a personal computer 145, a server 150, a remote user computer 155, or a mobile device 160 and may take the form of, for example, a remote representational state transfer (REST) ​​request using JavaScript object notation (JSON) as the data exchange format, or a simple object access protocol (SOAP) request to / from an XML server. The REST or SOAP requests may include API calls to components of the cloud infrastructure system 105. For example, a computer 145, 150, 155 in the enterprise network 115 may request the creation or deletion of a notebook paragraph via a notebook interface, or the creation or deletion of a pipeline node via a pipeline interface.

[0016] An enterprise network 115 may be associated with a business. For simplicity and clarity of explanation, the enterprise network 115 is represented by an on-site local area network 140 to which one or more personal computers 145 or servers 150 are operatively connected, along with one or more remote user computers 155 or mobile devices 160 connected to the enterprise network 115 via network 110. Each personal computer 145, remote user computer 155, or mobile device 160 is generally, but not necessarily, dedicated to a particular end user, such as an employee or contractor associated with the business. The personal computers 145 and remote user computers 155 may be, for example, desktop computers, laptop computers, tablet computers, or other devices capable of connecting to the local area network 140 or the Internet 110. The mobile devices 160 may be, for example, smartphones, tablet computers, mobile phones, or other devices capable of connecting to the local area network 140 or network 110 via a wireless network, such as a cellular network or Wi-Fi. Users of enterprise network 115 interface with cloud infrastructure system 105 through network 110 .

[0017] In one embodiment, data store 130 is a computing stack for structuring and storing and retrieving one or more collections of information or data in a non-transitory computer-readable medium, e.g., as one or more data structures. In one embodiment, data store 130 includes one or more databases configured to store and provide information used by cloud infrastructure system 105. In one embodiment, data store 130 includes one or more notebook databases configured to store and provide computational notebooks, e.g., as Jupyter-formatted data structures. In one embodiment, data store 130 includes one or more pipeline databases configured to store and provide information defining the sequencing and execution of discrete tasks, e.g., as a graph data structure of metadata describing tasks as nodes and the order as links in the graph. In one embodiment, data store 130 includes one or more Oracle® databases configured to store and provide notebook and pipeline data structures. In some example configurations, data store 130 may be implemented using one or more Oracle® Exadata compute, network-attached storage (NAS) devices, and / or other dedicated server devices.

[0018] In one embodiment, the data integration and modeling service is a model management and orchestration application offered by Oracle Financial Services. In one embodiment, the data integration and modeling service 120 includes one or more components configured to implement the methods, functions, and other embodiments described herein related to automatic bidirectional generation and synchronization of notebooks and pipelines. In one embodiment, the data integration and modeling service 120 is configured with logic (such as the automatic bidirectional notebook and pipeline generation and synchronization logic 830 shown and described with reference to FIG. 8 ) to implement the methods, functions, and other embodiments described herein. For example, the data integration and modeling service 120 may include a pipeline interface subsystem 165, a notebook interface subsystem 175, and a pipeline and notebook synchronizer subsystem 180. In one embodiment, the pipeline interface subsystem 165 maintains one or more pipelines representing application logic as a set of task nodes linked in order of execution, executes the pipelines (or pipeline segments), and presents the pipelines for user review and editing through a canvas-style graphical user interface, as shown and described in further detail herein. In one embodiment, the notebook interface subsystem maintains one or more notebooks representing application logic as a set of notebook paragraphs containing executable code or scripts, executes the notebooks (or individual notebook paragraphs), and presents the notebooks for user review and editing through a notebook-style graphical user interface.In one embodiment, the pipeline notebook synchronizer subsystem 180 causes changes made to the notebook representation of the application logic to be represented in the pipeline representation of the application logic, and causes changes made to the pipeline representation of the application logic to be represented in the notebook representation of the application logic, ensuring automatic, real-time, bidirectional synchronization between the notebook representation and the pipeline representation.

[0019] Notebook and pipeline interfaces to application logic 2 shows a conceptual diagram 200 of one embodiment of a notebook interface and a pipeline interface for application logic related to the automatic bidirectional creation and synchronization of notebooks and pipelines. In one embodiment, there are two interfaces for viewing and editing application logic available to an analyst (or user) 205, depending on the analyst's 205 preferences.

[0020] One interface, pipeline interface 210 (also referred to as a canvas), is targeted to analysts 205, who are business or domain analysts (i.e., specialists in a business area), and provides a visual pipeline representation of the underlying application logic 215. In one embodiment, the pipeline representation shows the application as a directed graph, with nodes representing fine-grained tasks or functions connected by links indicating the inputs and outputs of the task nodes. This visual pipeline representation may hide many of the operational details of the tasks to provide a visually organized representation of the application logic, while allowing the underlying operational details to be viewed and edited by selecting nodes and links. Thus, pipeline interface 210 provides a low-code (or no-code) data science platform. Pipeline interface 210 is presented primarily for non-technical analysts, allowing analysts to see each object and visually confirm that it is for a particular type of data process, such as data ingestion, data profiling, etc.

[0021] Another interface, notebook interface 220 (also referred to as Data Studio or rule or model authoring interface), is targeted to the data scientist or modeler analyst 205 and provides a computational notebook representation of the underlying application logic 215. In one embodiment, the notebook representation shows the application as a series of paragraphs 221, also referred to as cells, in a draft notebook 222. The paragraphs 221 represent fine-grained tasks or functions written in computer-executable code or script. The notebook representation displays the executable code for the tasks (also referred to herein as function scripts) to provide a complete representation of the application logic and allow for direct editing. In one embodiment, in addition to executable code paragraphs, such as results, formatting, graphics, or non-executable text paragraphs, which provide an interactive environment that presents code, results, visualization, and documentation in an integrated document, additional types of paragraphs may also be included in the notebook.

[0022] In one embodiment, notebook interface 220 includes a kernel for executing the code in a paragraph. The code in a notebook paragraph can be written in a variety of interpreted languages. A notebook can be multilingual, containing paragraphs written in different languages. The kernel includes interpreters for these languages. For example, these languages ​​may include Java, Scala, R, Python, PySpark, JavaScript, structured query language (SQL), and property graph query language (PGQL). The kernel may also include interpreters for application-specific content, such as an interpreter for Oracle Financial Services Analytical Applications (OFSAA) commands. In this paragraph, the code begins with a call to a magic function (a predefined function in the kernel that executes the supplied command) that selects an interpreter for the code. For example, the "%ofsaa" magic at the beginning of a paragraph indicates that the following content of the paragraph should be interpreted by the OFSAA interpreter, the "%pgq1" magic indicates that the following content of the paragraph should be interpreted by the PGQL interpreter, and the "%Python" magic indicates that the content should be interpreted by the Python interpreter. In response to these magic commands selecting an interpreter, the notebook kernel also provides an environment for executing the paragraph content.For example, for a Python script, the kernel will create a Python session to run the script, for a PGQL query, the kernel will request that the code be executed in a property graph server (PGX) session, and for an SQL query, the kernel will request a Java database connectivity (JDBC) connection to the SQL server to execute the command.

[0023] Thus, in one embodiment, notebook interface 220 supports scripting for advanced users (e.g., data scientists), while pipeline interface 210 presents a de-generated visual pipeline that represents code (or script) as a series of fine-grained tasks. The systems, methods, and other embodiments described herein for automatic, bidirectional notebook and pipeline generation and synchronization can automatically generate code (e.g., notebook paragraphs) from pipelines, automatically generate pipelines from code, and keep both the pipeline in the canvas and the model code in the notebook synchronized. Using the systems, methods, and other embodiments described herein for automatic, bidirectional notebook and pipeline generation and synchronization enables the two interfaces 210, 220 to be used collaboratively in real time to develop application logic 215. In one embodiment, the features of the pipeline interface are implemented by pipeline interface subsystem 165, and the features of the notebook interface 220 are implemented by notebook interface subsystem 175.

[0024] In one embodiment, the modeling metastore and registration repository 225 is a data structure, such as a graph database, that contains information defining a pipeline. The repository 215 may include information that defines the connections (or links or edges) between nodes in the pipeline, such as edge and node catalog tables. The repository 215 may include information that describes the content of the pipeline nodes, such as the node's task type and links (e.g., API calls) to notebook paragraphs that define the node's functionality. In one embodiment, the modeling metastore and registration repository 225 is a database data structure within the data store 130.

[0025] In one embodiment, a notebook is a draft that can be continually revised. The current draft 222 of a notebook can be stored as a fixed, static, or read-only copy 230 of the notebook by executing the "Publish" function of the notebook interface 220. Subsequent changes made to the draft notebook 222 do not modify the static copy 230.

[0026] In one embodiment, application logic 215 is stored as one or more data structures representing nodes and links of notebook pipelines or paragraphs in repository 225. Application logic 215 defines inputs and outputs of tasks (e.g., models and other functions) based on pipeline links and nodes using application programming interface (API) calls (e.g., REST API calls) to paragraphs that describe the functionality of the pipeline nodes defined in either draft notebook 222 or static copy 230. Analyst 205 may select draft version 222 or static version 230 before execution. Application logic 215 is executed 235 or performed by a computing system. Application logic 215 may be executed in various contexts, such as ML model training, where an ML model is included in application logic 215, testing and experimentation to evaluate the performance of application logic 215, or production, where application logic 215 operates on production data. The results 240 of the execution of the application logic 215 may be written to storage 240, for example as a data structure in data store 130, may be output or sent to another application 245 that uses the output results, or may be presented to an analyst 205 in a graphical user interface (GUI) 250 for making decisions and visualizing the output results. In one embodiment, there are endpoints (e.g., REST APIs) created for the pipeline that can accept input to the pipeline and provide results from the pipeline, and the output 245 is sent through these endpoints. Another example application is for online monitoring of model performance, alerting upon detecting a significant deviation in current model performance compared to initial performance, and providing feedback and / or initiating a retuning process in response to the alert.In one embodiment, output results may be collected during actual operation or retrieved from storage 240 for presentation through visualization GUI 250. The example model logic 215 may be executed by a trigger 255. The trigger 255 may be in response to a call from another application. The trigger 255 may be in response to a scheduled operation, for example, a menu may be presented to a user in the pipeline interface 210 that allows the user to schedule execution of the pipeline (e.g., daily, monthly, at a specific time, etc.). Scheduled execution may be performed at the pipeline level or for a subsection of the pipeline containing one or more nodes, and execution of various parts of the pipeline may be scheduled differently, for example, a scoring workflow may be scheduled to run daily and a retraining node may be scheduled to run monthly. The trigger 255 may be in response to initiation of interaction of the model logic 215 (or components thereof) by a user 205 interaction with the pipeline interface (canvas) 210, such as a user selecting a run button in the pipeline interface 210.

[0027] In response to analyst 205 making changes to model logic 215 by adding or deleting nodes in the pipeline representation of model logic 215 in pipeline interface 210, the processor may generate notebook changes 260, such as adding notebook paragraphs for added nodes or deleting paragraphs for deleted nodes, to make notebook 220 consistent with the pipeline representation displayed in pipeline interface 210. In response to analyst 205 making changes to model logic 215 by adding or deleting notebook paragraphs in the notebook representation of model logic 215 in notebook interface 220, the processor may generate pipeline changes 265, such as adding pipeline nodes for added notebook paragraphs or deleting pipeline nodes for deleted paragraphs, and in either case reconnecting links between nodes, to make canvas 210 consistent with the notebook representation displayed in notebook interface 220.

[0028] -Notebook and pipeline synchronization service- FIG. 3 illustrates one embodiment of a system component 300 for automatic, bidirectional creation and synchronization of notebooks and pipelines. As used herein, the terms “synchronize” or “synchronization” refer to having a notebook have paragraphs that correspond to nodes in a pipeline, and having a pipeline have nodes that correspond to notebook paragraphs. In one embodiment, there is a one-to-one relationship, association, or correspondence between notebook paragraphs and pipeline nodes. This relationship between nodes and paragraphs indicates that the corresponding paragraph in the notebook contains code for performing the task represented by the node in the pipeline. System component 300 includes an exemplary data studio or notebook interface 305, such as notebook interface 220. System component 300 includes an exemplary model canvas or pipeline interface 310. Notebook interface 305 presents notebook 315 for user interaction. Pipeline interface 310 presents pipeline 320 for user interaction. Notebook 315 includes notebook paragraphs 321-327. Pipeline 320 includes pipeline nodes 331-337. The notebook 315 and pipeline 320 are automatically synchronized through the operation of various synchronization components 340. In one embodiment, the synchronization components 340 are modules of a synchronization pipeline, namely, the notebook synchronizer subsystem 180. The synchronization components 340 include a synchronizer service 345, a link / order / dependency evaluator 350, a widget service 355, a paragraph template library 360, a studio driver 365, and a pipeline service 370. In one embodiment, each of these components communicates with each other via a REST API. The library 360 includes templates for algorithms, data sourcing, filters, transformations, adjustments, and other tasks. In one embodiment, the synchronization components 340 are executed by the pipeline and notebook synchronizer subsystem 180 of the cloud infrastructure system 105.

[0029] -Notebook and pipeline synchronization service: Between notebooks and pipelines - In one embodiment, synchronization component 340 synchronizes pipeline 320 with notebook 315 to which new paragraphs not registered by the pipeline have been added. In one embodiment, synchronizer service 345 converts code in notebook paragraphs into pipeline objects. In one embodiment, synchronizer service 345 operates to discover (373) new unregistered paragraphs in notebook 315. Studio driver 365 includes APIs for create, read, update, and delete (CRUD) operations on notebook 315, such as for reading identifiers for notebook paragraphs in notebook 315. In one embodiment, studio driver 365 parses notebook 315 to identify individual paragraphs, for example, by identifying paragraph boundaries within notebook 315. In one embodiment, synchronizer service 345 repeatedly polls studio driver 365 to determine whether new unregistered paragraphs 375 have been added to notebook 315. The polling may be repeated every few seconds, for example, about every 10 seconds, or, for example, every 30 seconds, allowing changes made to notebook 315 to be detected (and propagated to pipeline 320) in real time as notebook paragraphs are created in notebook 315. To increase concurrency, allowing simultaneous editing of notebook 305 and pipeline 310 interfaces, the polling may be repeated more quickly, for example, once per second, although in practice, polling at intervals of up to 600 seconds provides acceptable performance. In this manner, the processor executing synchronization component 340 repeatedly polls the notebook representation of the application logic to synchronize the pipeline and notebook representations in real time. In one embodiment, during a poll, synchronizer service 345 retrieves a list of unique identifiers for paragraphs in notebook 315 via studio driver 365.The synchronizer service 345 compares the most recently retrieved list with a previously retrieved list to determine whether there are any additional paragraphs that are not in the previously retrieved list. In one embodiment, the synchronization service 345 receives an alert indicating that a change, such as the addition of a new paragraph, has been made to the notebook 315. If there is a new paragraph 375, the synchronizer 345 further determines whether the paragraph is “unregistered” (“unregistered” means that the new paragraph 375 does not have a corresponding node in the pipeline 320), for example, by comparing the unique identifier of the new paragraph 375 with a list of identifiers for associated paragraphs of nodes in the pipeline 320 to determine whether there is a match. If there is no match, the new paragraph 375 is not registered, and a new synchronized node 376 corresponding to the new paragraph 375 must be added to the pipeline 320. In response to finding an unregistered new paragraph 375, the synchronizer service 345 may retrieve the unregistered new paragraph 375 for subsequent processing (by issuing a read request to the studio driver 365 and recording the response). If there is a match, the new paragraph 375 is already registered with the pipeline 320 (e.g., because the new paragraph 375 was dynamically generated in response to the creation of a new pipeline node in the pipeline 320), and no new pipeline node needs to be registered with the pipeline 320.

[0030] In one embodiment, in response to the detection, the synchronizer service 345 operates to register 377 the new node 376 in the pipeline 320. The pipeline service 370 includes an API for CRUD operations on the pipeline 320 (or on the modeling metastore and registration repository 225 of metadata describing the pipeline 320), such as to create a new node in the pipeline 320. In one embodiment, the synchronizer service 345 registers the new pipeline node 376 with the pipeline 320 immediately upon detecting an unregistered new notebook paragraph 375 in the notebook 315, thereby enabling synchronization of the pipeline 320 to include the new pipeline node 376 corresponding to the new notebook paragraph 375 in real time. In one embodiment, the synchronizer service 345 requests that the link / order / dependency evaluator 350 determine the placement or location of the new pipeline node 376 in the pipeline 320. In one embodiment, the synchronizer service requests that the widget service 355 determine a task type to associate with a new pipeline node 376 that corresponds to the new, unregistered paragraph 375. In one embodiment, the synchronizer service 345 inserts the new pipeline node 376 into the pipeline 320 at the location determined by the evaluator 350 and with the task type determined by the widget service 355.

[0031] In one embodiment, evaluator 350 makes its placement decision in pipeline 320 based on the placement of new paragraph 375 in the order of paragraphs in notebook 315. In one embodiment, evaluator 350 sorts notebook paragraphs by identifying the order of execution of the notebook paragraphs within the notebook. In one embodiment, evaluator 350 may analyze the contents of the notebook to identify the order in which the paragraphs appear. Evaluator 350 may store the identified order of execution by mapping the unique identifiers of the notebook paragraphs to positions according to the order of execution, for example, in a key-value data structure. In one embodiment, the order of execution is simply the order in which the notebook paragraphs appear in notebook 315. In one embodiment, evaluator 350 determines the notebook paragraph that immediately precedes unregistered new paragraph 375 in the order, in this example, the previous (order-1) paragraph 326. In one embodiment, evaluator 350 also determines the notebook paragraph that immediately follows unregistered new paragraph 375 in the order, in this example, the next (order+1) paragraph 327. Evaluator 350 identifies the pipeline nodes corresponding to the previous (order -1) and next (order +1) paragraphs, in this example, pipeline nodes 336 and 337, respectively. Thus, evaluator 350 determines that new pipeline node 376 should be placed between pipeline nodes 336 and 337. Evaluator 350 stores or returns the determined placement location for later use in inserting new node 376 into pipeline 320.

[0032] In one embodiment, the notebook has no way (functionally outside of comments) to distinguish the type of task or function performed by a notebook paragraph's function script. When a new paragraph is written to the notebook interface and does not use the pipeline interface, an analysis may be performed to determine the task type for which the pipeline object for the new paragraph should be displayed. In one embodiment, the synchronizer service 345 operates to find 380 the closest task type to the unregistered new paragraph 375. In one embodiment, the synchronizer service 345 requests the widget service 355 to parse the content of the unregistered new paragraph 375 to understand the type of task the paragraph performs. The widget service 355 scans the paragraph content of the unregistered new paragraph 375 to extract the actions described in the paragraph and compares those actions to a library of paragraph templates 360. In the library 360, paragraph templates are categorized by task type, such as algorithms, data hydration / dehydration, data preparation, data partitioning, data sourcing, data quality checks, experiments, exploratory data analysis (EDA), event coding, feature extraction, filters, imputation, missing value handling, prediction (e.g., model-based prediction), model training, statistical techniques, transformations (e.g., deterministic transformations), tuning, and validation. The library 360 may continue to grow through user curation. The library 360 may also be used as a library of template code segments or template notebook paragraphs associated with node placements in the pipeline interface 310, enabling rapid code generation for existing task types. The widget service compares actions in the unregistered new paragraph 375 with template paragraphs to determine the most likely task type for the unregistered new paragraph 375 based on exceeding some probabilistic confidence threshold, such as 60% confidence.For example, when the widget service 355 detects an action such as reading a .csv or calling a database to refresh data in the contents of the unregistered new paragraph 375, the unregistered new paragraph 375 is most likely performing a data sourcing type task. In one embodiment, the widget service runs a machine learning (ML) classifier algorithm trained on the paragraph templates and associated task types in the library 360 to classify the unregistered new paragraph 375 as being of a specific task type. In one embodiment, the ML classifier algorithm is a naive Bayes algorithm, a support vector machine, or a convolutional neural network. If no task type is determined by the widget service, for example, if a confidence threshold is not met for any task type in the library 360, a generic or undefined task type is selected, and the user of the canvas 310 may update the task type they perceive to be a match. The widget service 355 returns the task type to the synchronizer service 345, which automatically tags the new pipeline node as being of that task type. In one embodiment, the pipeline interface GUI (in canvas) 310 has distinct node icons and / or color coding associated with task types. Nodes representing particular types of tasks are displayed using distinct icons or color coding for the task type. In such an aspect, generating a new pipeline node may include (i) analyzing a notebook paragraph that does not have a corresponding pipeline node to identify the type of task performed by the notebook paragraph, (ii) configuring a new pipeline node to be of the identified type, and (iii) displaying the new node in the pipeline with an icon associated with the type of task.

[0033] In one embodiment, multiple new paragraphs or entire new notebooks may be added to the notebook interface 305 over the course of a polling cycle. In response to multiple new paragraphs, multiple nodes are added in the same manner as described above for adding a single node in response to the creation of a single new paragraph. In one embodiment, when generating multiple pipeline nodes from multiple new notebook nodes or from an entire new notebook, the generated pipeline may represent a linear format of the notebook, where each node may be linked to the next node in a linear sequence. This linear pipeline may then be adjusted by a user in the pipeline interface 310 to form diverging, converging, and parallel paths in the pipeline. In one embodiment, links or edges are not part of the notebook representation of the application logic, but links between nodes are maintained in the metadata of the pipeline 320 representation of the application logic. In this manner, notebook paragraphs are automatically extracted as pipeline nodes or tasks (i.e., as named objects), and the notebook code is encapsulated and callable as a REST API.

[0034] -Notebook and pipeline synchronization service: Between notebooks and pipelines - In one embodiment, the synchronization component 340 synchronizes the notebook 315 with the pipeline 320 to which a new node not registered by the notebook has been added. In one embodiment, the synchronizer service 345 converts pipeline nodes into notebook paragraph code. In one embodiment, the synchronizer service 345 operates to discover (383) new pipeline nodes in the pipeline 320 and write (385) corresponding new paragraphs to the notebook 315. Creating a new node in the pipeline 320 involves selecting a node type and placing the node in the pipeline 320 using the canvas 310. Each node type is associated with a template notebook paragraph that contains the functional script for the node. The template paragraph is initially stored in the modeling metastore and registration repository 225 in association with the new pipeline node. Placeholders in the template paragraph may be populated based on interactions with the new node. Each node type has an associated task type that controls the visual representation of the node in the canvas 310. Links in the pipeline 320 can be added or modified to connect the new node to the pipeline. Each node in the pipeline 320 may have a unique identifier.

[0035] In one embodiment, synchronizer service 345 repeatedly polls pipeline 320 through pipeline service 370 to determine whether a new node has been created in pipeline 320. In one embodiment, a processor executing synchronization component 340 repeatedly polls the pipeline representation of application logic to synchronize the pipeline and notebook representations in real time in a manner similar to that described above for notebook polling. For example, synchronizer service 345 determines whether a node that was not present at the last poll is present in pipeline 320 and whether a corresponding notebook paragraph for that node is present in notebook 315. In one embodiment, synchronizer service 345 compares a list of nodes currently present in pipeline 320 (e.g., a list of unique node IDs) with the list of nodes in pipeline 320 at the last poll. If a node was not present in the list of nodes at the last poll, a new node is present. The new node may need to have a corresponding notebook paragraph added to notebook 315. In one embodiment, synchronization service 345 receives an alert indicating that a change, such as the addition of a new node, has been made to pipeline 320. The new node may need to have a corresponding notebook paragraph added to notebook 315. Synchronizer service 345 then determines whether a notebook paragraph corresponding to the new node should be added to notebook 315 by determining whether a corresponding paragraph already exists in notebook 315. In one embodiment, synchronizer service 345 determines whether a corresponding paragraph already exists by examining the metadata representing the new node in modeling metastore and registration repository 225.The synchronizer service 345 determines whether the underlying paragraph for the node is held in the metastore 345 in the metadata representing the new node, or whether there is a reference or API call to a notebook paragraph in the metastore 225 in the metadata representing the new node. If the paragraph itself is held in the metadata for the new node, this indicates that the new node was created using the pipeline interface 310 and that a notebook paragraph corresponding to the new node has not yet been created in the notebook 315 and should be added to the notebook 315. In one embodiment, the synchronizer service 345 operates to remove the paragraph representing the new node in the metastore 225 from the metadata and add a link, reference, or API call to the new paragraph in the notebook 315 as a new paragraph into the notebook 315 to the metadata representing the new node. In one embodiment, the synchronizer service 345 requests that the link / order / dependency evaluator 350 determine a placement position for the new paragraph in the notebook 315. If a reference or API call to the notebook paragraph is kept in the metadata, the notebook paragraph for the new node already exists in the notebook 315 and does not need to be added to the notebook 315.

[0036] In one embodiment, evaluator 350 bases its decision on the placement of a new paragraph in notebook 315 on the placement of the new node in pipeline 320. In the illustrated example notebook 315 and pipeline 320, paragraph 321 corresponds to node 331, paragraph 322 corresponds to node 332, paragraph 323 corresponds to node 333, paragraph 324 corresponds to node 334, paragraph 325 corresponds to node 335, paragraph 326 corresponds to node 336, and paragraph 327 corresponds to node 337. In one embodiment, evaluator 350 backtraces pipeline 320 (i.e., traces a path from a node through preceding nodes to one or more starting points) to identify all preceding nodes in pipeline 320 that precede new node 376. These preceding nodes (in this example, nodes 331-336) are executed before the new node in pipeline 320. Evaluator 350 identifies corresponding paragraphs in notebook 315 (in this example, nodes 321-336), for example, by identifying unique identifiers for these corresponding paragraphs from metadata describing the preceding nodes. Evaluator 350 returns the list of paragraph identifiers to synchronizer service 345, which instructs studio driver 365 to insert a new paragraph 375 into notebook 315 that follows all identified paragraphs corresponding to the preceding nodes (in this example, following paragraph 326). Thus, generating a new notebook paragraph may include (i) determining or identifying a set of notebook paragraphs on which execution of the new notebook paragraph depends, and (ii) inserting the new notebook paragraph into the notebook, the new paragraph following the set of paragraphs.

[0037] In one embodiment, the notebook interface 305 assigns a unique identifier to the new paragraph 375. The studio driver 365 receives or retrieves the identifier for the new paragraph 375 and provides it to the synchronizer service 345. The synchronizer service 345 instructs the pipeline service 370 to add a reference or API request to the new paragraph 375 (using the unique identifier) ​​to the metadata describing the new node 376. This allows the pipeline interface 310 to execute the new paragraph 375 when the new node 376 is executed. This allows the contents of the new paragraph 375 to be accessed and edited through user interaction with the new node 376 in the pipeline interface 310. In one embodiment, a notebook (such as notebook 315) has API endpoints for specific functions related to the notebook. For example, there are API endpoints for executing the entire notebook; for executing, viewing, editing, or deleting specific paragraphs in the notebook; for creating a new paragraph in the notebook; for publishing the notebook; etc. In one embodiment, an API request to run, view, edit, or delete a particular paragraph in a notebook, such as new paragraph 375, uses a unique identifier to select the paragraph for run, view, edit, or deletion. In this manner, notebook paragraphs are encapsulated in an application programming interface, and further, notebook paragraphs may be run, viewed, edited, or deleted in response to application programming interface calls (e.g., REST API requests) from the pipeline interface.

[0038] Thus, the systems, methods, and other embodiments described herein enable automatic synchronization of notebook changes (code) to a visual pipeline (which represents and encodes models or other application logic) and vice versa, with changes to either the pipeline or the notebook triggering a refresh alert that can cause the pipeline or notebook to be regenerated.

[0039] In one embodiment, the synchronizer service 345 may check notebook code changes within notebook paragraphs for validity within the pipeline. Any attempt to change notebook code in a manner that destroys the integrity of the pipeline automatically triggers a warning. The synchronizer service 345 may also check pipeline changes for validity within the notebook. In some embodiments, because paragraphs are executed sequentially by the notebook interface, the order of operations defined by the pipeline is not feasible in the notebook. Therefore, application logic can only be executed in the proper order from the pipeline interface. Such a situation would also trigger a warning. The warning may be presented in both the notebook interface 305 and the pipeline interface 310, for example, as a pop-over window containing a message describing the problematic change or order of operations. In one embodiment, the warning is also sent in a message to an address associated with a user who has permission to review the application logic. In this manner, a warning message may be generated in response to identifying a difference between the pipeline representation and the notebook representation of the application logic. The warning message may be sent for display in at least one of the pipeline user interface and the notebook user interface.

[0040] Advantageously, synchronization in both directions occurs in real time as and when pipeline nodes or notebook paragraphs are added, without the use of batch processes or stop-and-restart cycles of the data integration and modeling service.

[0041] -Example Method- In one embodiment, each step of the computer-implemented methods described herein may be performed by a processor (such as processor 810 shown and described with reference to FIG. 8 ) of one or more computing devices configured with logic to (i) access memory (such as memory 815 and / or other computing device components shown and described with reference to FIG. 8 ) and (ii) cause the system (such as automatic interactive notebook and pipeline generation and synchronization logic 830 shown and described with reference to FIG. 8 ) to perform the method steps. For example, the processor accesses, reads from, or writes to memory to perform the computer-implemented method steps described herein. These steps may include (i) retrieving any necessary information, (ii) calculating, determining, generating, classifying, or creating any data, and (iii) storing any calculated, determined, generated, classified, or created data for later use. When referring to storage or storing, it refers to storage as a data structure in the memory or storage / disk of a computing device (such as memory 815 or storage / disk 835 of computing device 805 or remote computer 865 as shown and described with reference to FIG. 8, or data store 130 as shown and described with reference to FIG. 1).

[0042] In one embodiment, each subsequent step of the method is initiated automatically in response to analyzing a received signal or retrieved and stored data that indicates that the previous step has been performed at least to the extent necessary to initiate the subsequent step. Generally, the received signal or retrieved and stored data indicates completion of the previous step.

[0043] FIG. 4 illustrates one embodiment of a method 400 relating to automatic bidirectional notebook and pipeline generation and synchronization. In one embodiment, the steps of method 400 are performed by pipeline notebook synchronizer subsystem 180 (shown and described with reference to FIG. 1). In one embodiment, pipeline notebook synchronizer subsystem 180 is a dedicated computing device (e.g., computing device 805) configured with automatic bidirectional notebook and pipeline generation and synchronization logic 830. In one embodiment, pipeline notebook synchronizer subsystem 180 is a module (or collection of modules, such as the module for synchronization component 340 shown and described with reference to FIG. 3) of a dedicated computing device configured with logic 830. In one embodiment, the steps of method 400 (as well as other methods, systems, and embodiments described herein) enable automatic real-time synchronization of notebook-style user interface inputs and pipeline-style user interface inputs on the same application logic. Such real-time synchronization has not previously been possible by computing devices. In one embodiment, the steps of method 400 (as well as other methods, systems, and embodiments described herein) advance the art of computer model development by integrating notebook-style and pipeline-style development of computer models (whereas these two styles of interface were previously incompatible), thereby enabling collaborative development of computer models by users of different styles of interface.

[0044] Method 400 may be initiated automatically based on various triggers, such as in response to (i) a user (or administrator) of system 100 initiating method 400, (ii) method 400 being scheduled to initiate at a defined time or time interval, such as the polling interval described above, or (iii) receiving a signal over a network indicating that method 400 should be initiated in response to a change in one of the pipeline representation of application logic and the notebook representation of application logic, or analyzing stored data indicative of (i), (ii), or (iii) above. Method 400 begins at start block 405 in response to analyzing a received signal or retrieved and stored data and determining that the signal or stored data indicates that method 400 should be initiated. Processing continues to process block 410.

[0045] At process block 410, the processor identifies differences between the pipeline and notebook representations of the application logic. In one embodiment, the differences are one of (i) notebook paragraphs without corresponding pipeline nodes and (ii) pipeline nodes without corresponding notebook paragraphs. In one embodiment, the processor polls the notebook and pipeline and then determines whether an update has occurred since the last poll, for example, by comparing a record of the notebook or pipeline state at the previous poll with the notebook or pipeline state at the current poll, or by requesting the timestamp of the most recent update to the notebook or pipeline and comparing it with the timestamp of the last poll. If an update has occurred, the processor retrieves an identifier of the notebook paragraph associated with the updated notebook or pipeline. Recall, as discussed above, that notebook paragraphs have unique identifiers within a notebook and that pipelines reference notebook paragraphs for their underlying logic. The processor then searches unupdated notebooks or pipelines for the unique identifier. If the unique identifier is not present, there is a difference between the pipeline and notebook representations of the application logic. In one embodiment, these steps are performed by the synchronization component 340 of the synchronizer subsystem 180. In one embodiment, differences are detected as described above with respect to finding unregistered paragraphs 373 and finding new nodes 383 as shown and described with reference to Figure 3. Once the processor has thus completed identifying differences between the pipeline representation and the notebook representation of the application logic, processing proceeds to decision block 415.

[0046] At decision block 415, the processor determines whether the difference is (i) a notebook paragraph without a corresponding pipeline node or (ii) a pipeline node without a corresponding notebook paragraph. In one embodiment, the processor determines that the difference is (i) a notebook paragraph without a corresponding pipeline node. In this case, a new paragraph exists in the notebook that is not referenced by any node in the pipeline, as described above with reference to FIG. 3 . In one embodiment, the processor determines that the difference is (ii) a pipeline node without a corresponding notebook paragraph. In this case, a new pipeline node exists that stores its function script in the modeling metastore and registration repository rather than in the notebook. In one embodiment, this determination is made by the synchronization component 340 of the synchronizer subsystem 180. Thus, if the processor determines that the difference is (i) a notebook paragraph without a corresponding pipeline node, processing at decision block 415 is complete, and processing proceeds to process block 420. If the processor determines that the difference is (ii) a pipeline node without a corresponding notebook paragraph, processing at decision block 415 is complete, and processing proceeds to process block 425.

[0047] At process block 420, the processor synchronizes the pipeline representation with the notebook representation by automatically generating a new pipeline node in the pipeline representation for any notebook paragraph that does not have a corresponding pipeline node. In one embodiment, the processor analyzes the notebook paragraph's script to detect keywords indicative of the task type performed by the notebook paragraph, or analyzes the notebook script with an ML classification algorithm, e.g., as described above with reference to search 380, widget service 355, and library 360 in FIG. 3, to determine the task type for the new pipeline node. In one embodiment, the processor evaluates the notebook paragraph's position within the notebook to determine the placement of the new pipeline node in the pipeline representation, e.g., as described above with reference to link / order / dependency evaluator 350 in FIG. 3. The processor then configures the new node to have the determined task type and sets the notebook paragraph's function script to the node. The processor inserts the new node into the pipeline representation at the determined placement location and links the new node to the pipeline representation. In one embodiment, these steps are performed by synchronization component 340 of synchronizer subsystem 180. Thus, for notebook paragraphs that do not have a corresponding pipeline node, once the processor completes synchronization between the pipeline representation and the notebook representation by automatically generating a new pipeline node in the pipeline representation, processing at process block 420 is complete and processing proceeds to process block 430.

[0048] At process block 425, for pipeline nodes that do not have a corresponding notebook paragraph, the processor synchronizes the pipeline representation with the notebook representation by automatically generating a new notebook paragraph in the notebook representation. In one embodiment, in response to creating a pipeline node in the pipeline representation, the processor retrieves a template function script for a node of a type selected for the created pipeline node and writes the template script to a modeling metastore and registration repository that describes the pipeline representation. The processor analyzes the location of the created node in the pipeline to determine a location in the notebook where a new notebook paragraph corresponding to the created node should be placed, for example, as described above with reference to evaluator 350 in FIG. 3 . The processor executes a query (e.g., a PGQL query) against the repository to retrieve the template script and remove the template script from the repository. The processor then executes an API request against the notebook to create a new notebook paragraph at the determined location and populate the new notebook paragraph with the template script. In one embodiment, these steps are performed by synchronization component 340 of synchronizer subsystem 180. Thus, for a pipeline node that does not have a corresponding notebook paragraph, once the processor has completed synchronization between the pipeline representation and the notebook representation by automatically generating a new notebook paragraph in the notebook representation, processing at process block 425 is complete and processing proceeds to process block 430.

[0049] At process block 430, the processor updates the pipeline user interface to show the new pipeline node or updates the notebook user interface to show the new notebook paragraph. In one embodiment, the processor forces a refresh in the pipeline GUI in response to the insertion of a new node or forces a refresh in the notebook GUI in response to the insertion of a new paragraph. The processor constructs a message, such as a REST request, to cause the refresh and sends the request for execution by a web browser or other application through which the user is viewing the notebook or pipeline GUI for data integration and modeling services 120. The message is sent immediately in response to the completion of the insertion to maintain actual real-time synchronization between the notebook user interface and the pipeline user interface. In one embodiment, these steps are performed by the synchronization component 340 of the synchronizer subsystem 180 and the web interface server 135. Thus, once the processor has completed updating the pipeline user interface to show the new pipeline node or the notebook user interface to show the new notebook paragraph, processing at process block 430 is complete and processing proceeds to end block 435, where process 400 ends.

[0050] FIG. 5 illustrates another embodiment of a method 500 relating to automatic bidirectional creation and synchronization of notebooks and pipelines. In one embodiment, the steps of method 500 are performed by pipeline notebook synchronizer subsystem 180 (shown and described with reference to FIG. 1) in a manner similar to that described for method 400 above. Method 500 may be automatically initiated based on a trigger similar to that described for method 400 above. Method 500 begins at start block 505 in response to analyzing a received signal or retrieved and stored data and determining that the signal or the stored data indicates that method 500 should be initiated. Processing proceeds to process block 510.

[0051] At process block 510, the processor compares the canvas (pipeline interface) data with the studio (notebook interface) data. In one embodiment, the processor compares the pipeline data with the notebook data to identify notebook paragraphs that are not registered as pipeline nodes, for example, as described in more detail elsewhere herein with reference to blocks 410 and 415 of Figures 3 and 4. In one embodiment, the processor compares the pipeline data with the notebook data to identify pipeline nodes that do not have corresponding paragraphs in the notebook, for example, as described in more detail elsewhere herein with reference to blocks 410 and 415 of Figures 3 and 4. Processing at process block 510 is complete and proceeds to process block 515.

[0052] At process block 510, the processor (i) creates nodes in the canvas (pipeline) for unsaved or unregistered paragraphs in the notebook and (ii) removes nodes from the canvas (pipeline) for paragraphs that are not in the notebook. In one embodiment, to create the nodes, the processor automatically generates pipeline nodes in the pipeline that correspond to unregistered notebook paragraphs, for example, as described in more detail elsewhere herein with reference to block 420 of Figures 3 and 4.

[0053] In one embodiment, the synchronizer service 345 operates to discover paragraphs that have been deleted from the notebook 315. In one embodiment, as described above, the studio driver 365 parses the notebook 315 to identify distinct paragraphs. In one embodiment, the synchronizer service 345 repeatedly polls the studio driver 365 to determine whether existing paragraphs have been deleted from the notebook 315. In one embodiment, upon polling, the synchronizer service 345 retrieves a list of unique identifiers for the notebook 315 paragraphs via the studio driver 365. The synchronizer service 345 compares the most recently retrieved list with previously retrieved lists to determine whether paragraphs that are not in the most recently retrieved list are present in the previously retrieved lists, thereby identifying removed or deleted paragraphs. The synchronizer service 345 then operates to determine whether a node corresponding to the deleted paragraph exists in the pipeline. In one embodiment, in response to the request generated by the synchronizer service 345, the pipeline service 370 queries the repository 225 for the identities of nodes that reference the deleted paragraph ("nodes to be deleted") and removes any such nodes to be deleted from the pipeline 320, if any. In one embodiment, the removal includes removing the link between the previous (order -1) node and the node to be deleted, removing the node to be deleted, and changing the origin of the link between the node to be deleted and the next (order +1) node to the previous (order -1) node, thereby maintaining the link type of the link from the node to be deleted.

[0054] Thus, in one embodiment, the synchronization process also propagates deletions from the pipeline to the notebook or from the notebook to the pipeline, determining that for a notebook paragraph that does not have a corresponding pipeline node, the corresponding pipeline node was not deleted through the pipeline user interface, and for a pipeline node that does not have a corresponding notebook paragraph, the corresponding notebook paragraph was not deleted through the notebook user interface. Processing at process block 515 is complete and processing continues to process block 520.

[0055] At process block 520, the processor links the most recently added nodes. In one embodiment, the processor links the created pipeline nodes to the pipeline as shown and described with reference to FIG. 6. FIG. 6 illustrates one embodiment of a method 600 for linking new pipeline nodes associated with automatic bidirectional notebook and pipeline creation and synchronization. In one embodiment, the steps of method 600 are performed by the pipeline notebook synchronizer subsystem 180 (shown and described with reference to FIG. 1) in a manner similar to that described for method 400 above. Method 600 may be initiated automatically, for example, in response to receiving a signal indicating that a new pipeline node has been added to the pipeline or in response to analyzing stored data indicating that a new pipeline node has been added to the pipeline. Method 600 begins at start block 605 in response to analyzing a received signal or retrieved and stored data and determining that the signal or the stored data indicates that method 500 should be initiated. Processing continues to process block 610.

[0056] At process block 610, the processor sorts the notebook paragraphs based on the order of execution of those paragraphs within the notebook. This is a preliminary step performed because the order of paragraph identifiers may not correspond to the order of execution, and paragraphs should be evaluated based on the order of execution rather than the order in which they are included in the notebook. In one embodiment, the processor parses the contents of the notebook to identify the paragraphs in the order in which they appear in the notebook, extracts the unique IDs of the paragraphs, and writes the unique IDs for the paragraphs in the order in which they appear to a list. The list of unique paragraph IDs, in the order in which the paragraphs identified by the IDs appear, is a sort of the notebook paragraphs based on the order of execution in the notebook because the notebook execution of the paragraphs is consecutive. Processing at process block 610 is complete, and processing continues to process block 615 within loop 620.

[0057] In one embodiment, there may be multiple unregistered paragraphs (paragraphs without a corresponding node in the pipeline) for which a new node needs to be linked to the pipeline. Therefore, processing loop 620 repeats the linking process for the unregistered paragraphs. In process block 615, the processor finds a paragraph that is before the unregistered paragraph in the execution order (order -1) and finds a paragraph that is after the unregistered paragraph in the execution order (order +1). In one embodiment, the processor parses the ordered list of unique paragraph IDs to locate the paragraph ID of the unregistered paragraph. The processor then selects the paragraph IDs that are immediately before and immediately after the paragraph ID of the unregistered paragraph in the list. The processor then selects the paragraph with the immediately preceding paragraph ID to be the previous (order -1) paragraph and the paragraph with the immediately following paragraph ID to be the next (order +1) paragraph. In one embodiment, identifying the paragraph order and finding the previous and next paragraphs is performed by evaluator 350 as shown and described with reference to Figure 3. Processing at process block 615 is complete and processing continues to decision block 625.

[0058] At decision block 625, the processor determines whether a first node corresponding to the previous (order -1) paragraph is connected to a second node corresponding to the next (order +1) paragraph at the pipeline interface. In one embodiment, the processor queries the repository 225 to determine whether any link connected to the first node connects to a second node. If so (625: YES), processing at decision block 625 is complete and processing continues to process block 630. If not (625: NO), processing at decision block 625 is complete and processing continues to decision block 635. In one embodiment, this determination is made by the evaluator 350 and other synchronization components 340, as shown and described with reference to FIG. 3 .

[0059] At process block 630, the processor removes the connecting link between the first node and the second node and notes the link type. The link may have a link type property that is used by the pipeline interface 310 at runtime to direct one or more aspects of pipeline execution. In one embodiment, the processor performs a query to the repository 225 to read the link type of the connecting link and records the retrieved link type value for subsequent processing. The processor then performs a query to the repository 225 to delete the connecting link. In one embodiment, the query is performed by the pipeline service 370 in response to a request by the synchronization service 345, as shown and described with reference to FIG. 3 . Processing at process block 630 then completes, and processing proceeds to decision block 635.

[0060] At decision block 635, the processor determines whether the first node (corresponding to the previous paragraph in order -1) is already connected to a node (URP node) corresponding to an unregistered paragraph in the pipeline. In one embodiment, the processor queries the repository 225 to determine whether any links connected to the first node connect to a URP node. If not (635: NO), processing at decision block 635 is complete and processing continues to process block 640. If connected (635: YES), processing at decision block 635 is complete and processing continues to decision block 645. In one embodiment, the query is performed by the pipeline service 370, and the determination is made by the synchronization service 345, as shown and described with reference to FIG. 3 .

[0061] At process block 640, the processor connects a first node in the pipeline to a URP node and sets the link between the first node and the URP node to a default link type. In one embodiment, the processor generates and executes one or more queries against repository 225 to make this connection. Executing the queries causes the processor to add a new link to a link table for the pipeline in repository 225, set the source node of the link to the first node, e.g., by writing the node ID of the first node in the source node ID field of the link, and set the destination node of the link to the URP node, e.g., by writing the node ID of the URP node in the destination node ID field of the link. In one embodiment, the queries are REST API requests from synchronization service 345 executed by pipeline service 370 (shown and described with reference to FIG. 3 ). Processing at process block 640 then completes.

[0062] At decision block 645, the processor determines whether the URP node is already connected to a second node in the pipeline (corresponding to the next order +1 paragraph). In one embodiment, the processor queries the repository 225 to determine whether any links connected to the URP node connect to a second node. If not (645: NO), processing at decision block 645 is complete and processing continues to process block 650. If connected (645: YES), processing at decision block 645 is complete. In one embodiment, the query is performed by the pipeline service 370 and the determination is made by the synchronization service 345, as shown and described with reference to FIG. 3 .

[0063] At process block 650, the processor connects a URP node in the pipeline with a second node and sets the link between the URP node and the second node to the annotated link type. In one embodiment, the processor retrieves the annotated link type and generates and executes one or more queries to repository 225 to make this connection. Executing the query causes the processor to add a new link to a link table for the pipeline in repository 225, set the link's source node to the URP node, e.g., by writing the URP node's node ID to the link's source node ID field, set the link's destination node to the second node, e.g., by writing the second node's node ID to the link's destination node ID field, and set the link type to the annotated link type. In one embodiment, the query is a REST API request from synchronization service 345 executed by pipeline service 370 (shown and described with reference to FIG. 3 ). Processing at process block 650 then completes.

[0064] The loop 620 is then repeated for the next unregistered paragraph until there are no more unregistered paragraphs remaining. The loop 620 is then completed and processing proceeds to end block 655 where the process 600 ends.

[0065] In this manner, generating a new pipeline node may include sorting the notebook paragraphs based on order of execution, identifying (i) a first paragraph that immediately precedes, in order of execution, a notebook paragraph that does not have a corresponding pipeline node, and (ii) a second paragraph that immediately follows, in order of execution, a notebook paragraph that does not have a corresponding pipeline node, removing a connecting link in the pipeline representation between a first node corresponding to the first paragraph and a second node corresponding to the second paragraph, recording the type of the connecting link, linking the first node to the new pipeline node with a new link of a default type, and linking the new pipeline node to the second node with a new link of the recorded type. Referring again to FIG. 5, processing at process block 520 is complete upon completion of method 600, and processing proceeds to end block 525, where method 500 ends.

[0066] -Example Pipeline Graphical User Interface- FIG. 7 illustrates one embodiment of a pipeline graphical user interface (GUI) 700 associated with the automatic, bidirectional creation and synchronization of notebooks and pipelines. The pipeline GUI 700 illustrates an example pipeline 705, which consists of a set of nodes (e.g., “load graph” node 710) and links (e.g., link 715) that interconnect the nodes in a particular order. The pipeline GUI 700 includes a task library menu 720. A user can select a pipeline node type from the task library menu 720 and create a new pipeline node of that type within a pipeline, for example, by clicking an icon for that node type within the task library menu 720, dragging the cursor to a location within the pipeline, and releasing it to form a new node of the selected type. Pipeline nodes are sometimes referred to herein as “widgets.” As used herein, a “widget” may also include pre-configured pipeline segments, in which multiple nodes are linked together to perform core composite tasks, which can also be dragged and dropped into a pipeline. The new node is recorded in a data structure, e.g., pipeline 705, within the modeling metastore and registration repository 225. The new node has an associated type. Each type has a template script paragraph that, when executed, performs the node's task function. In response to the creation of a new node, the template script paragraph is first created, for example, in repository 225, and then synchronized to the notebook as a notebook paragraph, as described above with reference to FIG. 3. In one embodiment, a user may add a generic node that does not have a template script. The generic node may then be tagged by type by the user, and the user may manually script the generic node. The manual script will then be synchronized to the notebook. Note that after synchronizing the script paragraphs for a new node to the notebook, the notebook paragraphs for the new node are the authoritative copies of the script contents.The scripts used to provide the functionality of a node are not stored in multiple places, but only in notebook paragraphs. The script note paragraphs for new nodes are accessed, edited, and executed through API calls to the notebook.

[0067] Links between nodes may be created by clicking a first node connection point (e.g., connection point 730) and dragging the cursor to a second node connection point (e.g., connection point 735) to form the new link. The new link is recorded in a data structure, e.g., pipeline 705, in repository 225. Nodes and links may also be deleted from example pipeline 705, for example, by selecting them with the cursor and selecting a delete icon, a delete menu option, or by pressing the delete key. Changes to example pipeline 705 may be saved in response to selecting save button 740. Example pipeline 705 may be run in response to selecting run pipeline button 745. One or more individual nodes in example pipeline 705 may be run independently of other pipeline nodes in response to a user selecting a run node button, such as run node button 750 to run load graph node 710. These run buttons allow example pipeline 705 to be run interactively, in whole or in part (acting as interactive trigger 255). In one embodiment, a menu may be launched in response to selecting menu button 755. The user can provide runtime parameters for the pipeline through a menu.

[0068] In one embodiment, selection of a node, such as selection of data capture node 760, may launch a menu 765 that displays details of the node. Menu 765 may include an activity description 770 that describes the task performed by the node, an activity type 775 that indicates the represented node type, a paragraph ID 780 that records a unique identifier of the notebook paragraph that provides the node's functionality, and a function script 785 as stored in the notebook paragraph. In one embodiment, activity description 770, activity type 775, and paragraph script 785 are user-editable. For example, a user may modify script 785 in a notebook through menu 765. In one embodiment, activity description 770 and activity type 775 are stored in repository 225, and script 785 is stored in the notebook and interacted with through API requests. In this manner, the pipeline interface provides an alternative interface for creating, modifying, and executing application logic that is recorded as notebook paragraphs. Thus, a user may access and edit notebook paragraphs in the pipeline user interface, and editing in the pipeline user interface modifies the notebook paragraph in the notebook representation of the application logic.

[0069] Content updates to notebook paragraphs from pipeline interfaces do not need to be synchronized as described herein because the contents of a notebook paragraph (i.e., the script) are not kept in multiple places but are stored in a single place, the notebook. Cases for synchronization apply when a new thing (node ​​or paragraph) appears or is added to one of the interfaces, or when an old or existing thing (node ​​or paragraph) disappears or is removed from one of the interfaces.

[0070] In one embodiment, a new notebook paragraph describing the data transformation is written by the user in the notebook corresponding to the example pipeline 705. The example pipeline 705 is then synchronized with the notebook, and a new data transformation node 790 automatically appears in the example pipeline 705. The new data transformation node 790 is automatically placed in the correct location, with links automatically generated and attached (e.g., as shown and described with reference to FIG. 6 ) to incorporate the new data transformation node 790 into the example pipeline 705.

[0071] -Selected Benefits- The systems, methods, and other embodiments for automatic, bidirectional generation and synchronization of notebooks and pipelines described herein enable several advantages and improvements, including: (1) automatic synchronization between a representation of application logic in a pipeline user interface (business user artifact) and a representation of application logic in a notebook user interface (modeler user artifact), (2) reverse generation and insertion of new nodes into linked pipelines based on user input creation of new paragraphs in the notebook, (3) integration of visual pipeline reverse generation of nodes with code generation, providing a dual, non-conflicting interface for creating and editing application logic (enabling a collaborative development process between business or domain analysts and modelers or data scientists), (4) a warning process when the pipeline and notebook representations of application logic are accidentally allowed to get out of sync (providing powerful checks and balances), and (5) automatic provision of artificial intelligence (AI) and model management and auditability without manual or additional effort.

[0072] -Software module embodiment- The software instructions may be designed to be executed by one or more appropriately programmed processors that access memory, such as by accessing CPU or GPU resources. These software instructions may include, for example, computer-executable code and source code that may be compiled into the computer-executable code. These software instructions may also include instructions written in an interpreted programming language, such as a scripting language.

[0073] In a complex system, such instructions may be arranged into program modules, each of which performs a particular task, process, function, or operation. The entire set of modules may be controlled or coordinated in their operation by a main program, operating system (OS), or other form of organizational platform for the system.

[0074] In one embodiment, one or more of the components described herein are configured as modules stored on a non-transitory computer-readable medium, the modules being comprised of stored software instructions that, when executed by at least a processor accessing memory or storage, cause the computing device to perform the corresponding functions described herein.

[0075] -Cloud or Enterprise Implementation- In one embodiment, the system (e.g., system 100) includes a computing / data processing system that includes a computing application or collection of distributed computing applications (e.g., notebook interface 305 or pipeline interface 310 to data integration and modeling service 120) that are accessed and used by other client computing devices associated with the enterprise (e.g., client devices 145, 150, 155, and 160 of enterprise network 115). The system and client computing devices communicate with each other over a network (e.g., network 110). The applications and computing system may be configured to operate in or implemented as a cloud-based network computing system, an infrastructure-as-a-service (IAAS), a platform-as-a-service (PAAS), or a software-as-a-service (SAAS) architecture, or other type of networked computing solution. In one embodiment, the system provides at least one or more of the functions disclosed herein and a graphical user interface for accessing and operating them, and implements a centralized server-side application accessed by many users through computing devices / terminals that communicate with the computing system (acting as a server) over a computer network. In one embodiment, cloud infrastructure system 105 (including data integration and modeling services 120) may be implemented on an on-premise infrastructure, such as a set of one or more dedicated servers. In one embodiment, the system provides at least one or more of the functionality disclosed herein and a graphical user interface for accessing and operating the functionality.

[0076] -Computing Device Embodiment- 8 illustrates an example computing device 800 configured and / or programmed as a special-purpose computing device with one or more of the example systems and methods described herein and / or the like. The example computing device 800 may be a computer 805 including at least one hardware processor 810, memory 815, and input / output ports 820 operably connected by a bus 825. In one example, the computer 805 may include automatic bidirectional notebook and pipeline generation and synchronization logic 830 configured to facilitate automatic bidirectional generation and synchronization of notebooks and pipelines, similar to the logic, systems, methods, and other embodiments shown and described with reference to FIGS. 1-7.

[0077] In various examples, logic 830 may be implemented in hardware, a non-transitory computer-readable medium 837 having instructions stored thereon, firmware, and / or combinations thereof. While logic 830 is shown as a separate hardware component attached to bus 825, it should be understood that in other embodiments, logic 830 may be implemented in processor 810, stored in memory 815, or stored on disk 835.

[0078] In one embodiment, logic 830 or a computer is a means (e.g., structure: hardware, non-transitory computer-readable medium, firmware) for performing the described actions. In some embodiments, the computing device may be a server operating in a cloud computing system, a server configured in a Software as a Service (SaaS) architecture, a smartphone, a laptop, a tablet computing device, etc.

[0079] The means may be implemented, for example, as an ASIC programmed to facilitate automatic bidirectional generation and synchronization of notebooks and pipelines, or as stored computer-executable instructions presented to the computer 805 as data 840 that are temporarily stored in memory 815 and executed by the processor 810.

[0080] Logic 830 may also provide means (e.g., hardware, non-transitory computer-readable media storing executable instructions, firmware) for performing automatic bidirectional generation and synchronization of notebooks and pipelines.

[0081] Generally describing an exemplary configuration of computer 805, processor 810 can be a wide variety of processors, including dual microprocessors and other multi-processor architectures. Memory 815 can include volatile memory and / or non-volatile memory. Non-volatile memory can include, for example, ROM, PROM, etc. Volatile memory can include, for example, RAM, SRAM, DRAM, etc.

[0082] Storage disk 835 may be operatively connected to computer 805, for example, via input / output (I / O) interface (e.g., card, device) 845 and input / output port 820, controlled by at least input / output (I / O) controller 847. Disk 835 may be, for example, a magnetic disk drive, solid state drive (SSD), floppy disk drive, tape drive, Zip drive, flash memory card, memory stick, etc. Additionally, disk 835 may be an optical drive, such as a CD-ROM drive, CD-R drive, CD-RW drive, DVD ROM, etc. Memory 815 may store, for example, processes 850 and / or data 840. Disk 835 and / or memory 815 may store an operating system that controls and allocates resources of computer 805.

[0083] Computer 805 may interact with, control, and / or be controlled by input / output (I / O) devices via input / output (I / O) controller 847, I / O interface 845, and input / output ports 820. The input / output devices may include, for example, one or more displays 870, printer 872 (such as an inkjet, laser, or 3D printer), audio output device 874 (such as a speaker or headphones), text input device 880 (such as a keyboard), cursor control device 882 for pointing and selection input (e.g., a mouse, trackball, touch screen, joystick, pointing stick, electronic stylus, electronic pen tablet, etc.), audio input device 884 (such as a microphone or external audio player), video input device 886 (such as a video and still camera or external video player), image scanner 888, video card (not shown), disk 835, network device 855, etc. The input / output ports 820 may include, for example, serial ports, parallel ports, and USB ports.

[0084] The computer 805 can operate in a networked environment and, as such, can be connected to a network device 855 via the I / O interface 845 and / or the I / O port 820. The computer 805 can interact with a network 860 through the network device 855. The computer 805 can be logically connected to a remote computer 865 through the network. Networks with which the computer 805 can interact include, but are not limited to, a LAN, a WAN, and other networks.

[0085] -Definitions and Other Embodiments- In another embodiment, the described methods and / or their equivalents may be embodied in computer-executable instructions. Thus, in one embodiment, a non-transitory computer-readable / storage medium is configured with stored computer-executable instructions for an algorithm / executable application that, when executed by a machine, causes the machine (and / or associated components) to perform the method. Exemplary machines include, but are not limited to, processors, computers, servers operating in a cloud computing system, servers configured in a Software as a Service (SaaS) architecture, smartphones, etc. In one embodiment, a computing device is implemented with one or more executable algorithms configured to perform any of the disclosed methods.

[0086] In one or more embodiments, the disclosed methods or equivalents thereof are performed by either computer hardware configured to perform the methods, or by computer instructions embodied in modules stored on a non-transitory computer-readable medium, where the instructions are configured as an executable algorithm configured to perform the methods when executed by at least a processor of a computing device.

[0087] For simplicity of explanation, the example method shown in the figures is illustrated and described as a series of algorithmic blocks, but it should be understood that the method is not limited by the order of the blocks. Some blocks may occur in a different order than illustrated and described and / or concurrently with other blocks. Furthermore, fewer than all of the illustrated blocks may be used to implement the example method. The blocks may be combined or separated into multiple operations / components. Furthermore, additional and / or alternative methods may employ additional operations not shown in the blocks.

[0088] The following contains definitions of selected terms used herein. These definitions include various examples and / or forms of components that fall within the scope of the terms and that may be used for implementation. These examples are not intended to be limiting. Both singular and plural forms of terms may be within the scope of the definitions.

[0089] References to "one embodiment," "an embodiment," "an example," "an example," etc., indicate that the embodiment or example so described may include a particular feature, structure, characteristic, property, element, or limitation, but not all embodiments or examples necessarily include that particular feature, structure, characteristic, property, element, or limitation. Furthermore, repeated use of the phrase "in one embodiment" does not necessarily refer to the same embodiment, although it may.

[0090] The acronyms and abbreviations used herein have the following meanings: API: Application Programming Interface ASIC: Application Specific Integrated Circuit CD: Compact Disc CD-R: Recordable CD CD-RW: Rewritable CD CPU: Central Processing Unit CRUD: Create, Read, Update, Delete DRAM: Dynamic RAM DVD: Digital Versatile Disc and / or Digital Video Disc GPU: Graphics Processing Unit GUI: Graphical User Interface HDD: Hard Disk Drive HPC: High Performance Computing I / O: Input / Output IAAS: Infrastructure as a Service ID: Identifier JDBC: Java Database Connectivity JSON: JavaScript Object Notation LAN: Local Area Network ML: Machine Learning MMG: Model Management and Supervision NAS: Network Attached Storage OFSAA: Oracle Financial Services Analytical Applications OS: Operating System PAAS: Platform as a Service Pgq1: A Property Graph Query Language PGX: Property Graph Server PROM: Programmable ROM RAM: Random Access Memory REST: Representational State Transfer ROM: Read-Only Memory SAAS: Software as a Service SOAP: Simple Object Access Protocol Sq1: Structured Query Language SRAM: Synchronous RAM SSD: Solid State Storage Device TCP / IP: Transmission Control Protocol / Internet Protocol USB: Universal Serial Bus WAN: Wide Area Network XML: Extensible Markup Language As used herein, a "data structure" is an organization of data within a computing system that is stored in memory, a storage device, or other computerized system. A data structure may be, for example, any one of a data field, a data file, a data array, a data record, a database, a data table, a graph, a tree, a linked list, etc. A data structure may be formed from and may contain many other data structures (e.g., a database contains many data records). Other examples of data structures are possible according to other embodiments.

[0091] As used herein, "computer-readable medium" or "computer storage medium" refers to a non-transitory medium that stores instructions and / or data that, when executed, are configured to perform one or more of the disclosed functions. Data may function as instructions in some embodiments. Computer-readable media may take forms including, but not limited to, non-volatile media and volatile media. Non-volatile media may include, for example, optical disks, magnetic disks, and the like. Volatile media may include, for example, semiconductor memory, dynamic memory, and the like. Common forms of computer-readable media may include, but are not limited to, floppy disks, flexible disks, hard disks, magnetic tape, other magnetic media, application specific integrated circuits (ASICs), programmable logic devices, compact disks (CDs), other optical media, random access memory (RAM), read-only memory (ROM), memory chips or cards, memory sticks, solid-state storage devices (SSDs), flash drives, and other media on which a computer, processor, or other electronic device can function. When selected for implementation in one embodiment, each type of media may include stored instructions for an algorithm configured to perform one or more of the disclosed and / or claimed functions.

[0092] As used herein, "logic" refers to components implemented in computer or electrical hardware, non-transitory media having executable application or program module instructions stored thereon, and / or combinations thereof, to perform any of the functions or operations disclosed herein and / or to cause other logic, methods, and / or systems disclosed herein to perform functions or operations. Equivalent logic may include firmware, a microprocessor programmed with an algorithm, discrete logic (e.g., an ASIC), at least one circuit, analog circuit, digital circuit, programmed logic device, memory device containing algorithmic instructions, etc., any of which may be configured to perform one or more of the disclosed functions. In one embodiment, logic may include one or more gates, combinations of gates, or other circuit components configured to perform one or more of the disclosed functions. Where multiple logics are described, it may be possible to incorporate the multiple logics into one logic. Similarly, where a single logic is described, it may be possible to distribute the single logic among multiple logics. In one embodiment, one or more of these logics is the corresponding structure associated with performing the disclosed and / or claimed functions. The choice of which type of logic to implement may be based on desired system requirements or specifications. For example, if higher speed is a consideration, hardware would be selected to implement the function. If lower cost is a consideration, stored instructions / executable applications would be selected to implement the function.

[0093] An "operable connection," or a connection by which entities are "operably connected," is a connection through which signals, physical communications, and / or logical communications may be transmitted and / or received. An operable connection may include physical interfaces, electrical interfaces, and / or data interfaces. An operable connection may include various combinations of interfaces and / or connections sufficient to enable operable control. For example, two entities may be operably connected to communicate signals to each other directly or through one or more intermediate entities (e.g., processors, operating systems, logic, non-transitory computer-readable media). Logical and / or physical communication channels may be used to create an operable connection.

[0094] As used herein, a "user" includes, but is not limited to, one or more people, computers or other devices, or combinations thereof.

[0095] Although the disclosed embodiments have been illustrated and described in considerable detail, it is not intended to restrict or in any way limit the scope of the appended claims to such details. Naturally, it is impossible to describe every conceivable combination of elements or methodologies for the purpose of describing various aspects of the subject matter. Therefore, the present disclosure is not limited to the specific details or specific examples shown and described. Therefore, the present disclosure is intended to embrace changes, modifications, and variations that fall within the scope of the appended claims and that satisfy the legal subject matter requirements of Japanese Patent Law.

[0096] To the extent the word "includes" or "including" is used in the detailed description or the appended claims, it is intended to be as inclusive as the word "comprising" when interpreted as such when used as a transitional term in a claim.

[0097] To the extent that the word "or" is used in the detailed description or claims (e.g., A or B), it is intended to mean "A or B or both." If applicant intended to indicate "only A or B, but not both," the phrase "only A or B, but not both" would be used. Thus, the use of the word "or" herein is inclusive, not exclusive.

Claims

1. 1. A computer-implemented method comprising: identifying differences between the pipeline representation and the notebook representation of the application logic, the differences being one of (i) notebook paragraphs without corresponding pipeline nodes, and (ii) pipeline nodes without corresponding notebook paragraphs, the computer-implemented method further comprising: For the notebook paragraphs that do not have a corresponding pipeline node, automatically generating a new pipeline node in the pipeline representation; and for said pipeline nodes that do not have a corresponding notebook paragraph, automatically generating a new notebook paragraph in said notebook representation. synchronizing the pipeline representation with the notebook representation; updating a pipeline user interface to indicate the new pipeline node or a notebook user interface to indicate the new notebook paragraph.

2. Creating the new pipeline node further comprises: Sorting notebook paragraphs based on order of execution; (i) identifying a first paragraph that immediately precedes, in the order of execution, the notebook paragraph that does not have a corresponding pipeline node, and (ii) a second paragraph that immediately precedes, in the order of execution, the notebook paragraph that does not have a corresponding pipeline node; removing a connecting link in the pipeline representation between a first node corresponding to the first paragraph and a second node corresponding to the second paragraph; recording the type of said connection link; linking the first node to the new pipeline node with a new link of default type; and linking the new pipeline node to the second node with a new link of the recorded type.

3. generating the new notebook paragraph further comprises: determining a set of notebook paragraphs on which execution of the new notebook paragraph depends; and inserting the new notebook paragraph into the notebook after the set of paragraphs.

4. The synchronizing further comprises: For the notebook paragraphs that do not have a corresponding pipeline node, determining that the corresponding pipeline node was not deleted via the pipeline user interface; and for the pipeline node that does not have a corresponding notebook paragraph, determining that the corresponding notebook paragraph was not deleted via the notebook user interface.

5. 2. The computer-implemented method of claim 1, further comprising repeatedly polling the pipeline representation and the notebook representation to synchronize them in real time.

6. Creating the new pipeline node further comprises: analyzing the notebook paragraphs that do not have a corresponding pipeline node to identify the type of task performed by the notebook paragraph; setting the new pipeline node to be of the identified type; and displaying the new node in the pipeline with an icon associated with the type of task.

7. generating a warning message in response to identifying a difference between the pipeline representation and the notebook representation of the application logic; 2. The computer-implemented method of claim 1, further comprising: sending the warning message for display on at least one of the pipeline user interface and the notebook user interface.

8. 2. The computer-implemented method of claim 1, wherein the notebook paragraph is encapsulated in an application programming interface, and further comprising executing the notebook paragraph in response to an application programming interface call from the pipeline interface.

9. accessing the notebook paragraph in the pipeline user interface; Editing the notebook paragraph in the pipeline user interface; 2. The computer-implemented method of claim 1, wherein editing within the pipeline user interface modifies the notebook paragraphs within the notebook representation.

10. 1. A computing system comprising: a processor; a memory operatively connected to the processor; and a non-transitory computer-readable medium operatively connected to the processor and the memory and storing computer-executable instructions, the computer-executable instructions, when executed by at least a processor of the computing system, causing the computing system to: identifying differences between the pipeline representation and the notebook representation of the application logic, the differences being one of (i) notebook paragraphs without corresponding pipeline nodes and (ii) pipeline nodes without corresponding notebook paragraphs; and for said notebook paragraphs that do not have a corresponding pipeline node, automatically generating a new pipeline node in said pipeline representation; For those pipeline nodes that do not have a corresponding notebook paragraph, by automatically generating a new notebook paragraph in the notebook representation. Synchronizing the pipeline representation with the notebook representation; A computing system causes a pipeline user interface to update to show the new pipeline node or a notebook user interface to update to show the new notebook paragraph.

11. The instructions for creating a new pipeline node further include: Sort the notebook paragraphs based on the order of execution, (i) identifying a first paragraph that immediately precedes, in the order of execution, the notebook paragraph that does not have a corresponding pipeline node, and (ii) a second paragraph that immediately precedes, in the order of execution, the notebook paragraph that does not have a corresponding pipeline node; removing a connecting link in the pipeline representation between a first node corresponding to the first paragraph and a second node corresponding to the second paragraph; Recording the type of said connection link; linking said first node to said new pipeline node with a new link of default type; The computing system of claim 10 , further comprising: linking the new pipeline node to the second node with a new link of the recorded type.

12. The instructions for generating a new notebook paragraph further include instructions for the computing system: determining a set of notebook paragraphs on which execution of said new notebook paragraph depends; The computing system of claim 10 , further comprising: causing the new notebook paragraph to be inserted into the notebook after the set of paragraphs.

13. The instructions for synchronizing further include instructions for the computing system to: for the notebook paragraph having no corresponding pipeline node, determining that the corresponding pipeline node was not deleted via the pipeline user interface; 11. The computing system of claim 10, wherein for the pipeline node that does not have a corresponding notebook paragraph, it is determined that the corresponding notebook paragraph was not deleted via the notebook user interface.

14. 11. The computing system of claim 10, wherein the instructions further cause the computing system to repeatedly poll the pipeline representation and the notebook representation to synchronize the pipeline representation and the notebook representation in real time.

15. The instructions for creating a new pipeline node further include: analyzing the notebook paragraphs that do not have a corresponding pipeline node to identify the type of task performed by the notebook paragraph; causing the new pipeline node to be of the identified type; The computing system of claim 10 , wherein the computing system causes new nodes in the pipeline to be displayed with an icon associated with the type of task.

16. The instructions further include: generating a warning message in response to identifying a difference between the pipeline representation and the notebook representation of the application logic; The computing system of claim 10 , further comprising: causing the warning message to be sent for display on at least one of the pipeline user interface and the notebook user interface.

17. 11. The computing system of claim 10, wherein the notebook paragraph is encapsulated in an application programming interface, and the instructions further cause the computing system to execute the notebook paragraph in response to an application programming interface call from the pipeline interface.

18. The instructions further include: accessing the notebook paragraph in the pipeline user interface; Editing the notebook paragraph in the pipeline user interface; A computing system according to any one of claims 10 to 17, wherein editing within the pipeline user interface modifies the notebook paragraphs within the notebook representation.

19. 1. A computer-implemented method comprising: comparing the pipeline data with the notebook data to identify notebook paragraphs that are not registered as pipeline nodes; automatically generating a pipeline node in the pipeline corresponding to the unregistered notebook paragraph; linking the generated pipeline node to the pipeline.

20. Linking the generated pipeline nodes further comprises: Sorting notebook paragraphs based on order of execution; (i) identifying a first paragraph that immediately precedes the unregistered notebook paragraph in the order of execution, and (ii) a second paragraph that immediately follows the unregistered notebook paragraph in the order of execution; removing a connecting link in the pipeline representation between a first node corresponding to the first paragraph and a second node corresponding to the second paragraph; recording the type of said connection link; linking the first node to the created pipeline node with a new link of default type; 20. The computer-implemented method of claim 19, comprising: linking the created pipeline node to the second node with a new link of the recorded type.

21. A program for causing a computer to execute the method according to any one of claims 1 to 9, 19 and 20.

Images