Job Scheduling and Monitoring

Abstract

Embodiments of the present disclosure relate to job scheduling and monitoring. A method performed by a system includes: for display on a display device, conveying a timeline view of a data transformation job schedule in terms of one or more data slices, the data slices comprising data segments of a continuum, the continuum comprising a data set, wherein each data slice is generated or consumed by a data transformation job and comprises a length and a start time and an end time; receiving a first signal from a user to select a data slice from the one or more data slices in the view; automatically determining one or more related data slices for the user-selected data slice based on data dependencies; and conveying, for display on the display device, a visualization of the user-selected data slice and the one or more related data slices that are visually different from one or more other data slices in the timeline view.

Description

[0001] This application is a divisional application of the Chinese invention patent application with application date of December 7, 2015, application number 201580068297.8, and invention name “Job Scheduling and Monitoring”. Technical Field

[0002] Embodiments of the present disclosure relate generally to information processing, and more particularly to job scheduling and monitoring. Background Art

[0003] Processing large amounts of data, or so-called Big Data, to gather valuable insights first involves transforming the data. By creating, scheduling, and executing one or more jobs, the data is transformed into a form that can be used for publication or consumption by a business intelligence endpoint, such as a dashboard. In this context, a job is a unit of work on data that includes one or more transformation operations. Typically, jobs are manually coded by data developers, data architects, business intelligence architects, etc. Subsequently, the jobs can be scheduled and executed. Summary of the invention

[0004] A simplified summary is presented below in order to provide a basic understanding of some aspects of the disclosed subject matter. This summary is not an extensive overview. It is not intended to identify key / critical elements, nor is it intended to delineate the scope of the claimed subject matter. Its sole purpose is to present some concepts in a simplified form as a prelude to a more detailed description that is presented later.

[0005] Briefly, the subject disclosure relates to job scheduling and monitoring. Jobs may be scheduled for execution, and visualizations may be generated that allow a user to view and interact with the job scheduling and execution. According to one aspect, in response to selection of a job, one or more related jobs may be automatically determined based on data dependencies. Subsequently, the selected job and related jobs may be highlighted. Execution status may also be determined and presented, for example, to enable differentiation between successful and failed job executions. In addition, in response to selection of a failed job run, one or more related failed job runs or job runs predicted to fail may be identified. The selected failed job run and related failed job runs or job runs predicted to fail may then be highlighted.

[0006] To achieve the foregoing and related purposes, certain illustrative aspects of the claimed subject matter are described herein in conjunction with the following description and accompanying drawings. These aspects indicate various ways in which the subject matter may be practiced, all of which are intended to be within the scope of the claimed subject matter. Additional advantages and novel features may become apparent from the following detailed description when considered in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is a block diagram of the operating system.

[0008] Figure 2 is a block diagram of representative user interface components.

[0009] Figure 3 is a block diagram of a representative scheduling component.

[0010] Figure 4 is a block diagram of representative scheduler components.

[0011] Figure 5 is a block diagram of a representative monitor component.

[0012] Figure 6 is an exemplary screenshot of an interface for job-driven scheduling and monitoring.

[0013] Figure 7 is an exemplary screenshot of an interface that distinguishes related jobs.

[0014] Figure 8 is an exemplary screenshot of a screen for distinguishing related job failures.

[0015] Fig. 9 is an exemplary screen shot of an interface that produces a chart view and distinguishes related jobs and data.

[0016] Fig.10 is an exemplary screenshot of an interface for data-driven scheduling and monitoring.

[0017] Fig.11 It is a block diagram of the operating system.

[0018] Fig.12 is a flow chart of a method for scheduling and monitoring jobs.

[0019] Fig.13 It is a flowchart of a method for distinguishing related tasks.

[0020] Fig.14 is a flow chart of a method for distinguishing between successful and failed job executions.

[0021] Fig.15 is a flow chart that supports a method for troubleshooting a scheduling chain.

[0022] Fig.16 It is a flowchart that distinguishes related jobs and data sets.

[0023] Fig.17 It is a flowchart of a method for data-driven processing of a job.

[0024] Fig.18 is a schematic block diagram illustrating a suitable operating environment for various aspects of the subject disclosure. DETAILED DESCRIPTION

[0025] The following details generally relate to job scheduling and monitoring. A job corresponds to at least one data conversion operation. One or more jobs can be presented on a display in an interface, for example, in conjunction with scheduling and monitoring the execution of a job. For example, a job can be presented on a timeline view to indicate when the job is to be executed, when the job has been executed, and the length of time the job was executed. A job can be selected, and then related jobs are automatically identified based on data dependencies between the jobs and presented to the user. Related jobs can include jobs that depend on the selected job and / or jobs that the selected job depends on. This helps users to efficiently obtain information about job scheduling and interact with the information. In addition to data dependency information, execution status can be captured so that jobs that are executed or predicted to be successfully executed can be distinguished from jobs that fail or are predicted to fail. In addition, a failed job run can be selected, and then related job runs that fail or are predicted to fail can be automatically identified based on data dependencies, status, and scheduling information. Subsequently, the selected failed job run, the related failed job run, or the job run predicted to fail can be highlighted. Therefore, a convenient method for troubleshooting a scheduling chain is provided. In addition, additional views can also utilize at least dependency data. For example, a graph of a job or job pipeline may employ dependency information to identify a job family (lineage), which includes jobs and data sets used by a selected job and jobs and data sources that depend on the selected job. These and other aspects at least help users to efficiently schedule and monitor job execution and reduce errors.

[0026] Various aspects of the subject disclosure are now described in more detail with reference to the accompanying drawings, wherein the same reference numerals generally refer to the same or corresponding elements throughout. However, it should be understood that the drawings and the detailed description related thereto are not intended to limit the claimed subject matter to the particular form disclosed. On the contrary, it is intended to cover all modifications, equivalents, and alternatives that fall within the spirit and scope of the claimed subject matter.

[0027] First reference Figure 1, showing a job system 100. The job system 100 provides a device for scheduling, executing and monitoring jobs including data conversion operations. The job system includes a user interface component 110, a data repository 120, a scheduler component 130, an execution component 140 and a monitor component 150. The user interface component 110 is configured to allow a user to view, specify and control jobs. In one example, the user interface 110 is configured to provide a mechanism for defining jobs or job pipelines. As part of the definition, one or more relationships can be specified, including one or more data sets on which the job depends and output data sets produced by the job. Jobs including data dependency information can be stored in a data repository 120 as a computer-readable storage medium. The scheduler component 130 is configured to schedule jobs for execution in view of the schedule specified about the job and the relationship between the jobs. The schedule can be provided to the user interface for display, and the user can interact with the schedule to obtain specific information and optionally modify the schedule. The scheduler component 130 starts the job processing by the execution component 140. The execution component 140 may include a collection of software and hardware resources that enable the job to be executed on a computer. The results of the job execution can be stored in the data repository 120. In addition, the monitor component 150 can monitor the job execution by the execution component 140. For example, the monitor component 150 can identify the job status, such as whether the execution failed or succeeded. In addition, the monitor component 150 can obtain the computing resource utilization related to the processing job. The data or information obtained by the monitor component 150 can be stored in the data repository 120 and made available to the user interface component 110 for presentation.

[0028] Figure 2 A representative user interface component 110 is depicted in more detail. The user interface component 110 includes an author component 210 and a scheduling component 220. The author component 210 is configured to allow a user to specify a job and a pipeline including a set of one or more related jobs, wherein the output of a first job optionally provides input to a second job. According to an embodiment, the author component 210 can provide an interactive visual workspace or canvas for graphically authoring jobs and pipelines. For example, a data set can be represented as a cylinder and connected by arrows to a cube representing a job, which consumes the data set and produces a modified data set. Basically, a user can draw a relationship diagram between the data set and the job. This results in an intuitive experience that can save time related to understanding the relationship and ultimately specifying the pipeline. According to other aspects, the author component 210 can be implemented as a code editor that accepts program code or specifies one or more dialog boxes between jobs and jobs. The scheduling component 220 is configured to present visualizations about scheduling jobs for execution and monitoring execution.

[0029] Turn your attention to Figure 3, a representative scheduling component 220 is shown in more detail for clarity and understanding. The scheduling component 220 includes a view component 310, an update component 320, a related job component 330, a modification component 340, and a log component 350. The view component 310 is configured to present at least one view of jobs scheduled for execution. According to one aspect, the view can correspond to a timeline that visualizes a set of jobs ordered by time of execution, the executions including scheduled executions and completed executions. Such a timeline can also enable the length of the job execution to be visualized.

[0030] The update component 320 is configured to update or bring up to date the visualization generated by the view component 310 with respect to job execution. For example, the update component 320 can change the visualization to reflect the jobs that have been executed. In one embodiment, the update component 320 can interact with the view component 310 to include a line representing the current time, with scheduled but unexecuted jobs grayed out or dimmed behind the line. In addition, the update component 320 can be used to identify and distinguish between jobs that have been successfully executed and jobs that have failed or may be predicted to fail. For example, successfully executed jobs can be green, while failures can be represented by red.

[0031] According to certain embodiments, the view component 310 and the update component 320 can cooperate to present a split view including a timeline and a visualization of computing resource utilization. In a first portion, a timeline of scheduled jobs can be presented as described above. In a second portion, for example, below the first portion, a visualization such as a graph can be presented that represents resources utilized in alignment with job execution. In this instance, the update component 320 can obtain data about resource utilization and job execution and update the graph accordingly.

[0032] The related jobs component 330 is configured to distinguish jobs based on the relationships between jobs. The relationships may include jobs and data that the selected job depends on or that are dependent on the selected job. In one instance, the selected job may initiate the determination of jobs related to the selected job and visually distinguish these related jobs in the visualization provided by the view component 310. As an example, after receiving a signal that a job or its representation is selected in the visualization, jobs related to the selected job may be determined based on data dependencies and highlighted. Related jobs may be identified from a data repository 120 that stores relationships between jobs. According to one embodiment, a directed graph may be stored that includes jobs as vertices and data sets as directed edges connecting the jobs. In order to determine related jobs, the graph may be traversed forward and backward from the vertex representing the selected job, wherein the forward traversal (e.g., downstream from the selected job) captures jobs and data sets that are dependent on the selected job, and the backward traversal (e.g., upstream from the selected job) identifies jobs and data sets that the selected job depends on. Job runs or instances of jobs may also be distinguished based on characteristics or attributes. For example, if a user selects a job run that has failed to execute successfully, other job runs that are related to the selected job run that has failed or is predicted to fail can be automatically identified and highlighted in the view. Such job runs can be identified based on information about data dependencies between jobs, states associated with the jobs (e.g., failed, successful, ...), and execution schedules. Here, if a job run depends on a job run that has failed to execute successfully, then the job run can be predicted to fail.

[0033] The modification component 340 is configured to implement modification of the job schedule. The modification component 340 is configured to work in conjunction with the view component 310 to obtain user input regarding the modification of the job execution schedule. For example, the modification component 340 can initiate the rescheduling of the job for execution based at least on one or more gestures received from the user as a signal with respect to the presented interactive view. According to a non-limiting example, the user can select an instance of the job that could not be successfully executed, or perform some additional gesture (such as right-click or drag and drop) to reschedule execution.

[0034] The log component 350 is configured to implement the acquisition and display of the execution log. According to one embodiment, the log component 350 can provide a search mechanism by means of an overlaid search pane or panel. In this scenario, the user can specify and submit a search, and receive results about the execution log. According to another embodiment, the log component 350 can automatically generate and submit a query, for example, based on the selection of a specific job run. In response to a specific gesture associated with log file acquisition, the results corresponding to the selected job run can be returned. In this way, the user can quickly reach, for example, the origin from the failure notification. In the event, making the execution log accessible helps troubleshooting, such as resolving failures.

[0035] Figure 4 A representative scheduler component 130 according to a specific implementation is shown. The scheduler component 130 includes an output slice selection component 410, a dependency period determination component 420 and a dependency evaluation component 430, and an execution start component 440. In this context, a data set is generated or consumed by a job or activity that captures a data processing operation, and corresponds to a collection of data on a timeline. Specifically, a data set associates a data segment (referred to as a data slice) with a time period. In other words, a data set consists of a continuum of data slices. Each data slice can have a specific slice length as well as a start time and an end time.

[0036] The output slice selection component 410 is configured to determine the data slices to be generated. The determination is based on the state in the activity period and an optional policy. Each data slice may have a state indicating the availability of the corresponding data. The state may take multiple values, including "pending", which indicates that the data has not yet been generated, "in progress", which indicates that the slice is being generated, "ready", which indicates that the data is ready for consumption, and "failed to execute", which indicates that one or more attempts to generate data have failed. The output slice selection component 410 identifies data slices with a "pending" state within the activity period of the time window that specifies the job to generate its output. These data slices can be identified by executing a query on the data slices filtered by the state and the activity period. In addition, the output slice selection component 410 can also sort the generation of slice execution or data slices based on a policy. Slices with a pending state can be compared with the wall clock time and picked up in a certain order. For example, the policy may indicate that the oldest slices are generated first, or the newest slices are generated first, for example, based on when the state is set to "pending". Finally, the output slice selection component 410 identifies a single slice to be generated, the output slice.

[0037] The dependency period determination component 420 is configured to determine the dependency period. The dependency period is the data time range of the input data required to generate the identified output slice. The dependency period is part of the dependency information defined about the job. For example, a job can be specified to run three hours of data from a first source and one hour of data from a second source. Therefore, if all data sets have an hourly schedule (e.g., the slices are one hour), generating one hour of output slice data requires three hours of data from the first source and one hour of data from the second source.

[0038] The dependency evaluation component 430 is configured to determine whether the dependency of the output slice is satisfied. The dependency evaluation component 430 can identify the input data slices within the previously determined dependency period. In addition, the status of each input slice is obtained, and it is determined whether the status of each slice is "ready", which means that the slice is ready for consumption (for example, the slice is successfully generated or made available externally). If the input slice within the dependency period has a "ready" status, the dependency has been satisfied. Otherwise, the dependency has not yet been satisfied.

[0039] Once the dependency conditions are met, the execution initiation component 440 is configured to initiate the execution of the job to produce the output slice. In other words, the activity execution is triggered on the execution component 140, and the generation of the selected output data slice begins. After the execution is initiated, the status of the output data slice changes from "to be executed" to "executing". If the final output slice is successfully generated, the output status can be set to "ready". This means that downstream activities that consume the slice as their input can now use the slice. If the execution fails, the execution can be retried multiple times based on a predetermined strategy, during which time the status can be set to "retry" to indicate the previous failure and execution retry. If the execution is subsequently unsuccessful, its status can be set to "execution failed". In addition, each execution can be recorded in the run record associated with the output data slice.

[0040] To facilitate further clarity and understanding of the operation of the scheduler component 130, several exemplary scenarios are described. First, consider a scenario in which a job including a data transformation operation has one input and one output. Here, execution is simple. As time goes by, data slices with a status of "to be executed" before the current runtime are picked up and generated when the dependency period is ready. Consider the next scenario in which the job operates on one input and multiple outputs. In this case, the union of data slices from all outputs will be used as the slice pool to be generated. The state change for the output data slice occurs on all output data sets. The rest of the execution is no different from the previous case. Next, consider a job with multiple inputs and multiple outputs. The union of data slices from all outputs is used as the slice pool to be generated again. In order to generate the "to be executed" output slice, the dependency period is determined, and if all slices within the active period have a "ready" state, the job can be executed for the output slice. The state change for the output data slice occurs on all output data sets, and the execution is the same as the first scenario.

[0041] Figure 5 A representative monitor component 150 is depicted in more detail. The monitor component 150 includes a state component 510 and an execution record component 520. The state component 510 monitors the state of jobs and / or the data they generate. For example, the state component can monitor whether output data is successfully generated by the job execution or whether the job execution fails to successfully generate output data. As previously described, failed or successful executions can be presented to the user with the aid of a user interface. The state component 510 can also monitor additional state information, including when the job that generates data is to be executed, in progress, or ready for consumption, etc. The execution record component 520 is configured to obtain data about computer resource utilization during job execution. In one instance, the information can be requested and obtained from a control mechanism such as an operating system. Resource utilization can then be utilized by a user interface to present a segmented view of data utilization including a timeline and aligned with the timeline.

[0042] Figure 6-10 1 are exemplary screenshots illustrating various visualizations generated by user interface component 110 regarding job scheduling and monitoring. These screenshots are intended to aid clarity and understanding of various aspects of the present disclosure and are not intended to limit the claimed subject matter thereto. It should be understood that the screenshots provided depict only one implementation. Various other combinations and arrangements of graphical elements and text are contemplated and are intended to fall within the scope of the appended claims. In addition, it should be understood that various sounds may also be employed in conjunction with the visualizations to assist the user in understanding the job scheduling. By way of example and not limitation, sounds may be played when a job or data is selected or when an execution failure is detected.

[0043] Figure 6 600 . As shown, the interface 600 includes three panels: a source panel 610, a publish panel 620, and a schedule panel 630. The source panel 610 presents multiple available data sets and enables sources to be added or deleted therefrom. It should be understood that the data sets depicted in the source panel 610 can be any data source. For example, some data sets can be associated with local data, while other data sources are associated with network or cloud data repositories. In addition, the data sets can have essentially any structure or format. The publish panel 620 provides a visual representation of the published or consumable data sources after the required transformations are performed.

[0044] The scheduling panel 630 visualizes the scheduling of job execution and the monitored execution results. More specifically, the scheduling panel 630 presents a split view, which includes a timeline view 640 displayed as a Gantt chart and a resource utilization view 650, the Gantt chart includes the start and stop execution time of the job, and the resource utilization view 650 includes a line graph depicting resource consumption aligned with the job execution. Job scheduling is easy to determine based on graphics. For example, here, the first job that performs the deduplication operation is scheduled to be executed every day, and the second job that performs the conditional segmentation is scheduled to be executed every other day. Note that line 660 represents the current time with respect to execution. The job before the line is a job that has been executed, and the job across the line is scheduled for execution at a certain time in the future. In order to highlight this difference, the operation that is scheduled but not yet executed is illustrated in gray, in other words, the operation is grayed out. Once the operation is executed, the operation will no longer be grayed out. In addition, the color difference about the job run can represent additional information. Here, for example, a black job indicates that the execution failed. According to one aspect, the user can select the black job run and reschedule the execution of the job.

[0045] Figure 7 is a screen shot of an interface 700 that may be generated by the user interface component 110. Figure 6Interface 600, interface 700 includes source panel 610, release panel 620, and scheduling panel 630 including timeline view 640 and resource utilization view 650, as described above. However, interface 700 shows the selection of jobs and the highlighting of jobs related to the selection. Here, the second job 710 that performs conditional segmentation is selected by the user, for example, by clicking, touching or other gestures. After the job is selected, the job related to the selected job will be automatically identified and presented visually distinguished from other jobs. In this example, the first job 712 that performs the removal of duplicate operations, the fifth job 714 that performs the cleaning of sales data, the seventh job 716 that performs the union, and the eighth job 718 that performs the sorting operation are identified as related jobs and are highlighted with respect to other jobs. Specifically, the selected job and the related jobs are presented with a white background, and all other non-related jobs are grayed out. Related jobs include jobs that depend on the selected job and jobs that the selected job depends on. Here, the fifth job 714, the seventh job 716, and the eighth job 718 are dependent on the selected job, and the selected job is dependent on the first job 712. These dependencies can be identified based on the positions of the jobs, where jobs presented after the selected job are dependent on the selected job, and jobs located before the selected job are jobs on which the selected job is dependent.

[0046] Figure 8 is a screen shot of an interface 800 that may be presented by the user interface component 110 in response to selection of a failed run of a job. Figure 7 and Figure 8, the interface 800 includes a source panel 610, a release panel 620, and a scheduling panel 630 including a timeline view 640 and a resource utilization view 650, as previously described. The interface 800 also indicates the selection of a specific run of a job that cannot be successfully executed, captured by solid black. Here, the run of the conditional split job is shown at 810. The selection of the failed job run triggers the identification of other related failed job runs that may have caused the selected job run to fail, or other job runs that may have failed, or other job runs that are predicted to fail based on the failure of the selected job run. These related jobs can be identified based on the recorded dependencies between the jobs, the execution status (e.g., failed, successful), and the schedule. After analyzing the dependencies, status, and scheduling information, as shown at 820, the run of the removal of duplicate jobs can be identified as a possible reason for the failure of the selected "conditional split" job run indicated at 810. In addition, as shown at 830, the failure of the run of the "clean up sales conversion" job can be predicted as a result of the failure of the "conditional split" job. In addition, as indicated at 840, the failure of the operation of the "join" job can be predicted based on the failure of one or both of the "conditional segmentation" job and the "clean sales conversion" job. Similarly, as shown at 850, the failure of the "sort" job can be predicted based on the failure of the "conditional segmentation" job, the "clean data conversion" job, or the "join" job. Interface 800 distinguishes the selected failed job run and the related failed job run from other job runs. In other words, the selected and related failed job runs are highlighted. Here, all job runs except the selected and related failed job runs will be grayed out. Of course, opposite, different colors, different fonts, different sizes, etc. can be used for differentiation purposes. This convenient mechanism helps users troubleshoot the scheduling chain. Further, upon selection or other gestures about one of the failed job runs, a dialog search pane (not shown) can be presented, which can enable searching the execution log or log file associated with the job execution. In this way, the user can quickly convert from the identification of the failure and the original point in the log file. In addition, the user can select or gesture about the failed run and reschedule the execution of the job. For example, the user can drag and drop the failed job run to another time point to reschedule execution. As another example, the user can right-click the failed job run to present a dialog box allowing the user to reschedule the job.

[0047] Fig. 9A screenshot of an interface 900 that can be generated by the user interface component 110 is shown. Similar to the previous screenshots, the interface 900 includes a source panel 610 and a publish panel 620, the source panel 610 presents multiple available data sets and enables sources to be added or deleted therefrom, and the publish panel 620 provides a visual representation of the published or consumable data set after the required transformation is performed. In addition, the interface 900 includes a workspace panel 910. The workspace panel 910 enables visual creation of jobs and pipelines with the help of charts. For example, a user can obtain a data set by dragging and dropping a visual representation of a data set from the source panel 610. Next, the data set can be connected to a previously authored job (e.g., automatically created using data preview and / or manually coded), such as by drawing an arrow from a data set representation to a cube representation of a job to indicate that the data set provides input consumed by the job and performs one or more data transformation operations (e.g., sorting, grouping, pivoting, segmenting, filtering...) on it. In addition, the representation of the transformed output can be linked to the representation of the job on the workspace. Thus, a diagram of a job that receives input from a data source and outputs a new data source is displayed, the new data source reflecting the application of one or more transformation operations of the job. Interface 900 allows conversion between this diagram view for job creation and a timeline view for job scheduling and monitoring. In addition, scheduling and monitoring data can be presented within this diagram view in the workspace panel 910. As shown in 920, the representation of the "sort" job includes a split view, which includes a graph of scheduling data and resource utilization. In addition, after receiving a signal to select the job representation presented in the workspace panel 910, or previously with respect to the timeline, the related jobs and data sets are determined and highlighted. Here, the screenshot depicts the selection of the "conditional split" job at 930. After receiving the selection, the dependency data collected about the job and data can be used to identify the job and data on which the selected job depends, and the job and data that depend on the selected job. In other words, a job family including upstream and downstream jobs and data can be determined and then displayed. In addition, the related data and jobs can be visually distinguished or highlighted with respect to other data and jobs. Here, the relevant jobs and data are white, while other data and jobs are gray or dimmed.

[0048] Fig.10 Depicts a screen shot of an interface 1000 that may be produced by the user interface component 110. Similar to Figure 6Interface 600, interface 1000 includes a source panel 610 and a publish panel 620, the source panel 610 presents multiple available data sets and enables sources to be added or deleted therefrom, and the publish panel 620 provides a visual representation of the published or consumable data sets after the required transformations are performed. Interface 1000 also includes a scheduling panel 630, the scheduling panel 630 includes a split view, the split view includes a timeline view and a resource utilization view 650. However, here, the timeline view 1020 is presented in terms of data slice time, which is relative to the processing time for job execution. In other words, the timeline view 1020 shows the time of data rather than the time of processing. For example, consider a situation where hourly processing is to be performed on events. The processing starts with one hour's worth of events, followed by some transformation operations. More specifically, one hour of processing from 9 am to 10 am can start at 10:15 am. Subsequently, some aggregations can be performed at 11 am, merges can be performed with other data at 12 pm, and the results can be completed at 1 pm. Thus, as opposed to the job execution time about how long the job is to run, the timeline view 1020 provides a view of the time of the data and how the data is processed throughout the time period (e.g., day, week, ...). The scheduling panel 630 also includes a line 660 representing the current time of the data processing. The data before the line represents the data that has been generated, and the data after the line represents the data that is scheduled for production at a future time. In order to highlight this difference, the data that is scheduled but not yet generated is shown in gray, in other words, the operation is grayed out. After the data is generated, the data will no longer be grayed out. In addition, the color difference about the data can represent additional information. For example, black data indicates that the generation of the associated data has failed. According to one aspect, the user can select the black data and reschedule the execution to generate the data. Although not shown, other information (including data status, such as "to be executed", "ready", "in progress" and "failed", etc.) can also be conveyed through the color or other visual features of the data.

[0049] Fig.11 A job system 1100 is depicted. The system 1100 includes a view generation device 1110 that receives input including data transformation jobs and provides a device for generating a view of the data transformation jobs. Hardware, software, or a combination of hardware and software can be used to generate a view, which is a visualization that includes or relates to data transformation jobs. The view generated by the view generation device 1110 can be provided to a display for presentation. In one example, the view generation device 1110 can be or form part of a graphical user interface. According to one embodiment, the generated view can be a diagram of a data transformation job or a job pipeline, which includes a set of one or more related jobs, where the output of a first job optionally provides input to a second job.

[0050] The selection device 1120 is a mechanism that generates a signal indicating the selection of an assignment in the view. The selection device 1120 may be implemented using hardware, software, or a combination of hardware and software. The user may employ an input mechanism to select or otherwise identify an assignment, and the selection device 1120 may receive the input and identify the selected assignment, such as by comparing the location of the user input with the location of the assignment in the view. The user may employ a variety of input mechanisms, including but not limited to a touch pad, a mouse, a touch screen, a camera, or a microphone.

[0051] The related job means 1130 provides a mechanism for automatically identifying jobs related to the selected job. Given the selected job provided by the selection means 1120, the related job means may receive, retrieve or otherwise obtain or acquire dependency information about the job. Subsequently, the related job means may identify the related jobs, for example, by finding the selected job and identifying jobs related to the selected job, wherein the related jobs are jobs that depend on the selected job or jobs that the selected job depends on. According to one aspect, dependencies may be captured in terms of job data dependencies, which include input data that the job depends on and output data generated by the job. In one embodiment, dependency data may be captured in a directed graph that includes jobs as vertices and data sets as directed edges connecting the jobs. In order to determine the related jobs, the graph may be traversed forward and backward from the vertex representing the selected job, wherein the forward traversal (e.g., downstream from the selected job) captures jobs and data sets that depend on the selected job, and the backward traversal (e.g., upstream from the selected job) identifies jobs and data sets that the selected job depends on. The related job device 1130 may be implemented in hardware, software, or a combination of hardware and software to identify the related jobs (and data sets), such as a computer executable software component or firmware.

[0052] The view update device 1140 is a mechanism that allows changes to be made to the view. In one example, the view update device includes hardware, software, or a combination of hardware and software, which receives, retrieves, or otherwise obtains or acquires the selected job, one or more related jobs, and optionally related data sets directly from the related job device 1130, or indirectly from a location (e.g., memory, storage device, etc.), and visually distinguishes the selected job and related jobs and data sources from other jobs and data sources presented in the view. In other words, the view update device 1140 can highlight the selected job and related jobs and data sources in the view.

[0053] The scheduling device 1150 provides scheduling functions for job execution. Implemented as hardware, software, or a combination of hardware and software, the scheduling device 1150 can receive, retrieve, or otherwise obtain or acquire one or more jobs, and generate a schedule for job execution based on the definition of the job or information associated with it that provides scheduling information. For example, if the job indicates execution every day, the job or job run can be scheduled every day. Similarly, if the job specifies execution every other day, the job or job run can be scheduled every other day. Of course, the scheduling can also resolve various data dependencies so that the job is scheduled to execute after its input data is available.

[0054] The scheduling means 1150 may make the schedule available to the view generating means. Thus, the view generating means may generate a view of the schedule as a timeline of the scheduled jobs. In this example, the selecting means 1120 may select a job from the timeline view of the schedule. Subsequently, the related job means may determine jobs related to the selected job, and the view updating means 1140 may update the timeline view of the schedule to highlight the selected job and the related jobs.

[0055] The execution device 1160 executes jobs provided or otherwise made available by the scheduling device 1150. The execution device 1160 may employ hardware, software, or a combination of hardware and software to execute a data transformation job on one or more input data sets and produce one or more output data sets reflecting the application of the data transformation operations specified by the job. The execution device 1160 may read input data from a first specified location and then write output data to a second specified location, where a location may be a memory or storage device location.

[0056] The monitoring device 1170 is a mechanism including hardware, software, or a combination of hardware and software that monitors the execution of the job by the execution device 1160. According to one embodiment, the monitoring device 1170 can detect or otherwise determine whether the job is successfully executed or cannot be successfully executed. This can be achieved by monitoring the output of the execution device 1160, and the output of the execution device 1160 includes potential messages about the success or failure of the execution. In one example, the monitoring device 1170 can determine whether the execution of the job is started according to the schedule, and whether the output of the job is generated for judging whether the execution is successful or unsuccessful. In another embodiment, the monitoring device 1170 can monitor the resource utilization related to the execution of the job. In one example, the monitoring device 1170 can be implemented with a mechanism for directly monitoring resources, and the resources include processors, memories, storage devices, and network utilization. Alternatively, the monitoring device 1170 can request and receive such information from an operating system or other system or component that monitors resource utilization.

[0057] The monitoring device 1170 can make the acquired data or information available to the view generation device 1110 for use in generating a view. In one example, the view generation device 1110 can generate a timeline view of the schedule indicating whether the scheduled job run is successful or failed based on the information provided by the monitoring device 1170. In this case, the selection device 1120 can provide a mechanism for selecting the execution run that failed to signal transmission. Subsequently, the related job device 1130 identifies the related job run, and can more specifically identify the related job run that failed or can be predicted to fail based on the dependency on the failed job run. According to one embodiment, in addition to the data dependency information, the related job device can receive operation information, and the operation information includes whether the job is successfully executed or cannot be successfully executed. In one example, the operation information can be stored together with the dependency information. Alternatively, the operation information can be stored and accessed separately. The view update device 1140 can then highlight the selected failed job run, and the related job run that failed or was predicted to fail. In addition, the view generation device 1110 can include generating a visualization of the resource utilization information provided by the monitoring device 1170 aligned with the job run. For example, the view generating means 1110 generates a graph such as a line graph representing resource utilization.

[0058] The above-mentioned systems, architectures, environments, etc. have been described with respect to the interaction between several components. It should be understood that such systems and components may include those components or subcomponents specified therein, some of the specified components or subcomponents, and / or additional components. Subcomponents may also be implemented as components that are communicatively coupled to other components that are not included in the parent component. In addition, one or more components and / or subcomponents may be combined into a single component to provide aggregate functionality. The communication between systems, components, and / or subcomponents may be implemented according to a push and / or pull model. For the sake of brevity, components may also interact with one or more other components that are not specifically described herein but are known to those skilled in the art.

[0059] In addition, the various parts of the above disclosed systems and the following methods may include or employ artificial intelligence, machine learning, or knowledge-based or rule-based components, subcomponents, processes, means, methods or mechanisms (e.g., support vector machines, neural networks, expert systems, Bayesian belief networks, fuzzy logic, data fusion engines, classifiers ...). Such components may, in particular, automate certain mechanisms or processes performed thereby to make parts of the systems and methods more adaptable and efficient and intelligent. As an example and not limitation, the scheduler component 130 and the user interface 110 may employ such mechanisms to determine or infer job scheduling and data presentation, respectively, based on previous interactions and other contextual information.

[0060] In view of the above exemplary system, reference can be made to Figure 12-17Flowcharts of the present invention are provided to better understand the methods that may be implemented in accordance with the disclosed subject matter. Although the methods are shown and described as a series of blocks for the purpose of simplicity of explanation, it is to be understood and appreciated that the claimed subject matter is not limited by the order of the blocks, as some blocks may occur in a different order and / or concurrently with other blocks other than those depicted and described herein. In addition, not all of the illustrated blocks may be required to implement the methods described below.

[0061] refer to Fig.12 , a method 1200 for job scheduling and monitoring is shown. At reference numeral 1210, a set of jobs are received, including dependencies for each job, the dependencies for each job including at least identification of data on which the job depends and data generated by the job. In other words, a set of jobs are received, including data dependencies specified for each job. At reference numeral 1220, a schedule for processing the set of jobs is generated based on the dependencies and other information about the execution time and frequency defined by each job. At reference numeral 1230, the generated schedule is displayed. According to one aspect, the schedule can be present in a Gantt chart, which includes the start and stop execution times of the job or the data generated by the job. At reference numeral 1240, execution of a job or job run can be initiated according to the schedule. For example, the job can be provided to the execution component for processing. At 1250, the success or failure of the processing is determined with respect to the job whose execution is initiated. Such a determination can be made only by receiving a notification about the execution status from the execution component or retrieving the execution status from a location such as a memory or disk. At reference numeral 1260, the success or failure of the job run is displayed with respect to the schedule. For example, a successfully executed job run can be green, and a failed job run can be red to provide a clear indication of the success or failure of the job run. In addition, jobs can be predicted to succeed or fail based on their dependencies and colored accordingly.

[0062] Fig.13A method 1300 for distinguishing related jobs is depicted. At reference numeral 1310, a signal is received from a schedule to select a job including one or more job runs. For example, a job such as "remove duplicates" can be selected from a timeline view showing one or more jobs and job runs scheduled for execution, for example by left clicking or touching the job. At reference numeral 1320, jobs that depend on the selected job are identified. At reference numeral 1330, jobs that the selected job depends on are identified. The identification of jobs can be based on an analysis of dependencies. According to one embodiment, dependencies can be captured as a directed graph with jobs as vertices and data sets as directed edges connecting jobs. In this instance, the selected job can be identified in the graph, and the analysis can proceed forward from the selected job to identify jobs that depend on the selected job and backward to identify jobs that the selected job depends on. At reference numeral 1340, the selected job and the identified job are highlighted. In other words, for example, on the schedule, the selected job and the identified job are visually distinguished from other jobs displayed. In this way, the user can quickly understand the relationship between jobs.

[0063] Fig.14A method 1400 for distinguishing between successful and failed job executions is shown. At reference numeral 1410, execution of a job run is initiated. Here, a job run is an instance of a job that is scheduled to run at a specific time. At reference numeral 1420, job runs that are dependent on the execution of the job run whose execution is initiated are identified. Dependency information can be used to identify dependent job runs. For example, dependency information can be recorded in a directed graph of jobs, the directed graph including jobs as vertices and data sets connected as directed edges connecting the jobs. The job whose execution is initiated can be identified in the graph, and the dependent jobs can be identified by traversing the graph downward from the job. Subsequently, if the scheduled execution of the dependent job is recorded for each job, the scheduled execution can be determined from the schedule or from the graph. At reference numeral 1430, it is determined whether there is a failure with respect to the execution of the job run. If there is no failure ("no"), the method proceeds to 1440, where successful execution is recorded for the job run. According to one aspect, operation dependency information is recorded, which includes the job status in addition to the dependency information. Successful execution is a work state that can be saved in this aspect. Next, at 1450, with respect to the executed job and optionally for the dependent jobs, a successful execution is displayed as a prediction of successful execution. For example, the job run displayed on the schedule can be green. If it is determined at 1430 that the execution failed ("yes"), the method continues at 1460, where the execution failure is recorded. As described above, failure is a work state that can be saved with respect to retaining operation dependency information. The method then proceeds to 1470, where a display of the failure of the executed job is indicated, for example, on the schedule. In addition, if the executed job fails, the jobs that depend on the executed job may also fail. Therefore, the dependent job runs can be displayed in a manner that indicates that the failure is predicted. For example, the executed job run and the dependent job run can be red to indicate failure and predicted failure, respectively.

[0064] Fig.15A method 1500 for supporting troubleshooting a scheduling chain is shown. At reference numeral 1510, a signal is received to select a job run that failed or was predicted to fail, or more specifically, where, for example, a failure is indicated with respect to the schedule. For example, the schedule may include multiple jobs, and for each job, the job run is scheduled to be executed at a certain time. In addition, for each job run, a visual indication may be provided that indicates that the job run succeeded or failed (if previously executed), or is predicted to succeed or fail (if not yet executed). The selection signal may be associated with one of the job runs indicating a failure, and is generated, for example, when a user clicks or touches a visual representation of the job run. At reference numeral 1520, a related job run that failed or was predicted to fail is identified. The related job runs include job runs that depend on the selected job run and job runs that the selected job run depends on. The related jobs may be identified by analyzing dependency information about the jobs. As an example, the selected job may be identified in a directed graph, the directed graph including jobs as vertices and data sets connected as directed edges connecting the jobs. Subsequently, the graph may be traversed from upstream of the selected job and from downstream of the selected job to identify the related jobs. According to one embodiment, job runs and status can be recorded in conjunction with dependency information. Thus, when identifying related jobs, job runs can be identified, and job runs that include actual or predicted failure status can be identified. Of course, some information including job runs can be saved with respect to scheduling and used to determine related job runs. At reference numeral 1530, the selected job run and related job runs are highlighted. In other words, the selected and related job runs can be visually distinguished from other job runs. This can allow a user to focus on failures and cascading failures for troubleshooting.

[0065] Fig.1616 is a flow chart of a method 1600 for distinguishing related jobs and data sets. At reference numeral 1610, a signal is received to select a job (or data set). For example, the signal may be received when a user selects a job by means of one or more predetermined gestures in a scheduling view or in a schematic diagram of jobs and data sets. At reference numeral 1620, related jobs and data sets are identified. According to an embodiment, dependency information about jobs and data sets may be saved and consulted to identify related jobs and data sets, including jobs and data sets that depend on the selected job (e.g., downstream) and jobs and data sets that the selected job depends on (e.g., upstream). In one example, a directed graph may include jobs as vertices and data sets as directed edges connecting jobs. The selected job (or data set) may be identified in a graph, and the graph is traversed to identify jobs and data sets that depend on the selected job and jobs and data sets that the selected job (or data set) depends on. At reference numeral 1630, the selected job (or data set) and related jobs and data sets are highlighted on a visual display. In other words, the selected and related jobs and data sets are visually distinguished from other jobs and data sets. For example, in a diagrammatic view of specified jobs and datasets, subsets of related jobs and datasets can be visually distinguished. This is particularly helpful for understanding relationships when there are a large number of jobs and datasets and complex pipelines or chains.

[0066] Fig.17A data-driven processing method 1700 for a job is depicted. At reference numeral 1710, an output data slice to be generated is determined, wherein a data slice is a data segment in a data set that is sliced ​​at a certain time period, and the data segment is associated with the time period. In a specific implementation, the output data slice can be identified to analyze the status of the data slice. If the status is "to be executed", etc., the data slice can be determined as the output data slice to be generated. At reference numeral 1720, an output data slice for generation is selected. Here, the selection from the multiple output data slices to be generated can be based on a policy. For example, the policy for job execution can specify that the oldest output data slice is selected first or the latest output data slice is selected first for generation based on when the status is set to "to be executed", etc. At reference numeral 1730, a dependency period is determined. The dependency period is the time range of the input data required to generate the output data slice. This can be determined for selecting the output data slice based on the dependency information defined on the job that generates the output data slice. For example, a job that runs three hours of data from a first source and one hour of data from a second source can be specified. Therefore, if all data sets have hourly schedules (e.g., slices are one hour), generating one hour of output slice data requires three hours of data from the first source and one hour of data from the second source. At reference numeral 1740, it is determined whether all required input data slices in the dependency period are ready. This can be determined by querying the status of the input data slices. If each input data slice has a "ready" status, which means that the data is ready for consumption, the method can proceed to reference numeral 1750. Otherwise, the method can continue to loop and wait for the input data slice to become ready. At reference numeral 1750, the execution of the job that generates the selected output slice is triggered or started. At reference numeral 1760, the status of the output slice can be set. When data is being generated, the status can be set to "in progress" to indicate that data slices are being generated for the job. Once the execution is completed (or cannot be executed), if the attempt to generate the output slice fails, the status can be changed to "failed", or if the attempt to generate is successful and the data slice is ready for consumption, the status can be changed to "ready". At reference numeral 1770, information about the execution is recorded. For example, the status of output slices and the start and end times of execution may be recorded. Such data may be used in connection with generating information and visualizations for presentation to a user to facilitate understanding and control of job execution.

[0067] The subject disclosure supports various products and processes that perform or are configured to perform various actions related to job scheduling and monitoring. One or more exemplary methods and systems follow.

[0068] A method includes presenting a view of one or more data transformation jobs on a display in an interface; receiving a first signal from a user to select a job from the one or more data transformation jobs by means of the interface; automatically determining one or more related jobs for the selected job based on data dependencies between the jobs; and presenting the selected job and the one or more related jobs visually distinct from one or more other jobs. The method also includes presenting one or more jobs scheduled for execution on a display in the interface. The method also includes presenting one or more previously executed jobs on a display in the interface. The method also includes presenting previously executed jobs visually distinct from scheduled jobs on a display in the interface. The method also includes determining a computational load during execution of the one or more jobs; and presenting a computational load aligned with the one or more jobs on a display in the interface. The method also includes presenting an indication of at least one job that failed to execute successfully on a display in the interface. The method also includes receiving a second signal selecting one of the at least one job runs that failed to execute successfully; automatically determining one or more related job runs related to the selected job run that failed to execute successfully or was predicted to fail based on a dependency on the job run that failed to execute successfully; and highlighting the selected job run and the one or more related job runs on a display in the interface. The method also includes: receiving a second signal selecting one of the at least one jobs that could not be successfully executed; and rescheduling execution of the at least one job in response to the second signal. The method also includes presenting the selected job and one or more related jobs in a graph including representations of the jobs and data sets and connections between the jobs and data sets.

[0069] A method includes performing the following actions using at least one processor configured to execute computer executable instructions stored in a memory: detecting a selection of a data conversion job by means of a graphical user interface; and in response to detecting the selection of the job, automatically determining one or more related jobs based on data dependencies between the jobs; and presenting the selected job and the one or more related jobs visually different from other jobs displayed on the graphical user interface. The method also includes monitoring computer resource utilization in conjunction with job execution; and presenting a visualization of resource utilization aligned with one or more corresponding jobs. The method also includes determining the success or failure of execution of a job run; and presenting a failed execution run different from a successful execution run. The method also includes rescheduling the execution of the job run based on a signal received about the job run that could not be successfully executed. The method also includes: after receiving a signal identifying the selection of a failed job run, highlighting the selection of the failed job run and one or more related failed job runs or runs predicted to fail based on dependencies on the failed job run. The method also includes presenting the selected job and the one or more related jobs in a diagram including representations of the job and data sets and connections between the job and the data sets.

[0070] A system includes a processor coupled to a memory, the processor configured to execute the following computer executable components stored in the memory: a first component configured to schedule data transformations for execution based on requirements specified by a job; a second component configured to present the job on a timeline diagram according to the schedule; and a third component configured to highlight one or more jobs related to the selected job based on data dependencies in response to a signal identifying a selected job on the timeline diagram. The third component is also configured to highlight one or more jobs that are dependent on the selected job. The third component is also configured to highlight one or more jobs that the selected job depends on. The system also includes a fourth component configured to detect whether the job execution is successful or unsuccessful. The system also includes a fifth component, the fifth component configured to highlight the related unsuccessful job execution or the job execution that is predicted to be unsuccessful based on the dependency on the unsuccessful job execution in response to the selection of a representation of an unsuccessful job execution on the timeline.

[0071] A system includes means for generating a view of a set of data conversion jobs; means for receiving a selection of a job, the selected job, from the set of data conversion jobs; means for automatically determining one or more related jobs of the selected job based on dependency information; and means for highlighting the selected job and the one or more related jobs in the view. The system also includes means for generating a schedule for executing the set of data conversion jobs. The means for generating the view generates a view of the schedule. The system also includes means for executing one of the set of data conversion jobs according to the schedule. The system also includes means for detecting the success or failure of the execution of one of the set of data conversion jobs. The means for generating the view generates a view of the schedule, the view of the schedule including an indication of the success or failure of the job execution. The means for receiving the selection receives a selection of a failed job execution, and the means for highlighting highlights one or more related job executions that failed or were predicted to fail based on a dependency on the failed execution.

[0072] Various aspects of the present disclosure relate to data conversion, or in other words, to placing data in a usable form for subsequent publication or consumption by an analysis tool. More specifically, various aspects relate to the technical problem of scheduling and monitoring data conversion jobs. Technical means for solving this problem include determining jobs related to a job selected by a user based on job dependencies, including jobs that depend on the selected job and / or jobs that the selected job depends on. The selection of jobs is implemented in conjunction with a graphical user interface, and the selected job and the determined related jobs are highlighted, which helps the user understand the relationship between jobs and troubleshooting. Therefore, the technical effects include but are not limited to improving user efficiency with respect to job scheduling and error reduction.

[0073] The word "exemplary" or its various forms is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be construed as being preferred or advantageous over other aspects or designs. In addition, examples are provided only for the purpose of clarity and understanding, and are not intended to limit or restrict the claimed subject matter or the relevant parts of the present disclosure in any way. It should be understood that numerous additional or alternative examples of various scopes may be proposed, but have been omitted for the sake of brevity.

[0074] As used herein, the terms "component" and "system" and their various forms (e.g., component, system, subsystem, ...) are intended to represent a computer-related entity, either hardware, a combination of hardware and software, software, or software execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an instance, an executable file, an execution thread, a program, and / or a computer. As an illustration, both an application running on a computer and a computer can be a component. One or more components can reside within a process and / or execution thread, and a component can be localized on one computer and / or distributed between two or more computers.

[0075] As used in this specification and the appended claims, the conjunction "or" is intended to mean an inclusive "or" rather than an exclusive "or," unless otherwise specified or clear from the context. In other words, "X" or "Y" is intended to mean any inclusive permutation of "X" and "Y." For example, if "'A' employs 'X'," "'A' employs 'Y'," and "'A' employs both 'X' and 'Y'," then in any of the foregoing cases, "'A' employs 'X' or 'Y'" satisfies.

[0076] Furthermore, where the terms “include,” “contain,” “has,” “having,” or variations thereof, are used in either the detailed description or the claims, these terms are intended to be inclusive in a manner similar to that the term “comprising” is interpreted when used as a transitional word in a claim.

[0077] To provide context for the claimed subject matter, Fig.18 The following discussion is intended to provide a brief general description of a suitable environment in which various aspects of the subject matter may be implemented. However, the suitable environment is merely an example and is not intended to imply any limitation on the scope of use or functionality.

[0078] Although the above disclosed systems and methods can be described in the general context of computer executable instructions of programs running on one or more computers, those skilled in the art will recognize that these aspects can also be implemented in conjunction with other program modules, etc. Typically, program modules include routines, programs, components, data structures, etc. that perform specific tasks and / or implement specific abstract data types. In addition, those skilled in the art will understand that various computer system configurations can be used to practice the above systems and methods, including single processors, multi-processor or multi-core processor computer systems, micro-computing devices, mainframe computers, and personal computers, handheld computing devices (e.g., personal digital assistants (PDAs), phones, watches...), microprocessor-based or programmable consumer or industrial electronic devices, etc. Various aspects can also be practiced in a distributed computing environment, where tasks are performed by remote processing devices linked through a communication network. However, some (if not all) aspects of the claimed subject matter can be practiced on independent computers. In a distributed computing environment, program modules can be located in one or both of local and remote memory devices.

[0079] refer to Fig.18 , an example general purpose computer or computing device 1802 (e.g., a desktop computer, a laptop computer, a tablet computer, a watch, a server, a handheld computer, a programmable consumer or industrial electronic device, a set-top box, a gaming system, a computing node, etc.) is shown. The computer 1802 includes one or more processors 1820, a memory 1830, a system bus 1840, a mass storage device 1850, and one or more interface components 1870. The system bus 1840 at least communicatively couples at least the above-mentioned system components. However, it should be understood that in its simplest form, the computer 1802 may include one or more processors 1820 coupled to the memory 1830, which execute various computer executable actions, instructions and / or components stored in the memory 1830.

[0080] One or more processors 1820 can be implemented with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination thereof designed to perform the functions described herein. A general purpose processor can be a microprocessor, but in an alternative, the processor can be any processor, controller, microcontroller, or state machine. One or more processors 1820 can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, a multi-core processor, one or more microprocessors in combination with a DSP core, or any other such configuration. In one embodiment, one or more processors can be a graphics processor.

[0081] Computer 1802 may include or otherwise interact with various computer-readable media to support control of computer 1802 to implement one or more aspects of the claimed subject matter. Computer-readable media can be any available media that can be accessed by computer 1802, and includes both volatile and nonvolatile media and removable and non-removable media. Computer-readable media can include two distinct and mutually exclusive types, namely computer storage media and communication media.

[0082] Computer storage media include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media include storage devices such as memory devices (e.g., random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM) ...), magnetic storage devices (e.g., hard disks, floppy disks, magnetic cassettes, tapes ...), optical disks (e.g., compact disks (CDs), digital versatile disks (DVDs) ...), and solid-state devices (e.g., solid-state drives (SSDs), flash drives, (e.g., cards, sticks, key drives ...) ...), or any other similar media that, as opposed to transporting or communicating, stores the desired information accessible to the computer 1802. Therefore, computer storage media excludes modulated data signals and things that include communication media.

[0083] Communication media implements computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" refers to a signal that has one or more of its characteristics set or changed in a manner capable of encoding information in the signal. By way of example, and not limitation, communication media include wired media such as a wired network or direct wired connection, and wireless media such as acoustic, RF, infrared and other wireless media.

[0084] The memory 1830 and the one or more mass storage devices 1850 are examples of computer-readable storage media. Depending on the exact configuration and type of computing device, the memory 1830 may be volatile (e.g., RAM), non-volatile (e.g., ROM, flash memory, etc.), or some combination of the two. As an example, a basic input / output system (BIOS) (including basic routines to transfer information between elements within the computer 1802, such as during startup) may be stored in non-volatile memory, while volatile memory may be used as external cache memory to support processing by one or more processors 1820, etc.

[0085] The one or more mass storage devices 1850 include removable / non-removable, volatile / non-volatile computer storage media for storing large amounts of data relative to the memory 1830. For example, the one or more mass storage devices 1850 include, but are not limited to, one or more devices such as a magnetic or optical disk drive, a floppy disk drive, a flash memory, a solid state drive, or a memory stick.

[0086] The memory 1830 and the one or more mass storage devices 1850 may include or store an operating system 1860, one or more applications 1862, one or more program modules 1864, and data 1866. The operating system 1860 is used to control and allocate resources of the computer 1802. The application 1862 includes one or both of the system and application software, and can utilize the management of resources by the operating system 1860 through the program module 1864 and the data 1866 stored in the memory 1830 and / or the one or more mass storage devices 1850 to perform one or more actions. Therefore, the application 1862 can convert the general-purpose computer 1802 into a special-purpose machine according to the logic provided thereby.

[0087] All or part of the claimed subject matter may be implemented using standard programming and / or engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computer to implement the disclosed functionality. By way of example and not limitation, the operating system 100 or a portion thereof may be or form part of an application 1862 and include one or more modules 1864 and data 1866 stored in memory and / or one or more mass storage devices 1850, the functionality of which may be implemented when executed by one or more processors 1820.

[0088] According to a specific embodiment, one or more processors 1820 may correspond to a system on chip (SOC) or similar architecture that includes or in other words integrates hardware and software on a single integrated circuit substrate. Here, one or more processors 1820 may include one or more processors and a memory that is at least similar to one or more processors 1820 and memory 1830, etc. Traditional processors include a minimum amount of hardware and software and rely extensively on external hardware and software. In contrast, the SOC implementation of the processor is more powerful because hardware and software are embedded therein, which enables specific functions to be implemented with minimal or no reliance on external hardware and software. For example, the operating system 100 and / or associated functions can be embedded in hardware in the SOC architecture.

[0089] The computer 1802 also includes one or more interface components 1870 that are communicatively coupled to the system bus 1840 and support interaction with the computer 1802. As an example, the interface component 1870 can be a port (e.g., serial, parallel, PCMCIA, USB, FireWire, ...) or an interface card (e.g., sound, video, ...), etc. In one example implementation, the interface component 1870 can be implemented as a user input / output interface to enable a user to input commands and information into the computer 1802, such as through one or more gestures or voice input, through one or more input devices (e.g., a pointing device such as a mouse, trackball, stylus, touchpad, keyboard, microphone, joystick, game pad, satellite dish, scanner, camera, other computer, ...). In another example implementation, the interface component 1870 can be implemented as an output peripheral interface to provide output to a display (e.g., LCD, LED, plasma, ...), a speaker, a printer, and / or other computers, etc. In addition, the interface component 1870 can be implemented as a network interface to enable communication with other computing devices (not shown), such as through a wired or wireless communication link.

[0090] The above description includes examples of various aspects of the claimed subject matter. Of course, for the purpose of describing the claimed subject matter, it is not possible to describe every conceivable combination of components or methods, but one of ordinary skill in the art will recognize that many additional combinations and permutations of the disclosed subject matter are possible. Therefore, the disclosed subject matter is intended to encompass all such changes, modifications, and variations that fall within the spirit and scope of the appended claims.

Claims

1. A method performed by a system, the system comprising at least one processor, the at least one processor being coupled to a memory storing machine executable instructions, the machine executable instructions, when executed by the processor, controlling the system to perform an action, the action include: for display on a display device, conveying a timeline view of a data transformation job schedule in terms of one or more data slices, the data slices comprising data segments of a continuum, the continuum comprising a data set, wherein each data slice is generated or consumed by a data transformation job and comprises a length and a start time and an end time; receiving a first signal from a user selecting a data slice from the one or more data slices in the view; Based on data dependencies, automatically determining one or more related data slices for the data slice selected by the user, wherein the one or more related data slices include at least one of the following: one or more data slices that depend on the data slice selected by the user or one or more data slices on which the data slice selected by the user depends; as well as For display on the display device, convey a visualization of the user-selected data slice and the one or more related data slices that is visually distinct from one or more other data slices in the timeline view. 2 . The method of claim 1 , wherein the actions further comprise, for display on the display device, communicating one or more data slices scheduled for generation by the job. 3 . The method of claim 1 , wherein the actions further comprise, for display on the display device, communicating one or more data slices previously generated by a job. 4 . The method of claim 1 , wherein the actions further comprise, for display on the display device, communicating a previously generated data slice that is visually distinct from a data slice scheduled for generation by the job.

5. The method according to claim 1, wherein the action further include: During generation of the one or more data slices, determining a computational load; as well as The computational load aligned with the one or more data slices is communicated for display on the display device. 6 . The method of claim 1 , wherein the actions further comprise, for display on the display device, communicating an indication that at least one data slice could not be successfully generated.

7. The method according to claim 6, wherein the action further include: receiving a second signal for selecting one of the at least one data slice that could not be successfully generated; Automatically determine one or more related data slices related to the selected data slice, wherein the related data slices include at least one of the following: one or more data slices that the selected data slice depends on and cannot be successfully generated or one or more data slices that are predicted to be unsuccessfully generated based on the dependency on the selected data slice; as well as For display on the display device, highlighted versions of the selected data slice and the one or more related data slices are communicated.

8. The method according to claim 6, wherein the action further include: receiving a second signal for selecting one of the at least one data slice that could not be successfully generated; as well as In response to the second signal, generation of the at least one data slice is rescheduled.

9. The method of claim 1, wherein the action further comprises, for display on the display device, conveying the data slice selected by the user and the one or more related data slices in a job authoring graph including representations of jobs and data sets and connections between the jobs and the data sets.

10. A computer system, include: A processor, coupled to the memory, the processor being configured to execute machine-executable instructions stored in the memory, the instructions when executed performing actions including: conveying, for display on a display device, a timeline view of a data transformation job schedule in terms of one or more data slices, the data slices comprising data segments of a continuum, the continuum comprising a data set, wherein each data slice is generated or consumed by a data transformation job and comprises a length and a start time and an end time; detecting a first signal from a user to select a data slice from the one or more data slices in the view; Based on data dependencies, automatically determining one or more related data slices for the data slice selected by the user, wherein the one or more related data slices include at least one of the following: one or more data slices that depend on the data slice selected by the user or one or more data slices on which the data slice selected by the user depends; as well as For display on the display device, a visualization of the user-selected data slice and the one or more related data slices is conveyed that is visually distinct from one or more other data slices in the view.

11. The system of claim 10, wherein the action further include: Combined with data slice generation, monitoring computer resource utilization; as well as A visualization of the resource utilization aligned with one or more corresponding data slices is conveyed for display on the display device.

12. The system of claim 10, wherein the action further include: Determine the success or failure of data slice generation; as well as Renders a failed data slice generation different from a successful data slice generation.

13. The system of claim 10, wherein the actions further comprise rescheduling generation of the data slice based on a received signal regarding failed data slice generation.

14. The system of claim 10, wherein the action further include: detecting a second signal for selecting one of the at least one data slices for which generation failed; automatically determining one or more dependent data slices that are related to the selected data slice and predicted to fail based on a dependency on the selected data slice; as well as For display on the display device, highlighted versions of the selected data slice and the one or more related data slices are communicated.

15. A system according to claim 10, wherein the action further includes, for display on the display device, conveying the data slice selected by the user and the one or more related data slices in a job authoring diagram, the diagram including representations of jobs and data sets and connections between the jobs and the data sets.

16. A computer-readable storage medium having instructions stored thereon, the instructions enabling at least one processor to perform a method when the instructions are executed, the method include: conveying, for display on a display device, a timeline view of a data transformation job schedule in terms of one or more data slices, the data slices comprising data segments of a continuum, the continuum comprising a data set, wherein each data slice is generated or consumed by a data transformation job and comprises a length and a start time and an end time; receiving a first signal from a user selecting a data slice from the one or more data slices in the view; Based on data dependencies, automatically determining one or more related data slices for the data slice selected by the user, wherein the one or more related data slices include at least one of the following: one or more data slices that depend on the data slice selected by the user or one or more data slices on which the data slice selected by the user depends; as well as For display on the display device, a visualization of the user-selected data slice and the one or more related data slices is conveyed that is visually distinct from one or more other data slices in the view.

17. The computer-readable storage medium of claim 16, wherein the method further comprises, for display on the display device, communicating a previously generated data slice that is visually distinct from a data slice scheduled for generation.

18. The computer-readable storage medium of claim 16, wherein the method further comprises: include: During generation of the one or more data slices, determining a computational load; as well as The computational load aligned with the one or more data slices is communicated for display on the display device.

19. The computer-readable storage medium of claim 16, wherein the method further comprises, for display on the display device, communicating an indication that at least one data slice could not be successfully generated.

20. The computer-readable storage medium of claim 16, wherein the method further comprises: include: detecting a second signal for selecting one of the at least one data slice that could not be successfully generated; Automatically determine one or more related data slices related to the selected data slice, wherein the related data slices include at least one of the following: one or more data slices that the selected data slice depends on and cannot be successfully generated or one or more data slices that are predicted to be unsuccessfully generated based on the dependency on the selected data slice; as well as For display on the display device, highlighted versions of the selected data slice and the one or more related data slices are communicated.

Images