Data processing method, device, equipment and storage medium
By generating and updating task operation charts, and automatically sorting out and managing the hierarchical relationships and cycle types of ETL tasks, the problems of low efficiency and poor accuracy of manual manual sorting in the existing technology are solved, and the operation and maintenance efficiency of re-running of ETL tasks historical data is improved. and accuracy.
Patent Information
- Application Number
- CN202011042041.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-28
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-09-28
AI Technical Summary
The existing scheme for re-running historical data of existing ETL tasks relies on manual manual sorting of tasks and upstream and downstream relationships, which are inefficient and prone to errors, making it difficult to ensure data accuracy.
By generating a task operation diagram, automatically sorting out the hierarchical relationships and cycle types of batch tasks, updating the diagram in response to task modification instructions, and running tasks based on the updated diagram, displaying the operation diagram for managers to view, realizing automated batch task management.
It improves the operation efficiency and management convenience of batch tasks, reduces manual intervention, enhances the accuracy of task relationships and overall operation and maintenance efficiency, and is suitable for ETL task historical data re-run scenarios.
Smart Images

Figure CN112131297B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology. Specifically, the present application relates to a data processing method, device, equipment and storage medium. Background Art
[0002] Existing solutions for re-running historical data of ETL (Extract-Transform-Load) tasks mostly use manual methods, where managers manually sort out the re-run tasks and their upstream and downstream relationships, and record the current business date of the tasks. This is inefficient and prone to errors in task relationship sorting and task omissions, and data accuracy cannot be guaranteed. Summary of the invention
[0003] In view of the shortcomings of the existing methods, the present application proposes a data processing method, device, equipment and storage medium to solve the technical problems in the existing technology that the manual method has low efficiency and difficulty in ensuring accuracy.
[0004] In a first aspect, an embodiment of the present application provides a data processing method, including:
[0005] In response to the task scope, cycle type and business date range of the set batch task, a task operation diagram is generated; the task operation diagram includes: the hierarchical relationship of each task in the batch task, and the cycle type and business date range of each task in the batch task;
[0006] In response to a task modification instruction for at least one task in the task execution diagram, updating the task execution diagram;
[0007] Run batch tasks according to the updated task running graph and display the updated task running graph.
[0008] In a second aspect, an embodiment of the present application provides a data processing device, including:
[0009] The operation diagram generation module is used to generate a task operation diagram in response to the task scope, cycle type and business date range of the set batch task; the task operation diagram includes: the hierarchical relationship of each task in the batch task, and the cycle type and business date range of each task in the batch task;
[0010] An operation diagram updating module, configured to update a task operation diagram in response to a task modification instruction for at least one task in the task operation diagram;
[0011] The information display module is used to run batch tasks according to the updated task operation diagram and display the updated task operation diagram.
[0012] In a third aspect, an embodiment of the present application provides a data processing device, including:
[0013] Memory;
[0014] A processor, electrically connected to the memory;
[0015] The memory stores a computer program, and the computer program is executed by the processor to implement the data processing method provided in the first aspect of the embodiment of the present application.
[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the data processing method provided in the first aspect of the embodiment of the present application.
[0017] The technical solution provided in the embodiments of the present application has at least the following beneficial effects:
[0018] The embodiment of the present application can automatically sort out the hierarchical relationship between tasks based on the task scope, cycle type and business date range of the set batch tasks, and automatically generate a task operation diagram in combination with the received modification instructions, and then batch run the tasks within the task scope according to the task operation diagram, and display the task operation diagram for management personnel to view and manage. There is no need for management personnel to perform manual sorting and control, which can improve the efficiency and preparedness of running batch tasks, as well as the convenience and efficiency of maintaining and managing batch tasks. It is more suitable for application scenarios where historical data of ETL tasks are re-run, and can improve the overall operation and maintenance efficiency.
[0019] Additional aspects and advantages of the present application will be partially given in the following description, which will become apparent from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0021] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;
[0022] Figure 2 It is an example diagram of some tasks in the task operation diagram of an embodiment of the present application;
[0023] Figure 3 It is another example diagram of some tasks in the task operation diagram of an embodiment of the present application;
[0024] Figure 4 It is another example diagram of some tasks in the task operation diagram of an embodiment of the present application;
[0025] Figure 5 A schematic diagram of the structural framework of a data processing device provided in an embodiment of the present application;
[0026] Figure 6 A schematic diagram of the structural framework of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0027] The present application is described in detail below, and examples of embodiments of the present application are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar components or components with the same or similar functions. In addition, if the detailed description of the known technology is unnecessary for the features of the present application shown, it will be omitted. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.
[0028] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.
[0029] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.
[0030] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments.
[0031] The present application embodiment provides a data processing method, such as Figure 1 As shown, the method includes:
[0032] S101 , generating a task operation diagram in response to a set task scope, cycle type and business date range of a batch task.
[0033] The task operation diagram includes: the hierarchical relationship of each task in the batch task (such as upstream and downstream relationship), and the cycle type and business date range of each task in the batch task.
[0034] The embodiment of the present application can combine the cycle type and the business date range to automatically sort out the hierarchical relationship of each task within the task range, thereby generating a task operation diagram without manual sorting.
[0035] The task scope, cycle type and business date range of the batch task in the embodiment of the present application can be set by the administrator in the displayed setting interface. The cycle type of the batch task can be at least one of year, month, day, hour (i.e., hour), minute, and second, and the cycle of each task in the batch task can be the same or different.
[0036] In an example, the administrator can select the interface task that needs to be re-run as Task A through the setting interface, and set the task scope of this batch task to Task A and all its downstream tasks. The cycle type of Task A is time, and the business date range of this batch task is 2019070201 (indicating 1:00 on July 2, 2019) - 2019070202 (indicating 2:00 on July 2, 2019).
[0037] The task operation graph in the embodiment of the present application may be a DAG (Directed acyclic graph).
[0038] S102: In response to a task modification instruction for at least one task in the task execution graph, update the task execution graph.
[0039] The task modification instruction in the embodiment of the present application may include at least one of a date modification instruction, a cycle modification instruction, a task addition instruction, and a task cancellation instruction.
[0040] In an optional embodiment, in response to a date modification instruction for at least one task in a task operation diagram, the business date range of a first task and the business date range of downstream tasks of the first task in the task operation diagram are updated; the first task includes: at least one task targeted by the date modification instruction.
[0041] In the embodiment of the present application, the upstream task of a task refers to another task on which the task depends. The task can be run only after the upstream task of the task is completed. Similarly, the downstream task of a task refers to another task that depends on the task. The downstream task can be run only after the upstream task of the task is completed.
[0042] In one example, if Figure 2As shown, for task AC, task A is the root node, task B depends on task A, and task C depends on task B. Then task A is the upstream task of task B, task B is the downstream task of task A, task B is the upstream task of task C, and task C is the downstream task of task B.
[0043] Reference Figure 2 In the example shown, if the date modification instruction is for task B, task B is the first task, and the business date range of task B in the DAG diagram is updated from the original date range (2019070100-2019070123) to the new date range (2019070122-2019070123) in the date modification instruction, and the original date range (2019070100-2019070123) of task A is maintained unchanged. According to the update of the business date range of task B, the business date range of task C is updated from the original date range (2019070100-2019070123) to the business date range (2019070122-2019070123) consistent with task B, that is, the modification of the business date range of task B only affects its downstream task, namely task C, but does not affect its upstream task, namely task A.
[0044] In another example, if the date modification instruction is for task A and task D, the business date ranges of task A and task D are updated respectively, and the business date ranges of downstream tasks of task A and downstream tasks of task B are updated respectively.
[0045] In another optional embodiment, in response to a cycle modification instruction for at least one task in the task execution diagram, the cycle type of the second task is updated; when the cycle type of the second task conflicts with the cycle type of an adjacent level task of the second task in the task execution diagram, the level of the second task or its adjacent level task is updated; the second task includes: at least one task targeted by the cycle modification instruction.
[0046] The adjacent-level task of the second task may be at least one of an upstream task and a downstream task of the second task.
[0047] Optionally, the conflict between the cycle type of the second task and the cycle type of the task at the adjacent level can be determined according to the set granularity dependency of the cycle type. In one example, if the granularity dependency of the cycle type is set as follows: a task of a coarse-grained type can depend on a task of a fine-grained type (e.g., a task with a daily cycle type can depend on a task with an hourly cycle type), or tasks of the same granularity type can depend on each other (e.g., a task with an hourly cycle type can depend on another task with an hourly cycle type).
[0048] Optionally, when updating the level of the second task or the adjacent level task, the update can be performed according to a preset update strategy, or the conflict between the cycle type of the second task and the cycle type of its adjacent level task can be displayed through an error interface, and the update can be performed in response to the received update instruction for the conflict.
[0049] In another optional embodiment, in response to a task cancel instruction for at least one task in the task execution graph, a third task and at least one of downstream tasks that only depend on the third task are canceled in the task execution graph; in response to a task add instruction for at least one task in the task execution graph, at least one task is added to the task execution graph, and the hierarchical relationship between the tasks in the added task execution graph is updated; the third task includes: at least one task targeted by the task cancel instruction.
[0050] In one example, if Figure 3 As shown, for task AC, task A depends on both task B and task C. If the task targeted by the task cancellation instruction is task C, then task C is the third task. Task C is canceled in the DAG graph. Since this example does not have any task that depends only on task C, only task C needs to be canceled.
[0051] In another example, Figure 2 As shown, if the task targeted by the task cancel instruction is task B, then task B is the third task, and task B is canceled in the DAG graph. Since task C in this example only depends on task B, task C also needs to be canceled.
[0052] In another example, Figure 4 As shown, for tasks AE, both task A and task B depend on task C, task D depends on both task A and task B, and task E depends only on task A. If the task targeted by the task cancellation instruction is task A, then task A is the third task. In the DAG diagram, task A and task E are canceled. Since task D depends not only on task A but also on task B, after canceling task A, task D can still run after task B is completed, so there is no need to cancel task D.
[0053] The above three implementation modes may be executed simultaneously or sequentially, and this embodiment of the present application does not limit this.
[0054] In the embodiment of the present application, when a task is canceled based on a task cancel instruction, the task may be deleted from the task execution diagram or a display mark of the task may be changed (for example, a display item of the task may be grayed out).
[0055] S103, running batch tasks according to the updated task running diagram, and displaying the updated task running diagram.
[0056] Specifically, tasks at each level are executed sequentially according to the hierarchical relationship in the task execution diagram.
[0057] Optionally, the updated task operation graph may be displayed in a task operation monitoring page, and the administrator may view the information of each task through the task operation monitoring page.
[0058] In an optional implementation, at least one level of tasks in the updated task operation diagram can be displayed through the task operation monitoring page. For example, for a task at a certain level, the manager can view the task and choose whether to view the upstream task or downstream task of the task. In response to the view instruction generated by the manager's selection, the upstream task or downstream task of the task is displayed. In this way, tasks of different levels can be displayed layer by layer according to needs.
[0059] Optionally, while displaying the updated task operation diagram, the task instance information and real-time operation status information of each task in the task operation diagram can be displayed; wherein, the task instance information may include information such as the content of the task, the cycle type, the business date range, etc., and the task instance information can be established according to the task operation diagram; the real-time operation status information can be generated according to the status of the task detected in real time, whether the operation has been started, whether the operation has been interrupted (indicating that the operation has been stopped when the task content has not been completed) or the operation has been ended (indicating that the operation has been stopped when the task content has been completed).
[0060] The data processing method provided in the embodiment of the present application can be applied to a BI (Business Intelligence) system, and step S103 and its optional implementation methods can be applied to an ETL system in the BI system to implement the re-running of historical data of an ETL task.
[0061] The embodiment of the present application may allow the administrator to operate the displayed interface or page in a variety of ways, such as clicking, touching, etc. The task modification instructions in the embodiment of the present application may be instructions generated based on the corresponding operations of the administrator.
[0062] In an optional implementation, the data processing method provided in the embodiment of the present application, based on the above steps S101-S103, further includes:
[0063] For each task in the updated task operation diagram, in response to the operation mode of the task being the business date asynchronous mode, when the task ends execution, the start date of the business date range of the task is reset to the current date.
[0064] In the business date asynchronous mode, resetting the start date of the business date range in the above way can ensure the independence of each task run. For the same task, the current run in the ETL system does not affect the next run in the ETL system.
[0065] In another optional implementation, the data processing method provided in the embodiment of the present application, based on the above steps S101-S103, further includes:
[0066] For each task that has been started, check whether the task has been interrupted; when it is detected that the task has been interrupted, display the operation log of the interrupted task; when receiving the restart instruction of the interrupted task, restart the task according to the cycle type and business date range of the task in the task operation diagram.
[0067] The administrator can analyze the reason for the interruption of the task through the displayed operation log, and restart the task through the displayed restart setting option when restart is needed. The task restart instruction can be an instruction generated according to the restart operation of the administrator.
[0068] In another optional implementation, the data processing method provided in the embodiment of the present application, based on the above steps S101-S103, further includes:
[0069] When at least one task in the task execution graph has been started, the received task modification instruction is shielded until all tasks in the task execution graph have finished running; in response to the received batch cancellation instruction, the started tasks in the task execution graph are stopped, and the running plans of the unstarted tasks in the task execution graph are canceled.
[0070] By means of the above method, batch tasks can be frozen during their operation to prevent any task in the batch tasks from being modified and causing data anomalies, thereby improving the stability of batch task operation and the isolation of the batch task movement.
[0071] Based on the same inventive concept, an embodiment of the present application provides a data processing device, such as Figure 5 As shown, the data processing device 500 includes: an operation diagram generation module 501, an operation diagram update module 502 and an information display module 503.
[0072] The operation diagram generation module 501 is used to generate a task operation diagram in response to the task scope, cycle type and business date range of the set batch tasks; the task operation diagram includes: the hierarchical relationship of each task in the batch task, and the cycle type and business date range of each task in the batch task.
[0073] The operation diagram updating module 502 is configured to update the task operation diagram in response to a task modification instruction for at least one task in the task operation diagram.
[0074] The information display module 503 is used to run the batch tasks according to the updated task operation diagram and display the updated task operation diagram.
[0075] In an optional embodiment, the operation diagram update module 502 is specifically used to: in response to a date modification instruction for at least one task in the task operation diagram, update the business date range of the first task and the business date range of the downstream tasks of the first task in the task operation diagram; the first task includes: at least one task targeted by the date modification instruction.
[0076] In another optional embodiment, the operation diagram update module 502 is specifically used to: update the cycle type of the second task in response to a cycle modification instruction for at least one task in the task operation diagram; when the cycle type of the second task conflicts with the cycle type of an adjacent level task of the second task in the task operation diagram, update the level of the second task or the adjacent level task; the second task includes: at least one task targeted by the cycle modification instruction.
[0077] In another optional embodiment, the operation diagram update module 502 is specifically used to: in response to a task cancel instruction for a third task in the task operation diagram, cancel the third task and at least one of the downstream tasks that only depend on the third task in the task operation diagram; in response to a task add instruction for at least one task in the task operation diagram, add at least one task in the task operation diagram, and update the hierarchical relationship between the tasks in the added task operation diagram; the third task includes: at least one task targeted by the task cancel instruction.
[0078] In some embodiments, the operation diagram updating module 502 can execute the above three implementation modes simultaneously.
[0079] Optionally, the data processing device 500 provided in the embodiment of the present application further includes: a date resetting module.
[0080] The date reset module is used to: for each task in the updated task operation diagram, in response to the task operation mode being the business date asynchronous mode, when the task ends operation, reset the start date of the task's business date range to the current date.
[0081] Optionally, the data processing device 500 provided in the embodiment of the present application further includes: a task restart module.
[0082] The information display module 503 is also used to: for each task that has been started, detect whether the task has been interrupted; when it is detected that the task has been interrupted, display the running log of the interrupted task.
[0083] The task restart module is used to: when receiving the restart instruction of the interrupted task, restart the task according to the cycle type and business date range of the task in the task operation diagram.
[0084] Optionally, the data processing device 500 provided in the embodiment of the present application further includes: a task cancellation module.
[0085] The task cancellation module is used to: when at least one task in the task execution graph has been started, shield the received task modification instructions until all tasks in the task execution graph have finished running; in response to the received batch cancellation instructions, stop running the tasks that have been started in the task execution graph, and cancel the running plans of the tasks that have not been started in the task execution graph.
[0086] The data processing device 500 of this embodiment can execute any data processing method provided in the embodiments of the present application, and the implementation principles thereof are similar. The contents not shown in detail in this embodiment can be referred to the method embodiments described above, and will not be repeated here.
[0087] Based on the same inventive concept, an embodiment of the present application provides a data processing device, which includes: a memory and a processor, and the memory is electrically connected to the processor.
[0088] A computer program is stored in the memory, and the computer program is executed by the processor to implement any data processing method provided in the embodiments of the present application.
[0089] Those skilled in the art will appreciate that the data processing equipment provided by the embodiments of the present application can be specially designed and manufactured for the desired purpose, or can also include known devices in general-purpose computers. These devices have computer programs stored therein, which are selectively activated or reconstructed. Such computer programs can be stored in a device (e.g., computer) readable medium or in a medium suitable for storing electronic instructions and coupled to any type of bus respectively.
[0090] In an optional embodiment, the present application provides a data processing device, such as Figure 6 As shown, the data processing device 600 includes: a memory 601 and a processor 602 , and the memory 601 and the processor 602 are electrically connected, such as connected via a bus 603 .
[0091] The memory 601 may be a ROM (Read-Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read-Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.
[0092] The processor 602 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor 602 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.
[0093] The bus 603 may include a path to transmit information between the above components. The bus may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0094] Optionally, the data processing device 600 may further include a transceiver 604. The transceiver 604 may be used for receiving and sending signals. The transceiver 604 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. It should be noted that in actual applications, the transceiver 604 is not limited to one.
[0095] Optionally, the data processing device 600 may further include an input unit 605. The input unit 605 may be used to receive input digital, character, image and / or sound information, or generate key signal input related to user settings and function control of the electronic device 600. The input unit 605 may include, but is not limited to, one or more of a touch screen, a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, a camera, a microphone, etc.
[0096] Optionally, the data processing device 600 may further include an output unit 606. The output unit 606 may be used to output or display information processed by the processor 602. The output unit 606 may include but is not limited to one or more of a display device, a speaker, a vibration device, and the like.
[0097] Although Figure 6 The data processing device 600 is shown with various devices, but it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead.
[0098] Optionally, the memory 601 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 602. The processor 602 is used to execute the application code stored in the memory 601 to implement any data processing method provided in the embodiments of the present application.
[0099] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, any data processing method provided in the embodiment of the present application is implemented.
[0100] The computer readable medium includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM, RAM, EPROM (Erasable Programmable Read-Only Memory), EEPROM, flash memory, magnetic card or optical card. In other words, the readable medium includes any medium that can store or transmit information in a readable form by a device (e.g., a computer).
[0101] The present application embodiment provides a computer-readable storage medium applicable to various optional implementations of any of the above data processing methods, which will not be described in detail here.
[0102] By applying the embodiments of the present application, at least the following beneficial effects can be achieved:
[0103] 1) The embodiment of the present application can automatically sort out the hierarchical relationship between tasks based on the task scope, cycle type and business date range of the set batch tasks, and automatically generate a task operation diagram in combination with the received modification instructions, and then batch run the tasks within the task scope according to the task operation diagram, and display the task operation diagram for management personnel to view and manage. There is no need for management personnel to perform manual sorting and control, which can improve the efficiency and preparedness of running batch tasks, as well as the efficiency of maintaining and managing batch tasks. It is more suitable for application scenarios where ETL task historical data is re-run, and can improve overall operation and maintenance efficiency.
[0104] 2) The embodiment of the present application can modify at least one parameter of the current task hierarchy, cycle type and business date range according to the received task modification instruction, or add or cancel corresponding tasks to meet usage requirements, with high modification efficiency and accuracy.
[0105] 3) The embodiment of the present application can batch run tasks in a business date asynchronous mode, which is more in line with the demand for re-running task history data.
[0106] 4) When a task is interrupted, the embodiment of the present application can display the task's running log to analyze the cause of the interruption, restart the interrupted task in time, and promote the normal operation of batch tasks.
[0107] 5) In order to ensure the isolation of the batch task running environment, the embodiment of the present application can automatically freeze each task in the batch task by shielding the task modification instruction during the batch task running, so as to prevent data anomalies caused by task modification, thereby ensuring the stability of task running and the accuracy of data; it can also stop the running tasks and cancel the running tasks that have not been started in response to the batch cancellation instruction, so as to realize the batch cancellation function and meet the cancellation requirements in special situations.
[0108] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.
[0109] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the feature. In the description of this application, unless otherwise specified, "plurality" means two or more.
[0110] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0111] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A data processing method, characterized in that: include: In response to the task scope, cycle type and business date range of the set batch task, a task operation diagram is generated; The task operation diagram includes: the hierarchical relationship of each task in the batch task, and the cycle type and business date range of each task in the batch task; In response to a task modification instruction for at least one task in the task execution diagram, updating the task execution diagram; Run the batch tasks according to the updated task running diagram, and display the updated task running diagram; For each of the tasks in the updated task operation diagram, in response to the operation mode of the task being a business date asynchronous mode, when the task ends operation, resetting the start date of the business date range of the task to the current date; Wherein, in response to a task modification instruction for at least one task in the task operation graph, updating the task operation graph includes at least one of the following: In response to a date modification instruction for at least one task in the task operation diagram, updating the business date range of a first task and the business date range of a downstream task of the first task in the task operation diagram; the first task includes: at least one task targeted by the date modification instruction; In response to a cycle modification instruction for at least one task in the task execution diagram, the cycle type of a second task is updated; the second task includes: at least one task targeted by the cycle modification instruction; when the cycle type of the second task conflicts with the cycle type of an adjacent level task of the second task in the task execution diagram, the level of the second task or the adjacent level task is updated according to the set granularity dependency of the cycle type.
2. The data processing method according to claim 1, characterized in that: In response to a task modification instruction for at least one task in the task execution diagram, updating the task execution diagram comprises: In response to a task cancellation instruction for at least one task in the task execution graph, canceling at least one of a third task and downstream tasks that are only dependent on the third task in the task execution graph; the third task includes: at least one task targeted by the task cancellation instruction; In response to a task adding instruction for at least one task in the task execution graph, at least one task is added to the task execution graph, and the hierarchical relationship between the tasks in the added task execution graph is updated.
3. The data processing method according to claim 1, characterized in that: Also includes: For each task that has been started, detecting whether the task has been interrupted; When it is detected that the task has been interrupted, displaying the running log of the task; When a task restart instruction is received, the task is restarted according to the cycle type and business date range of the task in the task operation diagram.
4. The data processing method according to claim 1, characterized in that: Also includes: When at least one task in the task running graph has started running, shielding the received task modification instruction until all tasks in the task running graph have finished running; In response to the received batch cancellation instruction, the tasks that have been started in the task running diagram are stopped, and the running plans of the tasks that have not been started in the task running diagram are canceled.
5. A data processing device, characterized in that: include: A running diagram generating module, used for generating a task running diagram in response to the task scope, cycle type and business date range of the set batch task; The task operation diagram includes: the hierarchical relationship of each task in the batch task, and the cycle type and business date range of each task in the batch task; An operation diagram updating module is used to update the task operation diagram in response to a task modification instruction for at least one task in the task operation diagram; update the business date range of a first task and the business date range of downstream tasks of the first task in the task operation diagram in response to a date modification instruction for at least one task in the task operation diagram; the first task includes: at least one task targeted by the date modification instruction; update the cycle type of a second task in response to a cycle modification instruction for at least one task in the task operation diagram; the second task includes: at least one task targeted by the cycle modification instruction; when the cycle type of the second task conflicts with the cycle type of an adjacent hierarchical task of the second task in the task operation diagram, update the hierarchy of the second task or the adjacent hierarchical task according to the set granularity dependency of the cycle type; An information display module, used to run the batch tasks according to the updated task operation diagram and display the updated task operation diagram; The date reset module, for each task in the updated task running diagram, in response to the task running mode being the business date asynchronous mode, resets the start date of the task's business date range to the current date when the task finishes running.
6. A data processing device, characterized in that: include: Memory; A processor, electrically connected to the memory; The memory stores a computer program, and the computer program is executed by the processor to implement the data processing method according to any one of claims 1 to 4.
7. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the data processing method according to any one of claims 1 to 4 is implemented.
