Data processing methods and data processing equipment

Abstract

This disclosure relates to the field of task scheduling technology for big data platforms, and in particular to a data processing method and a data processing device. The method includes: when a preset directed acyclic tree is being executed at the current time, obtaining a first task identifier corresponding to the preset identifier based on a preset identifier of the preset directed acyclic tree; obtaining one or more scheduled tasks contained in a first task list corresponding to the first task identifier based on the first task identifier; inputting the first configuration parameters of each scheduled task into the preset directed acyclic tree respectively, creating a first running instance corresponding to each scheduled task; and running according to the first running instance corresponding to each scheduled task to obtain the first running result corresponding to each scheduled task.

Description

Technical Field

[0001] This disclosure relates to the field of task scheduling technology for big data platforms, and in particular to a data processing method and a data processing device. Background Technology

[0002] In the core architecture of current big data platforms, efficient data scheduling is a crucial link supporting massive data processing. The core functions of a big data platform encompass data collection, storage, computation, application, and management, and its efficient operation depends on the accuracy and timeliness of task scheduling. To achieve this goal, the industry widely adopts workflow scheduling tools like Apache Airflow, which uses its Directed Acyclic Graph (DAG) model to define, schedule, and monitor task execution processes. This design significantly improves the flexibility and maintainability of task scheduling, becoming the standard solution for handling complex dependencies in big data platforms.

[0003] However, with the rapid expansion of big data platform applications, especially when handling massive data tasks, traditional scheduling models face significant challenges. Current mainstream integration methods primarily rely on the platform's scheduling task to unidirectionally generate Airflow job tasks, meaning that an external scheduling system directly creates and manages Airflow DAG instances. While this approach has the advantage of synchronizing the lifecycle of Airflow jobs with that of scheduling tasks, facilitating unified control and resource coordination, its limitations become glaringly apparent in large-scale scenarios. Specifically, when the platform's workload grows exponentially, the centralized scheduling of massive Airflow jobs triggers system-level bottlenecks. On one hand, the scheduler needs to frequently process task requests, leading to a surge in scheduling pressure; on the other hand, intensified resource competition results in a shortage of computing nodes and storage resources. This resource scarcity further slows down task response speed, creating a vicious cycle that ultimately results in the platform's inability to respond to business needs promptly, impacting data processing timeliness and overall efficiency.

[0004] Therefore, improving the task response time of big data platforms has become an urgent problem to be solved. Summary of the Invention

[0005] To address the aforementioned technical problems, this disclosure provides a data processing method and a data processing device.

[0006] In a first aspect, this disclosure provides a data processing device, comprising: a communicator configured to communicate with a client; and a controller configured to: when a preset directed acyclic tree is being executed at the current time, obtain a first task identifier corresponding to a preset identifier based on a preset identifier of the preset directed acyclic tree; wherein one preset identifier corresponds to one first task identifier; obtain one or more timed tasks contained in a first task list corresponding to the first task identifier based on the first task identifier; wherein the timing parameters of timed tasks in the same first task list are matched; input the first configuration parameters of each timed task into the preset directed acyclic tree respectively to create a first running instance corresponding to each timed task; and run according to the first running instance corresponding to each timed task to obtain the first running result corresponding to each timed task.

[0007] Secondly, this disclosure provides a data processing method, comprising: when a preset directed acyclic tree is being executed at the current time, obtaining a first task identifier corresponding to the preset identifier based on the preset identifier of the preset directed acyclic tree; wherein, one preset identifier corresponds to one first task identifier; obtaining one or more timed tasks contained in the first task list corresponding to the first task identifier based on the first task identifier; wherein, the timing parameters of the timed tasks in the same first task list are matched; inputting the first configuration parameters of each timed task into the preset directed acyclic tree respectively, creating a first running instance corresponding to each timed task; running according to the first running instance corresponding to each timed task, and obtaining the first running result corresponding to each timed task.

[0008] Thirdly, this disclosure provides a computer-readable storage medium, comprising: storing a computer program on the computer-readable storage medium, the computer program being executed by a controller using a data processing method as provided in any of the second aspects.

[0009] Fourthly, this disclosure provides a computer program product that, when run on a computer, causes the computer to perform any of the data processing methods provided in the second aspect.

[0010] It should be noted that the aforementioned computer instructions may be stored, in whole or in part, on the first computer-readable storage medium. The first computer-readable storage medium may be encapsulated together with the controller of the data processing device, or it may be encapsulated separately from the controller of the data processing device; this disclosure does not impose any limitations on this.

[0011] The descriptions of the second, third, and fourth aspects in this disclosure can be referenced to the detailed description of the first aspect; and the beneficial effects described in the second, third, and fourth aspects can be referenced to the analysis of the beneficial effects of the first aspect, which will not be repeated here.

[0012] In this disclosure, the names of the aforementioned data processing devices do not limit the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear under other names. As long as the functions of each device or functional module are similar to those of this disclosure, they fall within the scope of this disclosure and its equivalents.

[0013] These or other aspects of this disclosure will become more readily apparent in the following description.

[0014] The technical solution provided in this disclosure has the following advantages compared with the prior art: The data processing device disclosed herein includes: a communicator configured to communicate with a client; and a controller configured to: when a preset directed acyclic tree is being executed at the current time, obtain a first task identifier corresponding to a preset identifier based on a preset identifier of the preset directed acyclic tree; wherein one preset identifier corresponds to one first task identifier; obtain one or more scheduled tasks contained in a first task list corresponding to the first task identifier based on the first task identifier; wherein the timing parameters of the scheduled tasks in the same first task list are matched; input the first configuration parameters of each scheduled task into the preset directed acyclic tree respectively, and create a first running instance corresponding to each scheduled task; run according to the first running instance corresponding to each scheduled task, and obtain the first running result corresponding to each scheduled task. Since the scheduled tasks in the same first task list share a preset directed acyclic tree, the data processing device does not need to deploy a DAG for each Airflow scheduling task when executing Airflow scheduling tasks, thus greatly reducing the number of DAGs and improving the task response time of Airflow scheduling tasks, thereby solving the problem of how to improve the task response time of big data platforms. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0016] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0017] Figure 1 One of the flowcharts illustrating the data processing method provided in the embodiments of this application; Figure 2 A second schematic flowchart illustrating the data processing method provided in this application embodiment; Figure 3The third schematic flowchart of the data processing method provided in the embodiments of this application; Figure 4 The fourth flowchart illustrating the data processing method provided in this application embodiment; Figure 5 Fifth flowchart illustrating the data processing method provided in the embodiments of this application; Figure 6 A flowchart illustrating the data processing method provided in this application embodiment is shown in Figure 6. Figure 7 The seventh flowchart illustrating the data processing method provided in the embodiments of this application; Figure 8 Eighth schematic flowchart of the data processing method provided in the embodiments of this application; Figure 9 A flowchart illustrating the data processing method provided in this application embodiment is shown in Figure 9. Figure 10 This is a schematic diagram of the structure of the data processing device provided in the embodiments of this application; Figure 11 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation

[0018] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0019] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0020] The data processing device provided in this application can have various implementation forms, such as a television, a smart television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Figure 1 and Figure 2 This is one specific implementation of the data processing device of this application.

[0021] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0022] In some examples, the cron provided in this disclosure is essentially a time scheduling tool for Unix-like systems. It defines the execution time rules of tasks (such as executing every minute or at midnight every day) through cron expressions. Its core function is to trigger tasks according to time, without involving the management of dependencies between tasks. For example, * / 10 * * * * means that the task is executed every 10 minutes, only focusing on "when to execute" and not caring about the order or dependency logic between tasks.

[0023] The data processing device provided in this disclosure can be a server. When the server executes the data processing method provided in this disclosure, it can be the server's processor.

[0024] In the following embodiments, the execution subject of the data processing method provided in the embodiments of this disclosure is the aforementioned server, which will be used as an example to illustrate the method of the embodiments of this application.

[0025] This application provides a data processing method, such as... Figure 1 As shown, the data processing method may include S11-S14.

[0026] S11. When a preset directed acyclic tree is being executed at the current time, the first task identifier corresponding to the preset identifier is obtained based on the preset identifier of the preset directed acyclic tree; wherein, one preset identifier corresponds to one first task identifier.

[0027] In some examples, by enabling Airflow's dynamic task matching feature, the server can dynamically search the database for the first task identifier corresponding to the preset identifier based on the preset identifier of the preset directed acyclic tree currently being executed, and then determine the list of tasks to be executed based on the first task identifier.

[0028] In some examples, the default identifier can be a DAG ID, representing the corresponding Airflow DAG ID; the first task identifier can be record Id1, representing the record ID; and the first task list can be cron-relate, representing the table of relationships between scheduled tasks (cron) and tasks.

[0029] In some examples, a user deactivates or deletes a pre-configured scheduled task or an immediately executed task on the server. In response to the user's target operation on the target task, the server obtains the task type of the target task; if the task type is a scheduled task, it obtains the second task identifier in the second task list where the target task is located; the target operation includes either a deactivation operation or a deletion operation; it removes the association between the target task and the second task identifier, and decrements the first total number (count) of scheduled tasks in the second task list by 1 to obtain the updated total number; if the updated total number is not equal to 0, it performs the target operation on the target task; or, if the updated total number is equal to 0, it calls the deactivation interface to deactivate the preset identifier corresponding to the second task identifier and performs the target operation on the target task. For example, the deactivation interface could be Airflow's DAG deactivation interface; or, if the task type is an immediately executed task, it performs the target operation on the target task.

[0030] In some examples, users can pre-configure tasks to be executed on the server, such as scheduled tasks or immediate tasks. In this case, the server's task creation module responds to the creation operation for creating a new task by obtaining the task type of the new task. If the task type of the new task is an immediate task, it obtains the third task identifier corresponding to the immediate task. Specifically, when the server executes an immediate task for the first time, to ensure that the immediate task can be executed, the third task identifier (e.g., recordId3) corresponding to the immediate task needs to be pre-configured in the server's memory. This way, the server can use recordId3 as the third task identifier for each immediate task executed each time. Simultaneously, when the server executes an immediately executed task for the first time, since the server has not configured a first directed acyclic tree (DAG) corresponding to the immediately executed task, and there is no first identifier corresponding to the third task identifier, a generation interface (e.g., the Airflow interface) is called to generate the first DAG and the first identifier corresponding to the first DAG, and an enable interface is called to enable the first DAG; wherein, the third task identifier corresponds one-to-one with the first identifier; the second configuration parameters of the first DAG are obtained; (the second configuration parameters corresponding to different immediately executed tasks can be the same or different, and can be freely configured by the user, which is not limited here). Based on the third task identifier, the first identifier corresponding to the third task identifier is determined, and the run interface is called to run the first DAG corresponding to the first identifier; the second configuration parameters are input into the first DAG to generate a second run instance (e.g., a task) corresponding to the newly added task; the second run instance corresponding to the newly added task is run to obtain the second run result corresponding to the newly added task. Thus, when the server executes an immediately executed task again, since a first directed acyclic tree for executing immediately executed tasks has already been established, it obtains the second configuration parameters of the first directed acyclic tree when a first identifier corresponding to the third task identifier exists. (Here, the second configuration parameters corresponding to the currently executed immediately executed task are obtained.) Based on the third task identifier, the server determines the first identifier corresponding to the third task identifier and calls the run interface to run the first directed acyclic tree corresponding to the first identifier. The second configuration parameters are input into the first directed acyclic tree to generate a second run instance corresponding to the newly added task. The server runs the second run instance corresponding to the newly added task to obtain the second run result corresponding to the newly added task.

[0031] In some examples, the second configuration parameter includes two categories: default parameters and dynamic parameters passed in at trigger time. Its core function is to ensure that newly added tasks are executed flexibly according to actual needs. Default parameters (params) are set during the DAG definition phase and serve as the basic configuration for task execution; they can be overridden at trigger time. Examples include specifying static configurations such as the default data source path and timeout, ensuring that the task can still run according to preset logic even without dynamic input. Dynamic parameters at trigger time (dag_run.conf) are passed in when triggering the DAG via CLI (such as Airflow dags trigger --conf '{"key": "value"}') or REST API. These parameters have higher priority than params and are used to convey real-time requirements. Examples include dynamically specifying the data processing date and target user ID, allowing the same DAG template to adapt to different scenarios.

[0032] In some examples, users can pre-configure tasks to be executed on the server, such as scheduled tasks or immediate tasks. In this case, the server's task creation module responds to the creation operation for creating a new task, obtaining the task type of the new task. When the task type is a scheduled task, based on the scheduled task's timing parameters (such as start time, number of loops, and loop period, or one or more of these), it obtains one or more fourth task identifiers that match the timing parameters (e.g., tasks with the same start time and the same loop period are considered a match, or tasks with the same start time, the same number of loops, and the same loop period are considered a match), and the total number of second tasks in the third task list corresponding to each fourth task identifier. If the number is less than a threshold... When the total number of the second task is less than the threshold, the new task is added to the third task list corresponding to the total number of the second task. When the total number of the second task is greater than 0, the actual directed acyclic tree corresponding to the fourth task identifier corresponding to the total number of the second task is waited to run periodically. Alternatively, when the total number of the second task is equal to 0 (meaning there is no scheduled task in the third task list corresponding to the fourth task identifier; to reduce the consumption of computing resources, the actual directed acyclic tree is usually disabled by calling the disable interface when the total number of tasks in the task list is equal to 0), since a new task has been added to the third task list corresponding to the fourth task identifier, and the actual directed acyclic tree corresponding to the fourth task identifier has been disabled, the enable interface needs to be called to enable the actual directed acyclic tree, and the actual directed acyclic tree is waited to run periodically. Alternatively, if there is no total number of second tasks less than the threshold, create a fifth task identifier, call the generation interface to generate the second directed acyclic tree corresponding to the fifth task identifier and the second identifier corresponding to the second directed acyclic tree, and call the enable interface to enable the second directed acyclic tree; wherein, the fifth task identifier and the second identifier correspond one-to-one; add the new task to the fourth task list corresponding to the fifth task identifier, and wait for the second directed acyclic tree to run periodically.

[0033] In some examples, the new task can be an Airflow scheduled task.

[0034] S12. Based on the first task identifier, obtain one or more timed tasks contained in the first task list corresponding to the first task identifier; wherein the timed parameters of the timed tasks in the same first task list are matched.

[0035] S13. Input the first configuration parameters of each scheduled task into the preset directed acyclic tree to create the first running instance corresponding to each scheduled task.

[0036] S14. Run each scheduled task according to its first running instance to obtain the first running result for each scheduled task.

[0037] In some examples, in data quality scenarios, there are 3570 tasks directly running in actual business operations, and the corresponding DAG number after executing the data processing method provided in this disclosure embodiment is 1. There are 5565 scheduled tasks, and the corresponding DAG number after executing the data processing method provided in this disclosure embodiment is 117. The deployment response time of Airflow scheduled tasks has decreased from 44 seconds to 173 milliseconds.

[0038] As described above, the data processing method provided in this embodiment, when a preset directed acyclic tree is being executed at the current time, obtains a first task identifier corresponding to the preset identifier based on the preset identifier of the preset directed acyclic tree; wherein, one preset identifier corresponds to one first task identifier; based on the first task identifier, obtains one or more timed tasks contained in the first task list corresponding to the first task identifier; wherein, the timing parameters of the timed tasks in the same first task list are matched; the first configuration parameters of each timed task are respectively input into the preset directed acyclic tree to create a first running instance corresponding to each timed task; and the first running instance corresponding to each timed task is run to obtain the first running result corresponding to each timed task. Since the timed tasks in the same first task list share a preset directed acyclic tree, the data processing device does not need to deploy a DAG for each Airflow scheduling task when executing Airflow scheduling tasks, thus greatly reducing the number of DAGs and improving the task response time of Airflow scheduling tasks.

[0039] In some feasible examples, combining Figure 1 ,like Figure 2 As shown, the data processing method provided in this embodiment of the disclosure further includes S15-S18.

[0040] S15. In response to the target operation on the target task, obtain the task type of the target task; S16. When the target task is a scheduled task, obtain the second task identifier of the second task list where the target task is located; wherein, the target operation includes either the deactivation operation or the deletion operation; S17. Remove the association between the target task and the second task identifier, and decrement the total number of first tasks in the second task list by 1 to obtain the updated total. S18. When the total number of updates is not equal to 0, perform the target operation on the target task.

[0041] As described above, the data processing method provided in this embodiment, in response to a target operation on a target task, obtains the task type of the target task; when the task type of the target task is a scheduled task, obtains the second task identifier of the second task list in which the target task is located; removes the association between the target task and the second task identifier, and decrements the total number of the first task in the second task list by 1 to obtain the updated total number; when the updated total number is not equal to 0, performs the target operation on the target task. Since each scheduled task shares a preset directed acyclic tree, the data processing device does not need to deploy a DAG for each Airflow scheduling task when executing Airflow scheduling tasks, thus greatly reducing the number of DAGs and improving the task response time of Airflow scheduling tasks.

[0042] In some feasible examples, combining Figure 2 ,like Figure 3 As shown in the embodiments of this disclosure, the data processing method further includes S19.

[0043] S19. When the total number of updates is equal to 0, call the deactivation interface to deactivate the preset identifier corresponding to the second task identifier, and perform the target operation on the target task.

[0044] As described above, the data processing method provided in this embodiment, in response to a target operation on a target task, obtains the task type of the target task; when the task type of the target task is a scheduled task, obtains the second task identifier of the second task list in which the target task is located; removes the association between the target task and the second task identifier, and decrements the total number of the first task in the second task list by 1 to obtain the update count; when the update count is not equal to 0, performs the target operation on the target task. When the update count is equal to 0, calls the disable interface to disable the preset identifier corresponding to the second task identifier, and performs the target operation on the target task. Since each scheduled task shares a preset directed acyclic tree, the data processing device does not need to deploy a DAG for each Airflow scheduling task when executing Airflow scheduling tasks, thus greatly reducing the number of DAGs and improving the task response time of Airflow scheduling tasks.

[0045] In some feasible examples, combining Figure 2 ,like Figure 4 As shown in the embodiments of this disclosure, the data processing method further includes S20.

[0046] S20. When the target task is an immediately executed task, perform the target operation on the target task.

[0047] As described above, the data processing method provided in this embodiment, in response to a target operation on a target task, obtains the task type of the target task; when the task type of the target task is a scheduled task, obtains the second task identifier of the second task list in which the target task is located; removes the association between the target task and the second task identifier, and decrements the total number of scheduled tasks in the second task list by 1 to obtain the update count; when the update count is not equal to 0, performs the target operation on the target task. When the update count is equal to 0, calls the disable interface to disable the preset identifier corresponding to the second task identifier, and performs the target operation on the target task. When the task type of the target task is an immediately executed task, performs the target operation on the target task. Since each immediately executed task shares a preset directed acyclic tree, the data processing device does not need to deploy a DAG for each Airflow scheduling task when executing Airflow scheduling tasks, thus greatly reducing the number of DAGs and improving the task response time of Airflow scheduling tasks.

[0048] In some feasible examples, combining Figure 1 ,like Figure 5 As shown, the data processing method provided in this embodiment of the present disclosure further includes S21-S27.

[0049] S21. In response to the creation operation used to create a new task, obtain the task type of the new task; S22. When the task type of the newly added task is an immediately executed task, obtain the third task identifier corresponding to the immediately executed task; S23. When there is no first identifier corresponding to the third task identifier, call the generation interface to generate the first directed acyclic tree and the first identifier corresponding to the first directed acyclic tree, and call the enable interface to enable the first directed acyclic tree; wherein, the third task identifier corresponds one-to-one with the first identifier; S24. Obtain the second configuration parameters of the first directed acyclic tree; S25. Based on the third task identifier, determine the first identifier corresponding to the third task identifier, and call the running interface to run the first directed acyclic tree corresponding to the first identifier; S26. Input the second configuration parameters into the first directed acyclic tree to generate the second running instance corresponding to the newly added task; S27. Run the second running instance corresponding to the newly added task to obtain the second running result corresponding to the newly added task.

[0050] As described above, the data processing method provided in this embodiment, in response to a creation operation for creating a new task, obtains the task type of the new task; when the task type of the new task is an immediately executable task, obtains the third task identifier corresponding to the immediately executable task; when there is no first identifier corresponding to the third task identifier, calls the generation interface to generate a first directed acyclic tree and the first identifier corresponding to the first directed acyclic tree, and calls the enable interface to enable the first directed acyclic tree; wherein, the third task identifier corresponds one-to-one with the first identifier; obtains the second configuration parameters of the first directed acyclic tree; determines the first identifier corresponding to the third task identifier according to the third task identifier, and calls the run interface to run the first directed acyclic tree corresponding to the first identifier; inputs the second configuration parameters into the first directed acyclic tree to generate a second running instance corresponding to the new task; runs according to the second running instance corresponding to the new task to obtain the second running result corresponding to the new task. Since each immediately executable task shares a preset directed acyclic tree, the data processing device does not need to deploy a DAG for each Airflow scheduling task when executing Airflow scheduling tasks, thereby greatly reducing the number of DAGs and improving the task response time of Airflow scheduling tasks.

[0051] In some feasible examples, combining Figure 5 ,like Figure 6 As shown, the data processing method provided in this embodiment of the disclosure further includes S28-S31.

[0052] S28. When there is a first identifier corresponding to the third task identifier, obtain the second configuration parameters of the first directed acyclic tree; S29. Based on the third task identifier, determine the first identifier corresponding to the third task identifier, and call the running interface to run the first directed acyclic tree corresponding to the first identifier; S30. Input the second configuration parameters into the first directed acyclic tree to generate the second running instance corresponding to the newly added task; S31. Run the second running instance corresponding to the newly added task to obtain the second running result corresponding to the newly added task.

[0053] As described above, the data processing method provided in this embodiment responds to a creation operation for creating a new task by obtaining the task type of the new task; when the task type of the new task is an immediately executed task, obtaining the third task identifier corresponding to the immediately executed task; when there is no first identifier corresponding to the third task identifier, calling the generation interface to generate a first directed acyclic tree and the first identifier corresponding to the first directed acyclic tree, and calling the enable interface to enable the first directed acyclic tree; wherein, the third task identifier corresponds one-to-one with the first identifier; obtaining the second configuration parameters of the first directed acyclic tree; determining the first identifier corresponding to the third task identifier according to the third task identifier, and calling the run interface to run the first directed acyclic tree corresponding to the first identifier; inputting the second configuration parameters into the first directed acyclic tree to generate a second running instance corresponding to the new task; running according to the second running instance corresponding to the new task to obtain the second running result corresponding to the new task. When a first identifier corresponding to a third task identifier exists, the second configuration parameters of the first directed acyclic tree are obtained; based on the third task identifier, the first identifier corresponding to the third task identifier is determined, and the running interface is called to run the first directed acyclic tree corresponding to the first identifier; the second configuration parameters are input into the first directed acyclic tree to generate a second running instance corresponding to the newly added task; the second running instance corresponding to the newly added task is run to obtain the second running result corresponding to the newly added task. Since each immediately executed task shares a preset directed acyclic tree, the data processing device does not need to deploy a DAG for each Airflow scheduling task when executing Airflow scheduling tasks, thus greatly reducing the number of DAGs and improving the task response time of Airflow scheduling tasks.

[0054] In some feasible examples, combining Figure 5 ,like Figure 7 As shown, the data processing method provided in this embodiment of the present disclosure further includes S32-S34.

[0055] S32. When the task type of the newly added task is a timed task, based on the timed parameters of the timed task, obtain one or more fourth task identifiers that match the timed parameters, and the total number of second tasks of the timed tasks contained in the third task list corresponding to each fourth task identifier. S33. When there is a total number of second tasks that is less than the quantity threshold, add the new task to the third task list corresponding to the total number of second tasks that is less than the quantity threshold. S34. When the total number of second tasks less than the quantity threshold is greater than 0, wait for the actual directed acyclic tree corresponding to the fourth task identifier corresponding to the total number of second tasks less than the quantity threshold to be run at a set time.

[0056] As described above, the data processing method provided in this embodiment responds to a creation operation for creating a new task by obtaining the task type of the new task; when the task type of the new task is an immediately executed task, obtaining the third task identifier corresponding to the immediately executed task; when there is no first identifier corresponding to the third task identifier, calling the generation interface to generate a first directed acyclic tree and the first identifier corresponding to the first directed acyclic tree, and calling the enable interface to enable the first directed acyclic tree; wherein, the third task identifier corresponds one-to-one with the first identifier; obtaining the second configuration parameters of the first directed acyclic tree; determining the first identifier corresponding to the third task identifier according to the third task identifier, and calling the run interface to run the first directed acyclic tree corresponding to the first identifier; inputting the second configuration parameters into the first directed acyclic tree to generate a second running instance corresponding to the new task; running according to the second running instance corresponding to the new task to obtain the second running result corresponding to the new task. When the newly added task is a scheduled task, based on the scheduled task's timing parameters, one or more fourth task identifiers matching the timing parameters are obtained, along with the total number of second tasks in the third task list corresponding to each fourth task identifier. If the total number of second tasks is less than a threshold, the newly added task is added to the third task list corresponding to the total number of second tasks less than the threshold. If the total number of second tasks less than the threshold is greater than 0, the actual directed acyclic tree corresponding to the fourth task identifier corresponding to the total number of second tasks less than the threshold is waited to be executed on a scheduled basis. Since all immediately executed tasks share a preset directed acyclic tree, the data processing device does not need to deploy a DAG for each Airflow scheduled task when executing Airflow scheduled tasks, thus greatly reducing the number of DAGs and improving the task response time of Airflow scheduled tasks.

[0057] In some feasible examples, combining Figure 7 ,like Figure 8 As shown in the embodiments of this disclosure, the data processing method further includes S35.

[0058] S35. When the total number of the second task, which is less than the quantity threshold, is equal to 0, call the enable interface to enable the actual directed acyclic tree and wait for the actual directed acyclic tree to run at regular intervals.

[0059] As described above, the data processing method provided in this embodiment, in response to a creation operation for creating a new task, obtains the task type of the new task; when the task type of the new task is an immediately executable task, obtains the third task identifier corresponding to the immediately executable task; when there is no first identifier corresponding to the third task identifier, calls the generation interface to generate a first directed acyclic tree and a first identifier corresponding to the first directed acyclic tree, and calls the enable interface to enable the first directed acyclic tree; wherein, the third task identifier corresponds one-to-one with the first identifier; obtains the second configuration parameters of the first directed acyclic tree; determines the first identifier corresponding to the third task identifier based on the third task identifier, and calls the run interface to run the first directed acyclic tree corresponding to the first identifier; inputs the second configuration parameters into the first directed acyclic tree to generate a second running instance corresponding to the new task; runs according to the second running instance corresponding to the new task to obtain the second running result corresponding to the new task. When the total number of second tasks less than the quantity threshold is equal to 0, calls the enable interface to enable the actual directed acyclic tree, and waits for the actual directed acyclic tree to run periodically. Since all immediately executed tasks share a pre-defined directed acyclic tree, the data processing device does not need to deploy a DAG for each Airflow scheduling task when executing Airflow scheduling tasks. This greatly reduces the number of DAGs and improves the task response time of Airflow scheduling tasks.

[0060] In some feasible examples, combining Figure 7 ,like Figure 9 As shown, the data processing method provided in this embodiment of the disclosure further includes S36 and S37.

[0061] S36. When there is no total number of second tasks less than the quantity threshold, create a fifth task identifier, call the generation interface to generate the second directed acyclic tree corresponding to the fifth task identifier and the second identifier corresponding to the second directed acyclic tree, and call the enable interface to enable the second directed acyclic tree; wherein, the fifth task identifier and the second identifier correspond one-to-one. S37. Add the new task to the fourth task list corresponding to the fifth task identifier, and wait for the second directed acyclic tree to run at a set time.

[0062] As described above, the data processing method provided in this embodiment responds to a creation operation for creating a new task by obtaining the task type of the new task; when the task type of the new task is an immediately executed task, obtaining the third task identifier corresponding to the immediately executed task; when there is no first identifier corresponding to the third task identifier, calling the generation interface to generate a first directed acyclic tree and the first identifier corresponding to the first directed acyclic tree, and calling the enable interface to enable the first directed acyclic tree; wherein, the third task identifier corresponds one-to-one with the first identifier; obtaining the second configuration parameters of the first directed acyclic tree; determining the first identifier corresponding to the third task identifier according to the third task identifier, and calling the run interface to run the first directed acyclic tree corresponding to the first identifier; inputting the second configuration parameters into the first directed acyclic tree to generate a second running instance corresponding to the new task; running according to the second running instance corresponding to the new task to obtain the second running result corresponding to the new task. If there is no second task with a total number less than the threshold, a fifth task identifier is created, and the generation interface is called to generate the second directed acyclic tree corresponding to the fifth task identifier and the second identifier corresponding to the second directed acyclic tree. The enable interface is then called to enable the second directed acyclic tree. The fifth task identifier and the second identifier are in one-to-one correspondence. The newly added task is added to the fourth task list corresponding to the fifth task identifier and waits for the second directed acyclic tree to run periodically. Since all immediately executed tasks share a preset directed acyclic tree, the data processing device does not need to deploy a DAG for each Airflow scheduling task when executing Airflow scheduling tasks, thus greatly reducing the number of DAGs and improving the task response time of Airflow scheduling tasks.

[0063] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0064] This application embodiment can divide the data processing device into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing unit. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0065] like Figure 10 As shown in the diagram, an embodiment of this application provides a schematic diagram of a data processing device. It includes a communicator 101 and a controller 102.

[0066] Communicator 101 is configured to communicate with a client; Controller 102 is configured as follows: When a preset directed acyclic tree is being executed at the current moment, the first task identifier corresponding to the preset identifier is obtained based on the preset identifier of the preset directed acyclic tree; wherein, one preset identifier corresponds to one first task identifier. Based on the first task identifier, obtain one or more timed tasks contained in the first task list corresponding to the first task identifier; wherein the timed parameters of the timed tasks in the same first task list are matched. Input the first configuration parameters of each scheduled task into the preset directed acyclic tree to create the first running instance corresponding to each scheduled task; Run each scheduled task according to its first running instance, and obtain the first running result for each scheduled task.

[0067] In some implementable examples, controller 102 is also configured as follows: In response to the target operation on the target task, obtain the task type of the target task; When the target task is a scheduled task, obtain the second task identifier in the second task list where the target task is located; wherein, the target operation includes either a deactivation operation or a deletion operation; Remove the association between the target task and the second task identifier, and decrement the total number of first tasks in the second task list by 1 to obtain the updated total; When the total number of updates is not equal to 0, perform the target operation on the target task.

[0068] In some implementable examples, controller 102 is also configured as follows: When the total number of updates is 0, the deactivation interface is called to deactivate the preset identifier corresponding to the second task identifier, and the target operation is performed on the target task.

[0069] In some implementable examples, controller 102 is also configured as follows: When the target task is an immediately executed task, perform the target operation on the target task.

[0070] In some implementable examples, controller 102 is also configured as follows: In response to the creation operation used to create a new task, obtain the task type of the new task; When the task type of the newly added task is an immediately executed task, obtain the third task identifier corresponding to the immediately executed task; If there is no first identifier corresponding to the third task identifier, the generation interface is called to generate the first directed acyclic tree and the first identifier corresponding to the first directed acyclic tree, and the activation interface is called to activate the first directed acyclic tree; wherein, the third task identifier corresponds one-to-one with the first identifier; Obtain the second configuration parameters of the first directed acyclic tree; Based on the third task identifier, determine the first identifier corresponding to the third task identifier, and call the running interface to run the first directed acyclic tree corresponding to the first identifier; Input the second configuration parameter into the first directed acyclic tree to generate the second running instance corresponding to the new task; Run the second instance corresponding to the newly added task to obtain the second running result corresponding to the newly added task.

[0071] In some implementable examples, controller 102 is also configured as follows: If a first identifier corresponding to a third task identifier exists, obtain the second configuration parameters of the first directed acyclic tree; Based on the third task identifier, determine the first identifier corresponding to the third task identifier, and call the running interface to run the first directed acyclic tree corresponding to the first identifier; Input the second configuration parameter into the first directed acyclic tree to generate the second running instance corresponding to the new task; Run the second instance corresponding to the newly added task to obtain the second running result corresponding to the newly added task.

[0072] In some implementable examples, controller 102 is also configured as follows: When the task type of the newly added task is a timed task, based on the timed parameters of the timed task, obtain one or more fourth task identifiers that match the timed parameters, and the total number of second tasks contained in the third task list corresponding to each fourth task identifier. If there is a total number of second tasks that is less than the quantity threshold, the new task will be added to the third task list corresponding to the total number of second tasks that is less than the quantity threshold. When the total number of second tasks less than the threshold is greater than 0, wait for the actual directed acyclic tree corresponding to the fourth task identifier corresponding to the total number of second tasks less than the threshold to be run at a set time.

[0073] In some implementable examples, controller 102 is also configured as follows: When the total number of the second task, which is less than the quantity threshold, is equal to 0, the enable interface is called to enable the actual directed acyclic tree, and the actual directed acyclic tree is waited to run at regular intervals.

[0074] In some implementable examples, controller 102 is also configured as follows: If there is no total number of second tasks less than the quantity threshold, create a fifth task identifier, call the generation interface to generate the second directed acyclic tree corresponding to the fifth task identifier and the second identifier corresponding to the second directed acyclic tree, and call the enable interface to enable the second directed acyclic tree; wherein, the fifth task identifier and the second identifier are in one-to-one correspondence. Add the new task to the fourth task list corresponding to the fifth task identifier, and wait for the second directed acyclic tree to run periodically.

[0075] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and their functions will not be repeated here.

[0076] Of course, the data processing device provided in this application embodiment includes, but is not limited to, the modules described above. For example, the data processing device may also include a memory 103. The memory 103 may be used to store the program code of the data processing device, and may also be used to store data generated by the data processing device during operation, such as data in write requests.

[0077] like Figure 11 As shown, this application embodiment also provides a chip system that can be applied to the data processing device in the foregoing embodiments. The chip system includes at least one processor 1501 and at least one interface circuit 1502. The processor 1501 may be the processor in the aforementioned data processing device. The processor 1501 and the interface circuit 1502 are interconnected via a line. The processor 1501 can receive and execute computer instructions from the memory of the aforementioned data processing device through the interface circuit 1502. When the computer instructions are executed by the processor 1501, the data processing device can perform the various steps executed by the data processing device in the foregoing embodiments. Of course, the chip system may also include other discrete devices, which are not specifically limited in this application embodiment.

[0078] This application also provides a computer-readable storage medium for storing computer instructions for operating the aforementioned data processing device.

[0079] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing device, characterized in that, include: The communicator is configured to communicate with the client. The controller is configured as follows: When a preset directed acyclic tree is being executed at the current moment, a first task identifier corresponding to the preset identifier is obtained based on the preset identifier of the preset directed acyclic tree; wherein, one preset identifier corresponds to one first task identifier. Based on the first task identifier, obtain one or more timed tasks contained in the first task list corresponding to the first task identifier; wherein, the timed parameters of timed tasks in the same first task list are matched. The first configuration parameters of each of the scheduled tasks are respectively input into the preset directed acyclic tree to create the first running instance corresponding to each of the scheduled tasks; Run each scheduled task according to its corresponding first running instance to obtain the first running result of each scheduled task.

2. The data processing device according to claim 1, characterized in that, The controller is also configured to: In response to a target operation on a target task, the task type of the target task is obtained; When the target task is a timed task, obtain the second task identifier of the second task list in which the target task is located; wherein, the target operation includes either a deactivation operation or a deletion operation; Remove the association between the target task and the second task identifier, and subtract 1 from the total number of the first tasks in the second task list to obtain the updated total; When the total number of updates is not equal to 0, the target operation is performed on the target task.

3. The data processing device according to claim 2, characterized in that, The controller is also configured to: When the total number of updates is equal to 0, the deactivation interface is called to deactivate the preset identifier corresponding to the second task identifier, and the target operation is performed on the target task.

4. The data processing device according to claim 2, characterized in that, The controller is also configured to: When the target task is an immediately executable task, the target operation is performed on the target task.

5. The data processing device according to claim 1, characterized in that, The controller is also configured to: In response to a creation operation for creating a new task, the task type of the new task is obtained; When the newly added task is of type "immediately executed task", obtain the third task identifier corresponding to the immediately executed task; If the first identifier corresponding to the third task identifier does not exist, the generation interface is called to generate the first directed acyclic tree and the first identifier corresponding to the first directed acyclic tree, and the activation interface is called to activate the first directed acyclic tree; wherein, the third task identifier corresponds one-to-one with the first identifier; Obtain the second configuration parameters of the first directed acyclic tree; Based on the third task identifier, determine the first identifier corresponding to the third task identifier, and call the running interface to run the first directed acyclic tree corresponding to the first identifier; The second configuration parameter is input into the first directed acyclic tree to generate a second running instance corresponding to the newly added task; Run the second running instance corresponding to the newly added task to obtain the second running result corresponding to the newly added task.

6. The data processing device according to claim 5, characterized in that, The controller is also configured to: When a first identifier corresponding to a third task identifier exists, obtain the second configuration parameters of the first directed acyclic tree; Based on the third task identifier, determine the first identifier corresponding to the third task identifier, and call the running interface to run the first directed acyclic tree corresponding to the first identifier; The second configuration parameter is input into the first directed acyclic tree to generate a second running instance corresponding to the newly added task; Run the second running instance corresponding to the newly added task to obtain the second running result corresponding to the newly added task.

7. The data processing device according to claim 5, characterized in that, The controller is also configured to: When the task type of the newly added task is a timed task, based on the timed parameters of the timed task, obtain one or more fourth task identifiers that match the timed parameters, and the total number of second tasks included in the third task list corresponding to each fourth task identifier; When there is a total number of second tasks that is less than the quantity threshold, the newly added task is added to the third task list corresponding to the total number of second tasks that is less than the quantity threshold. When the total number of the second tasks that is less than the threshold is greater than 0, wait for the actual directed acyclic tree corresponding to the fourth task identifier of the total number of the second tasks that is less than the threshold to be run at regular intervals.

8. The data processing device according to claim 7, characterized in that, The controller is also configured to: When the total number of the second task, which is less than the quantity threshold, is equal to 0, the enable interface is called to enable the actual directed acyclic tree, and the actual directed acyclic tree is waited to run at a timed interval.

9. The data processing device according to claim 7, characterized in that, The controller is also configured to: If there is no second task total number less than the quantity threshold, a fifth task identifier is created, and the generation interface is called to generate the second directed acyclic tree corresponding to the fifth task identifier and the second identifier corresponding to the second directed acyclic tree, and the enable interface is called to enable the second directed acyclic tree; wherein, the fifth task identifier and the second identifier correspond one-to-one. The newly added task is added to the fourth task list corresponding to the fifth task identifier, and the second directed acyclic tree is waited to run periodically.

10. A data processing method, characterized in that, include: When a preset directed acyclic tree is being executed at the current moment, a first task identifier corresponding to the preset identifier is obtained based on the preset identifier of the preset directed acyclic tree; wherein, one preset identifier corresponds to one first task identifier. Based on the first task identifier, obtain one or more timed tasks contained in the first task list corresponding to the first task identifier; wherein, the timed parameters of timed tasks in the same first task list are matched. The first configuration parameters of each of the scheduled tasks are respectively input into the preset directed acyclic tree to create the first running instance corresponding to each of the scheduled tasks; Run each scheduled task according to its corresponding first running instance to obtain the first running result of each scheduled task.

Images