A task processing method, device, equipment, and storage medium
By introducing a message-driven method that declares dependent components and refers to dependent components in the ETL system, the problem of insufficient process-level dependence and large database load in the ETL system is solved, and more fine-grained dependency control and database load reduction are achieved. Improves operational efficiency and concurrent execution capabilities.
Patent Information
- Application Number
- CN202010758652.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-07-31
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-07-31
AI Technical Summary
The granularity of the existing ETL systems at the process level's dependency settings control at the process level is not fine enough, and polling the database causes excessive database load, making it difficult to take into account the timeliness of the dependency state and the reduction of database load.
By declaring the message-driven method between dependent components and referenced dependent components, the process-based dependency is transformed into component-based dependency, and the message system and distributor are used to transmit dependency status check messages asynchronously to reduce database access.
It improves the granularity of dependency control, reduces database load, improves operational efficiency and concurrent execution capabilities, and avoids database overload.
Smart Images

Figure CN114064223B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to, but are not limited to, data processing technologies, and in particular, to a task processing method and apparatus, device, and storage medium. Background Art
[0002] The data warehouse (Extract-Transform-Load, ETL) system integrates data through the interdependence between processes. However, the existing ETL system has the following problems: 1) The control granularity of process-level dependency settings is not fine enough, and sometimes users may need component-level dependencies. 2) Polling the database will cause the database to bear a large load. In actual use, dependency scenarios are used in large quantities. During operation, there will be a large number of dependent processes that need to poll the database to check whether the executable conditions are met. At this time, the database will be under a large load. In order to reduce the load pressure on the database caused by data queries, the time interval for polling the database can be set to be longer. However, if the polling time is set too long, the timeliness of obtaining the execution status of the dependent process is not enough. If the polling time interval is set too short, the database load pressure will be very large, and it is difficult to take both into account. Summary of the invention
[0003] In view of this, embodiments of the present application provide a task processing method and apparatus, device, and storage medium.
[0004] The technical solution of the embodiment of the present application is implemented as follows:
[0005] In a first aspect, an embodiment of the present application provides a task processing method, the method comprising:
[0006] By declaring a dependency component, after the first file is generated, a notification message is sent to a message system, wherein the declared dependency component is a component associated with the first file; the notification message is read from the message system by a distributor, and the notification message is forwarded to a checker, wherein the checker is created by the distributor according to pre-configured dependency conditions; by the checker, when the checker determines that the pre-configured dependency condition is met according to the content in the notification message, a condition satisfaction message corresponding to the pre-configured dependency condition is sent to a reference dependency component associated with the second file; the generation of the second file depends on the generation of the first file, wherein the pre-configured dependency condition includes at least one dependency condition; the reference dependency component responds to the condition satisfaction message, and the second file is generated using the first file.
[0007] In a second aspect, an embodiment of the present application provides a task processing device, comprising: a control module, wherein the control module is used to: send a notification message to a message system after a first file is generated by declaring a dependency component, wherein the declared dependency component is a component associated with the first file; read the notification message from the message system by a distributor, and forward the notification message to a checker, wherein the checker is created by the distributor according to a pre-configured dependency condition; through the checker, when the checker determines that the pre-configured dependency condition is met based on the content in the notification message, send a condition satisfaction message corresponding to the pre-configured dependency condition to a reference dependency component associated with the second file; generation of the second file depends on generation of the first file, wherein the pre-configured dependency condition includes at least one dependency condition; respond to the condition satisfaction message by the reference dependency component, and generate the second file using the first file.
[0008] In a third aspect, an embodiment of the present application provides a task processing device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps in the above-mentioned task processing method are implemented.
[0009] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above-mentioned task processing method when executed by a processor.
[0010] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:
[0011] In an embodiment of the present application, the original process-based dependency between the first file and the second file is converted into a component-based dependency using a declared dependency component and a referenced dependency component, so that the dependency control granularity is finer. At the same time, by using a message system, a distributor, and a checker to pass messages from the declared dependency component to the referenced dependency component, the notification message of the successful execution of the declared dependency component can be asynchronously transmitted to the referenced dependency component. This message-driven dependency status check can avoid database access to determine the execution status of the process during process execution, reduce database pressure and improve operating efficiency, and can effectively reduce the database load when a large number of dependent components are running at the same time, thereby improving the concurrent execution capability of dependent components. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which:
[0013] Figure 1 A flowchart of a task processing method provided in an embodiment of the present application;
[0014] Figure 2 A flowchart of a task processing method provided in an embodiment of the present application;
[0015] Figure 3 A flowchart of a task processing method provided in an embodiment of the present application;
[0016] Figure 4 A schematic diagram of a configuration of a reference dependent component provided in an embodiment of the present application;
[0017] Figure 5 A schematic diagram of the structure of a task processing component provided in an embodiment of the present application;
[0018] Figure 6 A schematic diagram of the structure of a task processing device provided in an embodiment of the present application;
[0019] Figure 7 A hardware entity diagram of a task processing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0020] The data warehouse (Extract-Transform-Load, ETL) system integrates data through the interdependence between processes. For example, in the ETL system, the daily report generation process usually depends on the generation results of each hourly report of the 24 hours of the day. The hourly report generation process is triggered once an hour, and it is generated 24 times before the process of generating the daily report is triggered. The existing ETL system solves the above scenario, usually only able to achieve process-level dependencies. For example, process A is a daily report, which is scheduled once every day at 23:30, and process B is hourly scheduling, which is scheduled once every hour. Process A depends on process B at the process level.
[0021] The process-level dependencies in the ETL system can be specified through the system-provided page. For example, specify that process A depends on process B through the page. When process A is scheduled at 23:30, if it is found that process B has been executed less than 24 times from 0:00 to 23:30 on the same day, process A will be suspended until process B is found to have been executed 24 times, then process A will start to execute.
[0022] In the ETL system, the execution status of dependent processes can be confirmed by periodically polling the database. For example, during the suspension of process A, the database can be periodically polled to confirm whether process B has run 24 times within this time period.
[0023] There are two problems with the existing technology: 1) The control granularity of the process-level dependency setting is not fine enough, and sometimes users may need component-level dependencies. 2) Polling the database will cause the database to bear a large load. In actual use, dependency scenarios are used in large quantities. During operation, there will be a large number of dependent processes that need to poll the database to check whether the executable conditions are met. At this time, the database will be under a large load. In order to reduce the load pressure of data queries on the database, the time interval for polling the database can be set longer. However, if the polling time is set too long, the timeliness of obtaining the execution status of the dependent process is not enough. If the polling time interval is set too short, the load pressure on the database will be very large, and it is difficult to take both into account.
[0024] In response to the above problems, an embodiment of the present application provides a message-driven approach to replace polling of the execution status in the database, and an embodiment of the present application provides a component-level dependency solution to solve the problem of insufficient dependency granularity.
[0025] The technical solution of the present application is further described in detail below with reference to the accompanying drawings and embodiments.
[0026] The present application embodiment provides a task processing method. Figure 1 A flowchart of a task processing method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the method includes:
[0027] Step 101, after the first file is generated, a notification message is sent to a message system by declaring a dependent component, wherein the declared dependent component is a component associated with the first file;
[0028] Here, the first file may be a report file, for example, a daily report, an hourly report, a monthly report, etc. The declared dependency component associated with the first file is responsible for notifying the referenced dependency component that references the first file that the generation of this report is complete. Here, the notification message may include a process execution instance ID, a process ID / referenced dependency component ID, and a scheduling time. For example, one process execution instance ID is generated each time a daily report or hourly report is generated; one process ID corresponds to one daily report process; and the time when the daily report expects to schedule the hourly report is the scheduling time. Here, the message system is responsible for delivering the notification message of the declared dependency component to the referenced dependency component. The message system may be a distributed message queue (kafka). Here, the message system stores the notification message for reading by the distributor;
[0029] Step 102, reading the notification message from the message system through a distributor, and forwarding the notification message to a checker, wherein the checker is created by the distributor according to a pre-configured dependency condition;
[0030] Here, the distributor is used to read notification messages declaring dependent components from the message system.
[0031] Here, it should be noted that the distributor needs to meet specific forwarding rules when forwarding the notification message to the corresponding checker. The specific forwarding rules can be when the execution instance ID of the notification message is a value within a specific digital range, or the process ID / reference dependency component ID is a specific value, or the scheduling time is within a specific time range.
[0032] Here, the pre-configured dependency condition is used to determine whether the reference dependency component sending condition is met.
[0033] Here, the dependency condition can be a 4-tuple (process ID / reference dependency ID, start scheduling time (startDataTime), end scheduling time (endDataTime), target number of times). Among them, the start scheduling time is the time when the declared dependency component starts to execute, the end scheduling time is the time when the declared dependency component finishes executing the first file, and the target number of times is the number of times the declared dependency component is expected to execute when the reference dependency component is executed. For example, it takes 24 hours to generate a daily report, and the hourly report needs to be submitted 24 times, so the target number of times is 24 times.
[0034] In some embodiments, the checker created for the distributor according to the preconfigured dependency conditions can be one of the following: creating a checker for the preconfigured dependency conditions; creating a corresponding checker for each of the preconfigured dependency conditions.
[0035] Step 103: When the checker determines that the pre-configured dependency condition is met according to the content in the notification message, the checker sends a condition meeting message corresponding to the pre-configured dependency condition to the reference dependency component associated with the second file; the generation of the second file depends on the generation of the first file, wherein the pre-configured dependency condition includes at least one dependency condition;
[0036] Here, the second file may be a daily report that needs to be generated based on an hourly report, or a monthly report that needs to be generated based on a daily report, or an annual report that needs to be generated based on a monthly report, etc.
[0037] Here, the reference dependency component generates an identifier for the second file that needs to reference the first file, which is used to indicate that the generation of the second file needs to reference the first file.
[0038] Here, the condition fulfillment message may be one of the following: a condition fulfillment message corresponding to the fulfillment of the pre-configured dependent condition; a condition fulfillment message corresponding to the fulfillment of each pre-configured dependent condition.
[0039] Here, the pre-configured dependency condition includes at least one dependency condition which may be one of the following: the pre-configured dependency condition is one dependency condition; the pre-configured dependency condition is two or more dependency conditions.
[0040] During the implementation process, after the checker that meets the forwarding rules receives the notification message sent by the distributor, it determines whether the dependency condition is met based on the content of the notification message. When the condition is met, it sends a condition meeting message to the reference dependency component associated with the second file, informing the reference dependency component that the current condition for generating the second file has been met and the second file can be generated.
[0041] Here, when the second file is associated with the reference dependent component, it is not necessary to wait for all dependent first files to be executed before starting the execution thread. The execution thread can be started synchronously when the execution thread of the first declared dependent component associated with the first file is started.
[0042] Step 104: Respond to the condition fulfillment message through the reference dependency component and generate the second file using the first file.
[0043] Exemplarily, when the first file is an hourly report and the second file is a daily report, after the reference dependency receives a condition fulfillment message sent by the checker, the daily report of that day is generated using the received 24-hour report.
[0044] In the embodiment of the present application, the original process-based dependency between the first file and the second file is converted into a component-based dependency by using a declared dependency component and a referenced dependency component, so that the dependency control granularity is finer. At the same time, by using a message system, a distributor, and a checker to pass messages from the declared dependency component to the referenced dependency component, the notification message of the successful execution of the declared dependency component can be asynchronously transmitted to the referenced dependency component. This message-driven dependency status check can avoid database access to determine the execution status of the process during process execution, reduce database pressure and improve operating efficiency, and can effectively reduce the database load when a large number of dependent components are running at the same time, thereby improving the concurrent execution capability of dependent components.
[0045] The present application embodiment provides a task processing method. Figure 2 A flowchart of a task processing method provided in an embodiment of the present application is shown as follows: Figure 2 As shown, the method includes:
[0046] Step 201, the reference dependency component receives the at least one dependency condition configured;
[0047] Here, the reference dependency component receives the dependency condition by receiving the configuration information of the dependency condition. The execution of the reference dependency component needs to rely on the pre-configured dependency condition. When all the dependency conditions are met, the second file corresponding to the reference dependency component is generated. That is to say, when the pre-configured dependency condition includes only one dependency condition, the second file can be generated when the one dependency condition is met; when the pre-configured dependency condition includes multiple dependency conditions, the second file can be generated when the multiple dependency conditions are met.
[0048] Here, the configuration information received by the reference dependency component may include an offset and a timeout. The offset refers to the time period that allows the user to offset the execution in units of the declared dependency scheduling cycle. The offset can provide greater freedom. The timeout allows the component to wait for a period of time, ignore the dependency conditions, declare the dependency component to automatically succeed, and send the corresponding notification message when the generation of the declared dependency component is completed to the message system, so that the process continues to execute.
[0049] For example, if the daily report depends on the hourly report, it depends on the data from 0:00 to 24:00 by default. When the offset is 1, it depends on the data from 1:00 a.m. of the current day to 1:00 a.m. of the next day. For example, the generation time of the hourly report at 0:00 is a scheduling cycle. Here, a scheduling cycle can be set to 1 hour. When the timeout is set to 2, after waiting for the hourly report at 0:00 for 1 hour, wait for another 2 hours. If the hourly report generation process at 0:00 has not been executed at this time, the process is set to success and the subsequent processes are continued.
[0050] Step 202, registering each dependency condition of the at least one dependency condition in the distributor through the reference dependency component;
[0051] Here, registering each dependency condition in the dependency condition list with the distributor can enable the distributor to determine how many dependency conditions are needed for the current reference dependency component, which is conducive to forming the dependency condition list. Here, the dependency condition list may include all dependency conditions in the reference dependency component.
[0052] Step 203, creating a corresponding checker for each dependency condition through the distributor;
[0053] In some embodiments, the distributor may create a corresponding checker for each dependency condition through a list of dependency conditions.
[0054] Step 204, after the first file is generated, the declared dependency component stores the execution record of the first file in a database, wherein the execution record includes at least three tuples: an execution ID, a first identifier, and a scheduling time, wherein the first identifier includes a process identifier or an identifier of a referenced dependency component;
[0055] Here, the database is responsible for persisting the execution records of the declared dependencies, which can be used to determine how many expected declared dependent components have been executed before the referenced dependent component is initialized. Here, the database can be a relational database such as MySQL, SQLServer, Oracle, or a non-relational database such as Redis.
[0056] Step 205, after the execution record is successfully stored, sending the notification message to the message system;
[0057] Step 206, reading the notification message from the message system through the distributor, and forwarding the notification message to the checker, where the checker is created by the distributor according to the pre-configured dependency conditions;
[0058] Step 207, after receiving the notification message, extract the execution ID in the notification message, and if the execution ID is successfully added to the created execution ID set, determine the number of currently generated execution IDs in the execution ID set; or query the database to determine the number of currently generated execution IDs;
[0059] The notification message includes a 3-tuple: execution ID, first identifier and scheduling time; the pre-configured dependency condition includes a 4-tuple: first identifier, start scheduling time, end scheduling time and target number of times;
[0060] Here, when the checker receives a notification message, it takes out the execution ID in each notification message, adds the execution ID to the created execution ID set, and if the execution ID is successfully added to the created execution ID set, determines the number of execution IDs currently generated in the execution ID set.
[0061] In some embodiments, if the current execution ID has been added to the execution ID set, the execution ID will fail to be added to avoid repeated receipt of notification messages.
[0062] Here, the execution ID set is constructed to ensure that during the notification message transmission process, even if the checker receives two identical notification messages, the two identical notification messages will not be added to the execution ID set, so as to avoid repeated judgment of dependency conditions.
[0063] Here, the checker needs to query the database to determine the number of currently generated execution IDs because some notification messages may have occurred before the referenced dependency is executed, and these messages need to be retrieved from the database to avoid adding the execution IDs of used notification messages to the execution ID set.
[0064] Here, it should be noted that each reference dependency will only query the database once during the entire execution process. After the query, it is only necessary to judge based on the execution ID in the execution ID set to query whether the dependency condition of the reference dependency component is met. This can avoid the need to continuously query the database due to the need to confirm the execution status of the dependency condition, thereby alleviating the database load pressure.
[0065] Step 208, the checker determines that the pre-configured dependency condition is met according to the content in the notification message, including: the checker determines that the pre-configured dependency condition is met when it is determined that the preset condition is met according to the content in the notification message;
[0066] The preset conditions include:
[0067] The number of the execution IDs currently generated is equal to the size of the target number of times; when the identifier of the referenced dependent component of the notification message is the same as the identifier of the referenced dependent component of the pre-configured dependent condition, or when the process identifier of the notification message is the same as the process identifier of the pre-configured dependent condition, the notification message is a notification message that matches the pre-configured dependent condition; the scheduling time in the notification message that matches the pre-configured dependent condition is between the start scheduling time and the end scheduling time;
[0068] In one embodiment, comID' is the identifier of the reference dependent component of the notification message, and comID is the identifier of the reference dependent component of the dependency condition; in another embodiment, comID is the identifier of the reference dependent component of the notification message, and comID' is the identifier of the reference dependent component of the dependency condition.
[0069] When the identifier of the reference dependency component of the notification message is the same as the identifier of the reference dependency component of the dependency condition, the notification message is determined to be a notification message of reference dependency. Furthermore, when the scheduling time dataTime in the notification message is between the start scheduling time startDataTime and the end scheduling time endDataTime, the notification message is determined to be a notification message required by the reference dependency component, and when the number of generated execution IDs reaches the target number of times, it is determined that the pre-configured dependency condition is met.
[0070] Step 209: When the checker determines that the pre-configured dependency condition is met according to the content in the notification message, the checker sends a condition meeting message corresponding to the pre-configured dependency condition to the reference dependency component associated with the second file; the generation of the second file depends on the generation of the first file, wherein the pre-configured dependency condition includes at least one dependency condition;
[0071] Step 210, when the checker determines according to the content in the notification message that the pre-configured dependency condition is met, the checker is automatically destroyed;
[0072] Step 211: Responding to the condition fulfillment message through the reference dependency component, and generating the second file using the first file;
[0073] In some embodiments, the step 211 includes the following steps: Step 211A, after receiving the condition fulfillment message, the reference dependency component determines that, except for the fulfilled dependency condition, no other dependency conditions are currently unfulfilled, using the first file to generate the second file;
[0074] In some embodiments, when the pre-configured dependency conditions are two or more, there may be a situation where one dependency condition is met while the other dependency conditions are not met. In this case, it is necessary to further determine that in addition to the dependency conditions that have been met, there are no other dependency conditions that have not been met.
[0075] Step 211B: when it is determined that there are other dependent conditions that have not been met, wait for condition meeting messages corresponding to the other dependent conditions until all dependent conditions are met.
[0076] In some embodiments, when the pre-configured dependency conditions are two or more, there may be a situation where one dependency condition is met while the other dependency conditions are not met. In this case, it is necessary to further wait for the condition meeting message corresponding to the other dependency conditions while other dependency conditions are not met.
[0077] In an embodiment of the present application, after receiving the notification message, the checker extracts the execution ID in the notification message; adds the execution ID to the created execution ID set, and when the execution ID is successfully added, determines the number of currently generated execution IDs in the execution ID set; or, the checker queries the database to determine the number of currently generated execution IDs. In this way, the execution ID is managed by creating an execution ID set and querying the database, so that the execution ID is guaranteed to appear once, and the transmitted notification message can run normally. In addition, it is only necessary to judge the execution ID in the execution ID set to query whether the dependency condition of the referenced dependent component is met, which can avoid the need to continuously query the database because of the need to confirm the execution status of the dependent condition, thereby alleviating the load pressure of the database and greatly improving the concurrency of the dependent component execution.
[0078] In order to solve the problem that the dependency granularity at the process level is not fine enough, the embodiment of the present application provides a dependency status checking method, which adds a declared dependency component and a referenced dependency component to the ETL system, wherein the referenced dependency component observes the number of successful executions of the declared dependency component to decide whether to continue waiting or execute the subsequent component. Figure 3 As shown, users can add a declaration dependency component to the hourly report process, add a reference dependency component to the daily report, and configure the reference dependency component to reference the declaration dependency component of the daily report process. In this way, the execution results of the daily report process can be observed. At the same time, the declaration dependency component can be connected after the daily report generation component to notify the monthly report process and the daily report is successfully executed.
[0079] The present application embodiment provides a task processing method, taking the first file as an hourly report and the second file as a daily report as an example, Figure 3 A flowchart of a task processing method provided in an embodiment of the present application is shown as follows: Figure 3 As shown, the method includes:
[0080] Step 301: Download every hour of data from the File Transfer Protocol (FTP) software;
[0081] Step 302: sort the hourly data and generate an hourly report;
[0082] Step 303: Adding a declaration dependency component to the hourly report;
[0083] Step 304: Back up the daily report data;
[0084] Step 305: adding the reference dependency component to the daily report, and configuring the reference dependency component to reference the declaration dependency component;
[0085] Step 306: Generate a daily report based on the declared dependency component;
[0086] Step 307: Add a declaration dependency component to the daily report to provide a dependency component for generating a monthly report.
[0087] The present application embodiment provides a task processing method. Figure 4 A configuration diagram of a reference dependent component provided in an embodiment of the present application, such as Figure 4 As shown, the configuration includes:
[0088] Offset 41: The offset 41 refers to the time period that allows the user to offset the execution based on the scheduling period of the declared dependent component. Offset 41 can provide greater freedom. The timeout period allows the component to wait for a period of time, ignore the dependent condition, and declare the dependent component to automatically succeed, so that the process continues to execute the subsequent components.
[0089] In order to reduce the load pressure caused by polling the database when the reference dependency component is waiting, the embodiment of the present application provides a dependency status checking method to reduce the load pressure caused by polling the database. This is a distributed message-driven solution. The reference dependency component and the declaration dependency component can run on multiple hosts at the same time to provide high availability capabilities for the reference dependency component and the declaration dependency component.
[0090] This application embodiment provides a task processing component, Figure 5 A structural diagram of a task processing component provided in an embodiment of the present application, such as Figure 5 As shown, the structure includes:
[0091] The dependent component 51 is referenced, and according to the dependent condition pre-configured by the user, when the dependent condition is met, the subsequent component is triggered to continue to execute, and the dependent condition is registered in the distributor.
[0092] The declared dependent component 55 is responsible for notifying the referenced dependent component that references it that the dependent component is executed successfully.
[0093] The distributor 52 reads the notification message of the declared dependent component from Kafka and distributes the notification message to the corresponding checker.
[0094] The checker 53 is responsible for checking whether a certain dependency condition is satisfied, and when the condition is satisfied, notifying the referenced dependency component that the condition is successful.
[0095] kafka54, responsible for notification message delivery of declarative dependent components.
[0096] The database 56 is responsible for persisting the execution records of the declared dependent components. When the reference dependent component is initially executed, it is determined how many expected declared dependent components have been executed before it.
[0097] In the embodiment of the present application, the declaration dependency component 55 executes and stores the execution record in the database 56 first, and then sends a notification message to kafka 54 after the execution record is successfully stored in the database 56. The stored and sent notification message is a 3-tuple: (process execution instance ID, process ID / reference dependency component ID, scheduling time).
[0098] The following describes the execution process of each component:
[0099] First, the reference dependent component execution process includes:
[0100] Step 11: a timer is set according to the timeout information configured by the user, so as to cancel the waiting and set the reference dependency component 51 to be successful after the timeout occurs;
[0101] Step 12: Generate a dependency list based on the dependency conditions pre-configured by the user;
[0102] Step 13: register the first dependency condition in the distributor 52;
[0103] Step 14, after receiving the dependency condition, the distributor 52 creates the corresponding checker 53;
[0104] Step 15, when the dependent condition is successfully executed, the corresponding checker 53 notifies the referenced dependent component 51 that the condition is successfully executed, and the corresponding checker 53 is automatically destroyed;
[0105] Step 16, reference the dependency component 51 to check whether there are any dependency conditions, if yes, jump to step 13, otherwise execute 17;
[0106] Step 17, referencing dependent component 51 is executed successfully.
[0107] In the second aspect, the distributor 52 performs the following process:
[0108] Step 21, after the distributor 52 reads the notification message from Kafka, it will create a corresponding checker 53 and generate a dependency condition based on the notification message, where the dependency condition is a 4-tuple (process ID / reference dependency component ID, start scheduling time, end scheduling time, target number of times).
[0109] Step 22: When the distributor 52 receives a notification message of the declared dependency component 55 from kafka 54, it forwards the notification message of the declared dependency component 55 to the corresponding checker 53. The forwarding rule is: suppose the notification message of the declared dependency component received is:
[0110] (execID,flowID / comID,dataTime)
[0111] Here, execID is the process execution instance ID (execution ID), flowID is the process ID, comID is the reference dependent component ID (component ID), and dataTime is the scheduling time. For example, when a daily report or hourly report is generated, a process execution instance ID is generated; a daily report process corresponds to a process ID; the time when the daily report expects to schedule the hourly report is the scheduling time.
[0112] Assume that the dependency condition is (flowID / comID, startDataTime, endDataTime, target), where target is the target number. For example, if it takes 24 hours to generate a daily report and 24 hourly reports are required, then the target number is 24. The distributor 52 will forward the notification message to all the users who meet the following conditions:
[0113] Checker 53 that equal(flowID / comID, flowID / comID') == true, and startDataTime<=dataTime<=endDataTime.
[0114] Thirdly, the execution process of the checker 53:
[0115] Step 31, the checker 53 is created by the distributor 52. In the initialization state, the checker 53 queries the database 56 according to the parameters in the dependency condition, such as the process ID / reference dependency component ID, the start scheduling time and the end scheduling time as conditions, and constructs a set of initial execution IDs. After the database 56 query is successful, it will first determine whether the size of the ID set is equal to the expected number of successful executions. If they are equal, the reference dependency component 51 is notified that the dependency condition is met. Otherwise, it enters the waiting state. In the initialization state, all notification messages received from the declared dependency components will be cached, and these notification messages will be processed when entering the waiting state.
[0116] Step 32, when the checker 53 receives the notification message, it takes out the execution ID in each notification message and tries to add it to the ID set. If the execution ID already exists in the ID set, the joining fails; otherwise, the joining succeeds. When the joining succeeds, it checks whether the ID set is greater than or equal to the target number. If they are equal, it notifies the corresponding reference dependency component 51, the dependency condition is met, and it is automatically destroyed. Otherwise, it continues to wait for the notification message.
[0117] The reason why the checker 53 queries the database 56 for the first time when it starts is that some notification messages may have occurred before the reference dependency component 51 is executed, and this part needs to be checked from the database 56. The execution ID set is constructed to ensure that even if the same notification message is consumed twice during the notification message transmission process, it will not affect the dependency condition judgment. Each reference dependency component 51 will only query the database 56 once during the entire execution process, which greatly alleviates the load pressure of the database 56.
[0118] Based on the foregoing embodiments, an embodiment of the present application provides a task processing device, which includes the units included and the modules included in the units, and can be implemented by a processor in the task processing device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.
[0119] Figure 6 A schematic diagram of the structure of a task processing device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the task processing device 600 includes a control module 601: the control module is used to control a declaration dependency component 610, a distributor 620, a checker 630 and a reference dependency component 640, wherein:
[0120] After the first file is generated, a notification message is sent to the message system by declaring a dependency component 610, wherein the declaring dependency component is a component associated with the first file;
[0121] Reading the notification message from the message system through the distributor 620, and forwarding the notification message to the checker, wherein the checker is created by the distributor according to the pre-configured dependency condition;
[0122] By means of the checker 630, when the checker determines according to the content in the notification message that the pre-configured dependency condition is met, a condition meeting message corresponding to the pre-configured dependency condition is sent to the reference dependency component associated with the second file; the generation of the second file depends on the generation of the first file, wherein the pre-configured dependency condition includes at least one dependency condition;
[0123] The reference dependency component 640 responds to the condition fulfillment message and generates the second file using the first file.
[0124] In some embodiments, the task processing device 600 also includes a storage module and a sending module, wherein: the execution record includes at least 3 tuples: execution ID, a first identifier, and a scheduling time, the first identifier includes a process identifier or an identifier of a referenced dependent component, the storage module is used for the declared dependent component to store the execution record of the first file into a database after the first file is generated; the sending module is used to send the notification message to the message system after the execution record is successfully stored.
[0125] In some embodiments, the task processing device 600 also includes a receiving module, a generating module and a registering module, wherein: the receiving module is used for the reference dependency component to receive the at least one dependency condition configured; the registering module is used to register each of the at least one dependency condition to the distributor through the reference dependency component; and the creating module is used to create a corresponding checker for each dependency condition through the distributor.
[0126] In some embodiments, the task processing device 600 also includes a generation module and a delay module, wherein: the generation module is used to generate the second file using the first file when the referenced dependency component determines that there are no other dependency conditions that have not been met except the dependency conditions that have been met after receiving the conditional satisfaction message; the delay module is used to wait for the conditional satisfaction message corresponding to the other dependency conditions when it is determined that there are other dependency conditions that have not been met until all dependency conditions are met.
[0127] In some embodiments, the notification message includes a 3-tuple: an execution ID, a first identifier and a scheduling time; the pre-configured dependency condition includes a 4-tuple: a first identifier, a start scheduling time, an end scheduling time and a target number of times; the checker determines whether the pre-configured dependency condition is met based on the content in the notification message, including: the checker determines that the pre-configured dependency condition is met when it determines that the preset condition is met based on the content in the notification message; wherein the preset condition includes: the number of the execution IDs currently generated is equal to the size of the target number of times; when the identifier of the referenced dependency component of the notification message is the same as the identifier of the referenced dependency component of the pre-configured dependency condition, or when the process identifier of the notification message is the same as the process identifier of the pre-configured dependency condition, the notification message is a notification message that matches the pre-configured dependency condition; the scheduling time in the notification message that matches the pre-configured dependency condition is between the start scheduling time and the end scheduling time.
[0128] In some embodiments, the task processing device 600 also includes a determination module, wherein: the determination module is used to extract the execution ID in the notification message after receiving the notification message, and if the execution ID is successfully added to the created execution ID set, determine the number of currently generated execution IDs in the execution ID set; or, query the database to determine the number of currently generated execution IDs.
[0129] In some embodiments, the task processing device 600 further includes a destruction module, wherein: the destruction module is used for the checker to automatically destroy itself when the checker determines, based on the content in the notification message, that the pre-configured dependency condition is met.
[0130] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.
[0131] It should be noted that in the embodiment of the present application, if the above-mentioned task processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable the automatic test line of the device containing the storage medium to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk.
[0132] Correspondingly, the embodiment of the present application provides a task processing device 700, such as a task processing device, Figure 7 A hardware entity diagram of a task processing device provided in an embodiment of the present application, such as Figure 7 As shown, the hardware entity of the device 700 includes: a processor 701, a communication interface 702 and a memory 703, wherein:
[0133] Processor 701 generally controls the overall operation of device 700 .
[0134] The communication interface 702 may enable the device 700 to communicate with other terminals or servers via a network.
[0135] The memory 703 is configured to store instructions and applications executable by the processor 701, and can also cache data to be processed or processed by the processor 701 and various modules in the device 700 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).
[0136] Correspondingly, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the task processing method provided in the above embodiment are implemented.
[0137] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.
[0138] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.
[0139] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0140] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0141] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0142] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0143] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.
[0144] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including a number of instructions to enable a task processing device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.
[0145] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A task processing method, characterized in that: The method comprises: By declaring a dependent component, after the first file is generated, a notification message is sent to the message system, wherein the declared dependent component is a component associated with the first file; the notification message includes a 3-tuple: an execution ID, a first identifier, and a scheduling time; the execution ID is a process execution instance ID; the first identifier includes a process ID or a reference dependent component ID; The notification message is read from the message system by a distributor, and the notification message is forwarded to a checker, where the checker is created by the distributor according to a pre-configured dependency condition; the pre-configured dependency condition includes a 4-tuple: the first identifier, the start scheduling time, the end scheduling time, and the target number of times; Through the checker, when the checker determines that the preset condition is met according to the content in the notification message, when it is determined that the pre-configured dependency condition is met, a condition satisfaction message corresponding to the pre-configured dependency condition is sent to the reference dependency component associated with the second file; the generation of the second file depends on the generation of the first file, wherein the pre-configured dependency condition includes at least one dependency condition; the preset condition includes: the number of the execution IDs currently generated is equal to the size of the target number of times; when the identifier of the reference dependency component of the notification message is the same as the identifier of the reference dependency component of the pre-configured dependency condition, or when the process identifier of the notification message is the same as the process identifier of the pre-configured dependency condition, the notification message is a notification message that matches the pre-configured dependency condition; the scheduling time in the notification message that matches the pre-configured dependency condition is between the start scheduling time and the end scheduling time; The reference dependency component responds to the condition fulfillment message and generates the second file using the first file.
2. The method according to claim 1, characterized in that The method of sending a notification message to a message system after the first file is generated by declaring a dependent component includes: After the first file is generated, the declaration dependency component stores the execution record of the first file in a database, wherein the execution record includes at least three tuples: the execution ID, the first identifier, and the scheduling time; After the execution record is successfully stored, the notification message is sent to the message system.
3. The method according to claim 1, characterized in that The method further comprises: The reference dependency component receives the at least one dependency condition configured; registering each dependency condition of the at least one dependency condition in the distributor through the reference dependency component; A corresponding checker is created for each dependency condition by the dispatcher.
4. The method according to claim 1, characterized in that The step of responding to the condition fulfillment message by the reference dependency component and generating the second file by using the first file includes: After receiving the condition fulfillment message, the reference dependency component generates the second file by using the first file when it is determined that, except for the fulfilled dependency condition, no other dependency conditions are currently unfulfilled; When it is determined that there are other dependent conditions that have not been met, wait for condition meeting messages corresponding to the other dependent conditions until all dependent conditions are met.
5. The method according to claim 1, characterized in that The method further comprises: After receiving the notification message, extract the execution ID in the notification message, and if the execution ID is successfully added to the created execution ID set, determine the number of currently generated execution IDs in the execution ID set; or, The database is queried to determine the number of the currently generated execution IDs.
6. The method according to claim 1, characterized in that The method further comprises: When the checker determines, according to the content in the notification message, that the pre-configured dependency condition is met, the checker is automatically destroyed.
7. A task processing device, characterized in that: The device comprises: a control module, the control module being used to: send a notification message to a message system after a first file is generated by declaring a dependent component, wherein the declared dependent component is a component associated with the first file; the notification message comprises a 3-tuple: an execution ID, a first identifier and a scheduling time; the execution ID is a process execution instance ID; the first identifier comprises a process ID or a reference dependent component ID; The notification message is read from the message system by a distributor, and the notification message is forwarded to a checker, where the checker is created by the distributor according to a pre-configured dependency condition; the pre-configured dependency condition includes a 4-tuple: the first identifier, the start scheduling time, the end scheduling time, and the target number of times; Through the checker, when the checker determines that the preset condition is met according to the content in the notification message, when it is determined that the pre-configured dependency condition is met, a condition satisfaction message corresponding to the pre-configured dependency condition is sent to the reference dependency component associated with the second file; the generation of the second file depends on the generation of the first file, wherein the pre-configured dependency condition includes at least one dependency condition; the preset condition includes: the number of the execution IDs currently generated is equal to the size of the target number of times; when the identifier of the reference dependency component of the notification message is the same as the identifier of the reference dependency component of the pre-configured dependency condition, or when the process identifier of the notification message is the same as the process identifier of the pre-configured dependency condition, the notification message is a notification message that matches the pre-configured dependency condition; the scheduling time in the notification message that matches the pre-configured dependency condition is between the start scheduling time and the end scheduling time; The reference dependency component responds to the condition fulfillment message and generates the second file using the first file.
8. A task processing device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps in the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 6 are implemented.
