Batch data processing method and device

Abstract

The present invention relates to the field of financial technology, and discloses a batch data processing method and device, the method comprising: after obtaining a batch data processing task request in a data management area, if it is determined that the task has not been executed in the current batch data processing cycle, then determining the task execution identification information for the task request, and modifying the task status to being executed; if it is determined that the task needs to be sent to a data communication network DCN area, then determining whether there is a DCN node in the DCN area that is executing the task according to the task execution identification information; if it is determined that there is no DCN node in the DCN area that is executing the task, then sending the task to at least one DCN node in the DCN area so that the DCN node processes the task, and modifying the execution status of the DCN node to being executed; if it is determined that the DCN nodes that execute the task in the DCN area have completed the task, then modifying the task status to task execution success. Effectively reducing development and operation costs.

Description

Technical Field

[0001] The present invention relates to the technical field of financial technology (Fintech), and in particular to a batch data processing method and device. Background Art

[0002] With the development of computer technology, more and more technologies are applied in the financial field. The traditional financial industry is gradually transforming to financial technology (Fintech), and batch data processing technology is no exception. However, due to the security and real-time requirements of the financial industry, higher requirements are also placed on technology.

[0003] The existing resource processing system actually consists of several subsystems. Each system is deployed separately and is forcibly integrated into one system in a modular manner. However, the batch processing framework of each module is different according to the interaction mode of the submodule. Since the batch processing requires higher consistency, if each subsystem has its own batch processing framework, the cost of developers and maintenance personnel will be very high.

[0004] In summary, the prior art lacks a batch data processing method that is easy for developers to maintain and reduces maintenance costs. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a batch data processing method and device, which at least solves the problems existing in the prior art.

[0006] On the one hand, an embodiment of the present invention provides a batch data processing method, which is applied to a distributed resource processing system, and the method includes:

[0007] After the data management area obtains the batch data processing task request, if it is determined according to the task identification information in the task request that the task has not been executed in the current batch data processing cycle, the task execution identification information is determined for the task request, and the status of the task is modified to being executed;

[0008] If it is determined that the task needs to be sent to the data communication network DCN area, determining whether there is a DCN node in the DCN area that is executing the task according to the task execution identification information;

[0009] If it is determined that no DCN node in the DCN area is executing the task, sending the task to at least one DCN node in the DCN area so that the DCN node processes the task, and changing the execution state of the DCN node to being executed;

[0010] If it is determined that all DCN nodes executing the task in the DCN area have completed the task, the status of the task is modified to task execution success.

[0011] In the embodiment of the present invention, after obtaining the batch data processing task request, it is necessary to first determine whether the task has been executed. If it is determined that the task has not been executed, the task execution identification information is established; by determining whether the task has been executed, it is possible to prevent the same task request from being executed multiple times, that is, to prevent the concurrency of the same task in data management; further, after determining that the task needs to be sent to the DCN area for execution, in order to prevent concurrency in the DCN area, it is necessary to first determine that there is no DCN node in the DCN area processing the same task, and then assign the task to the DCN node for execution, and after determining that the DCN nodes have completed the task, it is determined that the task is successful. In the embodiment of the present invention, different resource processing subsystem modules use the same set of batch data processing methods, unified processing, and reduce the cost of development and operation and maintenance personnel.

[0012] Optionally, determining, according to the task identification information in the task request, that the task has not been executed in the current batch data processing cycle includes:

[0013] If it is determined that the task identification information is not found in the first data table in the current batch data processing cycle, it is determined that the task has not been executed in the current batch data processing cycle. The first data table is used to record the task identification information of batch data processing tasks executed in different batch data processing cycles.

[0014] In an embodiment of the present invention, by setting a first data table in the data management area, concurrency of the same task in data management can be prevented, that is, by the presence or absence of the task identification information in the first data table, it can be determined whether the same task exists in the current data management area.

[0015] Optionally, the determining that no DCN node in the DCN area is executing the task includes:

[0016] If the task execution identification information is not found in the second data table of the data management area, and the task execution identification information is not found in the third data table of the data management area, it is determined that there is no DCN node in the DCN area that is executing the task, and the second data table is used to record the task execution identification information executed in different batch data processing cycles, the DCN nodes corresponding to the task execution identification information, and the task execution status of each DCN node, and the third data table is used to record the task execution identification information executed in different batch data processing cycles and the number of batch data processing task notifications sent to each DCN node corresponding to the task execution identification information.

[0017] In the embodiment of the present invention, it can be determined through the second data table that no DCN node has executed the task corresponding to the task execution identification information during the current data processing cycle. Similarly, it can be further determined through the third data table that no DCN node has executed the task corresponding to the task execution identification information during the current data processing cycle, thereby ensuring that there is no concurrency problem of the same task in the DCN area.

[0018] Optionally, if the task execution identification information is not found in the second data table of the data management area, and the task execution identification information is not found in the third data table of the data management area, before determining that no DCN node in the DCN area is executing the task, the method further includes:

[0019] It is determined that the task execution identification information is not found in a fourth data table of each DCN node in the DCN area, wherein the fourth data table is used to record the task execution status of each DCN node for batch data processing tasks corresponding to different task execution identification information.

[0020] In the embodiment of the present invention, in order to prevent communication delays between the data management area and the DCN area, resulting in the fact that the task actually exists in the DCN area, but the data management area fails to query the task, it is necessary to query in the fourth table of the DCN area. If the task is not found, it can be considered that no DCN node has processed the task, thereby preventing the concurrency of the task.

[0021] Optionally, the method further includes:

[0022] If it is determined that the task identification information can be queried in the first data table within the current batch data processing cycle, determining the task execution status corresponding to the task identification information;

[0023] If it is determined that the task execution status is task failure, the step of determining whether the batch data processing task needs to be sent to the data communication network DCN area is performed, and the status of the batch data processing task is modified to being executed.

[0024] In an embodiment of the present invention, if it is determined that the task exists in the first data table in the current batch data processing cycle, the execution status of the task is determined. If the execution status is failure, the task continues to be executed, and in order to prevent concurrency, the status of the task is modified to being executed.

[0025] Optionally, the method further includes:

[0026] If the task execution identification information is found in the second data table of the data management area, a DCN node whose task execution status is a failure status is determined among the DCN nodes corresponding to the task execution identification information;

[0027] The task execution status of the DCN node whose task execution status in the second data table is a failed status is modified to being executed, and the step of determining whether the task execution identification information can be found in the third data table in the data management area is performed.

[0028] In the embodiment of the present invention, if the DCN node has executed the task and some or all DCN nodes are in a failed state, the state of the DCN node in the failed state is changed to being executed, the task is executed again, and the task is prevented from being concurrent.

[0029] Optionally, if it is determined that all nodes executing the batch data processing task in the DCN area have completed the task, then changing the status of the batch data processing task to task execution success includes:

[0030] If it is determined that any DCN node executing the batch data processing task in the DCN area has completed the task, the status of the DCN node in the fourth data table is modified to completed, and the number of batch data processing task notifications sent to each DCN node corresponding to the task execution identification information in the third data table is decremented;

[0031] If it is determined that the number of batch data processing task notifications sent to each DCN node corresponding to the task execution identification information in the third data table is zero, it is determined that all DCN nodes executing the batch data processing task in the DCN area have completed the task.

[0032] In the embodiment of the present invention, after the number in the third data table is reduced to zero, it can be determined that the DCN node processing the task has completed the task.

[0033] In one aspect, an embodiment of the present invention provides a batch data processing device, comprising:

[0034] A task execution identification information determining unit is used to determine the task execution identification information for the task request and modify the status of the task to being executed after obtaining the batch data processing task request in the data management area and determining that the task has not been executed in the current batch data processing cycle according to the task identification information in the task request;

[0035] A DCN node task processing unit is configured to, if it is determined that the task needs to be sent to a data communication network DCN area, determine whether there is a DCN node in the DCN area that is executing the task according to the task execution identification information; if it is determined that there is no DCN node in the DCN area that is executing the task, send the task to at least one DCN node in the DCN area so that the DCN node processes the task, and modify the execution state of the DCN node to being executed;

[0036] The task determination unit is configured to modify the status of the task to successful execution of the task if it is determined that all DCN nodes executing the task in the DCN area have completed the task.

[0037] Optionally, the task execution identification information determining unit is specifically used to:

[0038] If it is determined that the task identification information is not found in the first data table in the current batch data processing cycle, it is determined that the task has not been executed in the current batch data processing cycle. The first data table is used to record the task identification information of batch data processing tasks executed in different batch data processing cycles.

[0039] Optionally, the DCN node task processing unit is specifically used to:

[0040] If the task execution identification information is not found in the second data table of the data management area, and the task execution identification information is not found in the third data table of the data management area, it is determined that there is no DCN node in the DCN area that is executing the task, and the second data table is used to record the task execution identification information executed in different batch data processing cycles, the DCN nodes corresponding to the task execution identification information, and the task execution status of each DCN node, and the third data table is used to record the task execution identification information executed in different batch data processing cycles and the number of batch data processing task notifications sent to each DCN node corresponding to the task execution identification information.

[0041] Optionally, the DCN node task processing unit is further used for:

[0042] It is determined that the task execution identification information is not found in a fourth data table of each DCN node in the DCN area, wherein the fourth data table is used to record the task execution status of each DCN node for batch data processing tasks corresponding to different task execution identification information.

[0043] Optionally, the task execution identification information determining unit is further used to:

[0044] If it is determined that the task identification information can be queried in the first data table within the current batch data processing cycle, determining the task execution status corresponding to the task identification information;

[0045] If it is determined that the task execution status is task failure, the step of determining whether the batch data processing task needs to be sent to the data communication network DCN area is performed, and the status of the batch data processing task is modified to being executed.

[0046] Optionally, the DCN node task processing unit is further used for:

[0047] If the task execution identification information is found in the second data table of the data management area, a DCN node whose task execution status is a failure status is determined among the DCN nodes corresponding to the task execution identification information;

[0048] The task execution status of the DCN node whose task execution status in the second data table is a failed status is modified to being executed, and the step of determining whether the task execution identification information can be found in the third data table in the data management area is performed.

[0049] Optionally, the task determination unit is specifically used to:

[0050] If it is determined that any DCN node executing the batch data processing task in the DCN area has completed the task, the status of the DCN node in the fourth data table is modified to completed, and the number of batch data processing task notifications sent to each DCN node corresponding to the task execution identification information in the third data table is decremented;

[0051] If it is determined that the number of batch data processing task notifications sent to each DCN node corresponding to the task execution identification information in the third data table is zero, it is determined that all DCN nodes executing the batch data processing task in the DCN area have completed the task.

[0052] On the one hand, an embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes steps of a batch data processing method implemented by the computer program.

[0053] On the one hand, an embodiment of the present invention provides a computer-readable storage medium storing a computer program executable by a computer device, which enables the computer device to execute the steps of a batch data processing method when the program is run on the computer device. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1A flow chart of a batch data processing method provided by an embodiment of the present invention;

[0055] Figure 2 A schematic diagram of the structure of a distributed resource processing system provided by an embodiment of the present invention;

[0056] Figure 3 A flow chart of a batch data processing method provided by an embodiment of the present invention;

[0057] Figure 4 A schematic diagram of the structure of a batch data processing device provided by an embodiment of the present invention;

[0058] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and beneficial effects of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] Based on the problems existing in the prior art, an embodiment of the present invention provides a batch data processing method applied to a distributed resource processing system, specifically, Figure 1 As shown, including:

[0061] Step S101, after the data management area obtains a batch data processing task request, if it is determined according to the task identification information in the task request that the task has not been executed in the current batch data processing cycle, the task execution identification information is determined for the task request, and the status of the task is modified to being executed.

[0062] Specifically, in an embodiment of the present invention, the distributed resource processing system may be a financial management system or sales system of a financial institution or other system that requires periodic batch data processing. For example, if the distributed resource processing system is a financial management system, periodic batch data processing may be understood as running batches.

[0063] In the embodiment of the present invention, the distributed resource processing system can be divided into a data management area and a data communication network DCN (Data Communication Network, data communication network) area, such as Figure 2 As shown, in an embodiment of the present invention, the data management area has multiple hosts for data processing, the data management area communicates with at least one node in a DCN area, and each node in the DCN area has multiple hosts for data processing.

[0064] After the data management area obtains the batch data processing task request, in order to prevent concurrency, that is, the task already exists and is being executed, or the task has been processed and the task result has been obtained, so it is first necessary to determine that the task has not been executed in the current batch data processing cycle. In an embodiment of the present invention, it can be considered that if the task has been executed in the previous batch data processing cycle, but the task has not been executed in the current batch data processing cycle, it can be considered that the task has not been executed. Exemplarily, if the distributed resource processing system is a financial management system, the batch data processing cycle is every day, that is, the batch task is run every day, and the task of determining the daily income for the batch running is assumed to be June 1, 2019, and the task was executed on May 31, 2019 in the previous batch data processing cycle, then the task can be considered that the task has not been executed in the current batch data processing cycle.

[0065] In an embodiment of the present invention, if it is determined that the task has not been executed within the current batch data processing cycle, it is necessary to establish unique identification information for the task, which is the task execution identification information. That is to say, after obtaining the batch data processing task request, the unique task execution identification information is determined based on the current batch data processing cycle and the batch data processing task.

[0066] In order to prevent the concurrency of the task, the state of the task is modified to being executed. When the same task is received, the concurrency of the same task can be prevented by determining that the task exists and is being executed.

[0067] Optionally, in an embodiment of the present invention, if it is determined that the task has been executed within the current batch data processing cycle, the execution status of the executed task is determined; if it is determined that the execution status of the executed task is task success, it can be determined that the task has been completed, and the batch data processing process is exited; if it is determined that the task has been executed within the current batch data processing cycle and the execution status of the task has failed, the task can be restarted, the status of the task can be changed to being executed, and the batch data processing process can continue to be executed.

[0068] Optionally, in an embodiment of the present invention, in order to facilitate the management of tasks and task status and better prevent the concurrency of the same tasks, the task identification information of batch data processing tasks executed in different batch data processing cycles can be recorded through a first data table. For example, as shown in Table 1, the structure of the first data table is as follows. The first data table includes at least task identification information and a batch data processing cycle, which can be expressed by a date, as well as task execution identification information determined based on the task identification information and the date, and the status of the task.

[0069] Table 1

[0070]

[0071] In Table 1, the task identification information executed on different dates, the task execution identification information determined by the task identification information and the date, and the task status corresponding to the task identification information are recorded. For example, as shown in Table 2, on May 31, 2019, the task with task identification information 1 was executed, and the task execution identification information A01 was determined based on the date and task identification information 1, and the execution status of the task was completed. On May 31, 2019, the task with task identification information 2 was executed, and the task execution identification information B01 was determined based on the date and task identification information 2, and the execution status of the task was in progress.

[0072] Table 2

[0073]

[0074] When a batch data processing task request is received in the data management area, the task identification information of the task is task identification information 1, and the date is June 1, 2019. A search is performed in Table 2 to determine that there is no task with task identification information 1 dated June 1, 2019.

[0075] Optionally, in an embodiment of the present invention, the first data table further includes a main step number and a settlement system identifier. The main step number indicates that the batch data processing task is completed through multiple steps. For example, the batch data processing task is to transfer resources to each resource processing subsystem and calculate the total number of resource transfers for the day. The batch data processing task can be divided into two steps, one main step is to transfer resources to each resource processing subsystem, and the other main step is to calculate the total number of resource transfers for the day. The settlement system identifier refers to the settlement system in which the batch data processing task is processed.

[0076] Therefore, it can be seen from the above content that in an embodiment of the present invention, after obtaining a batch data processing task request, it is necessary not only to determine whether the task has been executed through the task identification information, but also to determine whether the task has been executed through the main step number and the settlement system identification.

[0077] Exemplarily, as shown in Table 3, when a batch data processing task request is received in the data management area, the task identification information of the task is task identification information 1, the date is June 1, 2019, the main step number of the task is 1, and the settlement system identification of the task is 01. A search is performed in Table 3 to determine that the task does not exist.

[0078] Table 3

[0079]

[0080] Step S102: If it is determined that the task needs to be sent to the data communication network DCN area, it is determined whether there is a DCN node in the DCN area that is executing the task according to the task execution identification information.

[0081] Specifically, in the embodiment of the present invention, after determining that no other host in the data management area is executing the same task, after determining that the task needs to be sent to the DCN area, it is determined whether there is a DCN node executing the task.

[0082] In an embodiment of the present invention, whether the task needs to be sent to the data communication network DCN area can be determined based on the configuration information of the task request. For example, tasks that need to be processed by the DCN area can be sent to the DCN area, such as transfers, savings, etc., while some services do not need to be sent to the DCN area, such as the business of total amount statistics.

[0083] In the embodiment of the present invention, it is possible to determine whether a DCN node is executing the task in a variety of ways, for example, by determining the status of the DCN node, such as whether it is occupied, or by querying and recording the tasks executed by the DCN node and the task status to determine whether a DCN node is executing the task.

[0084] Optionally, in an embodiment of the present invention, two data tables can be used to determine whether there is a DCN node executing the task. Specifically, if the task execution identification information is not found in the second data table of the data management area, and the task execution identification information is not found in the third data table of the data management area, it is determined that there is no DCN node in the DCN area executing the task. The second data table is used to record the task execution identification information executed in different batch data processing cycles, the DCN nodes corresponding to the task execution identification information, and the task execution status of each DCN node. The third data table is used to record the task execution identification information executed in different batch data processing cycles and the number of batch data processing task notifications sent to each DCN node corresponding to the task execution identification information.

[0085] That is to say, in an embodiment of the present invention, the execution identification information of the executed tasks and the DCN number of the task corresponding to the task identification information are saved in the second data table, and the execution status of each DCN node for the task is recorded, as shown in Table 4. Table 4 is the second data table, which includes the task execution identification information, the DCN number and the date of execution of the task.

[0086] Table 4

[0087]

[0088] By querying in Table 4 whether there is a batch data processing cycle, that is, whether the same task execution identification information has been executed on the set date, if not, it can be considered that no DCN node executes the task.

[0089] In an embodiment of the present invention, in order to avoid the situation where there is no DCN node executing the task in the second data table due to communication delay, but there is actually a DCN node executing the task, a third data table is introduced. By finding the task execution identification information in the third data table, it is determined that there is indeed no DCN node executing the task. The structure of the third data table is shown in Table 5. In Table 5, the number of DCNs corresponding to each task execution identification information represents the number of task execution notifications sent by the data management area to the DCN area. For example, the data management area needs to send task processing notifications to three nodes in the DCN area. In Table 5, the number of DCNs should be 3.

[0090] Table 5

[0091]

[0092] Optionally, in an embodiment of the invention, the DCN number is submitted after the DCN node completes the task. For example, there are three DCN nodes in the DCN area processing tasks, and the DCN number increases from 1 to 3. When the first DCN node has completed the task, the DCN number decreases to indicate that there are still two DCN nodes that have not completed the task.

[0093] Optionally, in an embodiment of the present invention, since the main step number and the settlement system identifier may be included in the first data table, the main step number and the settlement system identifier may also be included in Table 4 and Table 5 to facilitate task query.

[0094] In the embodiment of the present invention, since the second data table and the third data table are both queried in the data management area, in order to prevent communication delays between the data management area and the DCN area, for example, the number of DCNs in the third data table is 2, which means that there are still two DCN nodes that have not completed the task, but in fact, the DCN node has completed the task. In order to send the task to these two DCN nodes, a fourth data table is added to each DCN node in the DCN area. The fourth data table serves as a mapping table of the second data table. The structure is shown in Table 6, and at least includes task execution identification information, the DCN number and date of executing the task.

[0095] Table 6

[0096]

[0097] Of course, Table 6 may also include the main step number and the settlement system identifier.

[0098] In an embodiment of the present invention, if the task identification information is found in the second data table, and at least one DCN node among the DCN nodes corresponding to the identification information has an execution status of execution failure, then in an embodiment of the present invention, the task with a failed status can be restarted, that is, the status of these DCN nodes is changed to being executed, and the step of querying the third data table is continued.

[0099] Step S103: If it is determined that no DCN node in the DCN area is executing the task, the task is sent to at least one DCN node in the DCN area so that the DCN node processes the task, and the execution state of the DCN node is modified to being executed.

[0100] In the embodiment of the present invention, when it is determined that the DCN node needs to process the task, the state of the DCN node processing the task is modified to being executed to avoid task concurrency. That is, through the above content, the task in the first data table is set to be executed, the state of the DCN executing the task in the second data table is modified to be executed, and the number of DCNs in the third data table is not 0 because it is still being executed, and the state of the DCN in the fourth data table is being executed, so when the same task is queried, concurrency can be effectively prevented.

[0101] Step S104: if it is determined that all DCN nodes executing the task in the DCN area have completed the task, the status of the task is modified to task execution success.

[0102] In an embodiment of the present invention, when a DCN node in the fourth data table completes a task, the execution status of the DCN node in the fourth data table will be modified to completed. After the data management area receives a notification of completed task sent by the DCN node, the number of DCNs in the third data table will be first reduced. When the number of DCNs is reduced to 0, it can be considered that each DCN node in the DCN area has completed the task.

[0103] In order to better explain the embodiments of the present application, a batch data processing method provided by the embodiments of the present application is described below in conjunction with a specific implementation scenario. In this method, the batch data processing cycle is one day, and the batch data processing task request is to increase a set amount of funds to the customer's account and count the total amount of funds increased to the customer on that day. In the embodiment of the present invention, the batch data processing task request is divided into two main steps, namely, main step 1, increasing a set amount of funds to the customer's account, the task identifier of which is 1, and main step 2 counting the total amount increased, and the task of main step 1 is assigned to three DCN nodes, namely DCN node 1, DCN node 2 and DCN node 3, as shown in FIG. Figure 3 As shown:

[0104] Step S301, obtaining a batch data processing task request, the main step of the request is 1, and the task identifier is 1;

[0105] Step S302, determining whether the task exists in the first data table, if yes, executing step S303, otherwise executing step S304;

[0106] Step S303, determine whether the task status is successful, if successful, execute step S305, otherwise execute step S306;

[0107] Step S304, determining the task execution identification information for the task, and executing step S307;

[0108] Step S305, determine that the task is successfully executed and exit the process;

[0109] Step S306, modify the execution status of the task in the first data table to being executed, and execute step S304;

[0110] Step S307, determining whether the task needs to be distributed to the DCN area, if so, executing step S308, otherwise executing step S309;

[0111] Step S308, querying the second data table whether the task has been completed by the DCN node, if so, returning to step S305, otherwise executing step S310;

[0112] Step S309, the data management area processes the task and executes step S311;

[0113] Step S310, querying the third data table whether the task has been completed by the DCN node, if so, returning to step S305, otherwise executing step 312;

[0114] Step S311, determine whether the data management area has completed processing the task, if yes, execute step S305, otherwise execute step S309;

[0115] Step S312, sending the task to DCN node 1, DCN node 2 and DCN node 3, and updating the status of DCN node 1, DCN node 2 and DCN node 3 in the second data table as being executed;

[0116] Step S313, querying the fourth data table at each DCN node;

[0117] Step S314: If it is determined that the task does not exist in the fourth data table, execute the task.

[0118] The above embodiment only introduces the process of issuing tasks. The embodiment of the present invention also includes a process for determining the completion of a task. The process is described as follows: when the DCN node completes the task, the DCN status in the fourth data table is updated; after the data management area receives the notification that the DCN node has completed the task, the DCN quantity in the third data table is decremented, and the DCN status in the second data table is updated at the same time; when the DCN quantity is decremented to 0, it can be determined that the task is completed, and the task status in the first data table is updated.

[0119] Based on the same technical concept, the embodiment of the present application provides a batch data processing device, such as Figure 4 As shown, including:

[0120] The task execution identification information determining unit 401 is used to determine the task execution identification information for the task request and modify the status of the task to being executed after obtaining the batch data processing task request in the data management area and determining that the task has not been executed in the current batch data processing cycle according to the task identification information in the task request;

[0121] The DCN node task processing unit 402 is configured to, if it is determined that the task needs to be sent to the data communication network DCN area, determine whether there is a DCN node in the DCN area that is executing the task according to the task execution identification information; if it is determined that there is no DCN node in the DCN area that is executing the task, send the task to at least one DCN node in the DCN area so that the DCN node processes the task, and modify the execution status of the DCN node to being executed;

[0122] The task determining unit 403 is configured to modify the status of the task to successful execution of the task if it is determined that all DCN nodes executing the task in the DCN area have completed the task.

[0123] Optionally, the task execution identification information determining unit 401 is specifically used to:

[0124] If it is determined that the task identification information is not found in the first data table in the current batch data processing cycle, it is determined that the task has not been executed in the current batch data processing cycle. The first data table is used to record the task identification information of batch data processing tasks executed in different batch data processing cycles.

[0125] Optionally, the DCN node task processing unit 402 is specifically configured to:

[0126] If the task execution identification information is not found in the second data table of the data management area, and the task execution identification information is not found in the third data table of the data management area, it is determined that there is no DCN node in the DCN area that is executing the task, and the second data table is used to record the task execution identification information executed in different batch data processing cycles, the DCN nodes corresponding to the task execution identification information, and the task execution status of each DCN node, and the third data table is used to record the task execution identification information executed in different batch data processing cycles and the number of batch data processing task notifications sent to each DCN node corresponding to the task execution identification information.

[0127] Optionally, the DCN node task processing unit 402 is further configured to:

[0128] It is determined that the task execution identification information is not found in a fourth data table of each DCN node in the DCN area, wherein the fourth data table is used to record the task execution status of each DCN node for batch data processing tasks corresponding to different task execution identification information.

[0129] Optionally, the task execution identification information determining unit 401 is further used to:

[0130] If it is determined that the task identification information can be queried in the first data table within the current batch data processing cycle, determining the task execution status corresponding to the task identification information;

[0131] If it is determined that the task execution status is task failure, the step of determining whether the batch data processing task needs to be sent to the data communication network DCN area is performed, and the status of the batch data processing task is modified to being executed.

[0132] Optionally, the DCN node task processing unit 402 is further configured to:

[0133] If the task execution identification information is found in the second data table of the data management area, a DCN node whose task execution status is a failure status is determined among the DCN nodes corresponding to the task execution identification information;

[0134] The task execution status of the DCN node whose task execution status in the second data table is a failed status is modified to being executed, and the step of determining whether the task execution identification information can be found in the third data table in the data management area is performed.

[0135] Optionally, the task determination unit 403 is specifically configured to:

[0136] If it is determined that any DCN node executing the batch data processing task in the DCN area has completed the task, the status of the DCN node in the fourth data table is modified to completed, and the number of batch data processing task notifications sent to each DCN node corresponding to the task execution identification information in the third data table is decremented;

[0137] If it is determined that the number of batch data processing task notifications sent to each DCN node corresponding to the task execution identification information in the third data table is zero, it is determined that all DCN nodes executing the batch data processing task in the DCN area have completed the task.

[0138] Based on the same technical concept, the embodiment of the present application provides a computer device, such as Figure 5 As shown, it includes at least one processor 501 and a memory 502 connected to the at least one processor. The specific connection medium between the processor 501 and the memory 502 is not limited in the embodiment of the present application. Figure 5 For example, the processor 501 and the memory 502 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0139] In the embodiment of the present application, the memory 502 stores instructions that can be executed by at least one processor 501. The at least one processor 501 can execute the steps included in the aforementioned batch data processing method by executing the instructions stored in the memory 502.

[0140] The processor 501 is the control center of the computer device, and can use various interfaces and lines to connect various parts of the terminal device, and obtain the client address by running or executing instructions stored in the memory 502 and calling data stored in the memory 502. Optionally, the processor 501 may include one or more processing units, and the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 501. In some embodiments, the processor 501 and the memory 502 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0141] The processor 501 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0142] The memory 502 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 502 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 502 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 502 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0143] Based on the same technical concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the program runs on the computer device, the computer device executes the steps of the batch data processing method.

[0144] A person skilled in the art can understand that: all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0145] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0146] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A batch data processing method, characterized in that: Applied to a distributed resource processing system, the method comprises: After the data management area obtains the batch data processing task request, if it is determined according to the task identification information in the task request that the task has not been executed in the current batch data processing cycle, the task execution identification information is determined for the task request, and the status of the task is modified to being executed; If it is determined that the task needs to be sent to the data communication network DCN area, determining whether there is a DCN node in the DCN area that is executing the task according to the task execution identification information; If the task execution identification information is not found in the second data table of the data management area, and the task execution identification information is not found in the third data table of the data management area, it is determined that there is no DCN node in the DCN area that is executing the task, the second data table is used to record the task execution identification information executed in different batch data processing cycles, each DCN node corresponding to the task execution identification information, and the task execution status of each DCN node, and the third data table is used to record the task execution identification information executed in different batch data processing cycles and the number of batch data processing task notifications sent to each DCN node corresponding to the task execution identification information; then the task is sent to at least one DCN node in the DCN area so that the DCN node processes the task, and the execution status of the DCN node is modified to being executed; If it is determined that all DCN nodes executing the task in the DCN area have completed the task, the status of the task is modified to task execution success.

2. The method according to claim 1, characterized in that The determining, according to the task identification information in the task request, that the task has not been executed in the current batch data processing cycle includes: If it is determined that the task identification information is not found in the first data table in the current batch data processing cycle, it is determined that the task has not been executed in the current batch data processing cycle. The first data table is used to record the task identification information of batch data processing tasks executed in different batch data processing cycles.

3. The method according to claim 1, characterized in that If the task execution identification information is not found in the second data table of the data management area, and the task execution identification information is not found in the third data table of the data management area, before determining that no DCN node in the DCN area is executing the task, the method further includes: It is determined that the task execution identification information is not found in a fourth data table of each DCN node in the DCN area, wherein the fourth data table is used to record the task execution status of each DCN node for batch data processing tasks corresponding to different task execution identification information.

4. The method according to claim 2, characterized in that: The method further comprises: If it is determined that the task identification information can be queried in the first data table within the current batch data processing cycle, determining the task execution status corresponding to the task identification information; If it is determined that the task execution status is task failure, the step of determining whether the batch data processing task needs to be sent to the data communication network DCN area is performed, and the status of the batch data processing task is modified to being executed.

5. The method according to claim 1, characterized in that The method further comprises: If the task execution identification information is found in the second data table of the data management area, a DCN node whose task execution status is a failure status is determined among the DCN nodes corresponding to the task execution identification information; The task execution status of the DCN node whose task execution status in the second data table is a failed status is modified to being executed, and the step of determining whether the task execution identification information can be found in the third data table in the data management area is performed.

6. The method according to claim 3, characterized in that If it is determined that all nodes executing the batch data processing task in the DCN area have completed the task, then the status of the batch data processing task is modified to task execution success, including: If it is determined that any DCN node executing the batch data processing task in the DCN area has completed the task, the status of the DCN node in the fourth data table is modified to completed, and the number of batch data processing task notifications sent to each DCN node corresponding to the task execution identification information in the third data table is decremented; If it is determined that the number of batch data processing task notifications sent to each DCN node corresponding to the task execution identification information in the third data table is zero, it is determined that all DCN nodes executing the batch data processing task in the DCN area have completed the task.

7. A batch data processing device, characterized in that: The device comprises: A task execution identification information determining unit is used to determine the task execution identification information for the task request and modify the status of the task to being executed after obtaining the batch data processing task request in the data management area and determining that the task has not been executed in the current batch data processing cycle according to the task identification information in the task request; A DCN node task processing unit, configured to, if it is determined that the task needs to be sent to a DCN area of ​​a data communication network, determine whether there is a DCN node in the DCN area that is executing the task according to the task execution identification information; if the task execution identification information is not found in a second data table in a data management area, and the task execution identification information is not found in a third data table in the data management area, determine that there is no DCN node in the DCN area that is executing the task, the second data table is used to record task execution identification information executed in different batch data processing cycles, each DCN node corresponding to the task execution identification information, and the task execution status of each DCN node, and the third data table is used to record task execution identification information executed in different batch data processing cycles and the number of batch data processing task notifications sent to each DCN node corresponding to the task execution identification information; then send the task to at least one DCN node in the DCN area, so that the DCN node processes the task, and changes the execution status of the DCN node to being executed; The task determination unit is configured to modify the status of the task to successful execution of the task if it is determined that all DCN nodes executing the task in the DCN area have completed the task.

8. The device according to claim 7, characterized in that The task execution identification information determination unit is specifically used for: If it is determined that the task identification information is not found in the first data table in the current batch data processing cycle, it is determined that the task has not been executed in the current batch data processing cycle. The first data table is used to record the task identification information of batch data processing tasks executed in different batch data processing cycles.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: The computer stores a computer program executable by a computer device. When the program is run on the computer device, the computer executes the method according to any one of claims 1 to 6.

Images