Batch data processing method, device, system, computer equipment and computer readable storage medium

Abstract

The embodiments of the present application provide a batch data processing method, device, system, computer equipment and computer-readable storage medium. It belongs to the field of data processing technology. When batch data processing is performed, the computer equipment corresponding to the dispatching control center receives the first data, obtains the computer equipment corresponding to the dispatching node that is pre-marked as the effective state as the first type of dispatching node, sends the first data to the first type of dispatching node, and the first type of dispatching node distributes the first data to the processing node corresponding to the computer equipment scheduled by itself according to the first preset allocation algorithm, and the processing node processes the first data according to the first preset processing method corresponding to the first data. Since there are multiple dispatching nodes, each dispatching node and each group composed of the processing nodes dispatched by it can work independently, batch data is switched between different groups for processing, external services are not interrupted, and service continuity is guaranteed, thereby improving the efficiency of batch data processing.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a batch data processing method, apparatus, system, computer equipment and computer-readable storage medium. Background Art

[0002] There are generally two types of cluster application machines included in IT systems, which are used to process real-time online transactions and asynchronous batch transactions. For real-time online transactions, the processing results need to be returned to the front-end system at a specified time. For example, when a customer initiates a transfer transaction in the online bank, the customer will be informed within 10 seconds whether the transaction is successful or failed. For some large-volume transactions, such as automatic credit card repayments, which may involve batch deductions from millions of customers on the same day, asynchronous batch transaction processing is more suitable. The transaction details data can be generated into batch files and sent to the core system for batch deductions.

[0003] For real-time online trading machines, each machine node is independent and unrelated. The traditional way to deploy versions for real-time online trading machines is to mark the specified machine as unavailable in the load balancing device, and wait a few minutes before starting to deploy versions after the existing transactions of these machines are processed (deploying versions means updating the software system). Since only some machines stop service each time, other machines are still providing services normally, so the system is continuously providing services and external services are not interrupted.

[0004] For asynchronous batch transaction machinery, see Figure 1 , Figure 1 A schematic architecture diagram of the asynchronous batch transaction machine workflow involved in the asynchronous processing of batch data in the traditional technology provided in the embodiment of the present application requires a scheduling node to schedule data files, dispatch the data to the processing node for processing, and then collect the processing results of each processing node, summarize the processing results and return them to the front end. Since the scheduling node and the processing node need to interact, the scheduling is unified by the scheduling node. If a version needs to be deployed on the scheduling node, the scheduling node needs to stop working, and the processing node cannot work normally. In the traditional way, all batch transaction machines need to stop service at the same time, and the service can only be provided again after the version deployment on the scheduling node is completed and the service is restarted. The system's external service is completely interrupted during the deployment of the version.

[0005] Therefore, in traditional technology, for asynchronous processing of online batch files, planned downtime means stopping external services. For example, if a new version is to be deployed, it takes a downtime of several hours, resulting in low efficiency of external services. Summary of the invention

[0006] The embodiments of the present application provide a batch data processing method, apparatus, system, computer equipment and computer-readable storage medium, which can solve the problem of low efficiency in asynchronous processing of batch files in traditional technologies.

[0007] In the first aspect, an embodiment of the present application provides a batch data processing method applied to a scheduling control center, wherein the scheduling control center controls a number of scheduling nodes, each of which schedules a number of processing nodes, and the method includes: receiving first data sent by an external data source; obtaining scheduling nodes that are pre-marked as effective as first-type scheduling nodes; sending the first data to the first-type scheduling nodes according to a first preset sending method, so that the first-type scheduling nodes allocate the first data to the processing nodes scheduled by themselves according to a first preset allocation algorithm, so that the processing nodes process the first data according to a first preset processing method corresponding to the first data.

[0008] In the second aspect, an embodiment of the present application also provides a batch data processing method applied to a scheduling node, wherein the scheduling node is controlled by a scheduling control center, which controls a number of scheduling nodes, and each scheduling node schedules a number of processing nodes. The method includes: sending an access request to a mutex lock contained in the scheduling control center, and the mutex lock is used to ensure that only one scheduling node is in an effective state at the same time; judging whether to obtain the mutex lock; if the mutex lock is obtained, receiving second data sent by the scheduling control center; and allocating the second data to the processing node scheduled by itself according to a second preset allocation algorithm, so that the processing node processes the second data according to a second preset processing method corresponding to the second data.

[0009] In a third aspect, an embodiment of the present application further provides a batch data processing device, comprising a unit for executing the batch data processing method applied to a scheduling control center, or a unit for executing the batch data processing method applied to a scheduling node.

[0010] In a fourth aspect, an embodiment of the present application further provides a batch data processing system, the system comprising a scheduling control center, the scheduling control center controlling a number of scheduling nodes, each of the scheduling nodes scheduling a number of processing nodes; wherein the scheduling control center is used to execute the steps of the batch data processing method applied to the scheduling control center, the scheduling nodes are used to execute the steps of the batch data processing method applied to the scheduling nodes, and the processing nodes are used to execute the steps of processing the allocated data according to a preset processing method corresponding to the received allocated data.

[0011] In the fifth aspect, an embodiment of the present application also provides a computer device, comprising a first memory and a first processor connected to the first memory, or comprising a second memory and a second processor connected to the second memory; the first memory is used to store a first computer program, and the second memory is used to store a second computer program; the first processor is used to run the first computer program to execute the steps of the batch data processing method applied to the scheduling control center, and the second processor is used to run the second computer program to execute the steps of the batch data processing method applied to the scheduling node.

[0012] In the sixth aspect, an embodiment of the present application also provides a computer-readable storage medium, which stores a first computer program and / or a second computer program. When the first computer program is executed by a processor, it can implement the steps of the batch data processing method applied to the scheduling control center. When the second computer program is executed by a processor, it can implement the steps of the batch data processing method applied to the scheduling node.

[0013] The embodiment of the present application provides a batch data processing method, device, system, computer equipment and computer-readable storage medium. When the embodiment of the present application implements batch data processing, the corresponding application system includes a scheduling control center, the scheduling control center controls a number of scheduling nodes, each of the scheduling nodes schedules a number of processing nodes, the scheduling control center receives the first data sent by the external data source, obtains the scheduling nodes pre-marked as effective as the first type of scheduling nodes, sends the first data to the first type of scheduling nodes according to the first preset sending method, the first type of scheduling nodes allocate the first data to the processing nodes scheduled by themselves according to the first preset allocation algorithm, the processing nodes process the first data according to the first preset processing method corresponding to the first data, and after the processing nodes complete the processing of the first data, the processing results are returned to the scheduling control center through the scheduling node, so that the scheduling control center summarizes all the processing results as the final result of the batch data processing, and outputs the final result to complete the batch data processing. Since there are multiple scheduling nodes, each scheduling node schedules several processing nodes, and all scheduling nodes are uniformly controlled and managed by the scheduling control center. Therefore, each scheduling node and each group composed of the processing nodes it schedules can work independently. If one of the groups is in a shutdown state, the batch data processing work can be switched to other backup groups for processing. For example, if a new version is to be deployed or a planned shutdown such as maintenance is to be carried out for one of the groups, or if a failure occurs in one of the groups, resulting in an unplanned shutdown, the batch data processing can be switched to other groups for processing. Thus, by switching the batch data between different groups for processing, the system can ensure that external services are not interrupted during planned shutdowns such as version deployment or unplanned shutdowns such as failures, and realize grayscale deployment of online batch files. Especially for application environments with relatively high requirements for data processing continuity, the continuity of system services is guaranteed and the efficiency of batch data processing is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0015] Figure 1 A schematic architecture diagram of an asynchronous batch transaction machine workflow involved in batch data processing in the conventional technology provided in an embodiment of the present application;

[0016] Figure 2 A schematic diagram of an application environment in the batch data processing method provided in an embodiment of the present application;

[0017] Figure 3 A schematic diagram of a flow chart of a batch data processing method applied to a dispatching control center provided in an embodiment of the present application;

[0018] Figure 4 A schematic diagram of a flow chart of a batch data processing method applied to a scheduling node provided in an embodiment of the present application;

[0019] Figure 5 A schematic block diagram of a batch data processing device applied to a dispatching control center provided in an embodiment of the present application;

[0020] Figure 6 A schematic block diagram of a batch data processing device applied to a scheduling node provided in an embodiment of the present application; and

[0021] Figure 7 A schematic block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0024] See also Figure 2 , Figure 2 This is a schematic diagram of an application environment in the batch data processing method provided in the embodiment of the present application. Figure 2 As shown, the application environment includes a scheduling control center, the scheduling control center includes scheduling node 1 and scheduling node 2, scheduling node 1 is configured to include processing node 1 and processing node 2, scheduling node 2 is configured to include processing node 3, processing node 4 and processing node 5, wherein the solid line with an arrow indicates that it is currently configured in a connected state, and the dotted line with an arrow indicates that it can be configured in a connected state. Specifically, the various subjects included are as follows:

[0025] 1) Dispatching control center. A computer device that implements dispatching control, controls several dispatching nodes, switches and manages dispatching nodes, sends received external data to the currently effective dispatching node, receives data processing results returned by the dispatching node, and outputs the data processing results, thereby completing data processing.

[0026] 2) Scheduling nodes. A computer device for implementing a scheduling processing node includes at least two scheduling nodes, wherein the currently available scheduling node that is pre-marked as effective is a main scheduling node for processing data and is set as a first-class scheduling node for receiving data sent by the scheduling control center and responsible for distributing the data to the processing nodes that the scheduling node is configured to schedule, receiving the data processing results returned by the processing nodes, and returning the data processing results to the scheduling control center. The currently available scheduling node that is not in an effective state is a backup scheduling node for processing data and is set as a second-class scheduling node to replace the first-class scheduling node at any time as needed. Whether the scheduling node is in an available state can be set manually through a visualization page for planned downtime, such as when deploying a version of the node. A scheduling node that is set to be unavailable does not receive data sent by the scheduling control center. The scheduling control center only sends data to available scheduling nodes that are in an effective state, thereby realizing switching between multiple scheduling nodes, and external services do not need to be interrupted. Grayscale deployment of the system can be realized, solving the problems of low efficiency and poor scalability in the current scheduling process;

[0027] Furthermore, the scheduling node 1 may be configured to include any of the processing nodes 1, 2, 3, 4 and 5. Figure 2 As shown, the scheduling node 1 is configured as the processing node 1 and the processing node 2. Similarly, the scheduling node 2 can also be configured to include any node among the processing node 1, the processing node 2, the processing node 3, the processing node 4 and the processing node 5. Figure 2 As shown, scheduling node 1 is configured as processing node 3, processing node 4 and processing node 5.

[0028] 3) Processing node: A computer device that implements specific data processing, used to receive data distributed by the scheduling node, process the distributed data to obtain processing results, and return the processing results to the scheduling node.

[0029] The physical machines corresponding to the dispatching control center, the dispatching nodes and the processing nodes may be the same, but the functions they implement are different. For example, they may all be hosts in computer devices.

[0030] Figure 1The working process of each subject in is as follows: scheduling node 1 and scheduling node 2 are set to available state, scheduling node 1 and scheduling node 2 each send their own node state to the scheduling control center, and then access the mutex lock contained in the scheduling control center. The scheduling control center marks the scheduling node that obtains the mutex lock as a first-class scheduling node, which is the main scheduling node, that is, the currently effective scheduling node. Other scheduling nodes are nodes in an ineffective state and are marked as second-class scheduling nodes, that is, backup scheduling nodes. The scheduling control center receives the first data sent by the external data source, obtains the scheduling node that is currently in a pre-marked effective state as the first type of scheduling node, and sends the first data to the first type of scheduling node. After receiving the first data, the first type of scheduling node distributes the first data to the processing node scheduled by itself, and the processing node processes the first data. After the processing node completes the processing of the first data, it obtains the data processing result, and returns the data processing result to the scheduling node. The scheduling node returns the data processing result to the scheduling control center, and the scheduling control center integrates all data processing results to obtain the final processing result of the batch data, and outputs the final processing result, thereby completing the processing of the batch data.

[0031] It should be noted that the application scenario of the above-mentioned batch data processing method is only used to illustrate the technical solution of the present application, and is not used to limit the technical solution of the present application. The above-mentioned connection relationship may also have other forms.

[0032] please Figure 2 and Figure 3 , Figure 3 This is a flow chart of a batch data processing method applied to a dispatching control center provided in an embodiment of the present application. Figure 2 As shown, the method is applied to a dispatching control center, the dispatching control center controls a number of dispatching nodes, each of the dispatching nodes dispatches a number of processing nodes, and the method is applied to the dispatching control center, such as Figure 3 As shown, the method includes the following steps S301-S303:

[0033] S301: Receive first data sent by an external data source.

[0034] Specifically, for the processing of batch data, a batch transaction machine is generally set up, that is, multiple processing nodes adopt distributed processing. In the embodiment of the present application, since the application environment includes multiple scheduling nodes, each scheduling node includes several processing nodes, it is first necessary to configure the processing nodes corresponding to the batch transaction machine to different scheduling nodes. The processing nodes can be set by personnel through the setting page to realize the configuration of the processing nodes to different scheduling nodes, so that each scheduling node contains some processing nodes, and each scheduling node and the several processing nodes contained therein are regarded as a group, thereby realizing the grouping of scheduling nodes and processing nodes, each group can process data separately, and multiple groups will be obtained. Multiple groups are uniformly managed and scheduled by the scheduling control center, which can realize the main scheduling node and the standby scheduling node in the embodiment of the present application. Work in rotation, so as to achieve uninterrupted external services through the main and standby switching, and realize the grayscale deployment of online batch files. Grayscale deployment means that when some machines included in the system are planned or unplanned, the system can still provide services continuously and will not suspend services.

[0035] After configuring multiple processing nodes to different scheduling nodes, since the application environment contains multiple scheduling nodes coexisting, and the scheduling control center manages the multiple scheduling nodes uniformly according to the status of each scheduling node, the status of each scheduling node can be registered with the scheduling control center to send its own status when starting the service initialization. After the scheduling node successfully registers with the scheduling control center, it is set to be in an available state by the scheduling control center. The scheduling node that has not successfully registered with the scheduling control center is regarded as an unavailable scheduling node. The unavailable scheduling node can be in a planned shutdown state or an unplanned shutdown state. Subsequently, each scheduling node can report the status of this node to the scheduling control center regularly according to a preset period. The scheduling node that has not reported the status of this scheduling node is regarded as an unavailable scheduling node, so that the scheduling control center can continuously obtain the scheduling node in the available state, so that the scheduling control center can continuously obtain whether the scheduling node is in an available state. When the scheduling control center distributes data, it can distribute it according to the status of all global machines to prevent the status of the scheduling node from being changed without the scheduling control center's knowledge, and ensure that the master-slave switch is in a sustainable state. The status of the scheduling node can be set manually through a visualization page. For example, for planned downtime, such as version deployment, the scheduling node can be set to an unavailable state by personnel. For unplanned downtime, such as machine failure, the status of the scheduling node can be automatically detected by computer equipment. For example, after the scheduling node detects the failure, the status information can be reported to the scheduling control center. When the machine fails, the maintenance personnel can be notified through machine alarms, etc., so that the maintenance personnel can set the status of the scheduling node to an unavailable state through the setting page, thereby continuously ensuring that the scheduling control center can obtain the latest status of the scheduling node.

[0036] After obtaining the status of the scheduling node, the scheduling control center marks all available scheduling nodes as available scheduling nodes in effective state and available scheduling nodes in ineffective state according to the pre-settings. The scheduling node in effective state is the current main scheduling node, that is, the first type of scheduling node, which is used to process batch data. The scheduling node in ineffective state is the backup scheduling node, that is, the second type of scheduling node, which is used for backup processing data. When the status of the main processing node changes and becomes unavailable, the scheduling control center switches the scheduling node in the current effective state to the backup scheduling node.

[0037] When performing batch data processing, the scheduling control center receives batches of first data sent by an external data source, and sends the first data to the currently effective first scheduling node for processing according to the node status of the scheduling node.

[0038] Furthermore, the scheduling control center receives batch data sent by an external data source, and before sending the batch data to the currently effective scheduling node, it can divide the batch data into several data blocks according to a pre-set block setting, and send the divided data blocks to the currently effective scheduling node.

[0039] Specifically, the received batch data can be divided into smaller data blocks according to the timeliness requirements of the batch data and the resource status of the available processing nodes, and then the divided data blocks are sent to the currently effective scheduling node. For example, if the online batch file corresponding to the batch data is 3M, the result is required to be returned in 1 minute, and there are N threads available, the online batch file can be divided into blocks according to the number of threads, the size of the online batch file and the timeliness requirements. Or the online batch file can be divided into blocks according to a fixed size, for example, with 5000 communication records as a unit, each online transaction information is a communication record, if the online batch file contains less than 5000 communication records, it is 1 block of data, if the online batch file contains 5001 to 10000 communication records, it is divided into 2 blocks of data, if the online batch file contains 10001 to 15000 communication records, it is divided into 3 blocks of data, and so on. By dividing the batch data into blocks, the efficiency of the scheduling node in allocating the data to the processing nodes can be improved, and the efficiency of the collaborative processing of the data by the multiple processing nodes can also be improved.

[0040] S302: Acquire a scheduling node that is pre-marked as being in a valid state as a first-type scheduling node.

[0041] Specifically, after the scheduling node is successfully registered in the scheduling control center, the scheduling control center will pre-mark all available scheduling nodes according to the preset settings, and mark all scheduling nodes into two categories. Some scheduling nodes will be pre-marked as currently in an effective state and can process batch data. They are the first type of scheduling nodes. Some scheduling nodes will be pre-marked as currently in an ineffective state and will not process batch data. They serve as backup nodes and are the second type of scheduling nodes.

[0042] The scheduling control center processes the received first data according to its own pre-set processing algorithm, for example, after cutting the data into blocks, it obtains the scheduling nodes that are pre-marked as effective as the first-type scheduling nodes, that is, the scheduling nodes that are currently in the effective state are regarded as the first-type scheduling nodes, and sends the first data to the first-type scheduling nodes so that the processing nodes in the group where the first scheduling nodes are located process the first data.

[0043] S303. Send the first data to the first type of scheduling node according to a first preset sending method, so that the first type of scheduling node allocates the first data to the processing node scheduled by itself according to a first preset allocation algorithm, so that the processing node processes the first data according to the first preset processing method corresponding to the first data.

[0044] Among them, the first preset sending method refers to a method for selecting the scheduling node for sending the first data, mainly referring to how to distribute batch data between multiple scheduling nodes in a valid state if there are multiple scheduling nodes that are currently pre-marked as valid.

[0045] Among them, the first preset allocation algorithm can be an algorithm for allocating data from a scheduling node to a processing node in traditional technology. For example, after the scheduling node cuts the first data into data blocks, it evenly distributes all the data blocks to all the processing nodes scheduled by itself, or it can allocate the allocated data blocks in an inverse correlation with the busyness according to the busy procedures of all the processing nodes scheduled by itself.

[0046] Specifically, the scheduling control center sends the first data to the first type of scheduling nodes according to the number of scheduling nodes included in the first type of scheduling nodes and in accordance with the first preset sending method, so that the scheduling nodes included in the first type of scheduling nodes process the first data. If the first type of scheduling nodes includes one scheduling node, the first data is sent to the scheduling node included in the first type of scheduling nodes. If the first type of scheduling nodes includes two or more scheduling nodes, the first data is sent to each scheduling node included in the first type of scheduling nodes, so that each node processes the data.

[0047] The dispatch control center sends the first data to the first type of dispatch node. After receiving the first data, the first type of dispatch node distributes the first data to the processing node dispatched by itself according to the first preset distribution algorithm preset by itself. The processing node dispatched by itself may be a plurality of processing nodes. After receiving the distributed first data, the processing node dispatched by itself processes the first data according to the first preset processing method corresponding to the first data, mainly according to the business related to the batch data. For example, for the batch transaction data of deduction, the processing node performs deduction processing, for the batch transaction data of payment, the processing node performs payment processing, for the batch transaction data of shopping, the processing node generates transaction-related transaction information, etc., and processes the data according to specific business needs. Among them, after the service is started, the processing node, for example, the processing node corresponding to each server in the distributed system, will be in a working state as long as it is in a normal state. After receiving the first data, it directly processes the first data according to the algorithm configuration corresponding to the first preset processing method. Each of the processing nodes returns its own data processing results to the scheduling node, and the scheduling node returns the data processing results to the scheduling control center. After the scheduling control center integrates all data processing results, it takes the final result of batch data processing and outputs the final result to complete the batch data processing.

[0048] In the embodiment of the present application, since multiple scheduling nodes are allowed to coexist, there are other available scheduling nodes in standby state, that is, the second type of scheduling nodes, which can replace the first type of scheduling nodes that are currently in effective state at any time, and since the scheduling control center is responsible for managing the scheduling nodes, the scheduling control center can switch between different scheduling nodes according to the effective state and busyness of the scheduling nodes, so that the received batch data, such as the batch data corresponding to the online asynchronous batch file, is sent to the currently effective scheduling node, and then the currently effective scheduling node allocates it to the processing node corresponding to the currently effective scheduling node for processing, so that the batch processing machine can provide services to the outside world, and the scheduling nodes in the standby state can be deployed for planned or unplanned shutdowns such as versions, so as to achieve grayscale deployment for batch data, especially for online asynchronous batch files. For example, please continue to refer to Figure 2, assuming that the machines in group A are ready to deploy a version, all machines in group A can be marked as unavailable, and the scheduling node currently in effect automatically switches from scheduling node 1 to scheduling node 2, scheduling node 1 becomes a standby node, and scheduling node 2 becomes the main scheduling node. All machines in group A only process existing data and no longer receive new data. After the existing data is processed, the version can be deployed. After the machines in group A deploy the version and restart the service, the machines in group A are marked as normal, allowing them to receive and process new data files, and then the version deployment operation of the machines in group A is repeated for the machines in group B. In the embodiment of the present application, the improved asynchronous batch transaction machine can also continue to provide services during version deployment, and there is no interruption in external services.

[0049] In summary, when the embodiment of the present application implements batch data processing, since there are multiple scheduling nodes, each scheduling node schedules several processing nodes, and all scheduling nodes are uniformly controlled and managed by the scheduling control center, each scheduling node and each group composed of the processing nodes it schedules can work independently. If one of the groups is in a shutdown state, the batch data processing work can be switched to other standby groups for processing. For example, if a new version is to be deployed or a planned shutdown such as maintenance is to be carried out for one of the groups, or if a failure occurs in one of the groups, resulting in an unplanned shutdown, the batch data processing can be switched to other groups for processing, thereby ensuring that the system is not interrupted in external services during planned shutdowns such as version deployment or unplanned shutdowns such as failures, and realizing grayscale deployment of online batch files, especially for application environments with relatively high requirements for data processing continuity, ensuring the continuity of system services and improving the efficiency of batch data processing.

[0050] In one embodiment, before the step of acquiring the scheduling node pre-marked as a valid state as the first type of scheduling node, the step further includes:

[0051] receiving an access request from the scheduling node to a mutex lock, wherein the mutex lock is used to ensure that only one scheduling node is in a valid state at the same time;

[0052] The scheduling node that obtains the mutex lock is marked as a scheduling node in a valid state, and the other scheduling nodes are marked as scheduling nodes in a non-valid state.

[0053] Among them, the mutual exclusion lock, in English Mutex lock, is a way to control access to shared resources by locking.

[0054] Specifically, a mutex lock is set in the dispatching control center. When a dispatching node processes data, it needs to check that it is in an available state first, and obtain a mutex lock before processing data, so as to ensure that only one dispatching node is effective at the same time, and other available dispatching nodes are in a standby state, that is, multiple dispatching nodes in an available state access the mutex lock from the dispatching control center, and the dispatching node that obtains the mutex lock is the effective dispatching node, and other dispatching nodes are in a standby state and can replace the currently effective dispatching node at any time, so that only one dispatching node is effective at the same time, which can avoid the confusion of batch data caused by the existence of multiple effective dispatching nodes, until the state of the dispatching node in the effective state is changed, and then the master-slave switch is adopted. For example, the dispatching node in the effective state fails, or the dispatching node is set to an unavailable state due to the need to deploy a new version, etc., which causes the state of the dispatching node to be changed.

[0055] In one embodiment, before the step of sending the first data to the first-type scheduling node in a first preset sending manner, the step further includes:

[0056] Receiving a node status sent by the first-type scheduling node according to a first preset period, the node status including busyness information of the first-type scheduling node;

[0057] The step of sending the first data to the first type of scheduling node according to the first preset sending mode includes:

[0058] Determining the number of the scheduling nodes scheduled by the first type of scheduling nodes;

[0059] If the scheduling node scheduled by the first-type scheduling node is one, sending the first data to the scheduling node scheduled by the first-type scheduling node in a second preset sending manner according to the node state;

[0060] If there are two or more scheduling nodes scheduled by the first type of scheduling node, on the basis of the third preset sending method, according to the state of each node, the first data is sent to each scheduling node scheduled by the first type of scheduling node in accordance with the second preset sending method.

[0061] Among them, the busyness of the scheduling node refers to the proportional relationship between the amount of data processed by the scheduling node and the performance of the scheduling node. If the proportional relationship is an inverse proportional relationship, the larger the inverse proportional relationship, the busier the scheduling node is. If the proportional relationship is a positive proportional relationship, the larger the positive proportional relationship, the more idle the scheduling node is and the stronger the processing capability will be. Among them, the inverse proportional relationship can also be called an inverse correlation relationship.

[0062] Specifically, the first-class scheduling node sends its own node status to the scheduling control center according to a first preset period, and the node status includes busyness information of the first-class scheduling node. The scheduling control center receives the node status sent by the first-class scheduling node, and sends the first data to the scheduling node scheduled by the first-class scheduling node according to the node status, so that the first-class scheduling node distributes the first data to the processing node scheduled by itself.

[0063] After the scheduling control center receives the node status sent by the scheduling node, it sends the first data according to different situations based on the node status of the scheduling node and the number of scheduling nodes included in the first category of scheduling nodes. That is, it determines the number of scheduling nodes included in the first category of scheduling nodes. If the first category of scheduling nodes includes only one scheduling node, according to the node status, the first data is sent to the scheduling node included in the first category of scheduling nodes according to the second preset sending method. If the first category of scheduling nodes includes two or more scheduling nodes, based on the third preset sending method, according to the node status of each scheduling node, the first data is sent to each scheduling node included in the first category of scheduling nodes according to the second preset sending method. It is specifically divided into the following situations:

[0064] (1) If the first type of scheduling node includes one of the scheduling nodes, according to the node state, the first data is sent to the scheduling node included in the first type of scheduling node in accordance with a second preset sending method. The second preset sending method specifically includes:

[0065] (1) If the busyness of the scheduling node is less than a first preset threshold, it is determined that the scheduling node is in a non-busy state, and the scheduling control center directly sends the first data to the scheduling node so that the group to which the scheduling node belongs processes the first data.

[0066] (2) If the busyness of the scheduling node is greater than the first preset threshold but less than the second preset threshold, it is determined that the scheduling node is in a state between busy and non-busy, and part of the first data may be sent to the scheduling node.

[0067] (3) If the busyness of the scheduling node is greater than the second preset threshold, it is determined that the scheduling node is in a busy state. The following two methods can be adopted:

[0068] 1) temporarily suspending sending the first data to the scheduling node, that is, temporarily sending the data amount to zero, and subsequently sending the first data to the scheduling node when the scheduling node meets the conditions for receiving data according to the received state of the scheduling node.

[0069] 2) If the scheduling node is very busy, the currently effective scheduling node can be forcibly switched to other backup scheduling nodes, allowing the scheduling node to process the existing data, while other backup scheduling nodes can process new batch data, thereby ensuring that the processing of batch data is not affected and can continue to provide services to the outside world, thereby ensuring the processing efficiency of batch data.

[0070] (ii) If the first type of scheduling nodes includes two or more scheduling nodes, based on the third preset sending mode, according to the state of each node, the data is sent to each scheduling node included in the first type of scheduling nodes in accordance with the second preset sending mode. The third preset sending mode may specifically include the following modes:

[0071] 1) The first data may be divided into blocks, and a mark may be set for each block of data, and the data blocks may be sent to multiple scheduling nodes on average. Subsequently, the data processing results processed by each scheduling node may be integrated according to the preset mark to obtain the data processing result.

[0072] 2) Multiple scheduling nodes can also be allocated according to the busyness of each scheduling node, with the number of data blocks allocated for processing being inversely correlated to the busyness. Busier scheduling nodes, that is, scheduling nodes with more backlogs of data to be processed, or scheduling nodes with fewer remaining available resources, are allocated less data. Conversely, more data is allocated, so that data can be reasonably distributed among multiple scheduling nodes to make full use of the performance of each scheduling node and improve the processing efficiency of batch data.

[0073] In one embodiment, the step of receiving the scheduling node's access request to the mutex lock includes:

[0074] Receive an access request from the scheduling node to the mutex lock according to a preset period.

[0075] Specifically, after a scheduling node obtains a mutex lock through access, it is marked as being in an effective state. For the scheduling node in the current effective state, in order to ensure the fairness of resource allocation through the lock strategy and make full use of the performance of the scheduling node, the currently effective scheduling node can release the mutex lock under preset conditions, thereby re-receiving access to the mutex lock by multiple scheduling nodes in the active and standby states, iterating the above process, and receiving the access to the mutex lock by the scheduling node according to the preset period, thereby cyclically reallocating the mutex lock, which can ensure the fairness of data resource allocation and improve the utilization rate of the scheduling node.

[0076] Furthermore, the method further includes: the scheduling control center receives a modification instruction for modifying the scheduling node to implement the modification of the scheduling node controlled by the scheduling control center, wherein the modification includes adding a scheduling node or deleting a scheduling node.

[0077] Specifically, each scheduling node can be called a group, and the batch transaction machine can be split into any number of groups according to the performance requirements of external services. The groups contained in the scheduling control center can also be modified according to the needs of external business. For example, when the batch data corresponding to the business increases, new batch transaction machines and new scheduling nodes are added, and the scheduling control center receives the addition instruction of adding scheduling nodes to achieve the addition of scheduling nodes to improve the business data processing capability. Similarly, when the batch data corresponding to the business decreases, the original batch transaction machine and the original new scheduling node are reduced, and the scheduling control center receives the deletion instruction of reducing scheduling nodes to achieve the reduction of scheduling nodes to save resources and equipment and reduce costs. Through the above modification of the nodes contained in the system corresponding to the group, the matching of the batch data corresponding to the business and the performance of the batch transaction machine can be improved, so as to more appropriately meet the needs of the batch data corresponding to the business. At the same time, for example, for a business, 5 machines are needed to meet the business needs. If 10 machines are configured for the group corresponding to this business, 5 of which are idle, it will cause a waste of machine resources. The splitting of batch transaction machines into groups is mainly done by human intervention. The splitting standard is that each group of machines can process data independently without the need for other grouped machines to coordinate processing. Figure 2 , Figure 2 It includes two groups, Group A and Group B. Group A corresponds to scheduling node 1, and Group B corresponds to scheduling node 2. Only one scheduling node is effective at the same time, that is, Group A is effective, or Group B is effective. Group A can be manually set according to business needs and is configured to call any processing node among processing node 1, processing node 2, processing node 3 and processing node 4. Figure 2 As shown, group A is configured to call four nodes, namely, processing node 1, processing node 2, processing node 3 and processing node 4. Group B can also be manually set according to business needs and configured to call any processing node among processing node 1, processing node 2, processing node 3 and processing node 4. Figure 2 As shown, group B is configured to call four nodes, namely processing node 1, processing node 2, processing node 3 and processing node 4.

[0078] In the embodiment of the present application, by grouping batch transaction machines, if some machines are abnormal and temporarily unable to provide services, these machines can be marked as unavailable online, and fault prompts can be given through alarms, so that maintenance personnel can repair the machines and reconnect them. The machines that are reconnected after repair will reappear in the connectable list, and can be manually set through the visual page to be selected to be configured in the groups corresponding to different scheduling nodes. By grouping batch transaction machines, they can be switched between multiple scheduling nodes without interruption of external services, thereby realizing grayscale deployment of the system, reducing the coupling between machines, and enhancing the robustness of the system. This solves the problems of low efficiency and poor scalability in scheduling processing in traditional technologies.

[0079] It should be noted that online transactions refer to real-time online transactions. Online transactions are generally handled by load balancing devices such as F5 as scheduling nodes. Each transaction will only be sent to one processing node. The transaction returns the processing results in real time. The load balancing device only needs to know whether a certain node is available. Among them, load balancing, whose English name is LoadBalance, is to balance and distribute the load (work task) to multiple operation units for execution, such as Web servers, FTP servers, enterprise key application servers and other key task servers, so as to complete the work tasks together. The batch data processing described in the embodiment of the present application is for the batch data corresponding to the online batch file. The processing method in the traditional technology is to send the batch data corresponding to the online batch file to a single scheduling node and split it into multiple threads for concurrent processing. Due to the different working principles of real-time online transactions and asynchronous batch transactions, the load balancing device used in real-time online transactions cannot solve the problems that need to be solved in the application environment corresponding to asynchronous batch transactions, and asynchronous batch transactions cannot adopt the working method of real-time online transactions. Specifically, if the working principle of real-time online transactions is adopted in asynchronous batch transactions, the load balancing device needs to have the ability to solve the following problems:

[0080] 1) When a load balancing device dispatches file processing, it needs to consider not only whether the processing node is available, but also the current busyness of each node;

[0081] 2) Due to the large amount of data, the processing results cannot be returned in real time. Generally, asynchronous processing is used. When the file processing is completed, a callback is required to notify the front-end system. The load balancing device needs to support reverse interface calls;

[0082] 3) If a file can only be processed on a single node, once the file data volume is large, all threads of the node may not be able to meet the demand and can only wait for processing passively;

[0083] 4) If a file needs to be processed across nodes, the load balancing device needs to be able to communicate with multiple processing nodes simultaneously, track processing progress, collect and feedback results, etc.

[0084] Therefore, asynchronous batch transactions cannot adopt the working mode of real-time online transactions. Since real-time online transactions and asynchronous batch transactions have different requirements for data processing real-time, processing procedures and corresponding hardware equipment, the application scenario of the asynchronous batch transaction machine in the embodiment of the present application is different from that of real-time online transactions. By manually setting multiple processing nodes through a visual page for multiple processing nodes, the processing nodes can be flexibly configured to the scheduling nodes, and the switching between scheduling nodes can be realized through the setting of the scheduling node status, so that the online batch files are distributed to the effective scheduling nodes for processing through the scheduling control node. At the same time, the scheduling nodes in the standby state can be shut down in a planned or unplanned manner, so that the asynchronous batch transaction machine can continue to provide services during planned shutdowns such as version deployment, and external services are not interrupted, thereby realizing the grayscale deployment of asynchronous online batch (files), ensuring the continuity of external services of asynchronous batch transaction machines, and improving the efficiency of online batch (file) processing.

[0085] please Figure 2 and Figure 4 , Figure 4 The following is a flow chart of a batch data processing method applied to a scheduling node provided in an embodiment of the present application. Figure 2 As shown, the method is applied to a scheduling node, the scheduling node is controlled by a scheduling control center, the scheduling control center controls a number of scheduling nodes, each of the scheduling nodes schedules a number of processing nodes, such as Figure 4 As shown, the method includes the following steps S401-S405:

[0086] S401. Send an access request to a mutex lock included in the scheduling control center, where the mutex lock is used to ensure that only one scheduling node is in a valid state at the same time.

[0087] S402: Determine whether the mutex lock is acquired.

[0088] S403: if the mutex lock is acquired, receiving the second data sent by the dispatching control center;

[0089] S404, allocating the second data to the processing node scheduled by itself according to a second preset allocation algorithm, so that the processing node processes the second data according to a second preset processing method corresponding to the second data;

[0090] S405: If the mutex lock is not obtained, the scheduling node is in a standby state.

[0091] Among them, the second data and the first data described in the above embodiment may be the same data, the second preset allocation algorithm and the first preset allocation algorithm described in the above embodiment may be the same algorithm, and the second preset processing method and the first preset processing method described in the above embodiment may be the same method. For the convenience of description, different description terms are used here.

[0092] Specifically, if the scheduling node is in an available state, after successfully registering with the scheduling control center, the scheduling node will access the mutex lock contained in the scheduling control center, and the mutex lock is used to ensure that only one scheduling node is in an effective state at the same time. The scheduling node determines whether to obtain the mutex lock. If the mutex lock is obtained, it will be marked by the scheduling control center as a first-class scheduling node currently in an effective state, so that it can subsequently receive the second data sent by the scheduling control center. After receiving the second data sent by the scheduling control center, the scheduling node allocates the second data to the processing nodes contained in itself according to the second preset allocation algorithm. After the processing node receives the second data allocated by the scheduling node, the processing node processes the second data according to the second preset processing method corresponding to the second data to obtain the corresponding data processing result. If the scheduling node does not obtain the mutex lock, the scheduling node will be marked by the scheduling control center as a second-class scheduling node in a standby state. The scheduling node is in a standby state until the state of the first-class scheduling node is changed. Through the active-standby switching, the second-class scheduling node in the standby state is used as the result of the effective state. The second-class scheduling node receives the second data sent by the scheduling control center, and then sends the received second data to the processing nodes contained in itself for corresponding processing, so as to realize the processing of batch data by the group to which it belongs. This ensures that the system can provide uninterrupted external services during planned downtime such as version deployment or unplanned downtime such as failures, and enables grayscale deployment of online batch files. This ensures the continuity of system services and improves the efficiency of batch data processing, especially for application environments with high requirements for data processing continuity.

[0093] In one embodiment, before the step of allocating the second data to the processing node scheduled by itself according to the second preset allocation algorithm, the method further includes:

[0094] receiving a modification instruction for the scheduled processing node, wherein the modification instruction includes deleting a processing node, adding a processing node, and changing a status of a processing node;

[0095] The scheduled processing node is modified according to the modification instruction.

[0096] Specifically, for asynchronous batch transaction machines, in the embodiment of the present application, since multiple scheduling nodes are allowed to coexist, the scheduling control center includes a number of scheduling nodes, and each of the scheduling nodes is configured to include a number of processing nodes. Therefore, all processing nodes are divided into corresponding groups consisting of scheduling nodes and processing nodes included in the scheduling nodes according to the scheduling nodes. Each scheduling node can be configured with a fixed number of processing nodes by default, and the number of processing nodes included in each scheduling node can also be manually set through a setting page according to actual needs. For example, each scheduling node can be manually set through a preset visual setting page to be configured with a number of processing nodes to achieve adjustment of the processing nodes included in each scheduling node, so that each scheduling node receives modification instructions for the included processing nodes, and the modification instructions include deleting processing nodes, adding processing nodes, and changing the status of processing nodes. According to the modification instructions, the included processing nodes are modified. For example, please continue to refer to Figure 2 , the scheduling node group configured by the scheduling node 1 corresponding to the A group includes processing node 1 and processing node 2. Processing node 1 and processing node 2 belong to a scheduling node group according to the pre-settings. The processing nodes included in the scheduling node 1 can also be manually set through the visualization page, so that the modified scheduling node includes more processing nodes, or includes fewer processing nodes. Since the modified scheduling node is in a planned downtime state and is unavailable, and the system processes batch data by the group where other scheduling nodes in an available state are located, the scalability of the scheduling node in the standby state can be achieved at the same time without interrupting external services, which can solve the problem of poor scalability of the processing nodes included in the scheduling node in the traditional technology.

[0097] Furthermore, for the grouped asynchronous batch trading machines, the asynchronous batch trading machines are split into multiple groups. Through the above-mentioned modification of the processing nodes contained in each group, each group can process data independently, or can be merged to process data at the same time, and external services will not be interrupted, so as to realize the grayscale deployment of the system. For example, there are two groups, A corresponds to three machines 123, and B corresponds to three machines 456. A can schedule 123456 at the same time, and B is a backup node. The way A connects to 123456 is the same as the way A connects to 123. Both are connected to the connectable list contained in the setting page, and 456 is connected through the respective interfaces of the 456 devices. The merging of groups is the reverse process of group splitting, and can be set through human intervention. The processing nodes configured by the scheduling node can be arbitrarily set manually through the visualization page, so that the combination and splitting of processing nodes can be realized to flexibly meet the performance requirements of processing data for processing nodes. For example, when the data has high performance requirements for processing nodes, all processing nodes can be configured to the same scheduling node for scheduling. If the data does not have high performance requirements for processing nodes, the processing nodes can be grouped, and one scheduling node can be configured as different processing nodes, solving the problems of low efficiency and poor scalability in the scheduling process in traditional technologies.

[0098] Furthermore, the scheduling node sends the key data contained in the data to the processing node via Socket. Specifically, the scheduling node sends the key data contained in the data to the processing node via Socket. The processing node obtains the full data to which the partial data belongs from the database based on the identifier of the partial data contained in the Socket transmitted by the scheduling node, which is similar to obtaining full-text data according to a directory. The transmission of full data can be avoided to reduce the amount of communication data in a timely manner and improve communication efficiency. For example, Socket transmission is implemented based on Netty, and the communication based on Netty maintains the association between the scheduling node and the processing node, which can reduce coupling and improve performance. Netty is an asynchronous communication framework based on the JAVANIO class library. Socket is an encapsulation of the TCP / IP protocol, an intermediate software abstraction layer for communication between the application layer and the TCP / IP protocol family, and a set of call interfaces (API functions of the TCP / IP network).

[0099] Furthermore, when the scheduling node is switched, the currently effective scheduling node automatically reloads the scheduling data of the previous effective scheduling node. Specifically, in the case where only one scheduling node is effective at the same time, when the scheduling node is switched, the new scheduling node automatically reloads the data scheduling related information. When the node needs to be switched, the context is automatically obtained, the currently effective scheduling node reloads the scheduling data, and the scheduling node that was originally in effect and now belongs to the backup node unloads the scheduling data, that is, when the scheduling node is switched, the currently effective scheduling node automatically obtains the scheduling data loaded on the previous machine, thereby realizing the switching between scheduling nodes.

[0100] The following is a specific embodiment to describe the interaction process between the various subjects, so as to more clearly describe the process of implementing asynchronous grayscale processing for online batch files, including a scheduling control center and multiple scheduling nodes, including the following steps:

[0101] Step 1: The scheduling node sends the node status information of the scheduling node to the scheduling control center according to a preset time period, and the node status information includes the status and busyness of the node;

[0102] Step 2: The dispatch control center receives the node status information of the dispatch node;

[0103] Step 3: The dispatch control center receives the online batch file sent by the external terminal, and divides the batch file into data blocks of preset sizes according to a preset setting;

[0104] Step 4: The scheduling control center sends the data block to the scheduling node according to the node status information of the scheduling node:

[0105] Step 5: The scheduling node sends the data block to different processing nodes according to the second preset allocation algorithm. The processing nodes configured by the scheduling node can be manually set from all processing nodes as needed through the visual page intervention;

[0106] Step 6: The processing node processes the received data block and processes the data block to obtain a sub-data processing result, and returns the sub-data processing result to the scheduling node;

[0107] Step 7: After receiving the sub-data processing results, the scheduling node takes all sub-data processing results as the total data processing result of the online batch file, and returns the total data processing result to the scheduling control center;

[0108] Step 8: After receiving the total data processing result, the dispatch control center outputs the total data processing result to complete the processing of the online batch file.

[0109] It should be noted that the batch data processing methods described in the above embodiments can recombine the technical features contained in different embodiments as needed to obtain a combined implementation plan, but they are all within the protection scope required by this application.

[0110] An embodiment of the present invention also provides a batch data processing system, which includes a scheduling control center, which controls a number of scheduling nodes, and each scheduling node schedules a number of processing nodes; wherein the scheduling control center is used to execute the steps of the batch data processing method applied to the scheduling control center described in the above embodiments, the scheduling nodes are used to execute the steps of the batch data processing method applied to the scheduling nodes described in the above embodiments, and the processing nodes are used to execute the steps of processing data.

[0111] See also Figure 5 , Figure 5 A schematic block diagram of a batch data processing device applied to a dispatching control center provided in an embodiment of the present application. Corresponding to the batch data processing method applied to a dispatching control center, an embodiment of the present application also provides a batch data processing device applied to a dispatching control center. Figure 5 As shown, the batch data processing device applied to the dispatching control center includes a unit for executing the batch data processing method applied to the dispatching control center. The batch data processing device applied to the dispatching control center can be configured in a computer device included in the server side. Figure 5 The batch data processing device 500 applied to the dispatching control center includes a first receiving unit 501, an acquiring unit 502 and a sending unit 503.

[0112] The first receiving unit 501 is used to receive first data sent by an external data source;

[0113] An acquisition unit 502, configured to acquire a scheduling node that is pre-marked as being in a valid state as a first-type scheduling node;

[0114] The sending unit 503 is used to send the first data to the first type of scheduling node according to a first preset sending method, so that the first type of scheduling node allocates the first data to the processing node scheduled by itself according to a first preset allocation algorithm, so that the processing node processes the first data according to the first preset processing method corresponding to the first data.

[0115] In one embodiment, the apparatus 500 further includes:

[0116] A second receiving unit, used to receive an access request from the scheduling node to a mutex lock, wherein the mutex lock is used to ensure that only one scheduling node is in a valid state at the same time;

[0117] The marking unit is used to mark the scheduling node that obtains the mutex lock as a scheduling node in a valid state.

[0118] In one embodiment, the apparatus 500 further includes:

[0119] A third receiving unit, configured to receive a node status sent by the first type of scheduling node according to a first preset period, wherein the node status includes busyness information of the first type of scheduling node;

[0120] The sending unit 503 includes:

[0121] A determining subunit, used to determine the number of the scheduling nodes scheduled by the first type of scheduling nodes;

[0122] A first sending subunit is configured to send the first data to the scheduling node scheduled by the first-type scheduling node in a second preset sending manner according to the node state if the scheduling node scheduled by the first-type scheduling node is one;

[0123] A second sending subunit is used to send the first data to each of the scheduling nodes scheduled by the first type of scheduling node in accordance with the second preset sending method based on the third preset sending method and according to the state of each node if there are two or more scheduling nodes scheduled by the first type of scheduling node.

[0124] In one embodiment, the second receiving unit is used to receive an access request from the scheduling node to the mutex lock according to a preset period.

[0125] See also Figure 6 , Figure 6 A schematic block diagram of a batch data processing device applied to a scheduling node provided in an embodiment of the present application. Corresponding to the batch data processing method applied to a scheduling node, an embodiment of the present application also provides a batch data processing device applied to a scheduling node. Figure 6 As shown, the batch data processing device applied to the scheduling node includes a unit for executing the batch data processing method applied to the scheduling node, and the batch data processing device applied to the scheduling node can be configured in a computer device included in the server side. Specifically, please refer to Figure 6 The batch data processing device 600 applied to the scheduling node includes an access unit 601, a judgment unit 602, a fourth receiving unit 603 and an allocation unit 604.

[0126] The access unit 601 is used to send an access request to a mutex lock included in the scheduling control center, and the mutex lock is used to ensure that only one scheduling node is in a valid state at the same time;

[0127] A judging unit 602 is used to judge whether the mutex lock is acquired;

[0128] A fourth receiving unit 603 is configured to receive second data sent by the dispatching control center if the mutex lock is acquired;

[0129] The allocating unit 604 is configured to allocate the second data to the processing node scheduled by itself according to a second preset allocation algorithm, so that the processing node processes the second data according to a second preset processing method corresponding to the second data.

[0130] In one embodiment, the apparatus 600 further includes:

[0131] A second receiving unit, configured to receive a modification instruction for the scheduled processing node, wherein the modification instruction includes deleting a processing node, adding a processing node, and changing a status of a processing node;

[0132] A modification unit is used to modify the scheduled processing node according to the modification instruction.

[0133] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned batch data processing device and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and brevity of description, it will not be repeated here.

[0134] Meanwhile, the division and connection method of each unit in the above-mentioned batch data processing device is only used for illustration. In other embodiments, the batch data processing device can be divided into different units as needed, and the units in the batch data processing device can also be connected in different orders and methods to complete all or part of the functions of the above-mentioned batch data processing device.

[0135] The batch data processing apparatus can be implemented in the form of a computer program. Figure 7 Runs on the computer device shown.

[0136] See also Figure 7 , Figure 7 700 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 700 may be a computer device such as a desktop computer or a server, or may be a component or part in other devices.

[0137] See also Figure 7The computer device 700 includes a processor 702 , a memory and a network interface 705 connected via a system bus 701 , wherein the memory may include a non-volatile storage medium 703 and an internal memory 704 .

[0138] The non-volatile storage medium 703 can store an operating system 7031 and a computer program 7032. When the computer program 7032 is executed, the processor 702 can execute the above-mentioned batch data processing method.

[0139] The processor 702 is used to provide computing and control capabilities to support the operation of the entire computer device 700 .

[0140] The internal memory 704 provides an environment for the operation of the computer program 7032 in the non-volatile storage medium 703. When the computer program 7032 is executed by the processor 702, the processor 702 can execute the above-mentioned batch data processing method.

[0141] The network interface 705 is used to communicate with other devices over the network. Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the present application solution, and does not constitute a limitation on the computer device 700 to which the present application solution is applied. The specific computer device 700 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. For example, in some embodiments, the computer device may only include a memory and a processor. In such an embodiment, the structure and function of the memory and the processor are the same as those of the present application solution. Figure 7 The embodiments shown are consistent and will not be described again here.

[0142] In one embodiment, taking the batch data processing method applied to the scheduling control center as an example, the processor 702 is used to run the computer program 7032 stored in the memory to implement the following steps: receiving first data sent by an external data source; obtaining a scheduling node that is pre-marked as an effective state as a first type of scheduling node; sending the first data to the first type of scheduling node according to a first preset sending method, so that the first type of scheduling node allocates the first data to the processing node scheduled by itself according to a first preset allocation algorithm, so that the processing node processes the first data according to the first preset processing method corresponding to the first data.

[0143] In one embodiment, before implementing the step of acquiring the scheduling node pre-marked as a valid state as the first-type scheduling node, the processor 702 further implements the following steps:

[0144] A request for access to a mutex lock from the scheduling node is received, wherein the mutex lock is used to ensure that only one scheduling node is in a valid state at the same time; and the scheduling node that obtains the mutex lock is marked as a scheduling node in a valid state.

[0145] In one embodiment, before implementing the step of sending the first data to the first-type scheduling node in the first preset sending manner, the processor 702 further implements the following steps:

[0146] Receiving a node status sent by the first-type scheduling node according to a first preset period, the node status including busyness information of the first-type scheduling node;

[0147] When the processor 702 implements the step of sending the first data to the first-type scheduling node in the first preset sending mode, the processor 702 specifically implements the following steps:

[0148] Determine the number of scheduling nodes scheduled by the first type of scheduling node; if the number of scheduling nodes scheduled by the first type of scheduling node is one, send the first data to the scheduling node scheduled by the first type of scheduling node according to the node status and in accordance with the second preset sending method; if the number of scheduling nodes scheduled by the first type of scheduling node is two or more, on the basis of the third preset sending method, send the first data to each scheduling node scheduled by the first type of scheduling node according to the second preset sending method and in accordance with the node status of each node.

[0149] In one embodiment, when implementing the step of receiving the access request of the scheduling node to the mutex lock, the processor 702 specifically implements the following steps:

[0150] Receive an access request from the scheduling node to the mutex lock according to a preset period.

[0151] In another embodiment, taking the batch data processing method applied to the scheduling node as an example, the processor 702 is used to run the computer program 7032 stored in the memory to implement the following steps: sending an access request to the mutex lock contained in the scheduling control center, the mutex lock is used to ensure that only one scheduling node is in an effective state at the same time; judging whether to obtain the mutex lock; if the mutex lock is obtained, receiving the second data sent by the scheduling control center; allocating the second data to the processing node scheduled by itself according to the second preset allocation algorithm, so that the processing node processes the second data according to the second preset processing method corresponding to the second data.

[0152] In one embodiment, before implementing the step of allocating the second data to the processing node scheduled by itself according to the second preset allocation algorithm, the processor 702 further implements the following steps:

[0153] A modification instruction for a scheduled processing node is received, wherein the modification instruction includes deleting a processing node, adding a processing node, and changing a status of a processing node; and the scheduled processing node is modified according to the modification instruction.

[0154] It should be understood that in the embodiment of the present application, the processor 702 may be a central processing unit (CPU), and the processor 702 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0155] Those skilled in the art will appreciate that all or part of the processes in the method for implementing the above embodiment can be completed by a computer program, which can be stored in a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0156] Therefore, the present application also provides a computer-readable storage medium. The computer-readable storage medium may be a non-volatile computer-readable storage medium, and the computer-readable storage medium stores a first computer program, and when the first computer program is executed by a processor, the processor executes the following steps:

[0157] A computer program product, when running on a computer, enables the computer to execute the steps of the batch data processing method applied to a dispatching control center described in the above embodiments.

[0158] The present application also provides another computer-readable storage medium. The computer-readable storage medium may be a non-volatile computer-readable storage medium, and the computer-readable storage medium stores a second computer program, and when the second computer program is executed by a processor, the processor executes the following steps:

[0159] A computer program product, when running on a computer, enables the computer to execute the steps of the batch data processing method applied to a scheduling node described in the above embodiments.

[0160] The computer-readable storage medium may be an internal storage unit of the aforementioned device, such as a hard disk or memory of the device. The computer-readable storage medium may also be an external storage device of the device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the device and an external storage device.

[0161] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0162] The storage medium is a physical, non-transient storage medium, for example, it can be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, etc., which can store computer programs.

[0163] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0164] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0165] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0166] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for an electronic device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0167] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A batch data processing method, characterized in that: The method is applied to a dispatching control center, the dispatching control center controls a plurality of dispatching nodes, each of the dispatching nodes dispatches a plurality of processing nodes, and the method comprises: Receiving first data sent by an external data source; Obtaining a scheduling node that is pre-marked as being in a valid state as a first-category scheduling node; The first data is sent to the first type of scheduling node according to a first preset sending method, so that the first type of scheduling node allocates the first data to the processing node scheduled by itself according to a first preset allocation algorithm, so that the processing node processes the first data according to a first preset processing method corresponding to the first data; the first preset processing method includes performing corresponding processing according to the business related to the first data; the first preset sending method is a method of selecting the scheduling node to send the first data, and if there are multiple scheduling nodes that are currently pre-marked as effective, a method of allocating batch data among multiple scheduling nodes in effective states; Before the step of sending the first data to the first-type scheduling node in a first preset sending manner, the method further includes: Receiving a node status sent by the first-type scheduling node according to a first preset period, the node status including busyness information of the first-type scheduling node; The step of sending the first data to the first type of scheduling node according to the first preset sending mode includes: Determining the number of the scheduling nodes scheduled by the first type of scheduling nodes; If the scheduling node scheduled by the first-type scheduling node is one, sending the first data to the scheduling node scheduled by the first-type scheduling node in a second preset sending manner according to the node state; If there are two or more scheduling nodes scheduled by the first type of scheduling nodes, based on the third preset sending method, according to the state of each node, the first data is sent to each scheduling node scheduled by the first type of scheduling nodes according to the second preset sending method; the third preset sending method includes dividing the first data into blocks, setting a mark for each block of data, sending the data blocks to multiple scheduling nodes on average, and subsequently integrating the data processing results processed by each scheduling node according to the mark to obtain the data processing results, or allocating the number of data blocks allocated for processing in an inverse correlation with the busyness of each scheduling node according to the busyness of each scheduling node, and allocating less data to the scheduling node with more backlog of data to be processed or less remaining available resources, and vice versa; the busyness of the scheduling node is the proportional relationship between the amount of data processed by the scheduling node and the performance of the scheduling node. If the proportional relationship is an inverse proportional relationship, the larger the inverse proportional relationship is, the busier the scheduling node is; if the proportional relationship is a positive proportional relationship, the larger the positive proportional relationship is, the idler the scheduling node is.

2. The batch data processing method according to claim 1, characterized in that: Before the step of obtaining the scheduling node pre-marked as a valid state as the first type of scheduling node, the method further includes: receiving an access request from the scheduling node to a mutex lock, wherein the mutex lock is used to ensure that only one scheduling node is in a valid state at the same time; The scheduling node that obtains the mutex lock is marked as a scheduling node in a valid state.

3. The batch data processing method according to claim 2, characterized in that: The step of receiving the access request of the scheduling node to the mutex lock comprises: Receive an access request from the scheduling node to the mutex lock according to a preset period.

4. A batch data processing method, characterized in that: The method is applied to a scheduling node, the scheduling node is controlled by a scheduling control center, the scheduling control center controls a number of scheduling nodes, each of the scheduling nodes schedules a number of processing nodes, and the method includes: Sending an access request to a mutex lock included in the scheduling control center, wherein the mutex lock is used to ensure that only one scheduling node is in a valid state at the same time; Determine whether to obtain the mutex lock; If the mutex lock is acquired, receiving the second data sent by the dispatching control center; Allocate the second data to the processing node scheduled by itself according to a second preset allocation algorithm, so that the processing node processes the second data according to a second preset processing method corresponding to the second data; the second preset processing method includes performing corresponding processing according to the business related to the second data; Before receiving the second data sent by the dispatching control center, the dispatching control center also receives a node status sent by the first type of scheduling node according to a first preset period, wherein the node status includes busyness information of the first type of scheduling node; The dispatch control center specifically sends the second data to the dispatch node through the following steps: Determining the number of the scheduling nodes scheduled by the first type of scheduling nodes; If the scheduling node scheduled by the first-type scheduling node is one, sending the second data to the scheduling node scheduled by the first-type scheduling node in a second preset sending manner according to the node state; If there are two or more scheduling nodes scheduled by the first type of scheduling node, based on the third preset sending method, according to the state of each node, the second data is sent to each scheduling node scheduled by the first type of scheduling node in accordance with the second preset sending method; the third preset sending method includes dividing the second data into blocks, setting a mark for each block of data, sending the data blocks to multiple scheduling nodes on average, and subsequently integrating the data processing results processed by each scheduling node according to the mark to obtain the data processing results, or allocating the number of data blocks allocated for processing in an inverse correlation with the busyness of each scheduling node according to the busyness of each scheduling node, and allocating less data to the scheduling node with more backlog of data to be processed or less remaining available resources, and vice versa; the busyness of the scheduling node is the proportional relationship between the amount of data processed by the scheduling node and the performance of the scheduling node. If the proportional relationship is an inverse proportional relationship, the larger the inverse proportional relationship, the busier the scheduling node is; if the proportional relationship is a positive proportional relationship, the larger the positive proportional relationship, the idler the scheduling node is.

5. The batch data processing method according to claim 4, characterized in that: Before the step of allocating the second data to the processing node scheduled by itself according to the second preset allocation algorithm, the method further includes: receiving a modification instruction for the scheduled processing node, wherein the modification instruction includes deleting a processing node, adding a processing node, and changing a status of a processing node; The scheduled processing node is modified according to the modification instruction.

6. A batch data processing device, characterized in that: The method comprises a unit for executing the method according to any one of claims 1 to 3, or comprises a unit for executing the method according to any one of claims 4 to 5.

7. A batch data processing system, characterized in that: The system includes a dispatch control center, the dispatch control center controls a plurality of dispatch nodes, and each of the dispatch nodes dispatches a plurality of processing nodes; The dispatching control center is used to execute the steps of the method as described in any one of claims 1 to 3, the dispatching node is used to execute the steps of the method as described in any one of claims 4 to 5, and the processing node is used to execute the step of processing the allocation data according to a preset processing method corresponding to the received allocation data.

8. A computer device, characterized in that: The computer device includes a first memory and a first processor connected to the first memory, or includes a second memory and a second processor connected to the second memory; the first memory is used to store a first computer program, and the second memory is used to store a second computer program; the first processor is used to run the first computer program to perform the steps of the method as described in any one of claims 1 to 3, and the second processor is used to run the second computer program to perform the steps of the method as described in any one of claims 4 to 5.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a first computer program and / or a second computer program. When the first computer program is executed by a processor, it can implement the steps of the method as described in any one of claims 1 to 3. When the second computer program is executed by a processor, it can implement the steps of the method as described in any one of claims 4 to 5.

Images