Batch task processing method, device and system

Abstract

The embodiment of the present invention discloses a batch task processing method, device and system, which can be used in the financial field or other technical fields. The method comprises: when a batch task is started, multiple task scheduling threads of a local CPU are called, wherein each of the task scheduling threads first obtains the task data to be processed, and then transfers the task data to be processed into the global memory of a local GPU, and then calls a GPU thread in the local GPU to process the task data to be processed, and after completing the data processing, the GPU thread sends the processing result data to the task scheduling thread; and calls the local CPU to store the processing result data in a local business database. The present invention helps to improve the efficiency of batch task processing.

Description

Technical Field

[0001] The present invention relates to the technical field of batch data processing, and in particular to a batch task processing method, device and system. Background Art

[0002] With the promotion of the "de-IOE" process, major banks are currently starting to build their own distributed systems. Therefore, based on current technical conditions, banks mainly start from two aspects to complete the calculation of batch tasks. First, in terms of technology, the computing nodes in the distributed cluster are used to process batch tasks in parallel. The basic idea is to route the customer's transaction data to the designated computing node for storage according to key parameters such as customer number. When the batch task is started, the data stored in each node completes the subsequent processing of the transaction in the node where it is located. Second, through the batch and session processing scheme, starting from the time, the batch processing time of the day is cut in multiple small batches, avoiding the peak of online transactions while improving the utilization rate of the machine. However, with the rapid development of online business, the increase in transaction volume, and the complex scenarios such as the linkage of upstream and downstream batch tasks. As a result, there are still a small number of batch tasks in production that cannot be successfully completed before the branch is open for business, which brings pressure and risks to the online transactions when the branch is open, and also requires the developers of batch programs to have sufficient control over the performance of the program.

[0003] Since NVIDIA released the Geforce 256 graphics accelerator in 1999 and took over the CPU to complete the computing task of image rendering, and released the parallel programming framework CUDA for its own graphics processor in November 2006, the highly parallel computing capabilities of graphics processors GPU have been widely used and promoted. Each transaction in the bank's batch tasks is actually a completely independent business response, so it has good parallelism. However, in order to ensure the security of transactions, banks carry a large number of verification parameters, which is a great pressure on the scarce memory resources of GPUs. In addition, the business database is required to meet the ACID characteristics, so it is impossible to operate the database in parallel at will. Now a reasonable way is needed to solve the scheduling of batch tasks in heterogeneous parallel systems to maximize the use of computing resources. Summary of the invention

[0004] In order to solve the technical problems in the above-mentioned background technology, the present invention proposes a batch task processing method, device and system.

[0005] In order to achieve the above object, according to one aspect of the present invention, a batch task processing method is provided, the method comprising:

[0006] When a batch task is started, multiple task scheduling threads of the local CPU are called, wherein each of the task scheduling threads first obtains the task data to be processed, and then transfers the task data to be processed to the global memory of the local GPU, and then calls the GPU thread in the local GPU to process the task data to be processed. After completing the data processing, the GPU thread sends the processing result data to the task scheduling thread;

[0007] Call the local CPU to store the processing result data in the local business database.

[0008] In order to achieve the above object, according to another aspect of the present invention, a batch task processing device is provided, the device comprising:

[0009] A task scheduling module is used to call multiple task scheduling threads of the local CPU when batch tasks are started, wherein each of the task scheduling threads first obtains the task data to be processed, and then transfers the task data to be processed to the global memory of the local GPU, and then calls the GPU thread in the local GPU to process the task data to be processed. After completing the data processing, the GPU thread sends the processing result data to the task scheduling thread;

[0010] The processing result storage module is used to call the local CPU to store the processing result data in the local business database.

[0011] In order to achieve the above object, according to another aspect of the present invention, a batch task processing system is provided, the system comprising: a distributed batch task processing cluster; the distributed batch task processing cluster comprises a plurality of processing nodes; each of the processing nodes comprises: a business database, a CPU and a GPU;

[0012] Each of the processing nodes is used to call multiple task scheduling threads of the local CPU when a batch task is started. Each of the task scheduling threads first obtains the task data to be processed, and then transfers the task data to be processed to the global memory of the local GPU, and then calls the GPU thread in the local GPU to process the task data to be processed; after completing the data processing, the GPU thread sends the processing result data to the task scheduling thread, and then calls the local CPU to store the processing result data in the local business database.

[0013] In order to achieve the above-mentioned purpose, according to another aspect of the present invention, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned batch task processing method when executing the computer program.

[0014] In order to achieve the above object, according to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed in a computer processor, the steps in the above batch task processing method are implemented.

[0015] The beneficial effects of the present invention are as follows: when starting batch task processing, the task scheduling thread of the CPU schedules the thread of the GPU to process the task data to be processed. After completing the data processing, the GPU thread returns the processing result data to the task scheduling thread, and then the CPU stores the processing result data in the local business database. The heterogeneous parallel scheduling scheme of the present invention combines the characteristics of batch data, fully utilizes the parallel capability of the GPU chip while ensuring the ACID characteristics of the business database, taps the computing resources of the GPU chip, and improves the efficiency of batch task processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0017] Figure 1 is a flow chart of a batch task processing method according to an embodiment of the present invention;

[0018] Figure 2 is a flow chart of storing processing result data into a local business database according to an embodiment of the present invention;

[0019] Figure 3 is a schematic diagram of a batch task processing system according to an embodiment of the present invention;

[0020] Figure 4 is a schematic diagram of the life cycle of a batch task according to an embodiment of the present invention;

[0021] Figure 5 is a flow chart of a heterogeneous parallel scheduling solution for batch tasks according to an embodiment of the present invention;

[0022] Figure 6 is a structural block diagram of a batch task processing device according to an embodiment of the present invention;

[0023] Figure 7 is a structural block diagram of a processing result storage module according to an embodiment of the present invention;

[0024] Figure 8 It is a schematic diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0026] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0027] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or apparatus.

[0028] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0029] It should be noted that the batch task processing method, device and system of the present invention can be used in the financial field, and can also be applied to other technical fields.

[0030] One aspect of the present invention provides a batch task processing system. Figure 3 Schematic diagram of a batch task processing system according to an embodiment of the present invention. Figure 3 As shown, the batch task processing system of the present invention includes: a terminal transaction system and a distributed batch task processing cluster (i.e. Figure 3 The distributed batch task processing cluster includes multiple processing nodes (i.e. Figure 3 Each of the processing nodes includes: a business database, a CPU, and a GPU.

[0031] The following is a detailed introduction to each component of the batch task processing system of the present invention:

[0032] 1.Terminal trading system

[0033] The terminal transaction system may include intelligent terminals, mobile banking, counter transaction systems, etc., which can provide various types of banking business transactions. If the transactions initiated by customers or tellers in the system need to be processed in batches after being judged by the system, the transaction request data (i.e., the task data to be processed) will be recorded in a preset register. When the batch task is started, the system loads the data recorded in advance to complete the business processing flow of the transaction.

[0034] 2. Distributed batch task processing cluster:

[0035] The network where the distributed cluster is located includes a registration center, a configuration center, a controller node, and an executor node for managing cluster computing nodes, and a node for processing batch tasks (generally, a node contains multiple executors and shares the same business database). In addition, the distributed cluster system will push the customer's data and requests to a specific computing node according to the customer number and the agreed routing rules, and store them in the business database of the node.

[0036] 3. Processing nodes:

[0037] This type of node is scheduled by the controller to start batch tasks to process transactions that need to be processed within the business data range of this node. Specifically, the calculation is completed through the computing chip (CPU+GPU) installed on it.

[0038] 4. Business database:

[0039] Used to store customer-related business data. The cluster will store the data in a specific database based on the customer number and certain routing rules.

[0040] 5. Processing CPU:

[0041] The CPU chip is a computing chip that directly processes and runs batch tasks. In this solution, the MPI parallel programming framework is needed to schedule the multi-threading of the CPU.

[0042] 6. GPU in Processing:

[0043] GPU chips rely on CPU control and scheduling, and need to be combined with NVIDIA's CUDA programming framework to write and load parallel programs. The CUDA programming framework currently supports JAVA, C++, and Python programming languages, and can be well integrated with the current distributed system development environment.

[0044] In an embodiment of the present invention, each of the processing nodes is used to call multiple task scheduling threads of the local CPU when a batch task is started. Each of the task scheduling threads first obtains the task data to be processed, and then transfers the task data to be processed to the global memory of the local GPU, and then calls the GPU thread in the local GPU to process the task data to be processed; after completing the data processing, the GPU thread sends the processing result data to the task scheduling thread, and then calls the local CPU to store the processing result data in the local business database.

[0045] In the embodiment of the present invention, the CPU includes multiple threads, specifically including a CPU main thread and multiple CPU sub-threads, and the CPU sub-threads are the above-mentioned task scheduling threads.

[0046] In one embodiment of the present invention, calling the local CPU to store the processing result data in the local service database specifically includes:

[0047] Each task scheduling thread that receives the processing result data stores the processing result data in a local business database, wherein the processing result data includes a primary key or an index.

[0048] In another embodiment of the present invention, calling the local CPU to store the processing result data in the local service database specifically includes:

[0049] The CPU main thread in the local CPU detects in real time whether there is a confirmation file in the temporary file of each task scheduling thread. If so, the processing result data in the temporary file is first stored in the local business database, and then the temporary file is deleted. When the task scheduling thread receives the processing result data sent by the GPU thread, it stores the processing result data in the temporary file and generates a confirmation file in the temporary file.

[0050] In one embodiment of the present invention, if the data returned by the GPU needs to be persisted in a local business database in real time, the CPU main thread can be used to scan the temporary files generated by the slave thread (i.e., the task scheduling thread) in real time and delete them after processing. Specifically, after each slave thread generates a temporary data file, a confirmation file is added. The main thread can process the data file of the slave thread only after detecting the confirmation file.

[0051] In another embodiment of the present invention, each of the processing nodes is further configured to obtain the to-be-processed task data corresponding to the processing node when the batch task is started and store the data in the local business database.

[0052] In one embodiment of the present invention, the pending task data generated by the terminal transaction system is aggregated and stored in a preset register / file. When a batch task is started, each processing node obtains the pending task data corresponding to the processing node from the register / file and stores it in its own local business database.

[0053] Each of the task scheduling threads first obtains the task data to be processed, specifically including:

[0054] Each of the task scheduling threads obtains to-be-processed task data from the local service database.

[0055] In another embodiment of the present invention, each of the processing nodes is further used to clear the global memory of the local GPU and terminate the task scheduling thread of the local CPU when the batch task processing is completed.

[0056] Figure 4 Schematic diagram of the life cycle of a batch task according to an embodiment of the present invention. Figure 4 As shown, in one embodiment of the present invention, the life cycle of a batch task of the present invention includes the following stages:

[0057] 1. Generation of batch tasks:

[0058] Batch tasks usually come from the regular plans of existing businesses. For example, the salary cards of employees of a company that receive salary on the 15th of each month will be associated with batch tasks for salary payment.

[0059] Batch tasks also have set plans from incremental business, such as batch card activation transactions initiated by various channels every day, which are set to be processed uniformly in the early morning every day.

[0060] The batch task data to be processed is stored in the business database in the distributed cluster and is downloaded, uploaded and processed according to the needs of the batch tasks.

[0061] 2. Triggering of batch tasks:

[0062] Batch tasks are generally triggered by a set time or by the upstream batch tasks on which they depend.

[0063] 3. Execution of batch tasks:

[0064] 3.1 After the controller in the cluster triggers a batch task, it mobilizes the executor to load business data to complete the batch task.

[0065] 3.2 Executor schedules heterogeneous chips to process batch tasks:

[0066] 3.2.1 The executor uses the MPI framework to enable multiple CPU threads.

[0067] 3.2.2 The CPU thread obtains the data to be processed recorded in the register from the business database in advance.

[0068] 3.2.3 The CPU thread transfers data to the GPU based on the CUDA programming framework and invokes the GPU thread.

[0069] 3.2.4 After the GPU completes the calculation and processing, it returns the data to the CPU.

[0070] 3.2.5 The CPU persists the processed data back into the business database.

[0071] Figure 5 : is a flow chart of a heterogeneous parallel scheduling solution for batch tasks according to an embodiment of the present invention. Figure 5 As shown, in a specific embodiment of the present invention, the specific process of performing batch task processing using the batch task processing system of the present invention includes the following steps:

[0072] Assume that the CPU has 10 computing cores (threads), the GPU has 1,000 computing cores (threads), and the batch task needs to process 1 million pending task data in the register.

[0073] 1. Call the built-in function of the MPI framework in the main thread of the CPU to wake up the remaining 9 sub-threads (i.e., task scheduling threads).

[0074] 2. Each CPU sub-thread reads 100 pieces of data from the register.

[0075] 3. Each CPU thread calls the built-in CUDA kernel function to transfer data to the GPU global memory.

[0076] 4. Each CPU thread calls a custom CUDA kernel function, allocates 100 GPU threads, and processes the corresponding 100 data.

[0077] 5. Each CPU thread continues to query the register for pending data and waits for the end of the GPU kernel function.

[0078] 6. After the custom GPU kernel function completes the calculation, it transmits 100 processing result data back to the CPU.

[0079] 7. After the CPU thread receives the data returned from the GPU, it writes it into a temporary file and transfers the next batch of data to be processed to the GPU global memory for further processing.

[0080] 8. After all the data in the register is processed, the GPU memory space is released and the 9 CPU sub-threads are terminated to keep only the main thread.

[0081] 9. The CPU main thread reads the returned results recorded in the temporary file and inserts / updates them into the database one by one.

[0082] 10. If you need to persist the data returned by the GPU to the local business database in real time, you can use the CPU main thread to scan the temporary files generated by the slave thread (i.e., the task scheduling thread) in real time and delete them after processing. Specifically, after each slave thread generates a temporary data file, a confirmation file is added. The main thread can process the data file of the slave thread only after detecting the confirmation file.

[0083] 11. In particular, if the data returned from the GPU has a primary key or index, the child thread (i.e., the task scheduling thread) can directly persist the returned data to the business database without causing the table to be locked.

[0084] 12. In particular, if the GPU has enough storage space to record transaction data, then MPI and CUDA can be used to open threads that exceed the number of computing cores on the chip (CPU / GPU), and these threads are scheduled for execution in a polling manner. Polling scheduling can effectively hide memory access latency and improve computing throughput.

[0085] In addition, the program implementation of the heterogeneous parallel solution for batch tasks of the present invention includes the following details:

[0086] 1. When programming in CPU parallel, you need to set the number of threads to be enabled before calling the MPI framework to start multithreading, and obtain the thread id (thread_id) returned by the built-in function. In the parallel code block, confirm the program branch that each thread needs to execute based on the thread_id.

[0087] For example, the main thread with thread_id == 0 is responsible for scanning the data files generated by the slave threads and loading them into the database, while all slave threads with thread_id == 0 are responsible for scheduling the GPU to complete the calculation of batch tasks.

[0088] 2. When programming in parallel with the CPU, it is necessary to agree on the names of the data file and confirmation file of the slave thread so that the main thread can locate the data file based on the confirmation file.

[0089] For example, if the data file is named data_[slave thread id]_[file number], then the confirmation file can be named check_[slave thread id]_[file number].

[0090] 3. During GPU parallel programming, CUDA does not return the thread ID. CUDA has built-in ternary variables (threadIdx, blockDim, blockIdx, gridDim) to describe the organizational structure of threads (Thread, Block, Grid). Therefore, the GPU thread ID needs to be calculated manually in the program.

[0091] 4. When programming GPU parallelism, CUDA sets corresponding usage restrictions based on various GPU models, including two-dimensional and three-dimensional thread structures, and a maximum limit on thread data in each dimension. Therefore, before scheduling the GPU, you need to use CUDA's kernel function to obtain the GPU's hardware parameters, so as to reasonably set the number and structure of parallel threads.

[0092] Based on the same inventive concept, an embodiment of the present invention also provides a batch task processing method, as described in the following embodiment. Since the principle of solving the problem by the batch task processing method is similar to that of the batch task processing system, the embodiment of the batch task processing method can refer to the embodiment of the batch task processing system, and the repeated parts will not be repeated. The batch task processing method of the present invention is applied to any processing node in a distributed batch task processing cluster, that is, the implementation subject of the present invention is any processing node in a distributed batch task processing cluster.

[0093] Figure 1 is a flow chart of a batch task processing method according to an embodiment of the present invention, and the implementation subject is any processing node in a distributed batch task processing cluster, such as Figure 1 As shown, the batch task processing method of this embodiment includes steps S101 to S102.

[0094] Step S101, when a batch task is started, multiple task scheduling threads of the local CPU are called, wherein each of the task scheduling threads first obtains the task data to be processed, and then transfers the task data to be processed to the global memory of the local GPU, and then calls the GPU thread in the local GPU to process the task data to be processed. After completing the data processing, the GPU thread sends the processing result data to the task scheduling thread.

[0095] Step S102: calling the local CPU to store the processing result data into the local business database.

[0096] In one embodiment of the present invention, the calling of the local CPU to store the processing result data in the local service database in step S102 specifically includes:

[0097] Each task scheduling thread that receives the processing result data is called to store the processing result data in a local business database, wherein the processing result data includes a primary key or an index.

[0098] Figure 2 is a flow chart of storing the processing result data into the local business database according to an embodiment of the present invention, such as Figure 2 As shown, in one embodiment of the present invention, the above step S102 of calling the local CPU to store the processing result data in the local business database specifically includes step S201 and step S202.

[0099] Step S201, calling the CPU main thread in the local CPU to detect in real time whether there is a confirmation file in the temporary file of each task scheduling thread.

[0100] Step S202, if it exists, first store the processing result data in the temporary file in the local business database, and then delete the temporary file. When the task scheduling thread receives the processing result data sent by the GPU thread, it stores the processing result data in the temporary file and generates a confirmation file in the temporary file.

[0101] In one embodiment of the present invention, the step S101 of transferring the task data to be processed into the global memory of the local GPU specifically includes:

[0102] The built-in CUDA kernel function is called to transfer the task data to be processed into the global memory of the local GPU.

[0103] In one embodiment of the present invention, calling the GPU thread in the local GPU in the above step S101 to process the task data to be processed specifically includes:

[0104] The custom CUDA kernel function is called to allocate GPU threads in the local GPU to process the task data to be processed.

[0105] In one embodiment of the present invention, the batch task processing method of the present invention further includes:

[0106] When a batch task is started, the task data to be processed corresponding to the processing node is obtained and stored in the local business database.

[0107] In one embodiment of the present invention, each of the task scheduling threads in the above step S101 first obtains the task data to be processed, which specifically includes: each of the task scheduling threads obtains the task data to be processed from the local business database.

[0108] In one embodiment of the present invention, the batch task processing method of the present invention further includes:

[0109] At the end of batch task processing, the global memory of the local GPU is cleared and the task scheduling thread of the local CPU is terminated.

[0110] It can be seen from the above embodiments that the present invention provides a heterogeneous parallel scheduling scheme for batch tasks. For batch tasks of the same processing node in a distributed cluster, multiple slave threads are first started from the main thread of the CPU. Each CPU thread obtains the batch data to be processed from the business database / upstream downloaded files, and then loads the batch data into the GPU memory, and starts a corresponding number of GPU threads. Each GPU thread completes a transaction corresponding to a data. The GPU transmits the processed data back to the CPU, and according to the characteristics of the business data, the data can be directly persisted to the database, or exported to a temporary file first. After all transactions are processed, the CPU main thread persists the data in the file to the database. This heterogeneous parallel scheduling scheme combines the characteristics of batch data, while ensuring the ACID characteristics of the business database, giving full play to the parallel capabilities of heterogeneous chips, tapping the computing resources of the chips, and improving the efficiency of bank batch task processing.

[0111] The effects and advantages of the present invention are as follows:

[0112] The present invention solves the problem of heterogeneous parallel implementation of bank batch tasks on a single processing node in a distributed cluster. Specifically, the present invention dynamically adjusts the number of parallel threads and the data persistence method according to the characteristics of batch data, thereby achieving an efficient parallel processing solution while satisfying the ACID characteristics of the business database.

[0113] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0114] Based on the same inventive concept, an embodiment of the present invention also provides a batch task processing device, which can be used to implement the batch task processing method described in the above embodiment, as described in the following embodiment. Since the principle of solving the problem by the batch task processing device is similar to that of the batch task processing method, the embodiment of the batch task processing device can refer to the embodiment of the batch task processing method, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements the predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0115] Figure 6 is a structural block diagram of a batch task processing device according to an embodiment of the present invention. Figure 6 As shown, the batch task processing device according to the embodiment of the present invention includes:

[0116] Task scheduling module 1, used for calling multiple task scheduling threads of the local CPU when batch tasks are started, wherein each of the task scheduling threads first obtains the task data to be processed, and then transfers the task data to be processed to the global memory of the local GPU, and then calls the GPU thread in the local GPU to process the task data to be processed, and after completing the data processing, the GPU thread sends the processing result data to the task scheduling thread;

[0117] The processing result storage module 2 is used to call the local CPU to store the processing result data in the local business database.

[0118] In one embodiment of the present invention, the processing result storage module is specifically used to call each task scheduling thread that receives the processing result data to store the processing result data in a local business database, wherein the processing result data includes a primary key or an index.

[0119] Figure 7 is a structural block diagram of a processing result storage module according to an embodiment of the present invention. Figure 7 As shown, in one embodiment of the present invention, the processing result storage module specifically includes:

[0120] A confirmation file detection unit 201 is used to call the CPU main thread in the local CPU to detect in real time whether there is a confirmation file in the temporary file of each task scheduling thread;

[0121] The storage processing unit 202 is used to store the processing result data in the temporary file in the local business database when a confirmation file exists, and then delete the temporary file. When the task scheduling thread receives the processing result data sent by the GPU thread, it stores the processing result data in the temporary file and generates a confirmation file in the temporary file.

[0122] In one embodiment of the present invention, the task scheduling module 1 specifically calls a built-in CUDA kernel function to transfer the task data to be processed into the global memory of the local GPU.

[0123] In one embodiment of the present invention, the task scheduling module 1 specifically calls a custom CUDA kernel function and allocates a GPU thread in a local GPU to process the task data to be processed.

[0124] In one embodiment of the present invention, the batch task processing device of the present invention further includes:

[0125] The pending task data acquisition module is used to obtain the pending task data corresponding to the processing node when the batch task is started and store it in the local business database.

[0126] In one embodiment of the present invention, each of the task scheduling threads first obtains the task data to be processed, which specifically includes: each of the task scheduling threads obtains the task data to be processed from the local business database.

[0127] In one embodiment of the present invention, the batch task processing device of the present invention further includes:

[0128] The end processing module is used to clear the global memory of the local GPU and terminate the task scheduling thread of the local CPU when the batch task processing is completed.

[0129] In order to achieve the above object, according to another aspect of the present application, a computer device is also provided. Figure 8 As shown, the computer device includes a memory, a processor, a communication interface and a communication bus. A computer program that can be executed on the processor is stored in the memory. When the processor executes the computer program, the steps in the above-mentioned embodiment method are implemented.

[0130] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0131] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and units, such as the corresponding program units in the above method embodiments of the present invention. The processor executes various functional applications of the processor and works data processing by running the non-transitory software programs, instructions and modules stored in the memory, that is, implementing the method in the above method embodiments.

[0132] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0133] The one or more units are stored in the memory, and when executed by the processor, the method in the above embodiment is performed.

[0134] The specific details of the above-mentioned computer device can be understood by referring to the corresponding related descriptions and effects in the above-mentioned embodiments, and will not be repeated here.

[0135] In order to achieve the above purpose, according to another aspect of the present application, a computer-readable storage medium is also provided, wherein the computer-readable storage medium stores a computer program, and the computer program implements the steps in the above batch task processing method when executed in a computer processor. Those skilled in the art can understand that the implementation of all or part of the process in the above embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and the program can include the process of the embodiment of each method as described above when executed. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0136] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0137] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A batch task processing method, characterized in that: The method is applied to any processing node in a distributed batch task processing cluster, the distributed batch task processing cluster includes multiple processing nodes, each processing node includes: a business database, a CPU and a GPU, the method includes: When a batch task is started, multiple task scheduling threads of the local CPU are called, wherein each of the task scheduling threads first obtains the task data to be processed from a preset register, and then transfers the task data to be processed to the global memory of the local GPU, and then calls the GPU thread in the local GPU to process the task data to be processed. After completing the data processing, the GPU thread sends the processing result data to the task scheduling thread; if the transaction initiated by the customer or the teller in the terminal transaction system needs to be processed in batches after being judged by the system, the task data to be processed is recorded in the register, and the terminal transaction system includes: intelligent terminal, mobile banking and counter transaction system, which can provide various types of banking business transactions; Calling the local CPU to store the processing result data in the local business database; the calling of the local CPU to store the processing result data in the local business database specifically includes: calling the CPU main thread in the local CPU to detect in real time whether there is a confirmation file in the temporary file of each of the task scheduling threads, wherein the local CPU includes a CPU main thread and multiple CPU sub-threads, and the CPU sub-thread is the task scheduling thread; if it exists, first storing the processing result data in the temporary file in the local business database, and then deleting the temporary file, wherein the task scheduling thread stores the processing result data in the temporary file when receiving the processing result data sent by the GPU thread, and generates a confirmation file in the temporary file, and in CPU parallel programming, agreeing on the names of the data file and the confirmation file of the CPU sub-thread so that the CPU main thread can locate the data file according to the confirmation file; in GPU parallel programming, manually calculating the thread id of the GPU, and setting the number and structure of parallel threads according to the hardware parameters of the GPU; After all data in the register are processed, the global memory of the local GPU is released, and the CPU sub-threads are terminated to keep only the CPU main thread.

2. The batch task processing method according to claim 1, characterized in that: Call the local CPU to store the processing result data in the local business database, including: Each task scheduling thread that receives the processing result data is called to store the processing result data in a local business database, wherein the processing result data includes a primary key or an index.

3. The batch task processing method according to claim 1, characterized in that: The transferring the task data to be processed into the global memory of the local GPU specifically includes: The built-in CUDA kernel function is called to transfer the task data to be processed into the global memory of the local GPU.

4. The batch task processing method according to claim 1, characterized in that: The calling of the GPU thread in the local GPU to process the task data to be processed specifically includes: The custom CUDA kernel function is called to allocate GPU threads in the local GPU to process the task data to be processed.

5. The batch task processing method according to claim 1, characterized in that: Also includes: When a batch task is started, the task data to be processed corresponding to the processing node is obtained and stored in the local business database; Each of the task scheduling threads first obtains the task data to be processed, specifically including: Each of the task scheduling threads obtains to-be-processed task data from the local service database.

6. A batch task processing device, characterized in that: The device is applied to any processing node in a distributed batch task processing cluster, the distributed batch task processing cluster includes multiple processing nodes, each processing node includes: a business database, a CPU and a GPU, the device includes: The task scheduling module is used to call multiple task scheduling threads of the local CPU when the batch task is started, wherein each of the task scheduling threads first obtains the task data to be processed from a preset register, and then transfers the task data to be processed to the global memory of the local GPU, and then calls the GPU thread in the local GPU to process the task data to be processed. After completing the data processing, the GPU thread sends the processing result data to the task scheduling thread; if the transaction initiated by the customer or the teller in the terminal transaction system needs to be processed in batches after being judged by the system, the task data to be processed is recorded in the register, and the terminal transaction system includes: intelligent terminal, mobile banking and counter transaction system, which can provide various types of banking business transactions; A processing result storage module is used to call the local CPU to store the processing result data in the local business database; The processing result storage module specifically includes: A confirmation file detection unit is used to call a CPU main thread in a local CPU to detect in real time whether there is a confirmation file in a temporary file of each of the task scheduling threads, wherein the local CPU includes a CPU main thread and multiple CPU sub-threads, and the CPU sub-threads are the task scheduling threads; The storage processing unit is used to store the processing result data in the temporary file in the local business database when the confirmation file exists, and then delete the temporary file. When the task scheduling thread receives the processing result data sent by the GPU thread, it stores the processing result data in the temporary file and generates a confirmation file in the temporary file. When the CPU is parallelly programmed, the names of the data file and the confirmation file of the CPU sub-thread are agreed upon so that the CPU main thread can locate the data file according to the confirmation file; when the GPU is parallelly programmed, the thread id of the GPU is manually calculated, and the number and structure of parallel threads are set according to the hardware parameters of the GPU; The end processing module is used to release the global memory of the local GPU after all the data in the register are processed, and terminate the CPU sub-thread to only retain the CPU main thread.

7. The batch task processing device according to claim 6, characterized in that: The processing result storage module is specifically used to call each task scheduling thread that receives the processing result data to store the processing result data in a local business database, wherein the processing result data includes a primary key or an index.

8. A batch task processing system, characterized in that: include: Distributed batch task processing cluster; The distributed batch task processing cluster includes multiple processing nodes; each of the processing nodes includes: a business database, a CPU and a GPU; Each of the processing nodes is used to call multiple task scheduling threads of the local CPU when a batch task is started. Each of the task scheduling threads first obtains the task data to be processed from a preset register, and then transfers the task data to be processed to the global memory of the local GPU, and then calls the GPU thread in the local GPU to process the task data to be processed; after completing the data processing, the GPU thread sends the processing result data to the task scheduling thread, and then calls the local CPU to store the processing result data in the local business database; if the transaction initiated by the customer or the teller in the terminal transaction system needs to be processed in batches after being judged by the system, the task data to be processed is recorded in the register. The terminal transaction system includes: intelligent terminal, mobile banking and counter transaction system, which can provide various types of banking business transactions; The calling of the local CPU to store the processing result data in the local business database specifically includes: calling the CPU main thread in the local CPU to detect in real time whether there is a confirmation file in the temporary file of each of the task scheduling threads, wherein the local CPU includes a CPU main thread and multiple CPU sub-threads, and the CPU sub-thread is the task scheduling thread; if it exists, first storing the processing result data in the temporary file in the local business database, and then deleting the temporary file, wherein the task scheduling thread stores the processing result data in the temporary file when receiving the processing result data sent by the GPU thread, and generates a confirmation file in the temporary file, and in CPU parallel programming, agreeing on the names of the data file and the confirmation file of the CPU sub-thread so that the CPU main thread can locate the data file according to the confirmation file; in GPU parallel programming, manually calculating the thread id of the GPU, and setting the number and structure of parallel threads according to the hardware parameters of the GPU; Each of the processing nodes is further configured to release the global memory of the local GPU after all the data in the register are processed, and terminate the CPU sub-threads to retain only the CPU main thread.

9. The batch task processing system according to claim 8, characterized in that: The calling of the local CPU to store the processing result data in the local service database specifically includes: Each task scheduling thread that receives the processing result data stores the processing result data in a local business database, wherein the processing result data includes a primary key or an index.

10. The batch task processing system according to claim 8, characterized in that: Each of the processing nodes is used to obtain the task data to be processed corresponding to the processing node when the batch task is started and store it in the local business database; Each of the task scheduling threads first obtains the task data to be processed, specifically including: Each of the task scheduling threads obtains to-be-processed task data from the local service database.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed in a computer processor, the computer program implements the method according to any one of claims 1 to 5.

Images