Data batch processing method and device

Abstract

The present application provides a method and device for batch data processing, wherein the method comprises: a business front-end system receives first batch request data sent by multiple branch front-end systems; the business front-end system slices the first batch request data to obtain multiple first data slices; the business front-end system sends the multiple first data slices to a stream processing system; the stream processing system performs distributed stream processing on the received multiple first data slices to generate a first processing result file. The above scheme can improve processing efficiency and realize real-time processing of large batch transaction requests.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and device for batch processing of data. Background Art

[0002] At present, the volume of real-time batch transaction business is increasing, such as branch batch repayment, branch batch query of due repayment, conversion of provident fund old and new modes, automatic deduction batch entry, etc. With the continuous growth of banking business, the batch proxy files sent by the front end continue to increase. There are two traditional batch transfer modes: single AP single thread cyclically sending transaction requests, single AP multi-thread cyclically sending transaction requests.

[0003] However, the traditional batch transfer mode has the problems of poor real-time performance and low data processing efficiency.

[0004] To address the above problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a data processing technology field to solve the problems of poor real-time performance and low processing efficiency of batch data processing in the prior art.

[0006] An embodiment of the present application also provides a data batch processing method, including: a business front-end system receives first batch request data sent by multiple branch front-end systems; the business front-end system slices the first batch request data to obtain multiple first data slices; the business front-end system sends the multiple first data slices to a stream processing system; the stream processing system performs distributed stream processing on the received multiple first data slices to generate a first processing result file.

[0007] An embodiment of the present application also provides a data batch processing method, which is applied to a business front-end system, including: receiving first batch request data sent by multiple branch front-end systems; slicing the first batch request data to obtain multiple first data slices; sending the multiple first data slices to a stream processing system, wherein the stream processing system performs distributed stream processing on the multiple first data slices to generate a first processing result file.

[0008] An embodiment of the present application also provides a data batch processing method, which is applied to a stream processing system, including: receiving multiple first data slices sent by a business front-end system, wherein the multiple first data slices are obtained by the business front-end system after slicing first batch request data received from multiple branch front-end systems; performing distributed stream processing on the multiple first data slices to generate a first processing result file; and sending the first processing result file to the business front-end system.

[0009] An embodiment of the present application also provides a data batch processing device, which is located in a business front-end system and includes: a receiving module, used to receive first batch request data sent by multiple branch front-end systems; a slicing module, used to slice the first batch request data to obtain multiple first data slices; a sending module, used to send the multiple first data slices to a stream processing system, wherein the stream processing system performs distributed stream processing on the multiple first data slices to generate a first processing result file.

[0010] An embodiment of the present application also provides a data batch processing device, located in a stream processing system, comprising: a receiving module, used to receive multiple first data slices sent by a business front-end system, wherein the multiple first data slices are obtained after the business front-end system slices first batch request data received from multiple branch front-end systems; a processing module, used to perform distributed stream processing on the multiple first data slices to generate a first processing result file; and a sending module, used to send the first processing result file to the business front-end system.

[0011] An embodiment of the present application also provides a computer device, including a processor and a memory for storing processor executable instructions, wherein when the processor executes the instructions, the steps of the data batch processing method described in any of the above embodiments are implemented.

[0012] An embodiment of the present application also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed, the steps of the data batch processing method described in any of the above embodiments are implemented.

[0013] In an embodiment of the present application, a data batch processing method is provided, in which a business front-end system receives first batch request data sent by multiple branch front-end systems, slices the first batch request data, obtains multiple first data slices, and sends the multiple first data slices to a stream processing system, and the stream processing system performs distributed stream processing on the received multiple first data slices to generate a first processing result file, thereby realizing real-time processing of batch data. In the above scheme, by slicing the batch request data into multiple data slices and sending them to the stream processing system for distributed stream processing, the processing efficiency can be improved, and real-time processing of large batch transaction requests can be realized. The above scheme solves the technical problems of poor real-time performance and low efficiency of existing batch request data processing, and achieves the technical effect of effectively improving the real-time performance and processing efficiency of batch data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present application, constitute a part of the present application, and do not constitute a limitation of the present application. In the drawings:

[0015] Figure 1A flow chart of a method for batch processing of data in one embodiment of the present application is shown;

[0016] Figure 2 A flow chart of a data batch processing method in an embodiment of the present application is shown;

[0017] Figure 3 A flow chart of a data batch processing method in an embodiment of the present application is shown;

[0018] Figure 4 An architectural diagram of the Apache Storm system in an embodiment of the present application is shown;

[0019] Figure 5 A relationship diagram between Spout and Bolt in an embodiment of the present application is shown;

[0020] Figure 6 A timing diagram of a data batch processing method in an embodiment of the present application is shown;

[0021] Figure 7 A flow chart of a data batch processing method in an embodiment of the present application is shown;

[0022] Figure 8 A schematic diagram of a data batch processing device in an embodiment of the present application is shown;

[0023] Fig. 9 A schematic diagram of a data batch processing device in an embodiment of the present application is shown;

[0024] Fig.10 A schematic diagram of a computer device in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0025] The principles and spirit of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present application, and are not intended to limit the scope of the present application in any way. On the contrary, these embodiments are provided to make the disclosure of the present application more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0026] Those skilled in the art know that the embodiments of the present application can be implemented as a system, device, method or computer program product. Therefore, the present application can be implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0027] In a scenario example of the present specification, multiple branch front-end systems may send first batch request data to the business front-end system. Among them, the multiple branch front-end systems may include front-end systems of branches in various places. The business front-end system may collect batch request data from branch front-end systems in various places for unified processing. Among them, the first batch request data may be various business request data, for example, branch batch repayment request data, branch batch query request data, provident fund old and new mode conversion request data, automatic deduction batch entry, transfer request data, etc. After receiving the batch request data sent by the branch front-end systems in various places, the business front-end system may slice the first batch request data to obtain multiple first data slices. Afterwards, the business front-end system may send the multiple first data slices to the stream processing system. The stream processing system performs distributed stream processing on the received multiple first data slices to generate a first processing result file.

[0028] In one embodiment, after sending the plurality of first data slices to the stream processing system, the business front-end system may receive the first processing result file sent by the stream processing system. The business front-end system may obtain the result file configuration information of each branch front-end system among the plurality of branch front-end systems, and generate the first result file corresponding to each branch front-end system according to the first processing result file and the result file configuration information of each branch front-end system. Afterwards, the business front-end system may send the first result file corresponding to each branch front-end system to the corresponding branch front-end system.

[0029] In one embodiment, the business front-end system can read the first batch request data into a database, create a scheduled processing task, and read the first batch request data from the database according to the scheduled processing task. After reading the first batch request data from the database, the business front-end system can slice the first batch request data to obtain multiple first data slices.

[0030] In one embodiment, after the business front-end system sends the first result files corresponding to each branch front-end system to the corresponding branch front-end system, the business front-end system may also receive second batch request data sent by each branch front-end system. The second batch request data is generated based on the first result files corresponding to each branch front-end system. In one embodiment, each branch front-end system may generate the second batch request data based on the corresponding first result file. In another embodiment, other business centers that send the first batch request data to each branch front-end system may generate the second batch request data based on the first result file. After receiving the second batch request data, the business front-end system slices the second batch request data to obtain multiple second data slices. The business front-end system may send the multiple second data slices to the stream processing system. The stream processing system performs distributed stream processing on the multiple second data slices to generate second processing result files.

[0031] In one embodiment, the first batch request data may include multiple first request data, each of which includes a customer number. The business front-end system may slice the first batch request data according to the customer number.

[0032] In one embodiment, the stream processing system performs distributed stream processing on the plurality of first data slices based on an Apache Kafka system and an Apache Storm system.

[0033] Figure 1 A flow chart of a method for batch processing of data in an embodiment of the present application is shown. Although the present application provides method operation steps or device structures as shown in the following embodiments or drawings, more or fewer operation steps or module units may be included in the method or device based on routine or no creative labor. In the steps or structures where there is no necessary causal relationship logically, the execution order of these steps or the module structure of the device is not limited to the execution order or module structure described in the embodiments of the present application and shown in the drawings. When the method or module structure is applied to an actual device or terminal product, it can be connected according to the method or module structure shown in the embodiment or drawings for sequential execution or parallel execution (for example, a parallel processor or a multi-threaded processing environment, or even a distributed processing environment).

[0034] Specifically, Figure 1 As shown, an embodiment of the present application provides a data batch processing method which may include the following steps.

[0035] Step S101: the business front-end system receives a first batch of request data sent by a plurality of branch front-end systems.

[0036] Specifically, the plurality of branch front-end systems may include the front-end systems of branches in various places. The business front-end system may collect batch request data of the front-end systems of branches in various places for unified processing. Among them, the first batch request data may be various business request data, for example, branch batch repayment request data, branch batch query due request data, provident fund new and old mode conversion request data, automatic deduction batch entry, etc., transfer request data, etc.

[0037] Step S102: the business front-end system slices the first batch of request data to obtain a plurality of first data slices.

[0038] Step S103: The service front-end system sends multiple first data slices to the stream processing system.

[0039] Specifically, the business front-end system can slice the first batch request data to obtain multiple first data slices, and send the multiple first data slices to the stream processing system. Among them, the business front-end system can slice the first batch request data according to actual needs. For example, the business front-end system can slice the first batch request data according to the branch numbers to obtain the first data slices corresponding to the branch numbers. For another example, the business front-end system can slice the first batch request data according to the areas where the branches are located to obtain the first data slices corresponding to the areas where the branches are located.

[0040] Step S104: the stream processing system performs distributed stream processing on the received multiple first data slices to generate a first processing result file.

[0041] After receiving the multiple first data slices, the stream processing system can perform distributed stream processing on the obtained multiple first data slices to generate a first processing result file. The data processed by the stream processing system is received continuously and in real time. Distributed stream processing can process the continuously generated dynamic data based on distributed memory, and the data processing is fast, efficient and low-latency.

[0042] In the data batch processing method in the above embodiment, the business front-end system receives the first batch request data sent by multiple branch front-end systems, slices the first batch request data to obtain multiple first data slices, and sends the multiple first data slices to the stream processing system to perform distributed stream processing on the received multiple first data slices, generate a first processing result file, and realize real-time processing of batch data. In the above scheme, by slicing the batch request data into multiple data slices and sending them to the stream processing system for distributed stream processing, the processing efficiency can be improved, and real-time and efficient processing of large batch transaction requests can be realized.

[0043] Figure 2FIG. 1 is a flow chart showing a method for batch processing of data in another embodiment of the present application, which method is applied to a business front-end system. Specifically, Figure 2 As shown, an embodiment of the present application provides a data batch processing method which may include the following steps.

[0044] Step S201, receiving first batch request data sent by multiple branch front-end systems.

[0045] Specifically, the plurality of branch front-end systems may include the front-end systems of branches in various places. The business front-end system may collect batch request data of the front-end systems of branches in various places for unified processing. Among them, the first batch request data may be various business request data, for example, branch batch repayment request data, branch batch query due request data, provident fund new and old mode conversion request data, automatic deduction batch entry, etc., transfer request data, etc.

[0046] Step S202: Slice the first batch of requested data to obtain a plurality of first data slices.

[0047] Step S203: Send multiple first data slices to the stream processing system.

[0048] Specifically, the business front-end system can slice the first batch request data to obtain multiple first data slices, and send the multiple first data slices to the stream processing system. Among them, the business front-end system can slice the first batch request data according to actual needs. For example, the business front-end system can slice the first batch request data according to the branch numbers to obtain the first data slices corresponding to the branch numbers. For another example, the business front-end system can slice the first batch request data according to the areas where the branches are located to obtain the first data slices corresponding to the areas where the branches are located.

[0049] After receiving the multiple first data slices, the stream processing system can perform distributed stream processing on the obtained multiple first data slices to generate a first processing result file. The data processed by the stream processing system is received continuously and in real time. Distributed stream processing can process the continuously generated dynamic data based on distributed memory, and the data processing is fast, efficient and low-latency.

[0050] In the data batch processing method in the above embodiment, the business front-end system receives the first batch request data sent by multiple branch front-end systems, slices the first batch request data to obtain multiple first data slices, and sends the multiple first data slices to the stream processing system to perform distributed stream processing on the received multiple first data slices, generate a first processing result file, and realize real-time processing of batch data. In the above scheme, by slicing the batch request data into multiple data slices and sending them to the stream processing system for distributed stream processing, the processing efficiency can be improved, and real-time and efficient processing of large batch transaction requests can be realized.

[0051] In some embodiments of the present application, the first batch request data may include multiple first request data, and the first request data includes a customer number; slicing the first batch request data to obtain multiple first data slices may include: slicing the first batch request data according to the customer number.

[0052] Specifically, the first batch request data may include multiple first request data, and the first request data may include a customer number. After receiving the first batch request data, the business front-end system may slice the first batch request data according to the customer number. For example, the customer numbers may be arranged in sequence, for example, the customer numbers are 0-99999, and the batch request data corresponding to every 1000 customer numbers may be sliced ​​into one data slice in sequence.

[0053] In some embodiments of the present application, after sending multiple first data slices to the stream processing system, it may also include: receiving a first processing result file sent by the stream processing system; obtaining result file configuration information of each branch front-end system among multiple branch front-end systems; generating a first result file corresponding to each branch front-end system based on the first processing result file and the result file configuration information of each branch front-end system; and sending the first result file corresponding to each branch front-end system to the corresponding branch front-end system.

[0054] Specifically, the stream processing system can send the generated first processing result file to the business front-end system. After receiving the first processing result file, the business front-end system can obtain the result file configuration information of each branch front-end system in the multiple branch front-end systems. The business front-end system can generate the first result file corresponding to each branch system according to the result file configuration information of each branch front-end system, and send the generated first result file corresponding to each branch system to the corresponding branch front-end system. Among them, the result file configuration information of each branch front-end system may include attribute information such as the format and layout of the result file. The result file configuration information of each branch front-end system can match the system connected to each branch front-end system. For example, when the provident fund center wants to query the mortgage to be repaid, it can send a query request to each branch front-end system, and the first result file received by each branch front-end system can be adapted to the corresponding provident fund center. In the above manner, the first result file corresponding to each branch front-end system can be generated, which can improve the user experience.

[0055] In some embodiments of the present application, slicing the first batch request data to obtain multiple first data slices may include: reading the first batch request data into a database; creating a scheduled processing task, and reading the first batch request data from the database according to the scheduled processing task; slicing the first batch request data to obtain multiple first data slices.

[0056] For some batch request data, it is required to be triggered at a specific time. For example, the mortgage repayment business is generally carried out at the early morning of the first day of each month, so a scheduled processing task can be set. Specifically, after receiving the first batch request data, the business front-end system can read the first batch request data into the database, and create a scheduled processing task based on the first batch request data. At the time required by the scheduled processing task, the business front-end system can read the first batch request data from the database according to the scheduled task, and perform slice processing to obtain multiple first data slices. The business front-end system can send the generated multiple first data slices to the stream processing system, so that the stream processing system can perform distributed stream processing on the multiple data slices. In the above manner, it is possible to achieve scheduled and real-time processing of batch request data.

[0057] In some embodiments of the present application, after sending the first result file corresponding to each branch front-end system to the corresponding branch front-end system, it may also include: receiving second batch request data sent by each branch front-end system, wherein the second batch request data is generated based on the first result file corresponding to each branch front-end system; slicing the second batch request data to obtain multiple second data slices; sending the multiple second data slices to the stream processing system, wherein the stream processing system performs distributed stream processing on the multiple second data slices to generate a second processing result file.

[0058] Specifically, after the business front-end system can send the first result file to the corresponding branch front-end system, each branch front-end system can send the second batch request data to the business front-end system. The second batch request data is generated according to the first result file corresponding to each branch front-end system. For example, each branch front-end system can generate the second batch request data according to the first result file. For another example, each branch front-end system can send the first batch request data of the first result file to the client system of each branch front-end system, and the client system can generate the second batch request data according to the first result file, and send the second batch request data to each branch front-end system.

[0059] After receiving the second batch request data sent by the front-end systems of each branch, the business front-end system can slice the second batch request data to obtain multiple second data slices, and send the multiple second data slices to the stream processing system. The stream processing system can perform distributed stream processing on the multiple second data slices to generate a second processing result file. The stream processing system can send the second processing result file to the business front-end system. In the above manner, for the associated transaction requests, fully automated processing can be achieved without manual intervention, which can improve processing efficiency and save labor costs.

[0060] Figure 3 FIG. 1 is a flow chart showing a method for batch processing of data in another embodiment of the present application, which method is applied to a stream processing system. Specifically, Figure 3 As shown, an embodiment of the present application provides a data batch processing method which may include the following steps.

[0061] Step S301: receiving a plurality of first data slices sent by a service front-end system.

[0062] Among them, the multiple first data slices are obtained by the business front-end system after slicing the first batch request data received from the multiple branch front-end systems. Specifically, the multiple branch front-end systems may include the front-end systems of branches in various places. The business front-end system may collect the batch request data of the branch front-end systems in various places for unified processing. Among them, the first batch request data may be various business request data, for example, branch batch repayment request data, branch batch query request data, provident fund new and old mode conversion request data, automatic deduction batch entry, etc., transfer request data, etc.

[0063] The business front-end system can slice the first batch request data to obtain multiple first data slices, and send the multiple first data slices to the stream processing system. The business front-end system can slice the first batch request data according to actual needs. For example, the business front-end system can slice the first batch request data according to the branch numbers to obtain the first data slices corresponding to the branch numbers. For another example, the business front-end system can slice the first batch request data according to the regions where the branches are located to obtain the first data slices corresponding to the regions where the branches are located.

[0064] Step S302: Perform distributed stream processing on multiple first data slices to generate a first processing result file.

[0065] Step S303: Send the first processing result file to the business front-end system.

[0066] After receiving multiple first data slices, the stream processing system can perform distributed stream processing on the obtained multiple first data slices, generate a first processing result file, and send the generated first processing result file to the business front-end system. The data processed by the stream processing system is received continuously and in real time. Distributed stream processing can process the continuously generated dynamic data based on distributed memory, and the data processing is fast, efficient, and low-latency.

[0067] In the data batch processing method in the above embodiment, the business front-end system receives the first batch request data sent by multiple branch front-end systems, slices the first batch request data to obtain multiple first data slices, and sends the multiple first data slices to the stream processing system to perform distributed stream processing on the received multiple first data slices, generate a first processing result file, and send the generated first processing result file to the business front-end system to realize real-time processing of batch data. In the above scheme, by slicing the batch request data into multiple data slices and sending them to the stream processing system for distributed stream processing, the processing efficiency can be improved, and real-time and efficient processing of large batch transaction requests can be realized.

[0068] In some embodiments of the present application, distributed stream processing is performed on multiple first data slices to generate a first processing result file, including: performing distributed stream processing on the multiple first data slices based on an Apache Kafka system and an Apache Storm system.

[0069] Specifically, the stream processing system can receive multiple first data slices sent by the business front-end system based on the Apache Kafka system, generate asynchronous messages according to the multiple first data slices, and send the asynchronous messages to the Apache Storm system. The Apache Storm system can perform distributed stream processing according to the asynchronous messages to achieve real-time and efficient processing of batch request data.

[0070] Please refer to Figure 4 , shows the architecture diagram of the Apache Storm system in an embodiment of the present application. Among them, Nimbus is the main node of the Storm cluster, responsible for resource allocation and task scheduling. Both task submission and task deadline are operated on Nimbus. A Storm cluster has only one Nimbus node.

[0071] Supervisor is a Storm cluster working node, accepts tasks assigned by Nimbus, and manages all Workers (also called APs, application servers). Workers are working processes, and each working process has multiple Tasks. Tasks are tasks, and each Spout and Bolt is a task, and each task is a thread. Spout is the source of a Stream (message stream) in a Topology (topology). All business processing in a Topology is completed in Bolt. The relationship between Spout and Bolt is as follows: Figure 5 shown.

[0072] The Zookeeper cluster is responsible for the communication between the Nimbus node and the Supervisor node, and monitors the status of each node. For example, when a task is submitted, it is usually executed on the Nimbus node. The Nimbus node distributes the task through the Zookeeper cluster, and the Supervisor is where the task is actually executed. The Nimbus node monitors the status of each Supervisor node through the Zookeeper cluster. When a Supervisor node fails, the Nimbus node will redistribute the tasks on that Supervisor node through the Zookeeper cluster and execute them on other Supervisor nodes. This means that the Storm cluster is a highly available cluster.

[0073] The above method is described below in conjunction with a specific embodiment. However, it should be noted that this specific embodiment is only for better illustrating the present application and does not constitute an improper limitation on the present application.

[0074] Please refer to Figure 6 and Figure 7 , Figure 6FIG. 1 shows a timing diagram of a data batch processing method in an embodiment of the present application. Figure 7 FIG. 1 is a flow chart showing a method for batch processing of data in an embodiment of the present application. Figure 4 and Figure 5 As shown, the batch data processing method in this specific embodiment may include the following steps.

[0075] Step 1: The provident fund management system in each region generates a request file No. 01 for querying provident fund repayments and sends it to the branch front-end system.

[0076] Step 2: When the front-end system of each branch determines that the request file No. 01 has arrived, it continues to send the request file No. 01 to the NLS front-end system.

[0077] Step 3: When the NLS front-end system determines that request file No. 01 has arrived, it reads the file data into the database and schedules a scheduled processing task. The scheduled task reads the provident fund deduction account information in the database, divides the slices by account number, and sends them to the Storm stream processing system.

[0078] Step 4: After receiving the data slice, the Storm stream processing system generates an asynchronous message based on Kafka. Storm receives the Kafka asynchronous message and performs distributed stream processing. That is, when it is determined that the request file is No. 01, it queries the provident fund repayment, summarizes the results, and generates a result file.

[0079] Step 5: The Storm stream processing system returns the generated result file to the NLS front-end system. The NLS front-end system can generate the corresponding result file according to the result file configuration information of the provident fund management system in each region, and return the generated result file to the corresponding branch front-end system.

[0080] Step 6: Each branch's front-end system returns the received result file to the provident fund management system of each region.

[0081] Step 7. The provident fund management system in each region deducts the provident fund account amount according to the result file, generates repayment file No. 03 according to the deduction result and the result file, and sends repayment file No. 03 to the front-end system of each branch.

[0082] Step 8: The front-end systems of local branches send the received No. 03 repayment file to the NLS front-end system.

[0083] Step 9: After the NLS front-end system determines that it has received the repayment file No. 03, it reads the file data into the database and schedules a scheduled processing task. The scheduled task reads the provident fund deduction account information in the database, divides it into slices by account, and sends it to the Storm stream processing system.

[0084] Step 10. After receiving the data slice, the Storm stream processing system generates an asynchronous message based on Kafka. Storm receives the Kafka asynchronous message and performs distributed stream processing. That is, when it is determined to be the repayment file No. 03, it determines whether the provident fund account is sufficient to repay the loan. If so, the repayment is completed. Otherwise, the amount in the associated bank card account is deducted, the internal accounts are deducted and settled, and the result file is generated after the results are summarized and statistically analyzed.

[0085] Step 11: The Storm stream processing system returns the generated result file to the NLS front-end system, and the NLS front-end system returns the result file to the corresponding branch front-end system.

[0086] Step 12: Each branch's front-end system returns the received result file to each regional provident fund management system.

[0087] The method in the above embodiment can process files No. 01 and No. 03 sent from the provident fund system, realize multi-node high-concurrency data processing within the stream processing module, and generate different result files according to different branch parameter settings when processing file No. 01. Branch parameters can be managed in a unified manner, and the process is coherent and fully automated when processing files No. 01 and No. 03, without the need for human intervention. In addition, during the stream processing process, the design of multi-level cache inside the Kafka component and the Bolt component greatly reduces the pressure on database access.

[0088] Based on the same inventive concept, a data batch processing device is also provided in the embodiment of the present application, and the device is located in the business front-end system, as described in the following embodiments. Since the principle of solving the problem by the data batch processing device is similar to that of the data batch processing method, the implementation of the data batch processing device can refer to the implementation of the data batch processing method, and the repeated parts are not repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived. Figure 8 is a structural block diagram of a data batch processing device according to an embodiment of the present application, such as Figure 8 As shown, it includes: a receiving module 801, a slicing module 802 and a sending module 803, and the structure is described below.

[0089] The receiving module 801 is used to receive first batch request data sent by multiple branch front-end systems.

[0090] The slicing module 802 is used to perform slicing processing on the first batch request data to obtain multiple first data slices.

[0091] The sending module 803 is used to send the multiple first data slices to the stream processing system, wherein the stream processing system performs distributed stream processing on the multiple first data slices to generate a first processing result file.

[0092] In some embodiments of the present application, the device may further include a generation module, which may be specifically used to: after sending multiple first data slices to the stream processing system, receive a first processing result file sent by the stream processing system; obtain result file configuration information of each branch front-end system among multiple branch front-end systems; generate a first result file corresponding to each branch front-end system based on the first processing result file and the result file configuration information of each branch front-end system; and send the first result file corresponding to each branch front-end system to the corresponding branch front-end system.

[0093] In some embodiments of the present application, the slicing module can be specifically used to: read the first batch request data into a database; create a scheduled processing task, and read the first batch request data from the database according to the scheduled processing task; and slice the first batch request data to obtain multiple first data slices.

[0094] In some embodiments of the present application, the device may further include a processing module, which may be used to: after sending the first result file corresponding to each branch front-end system to the corresponding branch front-end system, receive second batch request data sent by each branch front-end system, wherein the second batch request data is generated based on the first result file corresponding to each branch front-end system; perform slicing processing on the second batch request data to obtain multiple second data slices; send the multiple second data slices to the stream processing system, wherein the stream processing system performs distributed stream processing on the multiple second data slices to generate a second processing result file.

[0095] In some embodiments of the present application, the first batch request data may include multiple first request data, and the first request data includes a customer number; the slicing module may be specifically used to: slice the first batch request data according to the customer number.

[0096] Based on the same inventive concept, a data batch processing device is also provided in the embodiments of the present application, and the device is located in the stream processing system, as described in the following embodiments. Since the principle of solving the problem by the data batch processing device is similar to that of the data batch processing method, the implementation of the data batch processing device can refer to the implementation of the data batch processing method, and the repeated parts are not repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived. Fig. 9is a structural block diagram of a data batch processing device according to an embodiment of the present application, such as Fig. 9 As shown, it includes: a receiving module 901, a processing module 902 and a sending module 903, and the structure is described below.

[0097] The receiving module 901 is used to receive multiple first data slices sent by the business front-end system, wherein the multiple first data slices are obtained by the business front-end system after slicing the first batch request data received from multiple branch front-end systems.

[0098] The processing module 902 is used to perform distributed stream processing on multiple first data slices to generate a first processing result file.

[0099] The sending module 903 is used to send the first processing result file to the business front-end system.

[0100] In some embodiments of the present application, the processing module can be specifically used to: perform distributed stream processing on multiple first data slices based on the Apache Kafka system and the Apache Storm system to generate a first processing result file.

[0101] From the above description, it can be seen that the embodiment of the present application achieves the following technical effects: the business front-end system receives the first batch request data sent by multiple branch front-end systems, slices the first batch request data, obtains multiple first data slices, and sends the multiple first data slices to the stream processing system, the stream processing system performs distributed stream processing on the received multiple first data slices, generates a first processing result file, and realizes real-time processing of batch data. In the above scheme, by slicing the batch request data into multiple data slices and sending them to the stream processing system for distributed stream processing, the processing efficiency can be improved and the real-time processing of large batch transaction requests can be realized.

[0102] The present application also provides a computer device, which can be found in Fig.10 The computer device structure diagram of the data batch processing method provided by the embodiment of the present application is shown, and the computer device may specifically include an input device 11, a processor 12, and a memory 13. Among them, the memory 13 is used to store processor executable instructions. When the processor 12 executes the instructions, the steps of the data batch processing method described in any of the above embodiments are implemented.

[0103] In this embodiment, the input device may specifically be one of the main devices for information exchange between the user and the computer system. The input device may include a keyboard, a mouse, a camera, a scanner, a light pen, a handwriting input board, a voice input device, etc.; the input device is used to input the original data and the program for processing these numbers into the computer. The input device can also obtain and receive data transmitted from other modules, units, and devices. The processor can be implemented in any appropriate manner. For example, the processor can take the form of a computer-readable medium, a logic gate, a switch, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a programmable logic controller, and an embedded microcontroller, etc., such as a microprocessor or a processor and a computer-readable program code (such as software or firmware) that can be executed by the (micro) processor. The memory may specifically be a memory device used to store information in modern information technology. The memory may include multiple levels. In a digital system, anything that can store binary data can be a memory; in an integrated circuit, a circuit with a storage function without a physical form is also called a memory, such as a RAM, a FIFO, etc.; in a system, a storage device with a physical form is also called a memory, such as a memory stick, a TF card, etc.

[0104] In this embodiment, the functions and effects specifically realized by the computer device can be explained in comparison with other embodiments and will not be described in detail here.

[0105] In an embodiment of the present application, a computer storage medium based on the data batch processing method is also provided, wherein the computer storage medium stores computer program instructions, and when the computer program instructions are executed, the steps of the data batch processing method described in any of the above embodiments are implemented.

[0106] In this embodiment, the storage medium includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a cache, a hard disk (HDD) or a memory card. The memory may be used to store computer program instructions. The network communication unit may be an interface for network connection communication set in accordance with the standard specified by the communication protocol.

[0107] In this embodiment, the functions and effects specifically implemented by the program instructions stored in the computer storage medium can be explained in comparison with other embodiments and will not be repeated here.

[0108] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0109] It should be understood that the above description is for illustration and not for limitation. By reading the above description, many embodiments and many applications beyond the examples provided will be apparent to those skilled in the art. Therefore, the scope of the present application should not be determined with reference to the above description, but should be determined with reference to the aforementioned claims and the full scope of equivalents possessed by these claims.

[0110] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for batch processing of data, characterized in that: include: The business front-end system receives the first batch of request data sent by the front-end systems of multiple branches; The business front-end system performs slicing processing on the first batch request data to obtain multiple first data slices; The business front-end system sends the multiple first data slices to the stream processing system; The stream processing system performs distributed stream processing on the received multiple first data slices to generate a first processing result file; The business front-end system receives a first processing result file sent by the stream processing system; Obtaining result file configuration information of each branch front-end system among the multiple branch front-end systems; The result file configuration information of each branch front-end system includes format information and layout information of the result file; Generate a first result file corresponding to each branch front-end system according to the first processing result file and the result file configuration information of each branch front-end system; Sending the first result file corresponding to each branch front-end system to the corresponding branch front-end system; The front-end system of each branch sends the received first result file to the corresponding housing provident fund management system of each region; the housing provident fund management system of each region deducts money from the housing provident fund account according to the received first result file, generates a second request according to the deduction result and the first result file, and sends the second request to the front-end system of each branch; The front-end system of each branch sends the received second request to the business front-end system; The business front-end system receives second batch request data; The second batch request data includes the second request sent by the front-end system of each branch; Slicing the second batch request data to obtain a plurality of second data slices; and sending the plurality of second data slices to a stream processing system; The stream processing system performs distributed stream processing on the multiple second data slices, generates a second processing result file, and returns the second processing result file to the business front-end system.

2. A method for batch processing of data, characterized in that: The method is applied to a business front-end system, comprising: Receiving first batch request data sent by multiple branch front-end systems; Slicing the first batch of requested data to obtain a plurality of first data slices; Sending the multiple first data slices to a stream processing system, wherein the stream processing system performs distributed stream processing on the multiple first data slices to generate a first processing result file; After sending the plurality of first data slices to the stream processing system, the method further includes: Receiving a first processing result file sent by the stream processing system; Acquire result file configuration information of each branch front-end system among the plurality of branch front-end systems; the result file configuration information of each branch front-end system includes format information and layout information of the result file; Generate a first result file corresponding to each branch front-end system according to the first processing result file and the result file configuration information of each branch front-end system; Sending the first result file corresponding to each branch front-end system to the corresponding branch front-end system; wherein each branch front-end system sends the received first result file to the corresponding provident fund management system of each region; the provident fund management system of each region deducts money from the provident fund account according to the received first result file, generates a second request according to the deduction result and the first result file, and sends the second request to the corresponding branch front-end system; receiving second batch request data; the second batch request data including the second request sent by the front-end system of each branch; slicing the second batch request data to obtain a plurality of second data slices; sending the plurality of second data slices to the stream processing system; wherein the stream processing system performs distributed stream processing on the plurality of second data slices to generate a second processing result file; Receive the second processing result file returned by the stream processing system.

3. The method according to claim 2, characterized in that The first batch of request data is sliced ​​to obtain a plurality of first data slices, including: Reading the first batch of request data into a database; Creating a scheduled processing task, and reading the first batch request data from the database according to the scheduled processing task; The first batch of requested data is sliced ​​to obtain a plurality of first data slices.

4. The method according to claim 2, characterized in that: The first batch request data includes a plurality of first request data, and the first request data includes a customer number; The first batch of request data is sliced ​​to obtain a plurality of first data slices, including: The first batch request data is sliced ​​according to the customer number.

5. A method for batch processing of data, characterized in that: The method is applied to a stream processing system, comprising: Receiving a plurality of first data slices sent by the business front-end system, wherein the plurality of first data slices are obtained after the business front-end system slices first batch request data received from a plurality of branch front-end systems; Performing distributed stream processing on the multiple first data slices to generate a first processing result file; The first processing result file is sent to the business front-end system; the business front-end system is used to receive the first processing result file sent by the stream processing system; it is also used to obtain the result file configuration information of each branch front-end system of the multiple branch front-end systems; the result file configuration information of each branch front-end system includes the format information and layout information of the result file; it is also used to generate the first result file corresponding to each branch front-end system according to the first processing result file and the result file configuration information of each branch front-end system; it is also used to send the first result file corresponding to each branch front-end system to the corresponding branch front-end system; each branch The front-end system sends the received first result file to the corresponding provident fund management systems of each region; the provident fund management systems of each region deduct the provident fund account according to the received first result file, generate a second request according to the deduction result and the first result file, and send the second request to the corresponding front-end systems of each branch; the front-end systems of each branch send the received second request to the business front-end system; the business front-end system receives the second batch request data; the second batch request data includes the second request sent by the front-end systems of each branch; the second batch request data is sliced ​​to obtain multiple second data slices; Receiving the multiple second data slices sent by the service front-end system; Distributed stream processing is performed on the multiple second data slices to generate a second processing result file, and the second processing result file is returned to the business front-end system.

6. The method according to claim 5, characterized in that Performing distributed stream processing on the plurality of first data slices to generate a first processing result file includes: Based on the Apache Kafka system and the Apache Storm system, distributed stream processing is performed on the multiple first data slices to generate a first processing result file.

7. A data batch processing device, characterized in that: The device is located in the business front-end system and includes: A receiving module, used for receiving first batch request data sent by multiple branch front-end systems; a slicing module, configured to perform slicing processing on the first batch of requested data to obtain a plurality of first data slices; A sending module, configured to send the plurality of first data slices to a stream processing system, wherein the stream processing system performs distributed stream processing on the plurality of first data slices to generate a first processing result file; A generation module, used for receiving a first processing result file sent by the stream processing system; also used for obtaining result file configuration information of each branch front-end system among the multiple branch front-end systems; the result file configuration information of each branch front-end system includes format information and layout information of the result file; also used for generating a first result file corresponding to each branch front-end system according to the first processing result file and the result file configuration information of each branch front-end system; also used for sending the first result file corresponding to each branch front-end system to the corresponding branch front-end system; wherein each branch front-end system sends the received first result file to the corresponding provident fund management system of each region; the provident fund management system of each region deducts money from the provident fund account according to the received first result file, generates a second request according to the deduction result and the first result file, and sends the second request to the corresponding branch front-end system; The receiving module is also used to receive second batch request data; the second batch request data includes the second request sent by the front-end system of each branch; the slicing module is also used to slice the second batch request data to obtain multiple second data slices; the sending module is also used to send the multiple second data slices to the stream processing system; wherein, the stream processing system performs distributed stream processing on the multiple second data slices to generate a second processing result file; the receiving module is also used to receive the second processing result file returned by the stream processing system.

8. A data batch processing device, characterized in that: The device is located in a stream processing system and includes: A receiving module, used for receiving a plurality of first data slices sent by a business front-end system, wherein the plurality of first data slices are obtained by the business front-end system after slicing first batch request data received from a plurality of branch front-end systems; A processing module, configured to perform distributed stream processing on the plurality of first data slices to generate a first processing result file; A sending module, used for sending the first processing result file to the business front-end system; the business front-end system is used to receive the first processing result file sent by the stream processing system; it is also used to obtain the result file configuration information of each branch front-end system of the multiple branch front-end systems; the result file configuration information of each branch front-end system includes the format information and layout information of the result file; it is also used to generate the first result file corresponding to each branch front-end system according to the first processing result file and the result file configuration information of each branch front-end system; it is also used to send the first result file corresponding to each branch front-end system to the corresponding branch front-end system; The front-end systems of each branch send the received first result file to the corresponding provident fund management systems of each region; the provident fund management systems of each region deduct money from the provident fund account according to the received first result file, generate a second request according to the deduction result and the first result file, and send the second request to the front-end systems of each branch; the front-end systems of each branch send the received second request to the business front-end system; the business front-end system receives second batch request data; the second batch request data includes the second request sent by the front-end systems of each branch; the second batch request data is sliced ​​to obtain multiple second data slices; The receiving module is also used to receive the multiple second data slices sent by the business front-end system; the processing module is also used to perform distributed stream processing on the multiple second data slices to generate a second processing result file; the sending module is also used to return the second processing result file to the business front-end system.

9. A computer device, characterized in that: The invention comprises a processor and a memory for storing processor-executable instructions, wherein the processor implements the steps of the method according to any one of claims 2 to 4 when executing the instructions.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instructions are executed, the steps of the method according to any one of claims 2 to 4 are implemented.

Images