Accounting data processing method and device

Abstract

The present invention discloses an account data processing method and device, which can be used in the field of big data technology, wherein the method comprises: obtaining account data of multiple bank systems, and transaction volume data and deployment distance data corresponding to each bank system, wherein the deployment distance data is the deployment distance data between the bank system and the big data platform; grouping the multiple bank systems according to the transaction volume data and deployment distance data to obtain multiple system sets; using a streaming computing cluster to read the account data corresponding to the multiple system sets to generate update stream data; and updating the local data of the account platform according to the update stream data. The present invention can process account data and improve performance while ensuring real-time performance.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to an accounting data processing method and device. Background Art

[0002] In a large banking system, which serves customers across the country and even the world, the transaction throughput is large and there are many internal systems. In order to calculate the internal accounts of the enterprise, an internal accounting system will be established. In order to ensure accurate accounting, accounting transactions of all systems are often collected and calculated at the end of the day.

[0003] In the above accounting process, since the cycle is every other day, that is, after the end-of-day processing is completed, the overall fund situation can be seen. Under the demand for faster and faster data support for corporate decision-making, the system must support the provision of efficient quasi-real-time accounting data statistics architecture in such large enterprises. This demand faces high data synchronization problems, statistical problems and multi-system synchronization problems.

[0004] If real-time processing is used, real-time performance can be guaranteed, but it may to some extent cause greater performance constraints on the original system, and the structure is more difficult to design. If multiple batch processing is used, it will lead to poor real-time performance and poor synchronization between multiple systems.

[0005] Therefore, there is an urgent need for an accounting data processing solution that can overcome the above problems. Summary of the invention

[0006] An embodiment of the present invention provides an accounting data processing method for processing accounting data to improve performance while ensuring real-time performance. The method includes:

[0007] Obtaining account data of multiple bank systems, as well as transaction volume data and deployment distance data corresponding to each bank system, wherein the deployment distance data is the deployment distance data between the bank system and the big data platform;

[0008] Grouping the plurality of banking systems according to the transaction volume data and the deployment distance data to obtain a plurality of system sets;

[0009] Using a streaming computing cluster to read the account data corresponding to the plurality of system sets, and generate update stream data;

[0010] The local data of the accounting platform is updated according to the update stream data.

[0011] The embodiment of the present invention provides an accounting data processing device for processing accounting data to improve performance while ensuring real-time performance. The device includes:

[0012] A data acquisition module, used to obtain account data of multiple bank systems, as well as transaction volume data and deployment distance data corresponding to each bank system, wherein the deployment distance data is the deployment distance data between the bank system and the big data platform;

[0013] A system grouping module, used for grouping the plurality of banking systems according to the transaction volume data and the deployment distance data to obtain a plurality of system sets;

[0014] A data generation module, used to use a stream computing cluster to read the account data corresponding to the plurality of system sets and generate update stream data;

[0015] The data updating module is used to update the local data of the accounting platform according to the update stream data.

[0016] An embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned accounting data processing method when executing the computer program.

[0017] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for executing the above-mentioned accounting data processing method.

[0018] The embodiment of the present invention obtains the account data of multiple bank systems, as well as the transaction volume data and deployment distance data corresponding to each bank system, wherein the deployment distance data is the deployment distance data between the bank system and the big data platform; the multiple bank systems are grouped according to the transaction volume data and deployment distance data to obtain multiple system sets; the account data corresponding to the multiple system sets are read using a streaming computing cluster to generate update stream data; and the local data of the account platform is updated according to the update stream data. The embodiment of the present invention groups multiple bank systems according to the transaction volume data and deployment distance data to obtain multiple system sets, and then uses a streaming computing cluster to read the account data corresponding to the multiple system sets to generate update stream data, which can effectively ensure real-time performance, achieve stability and scalability, and can achieve smooth connection for various systems and various deployment methods, thereby improving performance while ensuring real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0020] Figure 1 A schematic diagram of an accounting data processing method in an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of another accounting data processing method in an embodiment of the present invention;

[0022] Figure 3 This is an overall architecture diagram of the accounting data processing in the embodiment of the present invention;

[0023] Figure 4 This is a structural diagram of an accounting data processing device in an embodiment of the present invention;

[0024] Figure 5 It is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0026] In order to process accounting data and improve performance while ensuring real-time performance, an embodiment of the present invention provides an accounting data processing method, such as Figure 1 As shown, the method may include:

[0027] Step 101: Obtain account data of multiple bank systems, as well as transaction volume data and deployment distance data corresponding to each bank system, wherein the deployment distance data is the deployment distance data between the bank system and the big data platform;

[0028] Step 102: grouping the plurality of banking systems according to the transaction volume data and the deployment distance data to obtain a plurality of system sets;

[0029] Step 103: Using a stream computing cluster to read the account data corresponding to the plurality of system sets, and generate update stream data;

[0030] Step 104: Update the local data of the accounting platform according to the update stream data.

[0031] Depend on Figure 1As shown, it can be known that the embodiment of the present invention obtains the account data of multiple bank systems, as well as the transaction volume data and deployment distance data corresponding to each bank system, wherein the deployment distance data is the deployment distance data between the bank system and the big data platform; the multiple bank systems are grouped according to the transaction volume data and deployment distance data to obtain multiple system sets; the account data corresponding to the multiple system sets are read using a streaming computing cluster to generate update stream data; and the local data of the account platform is updated according to the update stream data. The embodiment of the present invention groups multiple bank systems according to the transaction volume data and deployment distance data to obtain multiple system sets, and then uses a streaming computing cluster to read the account data corresponding to the multiple system sets to generate update stream data, which can effectively ensure real-time performance, achieve stability and scalability, and can achieve smooth connection for various systems and various deployment methods, thereby improving performance while ensuring real-time performance.

[0032] In an embodiment, account data of multiple bank systems, as well as transaction volume data and deployment distance data corresponding to each bank system are obtained, wherein the deployment distance data is the deployment distance data between the bank system and a big data platform; the multiple bank systems are grouped according to the transaction volume data and deployment distance data to obtain multiple system sets.

[0033] In this embodiment, Figure 2 As shown, the multiple bank systems are grouped according to the transaction volume data and the deployment distance data to obtain multiple system sets, including:

[0034] Step 201: Compare the transaction volume data corresponding to each bank system with a set threshold;

[0035] Step 202: if the corresponding transaction volume data of the bank system is greater than the set threshold, the bank system is classified into the first system set;

[0036] Step 203: If the corresponding transaction volume data of the bank system is less than or equal to the set threshold, the bank system is classified into the second system set;

[0037] Step 204: Process the bank systems in the second system set according to the deployment distance data.

[0038] In this embodiment, processing the bank system in the second system set according to the deployment distance data includes:

[0039] Clustering the bank systems in the second system set according to the deployment distance data to obtain a plurality of system subclasses;

[0040] For each system subclass, the account data of the banking system in the system subclass is integrated;

[0041] A data synchronization method is determined based on the deployment distance data and the accounting data after data integration.

[0042] In the embodiment, a streaming computing cluster is used to read the account data corresponding to the plurality of system sets to generate update stream data.

[0043] In this embodiment, the stream computing cluster is used to read the account data corresponding to the plurality of system sets to generate update stream data, including:

[0044] Perform distributed computing on the accounting data read using the streaming computing cluster;

[0045] Merge the results of the distribution calculations within a set time period;

[0046] Based on the result of the merge processing, update stream data is generated.

[0047] In the embodiment, the local data of the accounting platform is updated according to the update stream data.

[0048] A specific embodiment is given below to illustrate the specific application of the accounting data processing in the embodiment of the present invention. Figure 3The figure shows the overall architecture of accounting data processing, in which the main accounting system generates large amounts of accounting data, such as the core system, account processing system, etc. The accounting data generated by this system can be synchronized to the accounting database for processing by the big data platform through logs at high speed; or it can be modified by writing the flow into the message queue (MQ, etc.) when writing the accounting data. The advantage of the former is that the original program does not need to be changed, but the dynamic account data update part needs to be processed. The advantage of the latter is that the format is neat and the message is pure dynamic account data. The big data platform processes the accounting data synchronized from the main system and processes it into data that can be read by streaming computing. The advantage of the big data platform is that it can cache peak data (which may cause the risk of streaming computing clusters to have difficulty in carrying) to reduce the pressure of subsequent processing. The accounting data generated by other system A is general, but it is deployed close to the big data platform. The accounting processing data is synchronized to the big data platform through the message queue. The accounting data generated by other system B is less, and may have the characteristics of a remote computer room layout and more systems. The data is synchronized to the big data platform through the data integration system. Data integration is to merge a small amount of data sent from multiple systems into a data update stream and send it to the big data platform. The streaming computing cluster reads the account update stream from the big data platform, performs appropriate distributed computing, merges at appropriate time segments, and sends the final processed update stream to the account platform. The account platform obtains the update stream from the streaming computing cluster and updates the local account system in real time. Customers can query the values ​​of various accounting items in the account system according to their needs, and even draw a curve chart of the account flow in a day based on the data; they can also monitor certain data.

[0049] Then proceed with the accounting data processing as follows:

[0050] 1. Obtain account data from each bank system and calculate the transaction volume. Also calculate the distance between each system and the big data platform.

[0051] 2. According to the specified threshold, select the systems whose account transaction volume is greater than the threshold, and then classify them into the first system set. The remaining systems are classified into the second system set.

[0052] 3. Perform separate account processing on each bank system in the first system set, and upload the subsequent account processing results that may cause abnormal account processing between systems to the blockchain.

[0053] 4. The banking systems in the second system set are clustered according to their distance from the big data platform to form several banking system subclasses.

[0054] 5. For each sub-category formed above, the data formed by the sub-category is integrated. According to the average distance between the banking system in the sub-category and the big data platform, the synchronization method of the integrated data and the big data platform is selected.

[0055] 6. The streaming computing cluster reads the accounting data of the big data platform in real time, performs appropriate distributed computing, and merges it at appropriate time segments, and sends the final processed update stream to the accounting platform.

[0056] 7. Based on the data from the streaming computing cluster, the accounting platform updates the local accounting data in real time. Customers can query the values ​​of each accounting item in the accounting system according to their needs, and even draw a curve chart of the accounting flow in a day based on the data; they can also monitor certain data.

[0057] 8. Update the corresponding data of the blockchain based on the real-time accounting data of the above streaming computing cluster and accounting platform.

[0058] In the embodiment of the present invention, a big data platform is used to buffer the account processing flow of a distributed system. Account processing is generally a relatively simple and regular text, but each transaction involving accounts will be generated, which is characterized by being simple but the amount of data may be particularly large. Using a big data platform to manage data can give full play to the management of massive data by the big data platform, and it has high convenience as a cache and processing of massive intermediate data. Using streaming computing to ensure real-time performance, streaming computing is mainly characterized by its sequential processing and response to input data. Using the underlying data synchronization mechanism to reduce the impact on the performance of the original system, the synchronization between databases based on the log level can greatly speed up the data synchronization speed, and there is no workload for the upper-level application to be transformed. Using the data flow integration method to reduce the interaction of multiple systems, if the system involving accounts is too scattered, the data transfer integration service is added in the middle, and multiple systems (the amount of account updates is not large) are integrated into a unified update flow to big data or streaming computing synchronization. This consideration is mainly due to the data imbalance between systems. A very small number of systems have a very large amount of data, while most systems generate very small amounts of data.

[0059] Based on the same inventive concept, the present invention also provides an accounting data processing device, as described in the following embodiments. Since the principles of solving these problems are similar to the accounting data processing method, the implementation of the accounting data processing device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0060] Figure 4 FIG. 1 is a structural diagram of an accounting data processing device in an embodiment of the present invention. Figure 4 As shown, the accounting data processing device includes:

[0061] The data acquisition module 401 is used to obtain the account data of multiple bank systems, as well as the transaction volume data and deployment distance data corresponding to each bank system, wherein the deployment distance data is the deployment distance data between the bank system and the big data platform;

[0062] A system grouping module 402, configured to group the plurality of banking systems according to the transaction volume data and the deployment distance data to obtain a plurality of system sets;

[0063] The data generation module 403 is used to use the stream computing cluster to read the account data corresponding to the plurality of system sets and generate update stream data;

[0064] The data updating module 404 is used to update the local data of the accounting platform according to the update stream data.

[0065] In one embodiment, the system grouping module 402 is further configured to:

[0066] Compare the transaction volume data corresponding to each bank system with the set threshold;

[0067] If the corresponding transaction volume data of the bank system is greater than the set threshold, the bank system is classified into the first system set;

[0068] If the corresponding transaction volume data of the bank system is less than or equal to the set threshold, the bank system is classified into the second system set;

[0069] The bank systems in the second system set are processed according to the deployment distance data.

[0070] In one embodiment, the system grouping module 402 is further configured to:

[0071] Clustering the bank systems in the second system set according to the deployment distance data to obtain a plurality of system subclasses;

[0072] For each system subclass, the account data of the banking system in the system subclass is integrated;

[0073] A data synchronization method is determined based on the deployment distance data and the accounting data after data integration.

[0074] In one embodiment, the data generation module 403 is further used to:

[0075] Perform distributed computing on the accounting data read using the streaming computing cluster;

[0076] Merge the results of the distribution calculations within a set time period;

[0077] Based on the result of the merge processing, update stream data is generated.

[0078] In summary, the embodiment of the present invention obtains the account data of multiple bank systems, as well as the transaction volume data and deployment distance data corresponding to each bank system, wherein the deployment distance data is the deployment distance data between the bank system and the big data platform; the multiple bank systems are grouped according to the transaction volume data and deployment distance data to obtain multiple system sets; the account data corresponding to the multiple system sets are read using a streaming computing cluster to generate update stream data; and the local data of the account platform is updated according to the update stream data. The embodiment of the present invention groups multiple bank systems according to the transaction volume data and deployment distance data to obtain multiple system sets, and then uses a streaming computing cluster to read the account data corresponding to the multiple system sets to generate update stream data, which can effectively ensure real-time performance, achieve stability and scalability, and can achieve smooth connection for various systems and various deployment methods, thereby improving performance while ensuring real-time performance.

[0079] Based on the above invention concept, Figure 5 As shown, the present invention also proposes a computer device 500, including a memory 510, a processor 520, and a computer program 530 stored in the memory 510 and executable on the processor 520, wherein the processor 520 implements the aforementioned accounting data processing method when executing the computer program 530.

[0080] Based on the aforementioned inventive concept, the present invention proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the aforementioned accounting data processing method is implemented.

[0081] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0082] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0083] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0085] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for processing accounting data, characterized in that: include: Obtaining account data of multiple bank systems, as well as transaction volume data and deployment distance data corresponding to each bank system, wherein the deployment distance data is the deployment distance data between the bank system and the big data platform; Grouping the plurality of banking systems according to the transaction volume data and the deployment distance data to obtain a plurality of system sets; Using a streaming computing cluster to read the account data corresponding to the plurality of system sets, and generate update stream data; Update local data of the accounting platform according to the update stream data; The plurality of banking systems are grouped according to the transaction volume data and the deployment distance data to obtain a plurality of system sets, including: Compare the transaction volume data corresponding to each bank system with the set threshold; If the corresponding transaction volume data of the bank system is greater than the set threshold, the bank system is classified into the first system set; If the corresponding transaction volume data of the bank system is less than or equal to the set threshold, the bank system is classified into the second system set; Processing the bank system in the second system set according to the deployment distance data; Processing the bank system in the second system set according to the deployment distance data includes: Clustering the bank systems in the second system set according to the deployment distance data to obtain a plurality of system subclasses; For each system subclass, the account data of the banking system in the system subclass is integrated; A data synchronization method is determined based on the deployment distance data and the accounting data after data integration.

2. The accounting data processing method according to claim 1, characterized in that: The stream computing cluster is used to read the account data corresponding to the plurality of system sets to generate update stream data, including: Perform distributed computing on the accounting data read using the streaming computing cluster; Merge the results of the distribution calculations within a set time period; Based on the result of the merge processing, update stream data is generated.

3. An accounting data processing device, characterized in that: include: A data acquisition module, used to obtain account data of multiple bank systems, as well as transaction volume data and deployment distance data corresponding to each bank system, wherein the deployment distance data is the deployment distance data between the bank system and the big data platform; A system grouping module, used for grouping the plurality of banking systems according to the transaction volume data and the deployment distance data to obtain a plurality of system sets; A data generation module, used to use a stream computing cluster to read the account data corresponding to the plurality of system sets and generate update stream data; A data updating module, used to update the local data of the accounting platform according to the update stream data; Among them, the system grouping module is further used for: Compare the transaction volume data corresponding to each bank system with the set threshold; If the corresponding transaction volume data of the bank system is greater than the set threshold, the bank system is classified into the first system set; If the corresponding transaction volume data of the bank system is less than or equal to the set threshold, the bank system is classified into the second system set; Processing the bank system in the second system set according to the deployment distance data; The system grouping module is further used to: Clustering the bank systems in the second system set according to the deployment distance data to obtain a plurality of system subclasses; For each system subclass, the account data of the banking system in the system subclass is integrated; A data synchronization method is determined based on the deployment distance data and the accounting data after data integration.

4. The accounting data processing device according to claim 3, characterized in that: The data generation module is further used to: Perform distributed computing on the accounting data read using the streaming computing cluster; Merge the results of the distribution calculations within a set time period; Based on the result of the merge processing, update stream data is generated.

5. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 2 is implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.

Images