Data processing method, apparatus, device, and medium

Abstract

The present application relates to the technical field of data processing, and particularly relates to a data processing method, device, equipment and medium. The method comprises the following steps: receiving original account detail data to be processed; aggregating the original account detail data to obtain aggregated account detail data and an aggregated restoration information table; sharding the aggregated account detail data to obtain a plurality of shard account detail data; performing data processing on the plurality of shard account detail data to obtain a plurality of shard processing results corresponding to the plurality of shard account detail data; and restoring the plurality of shard processing results based on the aggregated restoration information table to obtain original processing results corresponding to the original account detail data. The present application can significantly reduce the account detail data to be processed, can fully utilize the data processing capacity of each processing unit, and can flexibly select the number of processing units to achieve flexible horizontal expansion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a data processing method, device, equipment and medium. BACKGROUND

[0002] Cash flow refers to cash inflow or cash outflow generated by a business subject through certain economic activities in a certain accounting period according to the cash receipt and payment realization system. Cash flow measurement is the core function of commercial bank risk management, especially liquidity risk management. Commercial bank inventory business cash flow splitting is a process of measuring the future cash flow of each business account according to different scenario rules for commercial bank inventory business. Since the business products and business elements to be split are various, and generally need to be combined with behavior models and consider parameters such as expiration business continuation and incremental business simulation, various types of cash flow are output, so the cash flow splitting module of the commercial bank generally needs to embed complex algorithms.

[0003] The commonly used cash flow splitting application at present can better support multi-process parallel processing (but not infinite multi-process) based on centralized transaction type database, but when the account data reaches a certain level, the efficiency of executing key data preprocessing and subsequent result data analysis processing is very low, resulting in poor overall efficiency of completing cash flow splitting. In real application, when the account is split under multiple scenarios and reaches hundreds of billions, and the split result reaches tens of billions, the overall time consumption is more than ten hours, which is difficult to meet the management requirements of domestic large commercial banks. SUMMARY

[0004] In view of the above problems of the prior art, the purpose of the present application is to provide a data processing method, device, equipment and medium, which can significantly reduce the account detail data to be processed, can fully utilize the data processing capacity of each processing unit, and can flexibly select the number of processing units to realize flexible horizontal expansion.

[0005] In order to solve the above problems, the present application provides a data processing method, which is used for an electronic device, and the method comprises:

[0006] Receiving original account detail data to be processed;

[0007] Aggregating the original account detail data to obtain aggregated account detail data and an aggregated restoration information table, wherein the aggregated restoration information table indicates the correspondence between the aggregated account detail data and the original account detail data;

[0008] Sharding the aggregated account detail data to obtain a plurality of sharded account detail data;

[0009] perform data processing on the plurality of sharded account detail data to obtain a plurality of sharded processing results corresponding to the plurality of sharded account detail data;

[0010] restore the plurality of sharded processing results based on the aggregation restoration information table to obtain an original processing result corresponding to the original account detail data.

[0011] Further, the original account detail data is aggregated based on a key account feature, wherein the original account detail data with the same key account feature is aggregated together.

[0012] Further, the key account feature includes at least one of a currency, an interest rate, a fixed-floating flag, a repayment method, a interest accrual date, and a due date.

[0013] Further, the aggregation account detail data is sharded based on a data processing capability of each processing unit in a plurality of processing units, each sharded account detail data in the plurality of sharded account detail data is sent to a corresponding processing unit, and the plurality of sharded account detail data is processed in parallel by the plurality of processing units.

[0014] Further, the data processing includes calculating a cash flow of each sharded account detail data in the plurality of sharded account detail data.

[0015] Another aspect of the present application provides a data processing device, the device comprising:

[0016] a receiving module configured to receive original account detail data to be processed;

[0017] an aggregation module configured to aggregate the original account detail data to obtain aggregation account detail data and an aggregation restoration information table, wherein the aggregation restoration information table indicates a correspondence between the aggregation account detail data and the original account detail data;

[0018] a sharding module configured to shard the aggregation account detail data to obtain a plurality of sharded account detail data;

[0019] a processing module configured to perform data processing on the plurality of sharded account detail data to obtain a plurality of sharded processing results corresponding to the plurality of sharded account detail data;

[0020] a restoration module configured to restore the plurality of sharded processing results based on the aggregation restoration information table to obtain an original processing result corresponding to the original account detail data.

[0021] Further, the original account detail data is aggregated based on key account features, wherein the original account detail data with the same key account features are aggregated together.

[0022] Further, the key account features include at least one of currency, interest rate, fixed-floating flag, repayment method, interest start date and due date.

[0023] Further, the aggregated account detail data is sharded based on data processing capacity of each processing unit in the plurality of processing units, each of the plurality of sharded account detail data is sent to a corresponding processing unit, and the plurality of sharded account detail data is processed in parallel by the plurality of processing units.

[0024] Further, the data processing includes calculating cash flow of each of the plurality of sharded account detail data.

[0025] Another aspect of the present application provides an electronic device, comprising a memory storing computer executable instructions and a processor configured to execute the instructions to implement the above-described data processing method.

[0026] Another aspect of the present application provides a computer storage medium encoded with a computer program, the computer program comprising instructions executed by a computer to implement the above-described data processing method.

[0027] Another aspect of the present application provides a computer program product comprising computer instructions, which when executed, implement the above-described data processing method.

[0028] Due to the above technical solutions, the present application has the following beneficial effects:

[0029] The data processing method according to the embodiments of the present application aggregates the original account detail data, which can significantly reduce the account detail data to be processed, and shards the aggregated account detail data, which can fully utilize the data processing capacity of each processing unit, and additionally, each sharded account detail data is processed by an independent processing unit, which no longer interacts with the distributed database except for receiving or reading in the corresponding sharded account detail data each time, and the sharded account detail data do not depend on each other, so that flexible horizontal expansion can be achieved, and as long as the computing resources of the server allow, a large number of parallel processing processes can be simply started to achieve high-speed operation. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to make the technical scheme of the present application clearer, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative effort based on these drawings.

[0031] Figure 1 is a flow chart of the data processing method provided by an embodiment of the present application;

[0032] Figure 2 is a schematic diagram of the receiving, aggregating and fragmenting functions provided by an embodiment of the present application;

[0033] Figure 3 is a schematic diagram of the processing functions provided by an embodiment of the present application;

[0034] Figure 4 is a schematic diagram of the restoring functions provided by an embodiment of the present application;

[0035] Figure 5 is a structural schematic diagram of the data processing apparatus provided by an embodiment of the present application;

[0036] Figure 6 is a structural schematic diagram of the electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the technical scheme of the present application clearer, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative effort based on these drawings.

[0038] It should be noted that the terms "first", "second", and the like in the description and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, device, product or apparatus that includes a series of steps or units does not necessarily have to include only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or apparatus.

[0039] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application are further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the present application and not to limit the embodiments of the present application.

[0040] Reference is made to the accompanying drawings Figure 1 which shows a flow chart of a data processing method provided by an embodiment of the present application, as Figure 1 shown, the method can include the following steps:

[0041] S110: receiving original account detail data to be processed.

[0042] The original account detail data to be processed can be various transaction account detail data obtained from upstream product systems about the commercial bank's inventory business. The original account data is huge, but a large number of account features are consistent. The embodiments of the present application take this into account, as will be described below, and accounts with consistent account features are combined.

[0043] S120: aggregating the original account detail data to obtain aggregated account detail data and an aggregation restoration information table, wherein the aggregation restoration information table indicates the correspondence between the aggregated account detail data and the original account detail data.

[0044] As already described above, the original account detail data is aggregated. In practice, after aggregation, the number of accounts is reduced from 1 billion to 10 million, thereby reducing the data volume by 2-3 orders of magnitude. At the same time, in order to facilitate subsequent account information restoration, an aggregation restoration information table can be generated to save the account information items that need to be restored.

[0045] More specifically, the original account detail data is aggregated based on key account features, wherein the original account detail data with the same key account features is aggregated together.

[0046] The embodiments of the present application find that a large number of accounts have consistent key account features in the original account detail data. Compared to processing each account, the accounts with consistent key account features are aggregated first, and then the aggregated accounts are processed, and the processing results are the same, which can significantly reduce the number of data processing and improve the efficiency of data processing.

[0047] More specifically, the key account features include at least one of currency, interest rate, fixed-floating flag, repayment mode, interest accrual date and due date, and the setting of such key account features is particularly suitable for cash flow calculation. For example, for accounts that are consistent in currency, interest rate, fixed-floating flag, repayment mode, interest accrual date and due date, they can be aggregated, i.e., the amount and other account features are combined, so that only one data processing is needed for the combined account. It can be understood that the key account features can also be set according to actual conditions, and the embodiments of the present application do not limit this.

[0048] S130: Sharding the aggregated account detail data to obtain a plurality of sharded account detail data.

[0049] In order to support parallel processing and horizontal expansion, the embodiments of the present application first shard the aggregated account detail data to obtain a plurality of sharded account detail data, which can be stored on a distributed database, and then each sharded account detail data is handed over to an independent processing unit for processing, further improving the efficiency of data processing.

[0050] More specifically, the aggregated account detail data is sharded based on the data processing capability of each processing unit in the plurality of processing units.

[0051] The plurality of processing units can be a plurality of processing processes in one or more servers, which are independent of each other. Depending on the data processing capability of each processing process (e.g., corresponding CPU, memory, etc.), the aggregated account detail data can be sharded in advance, and the corresponding sharded account detail data is handed over to the corresponding processing process to make full use of the data processing capability of each processing process, further improving the efficiency of data processing.

[0052] Figure 2 is a schematic diagram of the receiving, aggregating and sharding functions provided by an embodiment of the present application. As shown in Figure 2 , first, the original account detail data to be processed is received, then the original account detail data is aggregated based on the key account features to obtain the aggregated account detail data and the aggregation restoration information table, and then the aggregated account detail data is sharded based on the data processing capability of each processing unit to obtain a plurality of sharded account detail data, which is stored on a distributed database for parallel processing by a plurality of processing units.

[0053] S140: Data processing on the plurality of sharded account detail data to obtain a plurality of sharded processing results corresponding to the plurality of sharded account detail data.

[0054] More specifically, each of the plurality of sharded account detail data is sent to a corresponding processing unit, and the plurality of sharded account detail data is processed in parallel by the plurality of processing units.

[0055] In addition to receiving or reading in the corresponding sharded account detail data each time, these processing units no longer interact with the distributed database, and each sharded account detail data is not dependent on each other, so that flexible horizontal expansion can be achieved. As long as the computing resources of the server allow, high-speed operation can be achieved by simply starting a large number of parallel processing processes, which not only improves the maintainability of the application logic, but also allows users to determine the most economical number of split processes in combination with their data volume and actual split performance requirements.

[0056] Figure 3 is a schematic diagram of the processing function provided by an embodiment of the present application. As shown in Figure 3 each processing unit reads in the corresponding sharded account detail data from the distributed database, and processes the sharded account detail data based on the required scenario parameters to obtain the corresponding sharded processing result. In the case of processing multiple sharded account detail data to calculate the cash flow of each sharded account detail data, the scenario parameters can include parameters required to measure the future cash flow of each transaction account. It can be understood that the scenario parameters can also be set according to actual conditions, and the embodiments of the present application do not limit this, for example, the scenario parameters can also include parameters required for expiration business continuation, incremental business simulation, and the like, so as to calculate various types of cash flow.

[0057] S150: restoring the plurality of sharded processing results based on the aggregation restoration information table to obtain the original processing result corresponding to the original account detail data.

[0058] The plurality of sharded processing results can be uploaded to the distributed database, and the aggregation restoration information table is associated, so that the original processing result can be quickly restored.

[0059] Figure 4 is a schematic diagram of the restoration function provided by an embodiment of the present application. As shown in Figure 4 the set of sharded processing results from each processing unit is the aggregation processing result of the aggregation account detail data, and the aggregation restoration information table indicates the correspondence between the aggregation account detail data and the original account detail data, so that by combining the two, the original processing result corresponding to the original account detail data can be restored.

[0060] In summary, the data processing method according to the embodiment of the present application can significantly reduce the account detail data to be processed by aggregating the original account detail data, and can fully utilize the data processing capacity of each processing unit by sharding the aggregated account detail data, and additionally, each shard account detail data is processed by an independent processing unit, which does not interact with the distributed database except for receiving or reading the corresponding shard account detail data each time, and there is no correlation and dependency between the shard account detail data, so that flexible horizontal expansion can be realized, and as long as the computing resources of the server allow, a large number of parallel processing processes can be started simply to realize high-speed operation.

[0061] Reference is made to the accompanying drawings Figure 5 which shows a structural schematic diagram of a data processing apparatus provided by an embodiment of the present application, as shown in Figure 5 The apparatus 500 can include the following modules:

[0062] The receiving module 510 is configured to receive original account detail data to be processed;

[0063] The aggregation module 520 is configured to aggregate the original account detail data to obtain aggregated account detail data and an aggregation restoration information table, wherein the aggregation restoration information table indicates the correspondence between the aggregated account detail data and the original account detail data;

[0064] The sharding module 530 is configured to shard the aggregated account detail data to obtain a plurality of shard account detail data;

[0065] The processing module 540 is configured to perform data processing on the plurality of shard account detail data to obtain a plurality of shard processing results corresponding to the plurality of shard account detail data;

[0066] The restoration module 550 is configured to restore the plurality of shard processing results based on the aggregation restoration information table to obtain original processing results corresponding to the original account detail data.

[0067] In one possible embodiment, the original account detail data is aggregated based on key account features, wherein the original account detail data with the same key account features is aggregated together.

[0068] In one possible embodiment, the key account features include at least one of currency, interest rate, fixed-floating flag, repayment method, interest accrual date and due date.

[0069] In a possible embodiment, the aggregated account detail data is sharded based on data processing capability of each processing unit in the plurality of processing units, each of the plurality of sharded account detail data is sent to a corresponding processing unit, and the plurality of sharded account detail data is processed in parallel by the plurality of processing units.

[0070] In a possible embodiment, the data processing comprises calculating cash flow of each of the plurality of sharded account detail data.

[0071] It should be noted that the apparatus provided in the above embodiments, when realizing its functions, is only exemplified by the above division of functional modules, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above described functions. In addition, the apparatus provided in the above embodiments and the corresponding method embodiments belong to the same concept, and the specific implementation process is detailed in the corresponding method embodiments, which will not be repeated here.

[0072] One embodiment of the present application further provides an electronic device, which comprises a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the data processing method provided in the above method embodiments.

[0073] The memory can be used to store software programs and modules, and the processor can execute various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required by functions, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory can also include a memory controller to provide access of the processor to the memory.

[0074] The method embodiments provided in the embodiments of the present application can be executed in a terminal, a server or a similar computing device, that is, the above electronic device can include a terminal, a server or a similar computing device. Taking the case of running on a server as an example, as Figure 6As shown, it shows a structural schematic diagram of a server for implementing the data processing method provided by the embodiments of the present application. The server 600 can be quite different due to different configurations or performances, and can include one or more central processing units (CPU) 610 (for example, one or more processors) and a memory 630, one or more storage media 620 (for example, one or more mass storage devices) for storing application programs 623 or data 622. Among them, the memory 630 and the storage medium 620 can be temporary storage or persistent storage. The programs stored in the storage medium 620 can include one or more modules, each of which can include a series of instruction operations in the server. Further, the central processing unit 610 can be configured to communicate with the storage medium 620 to execute a series of instruction operations in the storage medium 620 on the server 600. The server 600 can also include one or more power supplies 660, one or more wired or wireless network interfaces 650, one or more input / output interfaces 640, and / or one or more operating systems 621, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0075] The input / output interface 640 can be used to receive or send data via a network. The above-mentioned specific examples of the network can include a wireless network provided by a communication provider of the server 600. In one example, the input / output interface 640 includes a network adapter (NIC) which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the input / output interface 640 can be a radio frequency (RF) module for communicating with the Internet in a wireless manner, and the wireless communication can use any communication standard or protocol, including but not limited to global system for mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short message service (SMS), etc.

[0076] Those skilled in the art can understand that Figure 6 The structure shown is merely schematic, the server 600 can further include more or less components than those shown, or have a different configuration from that shown. Figure 6 The structure shown is merely schematic, the server 600 can further include more or less components than those shown, or have a different configuration from that shown. Figure 6 The structure shown is merely schematic, the server 600 can further include more or less components than those shown, or have a different configuration from that shown.

[0077] One embodiment of the present application further provides a computer readable storage medium, which can be arranged in an electronic device to store at least one instruction or at least one program for implementing a data processing method, the at least one instruction or the at least one program is loaded and executed by the processor to implement the data processing method provided by the above method embodiment.

[0078] Optionally, in the embodiment of the present application, the storage medium can include but is not limited to: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0079] One embodiment of the present application further provides a computer program product or a computer program, which includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the data processing method provided in the above various optional implementation examples.

[0080] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description, not representing the advantages and disadvantages of the embodiments. And the above-mentioned embodiments of the present application are described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be executed in different order from that in the embodiments and still can achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are possible or can be advantageous.

[0081] Each of the embodiments in the present specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other, and each embodiment mainly describes the difference from other embodiments. Especially, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts can be referred to the part of the method embodiment.

[0082] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or can be instructed to relevant hardware by program. The program can be stored in a computer readable storage medium, and the storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0083] The above description is merely preferred embodiments of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method is for an electronic device, and the method comprises: Receive original account details data to be processed; aggregating the original account detail data to obtain aggregated account detail data and an aggregate restoration information table, wherein the aggregate restoration information table indicates a correspondence between the aggregated account detail data and the original account detail data, so that the aggregated account detail data can be restored to the original account detail data; Slicing the aggregated account detail data to obtain multiple slicing account detail data; Performing data processing on the plurality of shard account detail data to obtain a plurality of shard processing results corresponding to the plurality of shard account detail data; The multiple shard processing results are restored based on the aggregate restoration information table to obtain original processing results corresponding to the original account detail data.

2. The method according to claim 1, characterized in that The original account detail data is aggregated based on key account features, wherein the original account detail data having the same key account features are aggregated together.

3. The method according to claim 2, characterized in that The key account characteristics include at least one of currency, interest rate, fixed-floating symbol, repayment method, value date and maturity date.

4. The method according to claim 1, wherein Based on the data processing capability of each of the multiple processing units, the aggregated account detail data is sharded, each of the multiple sharded account detail data is sent to the corresponding processing unit, and the multiple processing units perform data processing on the multiple sharded account detail data in parallel.

5. The method according to claim 1, wherein The data processing includes calculating the cash flow of each shard account detail data in the plurality of shard account detail data.

6. A data processing device, characterized in that: The device comprises: a receiving module configured to receive original account detail data to be processed; an aggregation module configured to aggregate the original account detail data to obtain aggregated account detail data and an aggregate restoration information table, wherein the aggregate restoration information table indicates a correspondence between the aggregated account detail data and the original account detail data, so that the aggregated account detail data can be restored to the original account detail data; a sharding module configured to shard the aggregated account detail data to obtain a plurality of sharded account detail data; a processing module configured to perform data processing on the plurality of shard account detail data to obtain a plurality of shard processing results corresponding to the plurality of shard account detail data; The restoration module is configured to restore the multiple shard processing results based on the aggregate restoration information table to obtain original processing results corresponding to the original account detail data.

7. The device according to claim 6, characterized in that The original account detail data is aggregated based on key account features, wherein the original account detail data having the same key account features are aggregated together.

8. The device according to claim 7, characterized in that The key account characteristics include at least one of currency, interest rate, fixed-floating symbol, repayment method, value date and maturity date.

9. The device according to claim 6, characterized in that Based on the data processing capability of each of the multiple processing units, the aggregated account detail data is sharded, each of the multiple sharded account detail data is sent to the corresponding processing unit, and the multiple processing units perform data processing on the multiple sharded account detail data in parallel.

10. The device according to claim 6, characterized in that The data processing includes calculating the cash flow of each shard account detail data in the plurality of shard account detail data.

11. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the data processing method according to any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by a processor to implement the data processing method according to any one of claims 1 to 5.

13. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed, the data processing method according to any one of claims 1 to 5 is implemented.

Images