A real-time indicator system for bank flow computing business based on multi-dimensional intermediate state aggregation

Abstract

The present invention discloses a real-time indicator system for bank flow computing business based on multi-dimensional intermediate state aggregation, in which a preprocessing module receives real-time transaction flow from a Kafka message queue and interacts with a full database Redis module. The feature processing module screens and combines the features of the preprocessed data to generate intermediate state records. The indicator aggregation module is composed of a plurality of aggregation computing nodes, and the intermediate state records are dispersed to different computing nodes for aggregation according to the aggregation method. The aggregation operation process uses a parallel computing method and can use the underlying computing resources for acceleration, and the final calculation results are stored in the intermediate state database MemDB module. The present invention can improve the efficiency of computing resource utilization, balance the computing load, and effectively reduce the system IO load, so as to meet the real-time requirements of bank flow computing business indicators.

Description

Technical Field

[0001] The present invention relates to the field of bank stream computing data processing, and in particular to a real-time indicator system for bank stream computing services based on multi-dimensional intermediate state aggregation. Background Art

[0002] With the advent of the big data era, multidimensional data will appear in many financial scenarios, such as bank flow data, stock trading data, credit card account data, etc. These data usually contain multiple features, and feature calculation systems often use different calculation methods to process these different features, such as summation, averaging, maximum and minimum values, etc. However, as the complexity of the business increases, the amount of data that the feature calculation system needs to process increases rapidly. Many financial scenarios require the establishment of a multi-dimensional feature system for business objects and the use of different processing methods for different features, which poses new challenges to the performance of the system.

[0003] As an important part of the financial field, the bank's stream computing business has high requirements for the real-time performance of the system. At present, most real-time indicator systems use the mainstream classic stream computing architecture. In complex business scenarios, this architecture exposes two problems: First, because the stream computing framework does not manage and optimize the underlying data storage, each business event processing needs to extract relevant data from the underlying database and input it into the computing framework, which will cause system IO load; second, the different feature processing methods caused by multi-dimensional data will cause additional data processing overhead and waste of computing resources, which also greatly increases the operation and maintenance cost of the feature computing system. The above two problems make the classic stream computing architecture unable to meet the real-time requirements of the bank's stream computing business indicator system. Summary of the invention

[0004] The purpose of the present invention is to address the deficiencies of the prior art and propose a real-time indicator system for bank flow calculation business based on multi-dimensional intermediate state aggregation, which realizes the bank flow calculation indicator business by processing the features of the full amount of data to generate intermediate state records and aggregate them, and then compares and updates them with the original records in the intermediate state database. The system can improve the calculation efficiency of multi-dimensional data, save computing resources, and reduce the IO load of the system, so as to meet the real-time requirements of the bank flow calculation business.

[0005] The object of the present invention is to achieve the following technical solutions: a real-time indicator system for bank flow computing business based on multi-dimensional intermediate state aggregation, the system comprising a preprocessing module, a full database Redis module, an ETL module, a feature processing module, an indicator aggregation module and an intermediate state database MemDB module;

[0006] The preprocessing module is used to receive real-time transaction flow data from the Kafka message queue, retrieve the full historical data of the specified time interval from the full database Redis module according to business needs, and save the new real-time transaction flow data, and send the historical data retrieved from the full database Redis module and the real-time transaction flow data of the Kafka message queue to the ETL module;

[0007] The full database Redis module is used for complete and unprocessed multi-dimensional data related to the bank flow computing business;

[0008] The ETL module is used to clean and standardize the real-time transaction flow data of the Kafka message queue sent by the preprocessing module and the historical data of the full database Redis module, align the data according to the dimension and unify the format, convert the historical data in the full database Redis module and the real-time transaction flow data from Kafka into standard data and send them to the feature processing module;

[0009] The feature processing module performs feature screening on the cleaned and standardized multidimensional data according to business needs, and groups the screened multidimensional features according to the aggregation method of the indicator aggregation module to form a multidimensional intermediate state record in a specified format; specifically: the feature processing module includes multiple lists, each list corresponds to an aggregation method of the indicator aggregation module, and each list contains multidimensional features; each list determines the number of features and the feature combination format according to business needs;

[0010] The index aggregation module is composed of multiple aggregation nodes, each of which executes an aggregation method, distributes the multidimensional intermediate state records obtained by the feature processing module to different aggregation nodes according to the corresponding aggregation method for aggregation operation, and sends the results to the intermediate state database MemDB module;

[0011] The intermediate database MemDB module is used to store the indicator calculation results after aggregation operations, and provide indicator judgment basis for bank flow calculation business.

[0012] Furthermore, the feature combination format in the feature processing module refers to arranging multiple features of a certain type of specific data in each specific business scenario in a specified order, and directly storing the specific value of each feature in the list.

[0013] Furthermore, the aggregation method of the multidimensional intermediate state records to be processed in each aggregation calculation node in the indicator aggregation module is the same, and parallel calculations are performed by first combining the aggregatable data and then uniformly aggregating them, thereby avoiding serial calculations when a single intermediate state record is aggregated and improving aggregation efficiency.

[0014] Furthermore, the aggregation method of the multi-dimensional features of the indicator aggregation module includes taking the maximum value, the minimum value, the sum and the average value.

[0015] Furthermore, the feature processing module obtains additional auxiliary data required for data aggregation, including the timestamp of the time series data to be aggregated and the total amount of data that needs to be recorded when calculating the average value.

[0016] The advantages and beneficial effects of the present invention are:

[0017] (1) Reduce the space required to store multi-dimensional intermediate records: By analyzing business needs and defining a specified feature combination format, the space required to store each feature field name can be saved.

[0018] (2) Reduce system IO load: The system will formulate a database read and write plan based on business needs. It only needs to read a small amount of un-aggregated data from the full database. At the same time, it uses the method of aggregating intermediate state records, which eliminates the need for frequent read and write operations on the full database, thereby reducing the system IO load.

[0019] (3) Improve the efficiency of system computing resource utilization: By adopting the method of grouping features according to the aggregation method, the intermediate state records can be distributed to the computing nodes of each specified aggregation method, thereby dispersing the system computing load; multi-dimensional intermediate state records can enable the system to operate on multiple features at the same time. In addition, the calculation method of first combining and then unifying the aggregation is adopted to convert serial calculations into parallel calculations. Computing resources such as GPU can be used to accelerate the calculation process, thereby greatly improving the efficiency of computing resource utilization.

[0020] (4) By reducing the system IO load and improving the system computing efficiency, the performance requirements of the real-time indicator system of the bank flow computing business can be well met. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a structural block diagram of a preferred embodiment of the present invention.

[0022] Figure 2 This is a structural diagram of the multi-dimensional intermediate state record. DETAILED DESCRIPTION

[0023] The present invention will be further described below in conjunction with the accompanying drawings.

[0024] like Figure 1 As shown, the present invention provides a real-time indicator system for bank flow computing business based on multi-dimensional intermediate state aggregation, including a preprocessing module, a full database Redis module, an ETL (Extract-Transform-Load) module, a feature processing module, an indicator aggregation module, and an intermediate state database MemDB module.

[0025] In this invention, credit card-related business in bank flow computing business is used for explanation. This business scenario is usually to count the maximum single consumption, maximum single repayment, cumulative total consumption and credit card usage times of all customers in the month. Therefore, the business requirements can be classified into the following categories according to the aggregation method:

[0026] 1. Find the maximum value: single consumption amount, single accumulated credit points, single repayment amount, etc.;

[0027] 2. Sum: total consumption, total repayment, consumption frequency, total credit points, etc.;

[0028] 3. Find the average: average daily expenditure, average monthly expenditure, average annual expenditure, etc.

[0029] In view of the above business requirements, a specified feature combination format can be defined for each aggregation method. For example, the feature combination of the "maximum value" business is (consumption amount - repayment amount - credit points). Therefore, when transmitting the intermediate record of this business requirement, there is no need to describe the feature field. Instead, the specific value of each feature can be directly transmitted in the above format.

[0030] The preprocessing module will formulate an interaction plan with the full database Redis module based on the real-time transaction flow from the Kafka message queue, the historical transaction data of the full database Redis module, and the data stored in the intermediate database MemDB module. If it is known that the intermediate database MemDB module has stored all intermediate data before a certain day, it is only necessary to export the full data after that day from the full database Redis module and participate in aggregation (the full data before that day has been aggregated and stored in the intermediate database MemDB module, so it does not need to be exported).

[0031] The full database Redis module is used for complete unprocessed multidimensional data related to bank flow computing business, including account opening information, transaction records, transaction limits, repayment records, limit change records and credit card validity period.

[0032] The ETL module cleans and standardizes real-time transaction flow and historical transaction data. It aligns the data by dimension and unifies the format, converts the historical data in the full database Redis module and the real-time transaction flow data from Kafka into standard data and sends it to the feature processing module.

[0033] After obtaining the standardized full data, the feature processing module generates an intermediate state record according to the format definition of the feature combination. The specific structure of the multidimensional intermediate state record is as follows: Figure 2As shown, it includes the primary key P_Key, the aggregation key W_Key, the feature combination key F_Key and the auxiliary data key D_Key, as follows:

[0034] (1) Primary key P_Key, associated with a specific business object, such as Figure 2 The primary key can be a credit card account ID, and the key value is globally unique. The primary key and auxiliary data are used together to determine the aggregability between multiple intermediate records. When the primary key and auxiliary data are consistent, it means that the two sets of data can be aggregated. Therefore, the combination process between aggregable data can be accelerated by hash mapping the primary key and auxiliary data.

[0035] (2) Aggregation method key W_Key, which is used to describe the aggregation method of the features contained in the intermediate state records, such as taking the maximum value, accumulation, averaging, etc. Figure 2 The aggregation method shown is to take the maximum value, which is represented by Max.

[0036] (3) Feature combination key F_Key, which is composed of a list of multiple features. The feature processing module determines the feature combination format according to business needs to omit the field name. Multiple features in the same intermediate record have the same aggregation method. The feature combination format defined according to business needs is for a certain type of specific data in a specific business scenario, and its multiple features are arranged in a specified order. This can save the space required to store the field name, and the specific value of each feature is directly stored in the list. Figure 2 One way to combine the features of credit card data based on the business scenario of taking the maximum value is: "Consumption amount - repayment amount - credit points". The list length of the feature combination key is 3, and each value corresponds to each field in the feature combination order.

[0037] (4) Auxiliary data key D_Key, which consists of two fields: the timestamp for time series data and additional auxiliary data required for data aggregation, such as the total amount of data required to calculate the average. If no auxiliary data is required, it is filled with null, such as Figure 2 If the maximum value calculation process shown does not require auxiliary data, the value of this field is null.

[0038] The aggregation method of multi-dimensional intermediate state records that each aggregation computing node in the indicator aggregation module needs to process is the same, and the method of combining the aggregatable data first and then aggregating them uniformly is adopted, which can avoid serial operations when aggregating a single intermediate state record. Instead, it uses underlying computing resources such as GPU to perform parallel operations on massive data and aggregate them uniformly, thereby improving aggregation efficiency. After obtaining the multi-dimensional intermediate state records, the indicator aggregation module will first group them according to the aggregation method, and assign the intermediate state records with the same aggregation method to the same computing node. The operation process of the aggregation computing node is as follows: the aggregation computing node takes out n multi-dimensional intermediate state records each time, first traverses n records, and first hashes the system data to the specified queue according to the primary key and auxiliary data key of the multi-dimensional intermediate state record. The mapping process does not perform aggregation calculations but directly merges the multi-dimensional intermediate state records. After traversing n data, a unified aggregation operation is performed on each queue that has been merged. The specific algorithm is shown below (expressed in Python syntax):

[0039]

[0040]

[0041] The parameter data in line 01 is a list containing all intermediate records. Line 02 defines the word typical variable that stores the final aggregation result. Line 03 traverses n multidimensional intermediate records. Line 04 first combines the primary key and the auxiliary data key of each record, and uses the composite key for hash mapping. Line 05 determines whether the above composite key exists in the dictionary. If not, a new list is created for the composite key in line 06. If the composite key exists in the dictionary or a new list is created for the composite key, the feature composite key of the intermediate record is directly added to the corresponding list in line 07. This process only involves the combination of data that can be aggregated and does not involve aggregation operations. Line 08 traverses the dictionary, and line 09 performs unified aggregation calculations on the lists under each composite key. Line 10 returns the final aggregation result.

[0042] After the index aggregation module completes the aggregation operation, it obtains the final aggregation result, aggregates the calculation result with the corresponding intermediate state record in the intermediate state database MemDB module, and updates the database. This process only requires one read and write operation of the intermediate state database MemDB module, which greatly reduces the IO load of the system. At the same time, since the index aggregation module adopts the method of merging all aggregatable records first and then performing aggregation calculation uniformly, and all data in each merged queue adopts the same aggregation method, it can use underlying computing resources such as GPU for parallel computing, accelerate the calculation process, and greatly improve the efficiency of computing resource utilization and reduce the time required for calculation.

[0043] The workflow of the system of the present invention is as follows:

[0044] (1) The preprocessing module receives real-time transaction flow from the Kafka message queue and interacts with the full database Redis module. After the interaction is completed, the real-time transaction flow and historical transaction data are sent to the ETL module.

[0045] (2) The ETL module cleans and standardizes the data and sends it to the feature processing module.

[0046] (3) The feature processing module selects features according to business needs and combines them in a specified format. Each combination format corresponds to an aggregation method. It then generates multidimensional intermediate state records and sends them to the indicator aggregation module.

[0047] (4) The indicator aggregation module distributes the intermediate state records to different aggregation nodes according to the aggregation method. The aggregation calculation node takes out n multidimensional intermediate state records each time, traverses the n records first, and hashes the intermediate state records to the specified queue according to the primary key and auxiliary data key of the multidimensional intermediate state records. The data mapped to the same queue is the aggregatable data. The mapping process does not perform aggregation calculations but directly merges the multidimensional intermediate state records. After traversing the n data, the aggregation operation is performed on each queue that has been merged.

[0048] (5) The final intermediate state records calculated by the indicator aggregation module are aggregated and saved with the corresponding intermediate state records in the intermediate state database MemDB.

[0049] The key point of the present invention is that by using multi-dimensional intermediate state records, the system does not need to read all data from the Redis module of the full database, nor does it need to perform frequent read and write operations on the intermediate state database, and it is not necessary to perform separate operations on each feature, which significantly reduces the system IO load. The definition of the feature format reduces the space required to save the intermediate state records. In addition, in the process of aggregate calculation, the method of allocating computing nodes according to the aggregation method and using GPU parallel computing not only disperses the computing load of the system, but also greatly improves the efficiency of computing resource utilization, thereby better meeting the real-time requirements of the real-time indicator system of the bank flow computing business.

[0050] The above-mentioned implementation cases are used to explain the present invention rather than to limit the present invention. Any modifications and changes made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A real-time indicator system for bank flow calculation business based on multi-dimensional intermediate state aggregation, characterized in that: The system includes a preprocessing module, a full database Redis module, an ETL module, a feature processing module, an indicator aggregation module and an intermediate database MemDB module; The preprocessing module is used to receive real-time transaction flow data from the Kafka message queue, retrieve the full historical data of the specified time interval from the full database Redis module according to business needs, and save the new real-time transaction flow data, and send the historical data retrieved from the full database Redis module and the real-time transaction flow data of the Kafka message queue to the ETL module; The full database Redis module is used for complete and unprocessed multi-dimensional data related to the bank flow computing business; The ETL module is used to clean and standardize the real-time transaction flow data of the Kafka message queue sent by the preprocessing module and the historical data of the full database Redis module, align the data according to the dimension and unify the format, convert the historical data in the full database Redis module and the real-time transaction flow data from Kafka into standard data and send them to the feature processing module; The feature processing module performs feature screening on the cleaned and standardized multidimensional data according to business needs, and groups the screened multidimensional features according to the aggregation method of the indicator aggregation module to form a multidimensional intermediate state record in a specified format; specifically: the feature processing module includes multiple lists, each list corresponds to an aggregation method of the indicator aggregation module, each list contains multidimensional features, and the aggregation method of the multidimensional features includes taking the maximum value, the minimum value, the sum and the average value; each list determines the number of features and the feature combination format according to business needs; the feature combination format refers to arranging multiple features of a certain type of specific data in each specific business scenario in a specified order, and directly storing the specific value of each feature in the list; the specific structure of the multidimensional intermediate state record includes the primary key P_Key associated with a specific business object, the aggregation method key W_Key describing the aggregation method of the features contained in the intermediate state record, the feature combination key F_Key composed of a list of multiple features, and the auxiliary data key D_Key including the timestamp field of the time series data and the additional auxiliary data field required for data aggregation; the primary key and the auxiliary data are used together to determine the aggregability between multiple intermediate state records; The indicator aggregation module is composed of multiple aggregation nodes, each of which executes an aggregation method, distributes the multidimensional intermediate state records obtained by the feature processing module to different aggregation nodes according to the corresponding aggregation method for aggregation operation, and sends the results to the intermediate state database MemDB module; the aggregation method of the multidimensional intermediate state records to be processed in each aggregation calculation node is the same, and the method of first combining the aggregatable data and then uniformly aggregating them is adopted for parallel operation, avoiding serial operation when a single intermediate state record is aggregated, thereby improving the aggregation efficiency; The intermediate database MemDB module is used to store the indicator calculation results after aggregation operations, and provide indicator judgment basis for bank flow calculation business.

2. According to claim 1, a real-time indicator system for bank flow computing business based on multi-dimensional intermediate state aggregation is characterized in that: The feature processing module obtains additional auxiliary data required for data aggregation, including the timestamp of the time series data to be aggregated and the total amount of data that needs to be recorded when calculating the average value.

Images