Data processing method and device based on big data warehouse, equipment and medium

Abstract

The invention relates to the technical field of finance, and relates to a data processing method and system based on a big data warehouse, and the method comprises the steps: obtaining a committee case relationship flow table and a repayment flow table, carrying out the granularity clustering and data association of the committee case relationship flow table and the repayment flow table, and obtaining a full-amount committee case relationship repayment flow table; performing time dimension clustering and mechanism dimension mapping on the total committed case relationship repayment flow table to obtain a mechanism dimension repayment table; index rule derivation is carried out on the organization dimension money return table to obtain a commission calculation rule; according to the commission calculation rule, commission parameter extraction is carried out on the organization dimension refunding table, and a commission parameter table is obtained; and performing data processing on the commission parameter table according to the commission calculation rule to obtain a commission table. The financial data processing efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of financial technology, and in particular to a data processing method, device, equipment and medium based on a big data warehouse. Background Art

[0002] Outsourcing commission settlement refers to the process of settling and paying commissions for the services provided by a third-party institution after an enterprise or financial institution outsources part of its business to the institution, in accordance with the commission rules agreed in the contract. With the rapid development of the financial industry, outsourcing business models are becoming increasingly widespread, especially in the fields of credit cards, consumer loans, and installment payments on e-commerce platforms. In order to reduce operating costs and improve collection efficiency, financial data processing has become increasingly common.

[0003] Existing outsourcing settlement methods are mostly based on manual offline calculations, performed offline by trained personnel using spreadsheet tools. In actual applications, settlement methods based on manual offline calculations are difficult to handle data source commission case data with monthly turnover exceeding 100 million, and the calculation rules for commission case settlement are complex and changeable, which may lead to low data processing efficiency. Summary of the Invention

[0004] The present invention provides an artificial intelligence-based data processing method, device, computer equipment and medium based on a big data warehouse, which improves the flexibility of commission calculation with configurable commission calculation rules, realizes low-code commission calculation with rule calculation scripts, improves the efficiency of commission calculation, and solves the technical problem of low efficiency in financial data processing during commission settlement.

[0005] In a first aspect, a data processing method based on a big data warehouse is provided, comprising: Obtain a commission-case relationship flow table and a repayment flow table, and cluster the commission-case relationship flow table into a monthly granularity commission-case relationship flow table and the repayment flow table into a monthly granularity repayment flow table according to a preset time period; Performing repayment time matching and data association on the monthly granularity commission relationship flow table and the monthly granularity repayment flow table to obtain a full commission relationship repayment flow table; Perform time dimension clustering and institution dimension mapping on the repayment flow sheet of all entrusted case relationships to obtain an institution dimension repayment sheet; Derivation of indicator rules for the institution-dimensional reimbursement table to obtain commission calculation rules; Extracting commission parameters from the institution-dimensional payment collection table according to the commission calculation rules to obtain a commission parameter table; The commission parameter table is processed according to the commission calculation rules to obtain a commission table.

[0006] In a second aspect, a data processing device based on a big data warehouse is provided, comprising: A granularity clustering module is used to obtain a commission-case relationship flow table and a repayment flow table, and cluster the commission-case relationship flow table into a monthly granularity commission-case relationship flow table and the repayment flow table into a monthly granularity repayment flow table according to a preset time period; A data association module is used to match the payment collection time and perform data association on the monthly granularity commission relationship flow table and the monthly granularity repayment flow table to obtain a full commission relationship repayment flow table; A dimension mapping module is used to perform time dimension clustering and institution dimension mapping on the repayment flow sheet of the full entrusted case relationship to obtain an institution dimension repayment sheet; A rule derivation module is used to derive indicator rules for the institution-dimensional reimbursement table to obtain commission calculation rules; A parameter extraction module, configured to extract commission parameters from the institution-dimensional payment collection table according to the commission calculation rule to obtain a commission parameter table; The financial data processing module is used to process the commission parameter table according to the commission calculation rules to obtain a commission table.

[0007] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned data processing method based on a big data warehouse are implemented.

[0008] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned data processing method based on a big data warehouse are implemented.

[0009] In the above-mentioned scheme implemented by the data processing method, device, computer equipment and storage medium based on big data data warehouse, the full commission relationship repayment flow table can be obtained by data association of the commission relationship flow table and the repayment flow table, which can realize data cleaning of the commission repayment data and provide support for monthly dimensional analysis for financial data processing, thereby improving the efficiency of financial data processing. By performing institutional dimension mapping on the full commission relationship repayment flow table, an institutional dimension repayment table can be obtained, which can measure the business performance of each institution in commission and repayment, and can serve as a direct basis for calculating the institution's commission, thereby improving the computational efficiency of financial data processing. By deriving indicator rules from the institutional dimension repayment table, a hierarchical processing indicator for the commission calculation rule is obtained, thereby ensuring that the logic of the indicator is clear, and enabling each calculation to be flexibly iterated, thereby improving the efficiency of financial data processing.

[0010] By extracting commission parameters from the institutional dimension collection table according to the commission calculation rules, a commission parameter table is obtained. The corresponding basic data can be filtered out from the institutional dimension collection table according to the calculation rules, thereby facilitating the subsequent calculation of the commission amount. By processing the commission parameter table according to the commission calculation rules, performance data and rule logic can be fully utilized to ensure the accurate generation of the commission table and improve the efficiency of financial data processing. Therefore, the data processing method and system based on big data warehouse proposed in the present invention can solve the problem of low efficiency when performing commission calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0012] Figure 1 This is a flow chart of a data processing method based on a big data warehouse in one embodiment of the present invention; Figure 2 yes Figure 1 A schematic flow chart of a specific implementation of step S20; Figure 3 yes Figure 1 A schematic flow chart of a specific implementation of step S30; Figure 4 This is a structural diagram of a data processing device based on a big data warehouse in one embodiment of the present invention; Figure 5 is a structural diagram of a computer device in one embodiment of the present invention; DETAILED DESCRIPTION The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0013] See also Figure 1 As shown, Figure 1 A flowchart of a data processing method based on a big data warehouse provided in an embodiment of the present invention includes the following steps: S10: Obtain a commission-case relationship flow table and a repayment flow table, and cluster the commission-case relationship flow table granularity into a monthly granularity commission-case relationship flow table and the repayment flow table granularity into a monthly granularity repayment flow table according to a preset time period.

[0014] In detail, the case entrustment relationship flow sheet records the basic information involved in the case entrustment process and is the core table of the case entrustment and business process. The repayment flow sheet records the customer's repayment behavior and its details and is a reflection of the subsequent behavior of the case entrustment business.

[0015] Specifically, the case relationship flow table includes parameters such as user name, queue name, case start time, case end time, institution name, collection officer name, case amount and remaining principal of the case. The repayment flow table includes user name, repayment time, repayment amount and overdue repayment amount, etc.

[0016] In detail, respectively obtaining the commission-case relationship flow table and the repayment flow table refers to importing the commission-case relationship flow data in the MySQL database into the Hive database to obtain the commission-case relationship flow table, and importing the repayment flow data in the MySQL database into the Hive database to obtain the repayment flow table. By importing the Hive database, subsequent calculations of the data can be facilitated.

[0017] In detail, the time period refers to a period with a monthly time interval, and the granularity clustering refers to statistics on the case amount and the remaining principal of the case in the case relationship flow table according to the time period to obtain a monthly granularity case relationship flow table, and statistics on the repayment amount and the overdue repayment amount in the repayment flow table according to the time period to obtain a monthly granularity repayment flow table.

[0018] S20: Perform repayment time matching and data association on the monthly granularity commission relationship flow table and the monthly granularity repayment flow table to obtain a full commission relationship repayment flow table.

[0019] Specifically, the full commission relationship repayment flow table is a full data table aggregated through the time dimension based on the commission relationship flow table and the repayment flow table. The full commission relationship repayment flow table includes user name, queue name, commission start time, commission end time, institution name, collection officer name, commission amount, commission remaining principal, repayment amount and overdue repayment amount.

[0020] In the embodiment of the present invention, referring to Figure 2 As shown, the monthly granularity commission relationship flow table and the monthly granularity repayment flow table are matched with the repayment time and data associated to obtain the full commission relationship repayment flow table, including: S21. Extract the commission start time and commission end time from the monthly commission relationship flow table; S22. Generate a case entrustment time interval according to the case entrustment start time and the case entrustment end time; S23. Extracting the repayment time from the monthly repayment flow sheet; S24. Match the payment time with the case commission time interval to obtain a matching payment time. S25. Extracting matching repayment flow from the monthly repayment flow table according to the matching repayment time; S26. Data association is performed on the monthly granularity commission relationship flow table based on the matched repayment flow to obtain a full commission relationship repayment flow table.

[0021] Specifically, the commission case start time refers to the start time of the commission case after being divided by month, the commission case end time refers to the end time of the commission case after being divided by month, and the commission case time interval is the time interval from the commission case start time to the commission case end time divided by month.

[0022] In detail, the time matching refers to matching the repayment time according to the user name to see whether it is within the corresponding case entrustment time interval. If so, the corresponding repayment time will be used as the matching repayment time. The matching repayment flow refers to the repayment amount corresponding to the matching repayment time and the overdue repayment amount and other flows.

[0023] In detail, the data association refers to adding the corresponding repayment amount and the overdue repayment amount to the table where the case time interval is located, and aggregating all corresponding tables into a full case relationship repayment flow table.

[0024] In an embodiment of the present invention, by matching the repayment time and associating the data of the monthly granularity commission relationship flow table and the monthly granularity repayment flow table, a full commission relationship repayment flow table is obtained, which can achieve data cleaning of the commission repayment data and provide support for monthly dimensional analysis of financial data processing, thereby improving the efficiency of financial data processing.

[0025] S30: Perform time dimension clustering and institution dimension mapping on the repayment flow sheet of all entrusted case relationships to obtain an institution dimension repayment sheet.

[0026] In detail, the institution-dimensional repayment table refers to a table that aggregates and compiles the repayment data in the full-case relationship repayment flow table according to the dimension of the institution name, which can reflect the overall business performance of each institution.

[0027] Specifically, the institution-dimensional collection table is a full-month granularity table aggregated by institution, reflecting the overall business performance of each institution. The institution-dimensional collection table includes the report month, queue name, institution name, cumulative case amount, cumulative remaining principal of entrusted cases, cumulative collection amount and cumulative overdue collection amount.

[0028] In the embodiment of the present invention, referring to Figure 3 As shown, the repayment flow sheet of the full entrusted case relationship is clustered in the time dimension and mapped in the institution dimension to obtain the institution dimension repayment sheet, including: S31. Cluster the repayment flow sheet of all entrusted cases in the time dimension to obtain a repayment sheet in the time dimension; S32. Extract queue names from the time dimension collection table, and cluster the time dimension collection table according to the queue names to obtain a queue dimension collection table; S33. Extract the institution name from the queue dimension collection table, and perform institution clustering on the queue dimension collection table according to the institution name to obtain an institution dimension collection table.

[0029] In detail, the time dimension clustering refers to aggregating and grouping the commission amount, remaining principal of the commission, repayment amount and overdue repayment amount in the repayment flow table of the full commission relationship according to the month corresponding to the commission start time and the commission end time, and aggregating the aggregated commission start time and commission end time into a report month to obtain a time dimension repayment table.

[0030] Specifically, the queue clustering refers to aggregating and grouping the entrusted case amount, the entrusted case remaining principal, the repayment amount and the overdue repayment amount in the time dimension repayment table according to the queue name to obtain the queue dimension repayment table.

[0031] In detail, the institutional clustering refers to aggregating and grouping the entrusted case amount, remaining principal of the entrusted case, repayment amount and overdue repayment amount in the queue dimension repayment table according to the institution name, and obtaining the institutional dimension repayment table corresponding to the cumulative entrusted case amount, cumulative remaining principal of the entrusted case, cumulative repayment amount and cumulative overdue repayment amount respectively.

[0032] In an embodiment of the present invention, by clustering the time dimension and mapping the institution dimension of the repayment flow sheet of the full commission relationship, an institution dimension repayment sheet is obtained, which can measure the business performance of each institution in commission and repayment, and can be used as a direct basis for calculating the institution's commission, thereby improving the computational efficiency of financial data processing.

[0033] S40: deriving indicator rules for the institution-dimensional payment collection table to obtain commission calculation rules.

[0034] Specifically, the commission calculation rules store specific logical rules for commission calculation, including various indicators, algorithm rules and corresponding combination sequences for commission calculation, and can complete commission calculation in combination with the commission parameter table.

[0035] In detail, the commission calculation rules include indicator name, indicator combination name, indicator type, rule type, rule input parameters, rule output parameters, rule script, data operation type and indicator combination order. The indicator types include Class I basic indicators, Class II basic indicators, Class I derivative indicators and Class II derivative indicators.

[0036] In the embodiment of the present invention, the indicator rule derivation of the institution-dimensional payment collection table to obtain the commission calculation rule includes: Extracting payment details from the institution-dimensional payment table; Performing index calculation on the detailed payment data according to preset calculation rules to obtain calculation index; A commission calculation rule is generated according to the calculation rule and the calculation index.

[0037] In detail, the collection data screening refers to screening out the collection details data such as the cumulative entrusted amount, the cumulative entrusted remaining principal, the cumulative collection amount and the cumulative overdue collection amount corresponding to each institution name and each queue name.

[0038] Specifically, the preset calculation rules are extracted from the rule calculation script. The rule calculation script is the calculation rules for various types of commissions pre-configured by relevant personnel, including repayment methods and repayment ratios corresponding to different commission types. The calculation rules include rule type, rule input parameters, rule output parameters and rule scripts, and the calculation rules can be extracted by database retrieval.

[0039] In detail, generating the commission calculation rule according to the calculation rule and the calculation index refers to adding the calculation index to the calculation rule to obtain the commission calculation rule.

[0040] Specifically, the calculation of the indicators of the payment details data according to the preset calculation rules to obtain the calculation indicators includes: Performing basic index calculation on the detailed payment data according to a preset calculation rule to obtain a class of basic indicators; Performing indicator combination and rule-dependent calculation on the first category of basic indicators to obtain the second category of basic indicators; Performing data expansion, indicator combination, and dependent rule calculation on the two types of basic indicators to obtain a type of derived indicators; Expanding the operation type of the first type of derivative indicators to obtain a second type of derivative indicators; A calculation indicator is generated according to the first type of basic indicator, the second type of basic indicator, the first type of derived indicator and the second type of derived indicator.

[0041] Specifically, the basic indicator calculation refers to the indicator calculation of the repayment details data based on the calculation rules and the database calculation algorithm, the indicator combination refers to the parallel calculation of each indicator of the same layer, and the dependent rule calculation refers to the serial calculation of indicators of different layers based on the dependency relationship in the calculation rules.

[0042] In detail, the first category of basic indicators can be basic indicators such as the total transaction amount, commission amount, and number of customer transactions, which are directly calculated based on detailed data. The second category of basic indicators can be indicators such as the transaction average or the commission rate average, which are calculated based on the first category of basic indicators. The first category of derivative indicators can be indicators such as the monthly commission ratio and customer activity, which are derived from the second category of basic indicators. The second category of derivative indicators can be calculated indicators such as the high-net-worth customer commission ratio and large-value transaction ratio, which are expanded based on the first category of derivative indicators.

[0043] In detail, the data expansion refers to expanding the repayment details data in different databases, and the operation type expansion refers to expanding data of different data operation types, wherein the different data operation types may be data stratification types, data clustering types, and data derived calculation types, etc., and the generating of calculation indicators based on the first category of basic indicators, the second category of basic indicators, the first category of derived indicators, and the second category of derived indicators refers to aggregating the first category of basic indicators, the second category of basic indicators, the first category of derived indicators, and the second category of derived indicators into calculation indicators.

[0044] In an embodiment of the present invention, by deriving indicator rules for the institution-dimensional collection table, a hierarchical processing indicator for commission calculation rules is obtained, thereby ensuring clear logic of the indicator and enabling flexible iteration of each calculation, thereby improving the efficiency of financial data processing.

[0045] S50: Extracting commission parameters from the institution-dimensional collection table according to the commission calculation rule to obtain a commission parameter table.

[0046] In detail, the commission parameter table is a table that stores key parameters designed in financial data processing, and the commission parameter table includes the report month, queue name, institution name and various commission parameters.

[0047] In the embodiment of the present invention, the commission parameter table is obtained by extracting the commission parameter from the institution-dimensional collection table according to the commission calculation rule, including: Extracting rule input parameters and indicator types from the commission calculation rule; Extracting a rule parameter table from the institution-dimensional payment collection table according to the rule input; Decomposing the indicator type into composite parameters to obtain indicator parameters; Extracting an indicator parameter table from the institution dimension payment collection table according to the indicator parameters; A commission parameter table is generated according to the rule parameter table and the indicator parameter table.

[0048] In detail, the rule input refers to the parameters that need to be input into the commission calculation rule, and the indicator types include a first-class basic indicator, a second-class basic indicator, a first-class derivative indicator, and a second-class derivative indicator.

[0049] In detail, the rule parameter table is a parameter table composed of parameters corresponding to the rule input parameters in the institution dimension collection table, and the composite parameter decomposition refers to decomposing the parameters corresponding to various indicators in the indicator type into the smallest data type items, and using the corresponding data type items as indicator parameters.

[0050] Specifically, the indicator parameter table is a parameter table composed of parameters corresponding to the indicator parameters in the institution dimension repayment table, and the commission parameter table is composed of the rule parameter table and the indicator parameter table.

[0051] In an embodiment of the present invention, by extracting commission parameters from the institution-dimensional collection table according to the commission calculation rules, a commission parameter table is obtained, and the corresponding basic data can be filtered out from the institution-dimensional collection table according to the calculation rules, thereby facilitating the subsequent calculation of the commission amount.

[0052] S60: Processing the commission parameter table according to the commission calculation rule to obtain a commission table.

[0053] In detail, the commission table stores the final calculated financial data processing results of the monthly settlement, including the report month, queue name, institution name, collection commission, collection commission of all queues and other commission results.

[0054] In the embodiment of the present invention, the data processing of the commission parameter table according to the commission calculation rule to obtain the commission table includes: Selecting calculation rules from the commission calculation rules one by one as target calculation rules, using the rule script corresponding to the target rule as the target rule script, and extracting target rule input parameters from the target calculation rule; Extracting target rule parameters from the commission parameter table according to the target rule input parameter; Performing data processing on the target rule parameters according to the target rule script to obtain a payment commission; Generate a primary commission table based on all the collection commissions corresponding to all the target calculation rules; The primary commission table is clustered according to the queue names to obtain a commission table.

[0055] In detail, the target rule script refers to the Hive database calculation script that implements the target rule. The target rule parameters can be extracted from the commission parameter table using a database matching algorithm. The queue clustering method is consistent with that in step S20 above and will not be repeated here.

[0056] In the embodiment of the present invention, by performing data processing on the commission parameter table according to the commission calculation rules, performance data and rule logic can be fully utilized to ensure accurate generation of the commission table and improve the efficiency of financial data processing.

[0057] It can be seen that in the above scheme, by performing data association on the commission-case relationship flow sheet and the repayment flow sheet, the full commission-case relationship repayment flow sheet is obtained, which can realize data cleaning of the commission-case repayment data, and can provide support for monthly dimensional analysis for financial data processing, thereby improving the efficiency of financial data processing. By performing institutional dimension mapping on the full commission-case relationship repayment flow sheet, the institutional dimension repayment sheet is obtained, which can measure the business performance of each institution in commissions and repayments, and can serve as a direct basis for calculating institutional commissions, thereby improving the computational efficiency of financial data processing. By performing indicator rule deduction on the institutional dimension repayment sheet, the hierarchical processing indicator of the commission calculation rule is obtained, thereby ensuring the clarity of the indicator logic, and enabling each calculation to be flexibly iterated, thereby improving the efficiency of financial data processing.

[0058] By extracting commission parameters from the institution-dimensional collection table according to the commission calculation rules, a commission parameter table is obtained. The corresponding basic data can be filtered out from the institution-dimensional collection table according to the calculation rules, thereby facilitating the subsequent calculation of the commission amount. By processing the commission parameter table according to the commission calculation rules, performance data and rule logic can be fully utilized to ensure the accurate generation of the commission table and improve the efficiency of financial data processing. Therefore, the data processing method based on big data warehouse proposed in the present invention can solve the problem of low efficiency when performing commission calculation.

[0059] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0060] In one embodiment, a data processing device based on a big data warehouse is provided, and the data processing device based on a big data warehouse corresponds one-to-one to the data processing method based on a big data warehouse in the above embodiment. Figure 4 As shown, the data processing device based on the big data warehouse includes a granularity clustering module 101, a data association module 102, a dimension mapping module 103, a rule derivation module 104, a parameter extraction module 105, and a financial data processing module 106. The functional modules are described in detail as follows: The granularity clustering module 101 is used to obtain the commission-case relationship flow table and the repayment flow table, and cluster the commission-case relationship flow table into a monthly granularity commission-case relationship flow table and the repayment flow table into a monthly granularity repayment flow table according to a preset time period. The data association module 102 is configured to perform payment collection time matching and data association on the monthly granularity commission relationship flow table and the monthly granularity repayment flow table to obtain a full commission relationship repayment flow table; The dimension mapping module 103 is used to perform time dimension clustering and institution dimension mapping on the repayment flow sheet of the full case entrustment relationship to obtain an institution dimension repayment sheet; The rule derivation module 104 is used to derive indicator rules from the institution-dimensional repayment table to obtain commission calculation rules; The parameter extraction module 105 is used to extract commission parameters from the institution-dimensional collection table according to the commission calculation rules to obtain a commission parameter table; The financial data processing module 106 is configured to process the commission parameter table according to the commission calculation rules to obtain a commission table.

[0061] In one embodiment, when the data association module 102 performs repayment time matching and data association on the monthly granularity commission relationship flow table and the monthly granularity repayment flow table to obtain the full commission relationship repayment flow table, it is used to: Extract the commission start time and commission end time from the monthly commission relationship flow table; Generate a case entrustment time interval according to the case entrustment start time and the case entrustment end time; Extracting the repayment time from the monthly repayment flow sheet; Match the payment time with the case commission time interval to obtain a matching payment time; Extracting matching repayment flow from the monthly repayment flow table according to the matching repayment time; The monthly granularity commission relationship flow table is data-associated according to the matching repayment flow to obtain a full commission relationship repayment flow table.

[0062] In one embodiment, when the dimension mapping module 103 performs time dimension clustering and institution dimension mapping on the repayment flow sheet of the full entrusted case relationship to obtain the institution dimension repayment sheet, it is used to: Perform time dimension clustering on the repayment flow sheet of all entrusted case relationships to obtain a time dimension repayment sheet; Extracting queue names from the time dimension collection table, and performing queue clustering on the time dimension collection table according to the queue names to obtain a queue dimension collection table; The institution name is extracted from the queue dimension collection table, and the queue dimension collection table is clustered according to the institution name to obtain the institution dimension collection table.

[0063] In one embodiment, when the rule derivation module 104 derives the indicator rules for the institution-dimensional reimbursement table to obtain the commission calculation rules, it is used to: Extracting payment details from the institution-dimensional payment table; Performing index calculation on the detailed payment data according to preset calculation rules to obtain calculation index; A commission calculation rule is generated according to the calculation rule and the calculation index.

[0064] In one embodiment, when the rule derivation module 105 performs an index calculation on the payment collection details data according to a preset calculation rule to obtain a calculation index, it is used to: Performing basic index calculation on the detailed payment data according to a preset calculation rule to obtain a class of basic indicators; Performing indicator combination and rule-dependent calculation on the first category of basic indicators to obtain the second category of basic indicators; Performing data expansion, indicator combination, and dependent rule calculation on the two types of basic indicators to obtain a type of derived indicators; Expanding the operation type of the first type of derivative indicators to obtain a second type of derivative indicators; A calculation indicator is generated according to the first type of basic indicator, the second type of basic indicator, the first type of derived indicator and the second type of derived indicator.

[0065] In one embodiment, when the parameter extraction module 105 extracts commission parameters from the institution-dimensional collection table according to the commission calculation rule to obtain the commission parameter table, it is used to: Extracting rule input parameters and indicator types from the commission calculation rule; Extracting a rule parameter table from the institution-dimensional payment collection table according to the rule input; Decomposing the indicator type into composite parameters to obtain indicator parameters; Extracting an indicator parameter table from the institution dimension payment collection table according to the indicator parameters; A commission parameter table is generated according to the rule parameter table and the indicator parameter table.

[0066] In one embodiment, when the financial data processing module 106 processes the commission parameter table according to the commission calculation rule to obtain the commission table, it is configured to: Selecting calculation rules from the commission calculation rules one by one as target calculation rules, using the rule script corresponding to the target rule as the target rule script, and extracting target rule input parameters from the target calculation rule; Extracting target rule parameters from the commission parameter table according to the target rule input parameter; Performing data processing on the target rule parameters according to the target rule script to obtain a payment commission; Generate a primary commission table based on all the collection commissions corresponding to all the target calculation rules; The primary commission table is clustered according to the queue names to obtain a commission table.

[0067] For the specific definition of the data processing device based on the big data warehouse, please refer to the definition of the data processing method based on the big data warehouse above, and will not be repeated here. Each module in the above-mentioned data processing device based on the big data warehouse can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0068] In one embodiment, a computer device is provided, wherein the internal structure of the computer device can be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of a data processing method based on a big data warehouse.

[0069] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed: Obtain a commission-case relationship flow table and a repayment flow table, and cluster the commission-case relationship flow table into a monthly granularity commission-case relationship flow table and the repayment flow table into a monthly granularity repayment flow table according to a preset time period; Performing repayment time matching and data association on the monthly granularity commission relationship flow table and the monthly granularity repayment flow table to obtain a full commission relationship repayment flow table; Perform time dimension clustering and institution dimension mapping on the repayment flow sheet of all entrusted case relationships to obtain an institution dimension repayment sheet; Derivation of indicator rules for the institution-dimensional reimbursement table to obtain commission calculation rules; Extracting commission parameters from the institution-dimensional payment collection table according to the commission calculation rules to obtain a commission parameter table; The commission parameter table is processed according to the commission calculation rules to obtain a commission table.

[0070] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: Obtain a commission-case relationship flow table and a repayment flow table, and cluster the commission-case relationship flow table into a monthly granularity commission-case relationship flow table and the repayment flow table into a monthly granularity repayment flow table according to a preset time period; Performing repayment time matching and data association on the monthly granularity commission relationship flow table and the monthly granularity repayment flow table to obtain a full commission relationship repayment flow table; Perform time dimension clustering and institution dimension mapping on the repayment flow sheet of all entrusted case relationships to obtain an institution dimension repayment sheet; Derivation of indicator rules for the institution-dimensional reimbursement table to obtain commission calculation rules; Extracting commission parameters from the institution-dimensional payment collection table according to the commission calculation rules to obtain a commission parameter table; The commission parameter table is processed according to the commission calculation rules to obtain a commission table.

[0071] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant description in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0072] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0073] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0074] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents. However, these modifications or replacements do not deviate from the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. It should be noted that if software tools or components other than those of the company appear in the embodiments of this application, they are only used for example and do not represent actual use.

Claims

1. A data processing method based on a big data warehouse, characterized in that: The method comprises: Obtain a commission-case relationship flow table and a repayment flow table, and cluster the commission-case relationship flow table into a monthly granularity commission-case relationship flow table and the repayment flow table into a monthly granularity repayment flow table according to a preset time period; Performing repayment time matching and data association on the monthly granularity commission relationship flow table and the monthly granularity repayment flow table to obtain a full commission relationship repayment flow table; Perform time dimension clustering and institution dimension mapping on the repayment flow sheet of all entrusted case relationships to obtain an institution dimension repayment sheet; Derivation of indicator rules for the institution-dimensional reimbursement table to obtain commission calculation rules; Extracting commission parameters from the institution-dimensional payment collection table according to the commission calculation rules to obtain a commission parameter table; The commission parameter table is processed according to the commission calculation rules to obtain a commission table.

2. The data processing method based on a big data warehouse according to claim 1, characterized in that: The monthly granularity commission relationship flow table and the monthly granularity repayment flow table are matched with the repayment time and data associated to obtain the full commission relationship repayment flow table, including: Extract the commission start time and commission end time from the monthly commission relationship flow table; Generate a case entrustment time interval according to the case entrustment start time and the case entrustment end time; Extracting the repayment time from the monthly repayment flow sheet; Match the payment time with the case commission time interval to obtain a matching payment time; Extracting matching repayment flow from the monthly repayment flow table according to the matching repayment time; The monthly granularity commission relationship flow table is data-associated according to the matching repayment flow to obtain a full commission relationship repayment flow table.

3. The data processing method based on a big data warehouse according to claim 1, characterized in that: The repayment flow sheet of the full entrusted case relationship is clustered in the time dimension and mapped in the institution dimension to obtain an institution dimension repayment sheet, including: Perform time dimension clustering on the repayment flow sheet of all entrusted case relationships to obtain a time dimension repayment sheet; Extracting queue names from the time dimension collection table, and performing queue clustering on the time dimension collection table according to the queue names to obtain a queue dimension collection table; The institution name is extracted from the queue dimension collection table, and the queue dimension collection table is clustered according to the institution name to obtain the institution dimension collection table.

4. The data processing method based on a big data warehouse according to claim 1, characterized in that: The indicator rule derivation of the institution-dimensional payment collection table to obtain the commission calculation rules includes: Extracting payment details from the institution-dimensional payment table; Performing index calculation on the detailed payment data according to preset calculation rules to obtain calculation index; A commission calculation rule is generated according to the calculation rule and the calculation index.

5. The data processing method based on a big data warehouse according to claim 4, characterized in that: The calculation of the payment details data according to the preset calculation rules to obtain the calculation indicators includes: Performing basic index calculation on the detailed payment data according to a preset calculation rule to obtain a class of basic indicators; Performing indicator combination and rule-dependent calculation on the first category of basic indicators to obtain the second category of basic indicators; Performing data expansion, indicator combination, and dependent rule calculation on the two types of basic indicators to obtain a type of derived indicators; Expanding the operation type of the first type of derivative indicators to obtain a second type of derivative indicators; A calculation indicator is generated according to the first type of basic indicator, the second type of basic indicator, the first type of derived indicator and the second type of derived indicator.

6. The data processing method based on a big data warehouse according to claim 1, characterized in that: The commission parameter table is obtained by extracting the commission parameters from the institution-dimensional repayment table according to the commission calculation rules, including: Extracting rule input parameters and indicator types from the commission calculation rule; Extracting a rule parameter table from the institution-dimensional payment collection table according to the rule input; Decomposing the indicator type into composite parameters to obtain indicator parameters; Extracting an indicator parameter table from the institution dimension payment collection table according to the indicator parameters; A commission parameter table is generated according to the rule parameter table and the indicator parameter table.

7. The data processing method based on a big data warehouse according to claim 3, characterized in that: The step of processing the commission parameter table according to the commission calculation rules to obtain a commission table includes: Selecting calculation rules from the commission calculation rules one by one as target calculation rules, using the rule script corresponding to the target rule as the target rule script, and extracting target rule input parameters from the target calculation rule; Extracting target rule parameters from the commission parameter table according to the target rule input parameter; Performing data processing on the target rule parameters according to the target rule script to obtain a payment commission; Generate a primary commission table based on all the collection commissions corresponding to all the target calculation rules; The primary commission table is clustered according to the queue names to obtain a commission table.

8. A data processing device based on a big data warehouse, characterized in that: include: A granularity clustering module is used to obtain a commission-case relationship flow table and a repayment flow table, and cluster the commission-case relationship flow table into a monthly granularity commission-case relationship flow table and the repayment flow table into a monthly granularity repayment flow table according to a preset time period; A data association module is used to match the payment collection time and perform data association on the monthly granularity commission relationship flow table and the monthly granularity repayment flow table to obtain a full commission relationship repayment flow table; A dimension mapping module is used to perform time dimension clustering and institution dimension mapping on the repayment flow sheet of the full entrusted case relationship to obtain an institution dimension repayment sheet; A rule derivation module is used to derive indicator rules for the institution-dimensional reimbursement table to obtain commission calculation rules; A parameter extraction module, configured to extract commission parameters from the institution-dimensional payment collection table according to the commission calculation rule to obtain a commission parameter table; The financial data processing module is used to process the commission parameter table according to the commission calculation rules to obtain a commission table.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the data processing method based on a big data warehouse as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the data processing method based on a big data warehouse as described in any one of claims 1 to 7 are implemented.

Images