Financial enterprise data quality evaluation method and system based on LSTM algorithm

Abstract

The invention relates to a financial enterprise data quality evaluation method and system based on an LSTM algorithm, and the method comprises a quality rule modeling and quality abnormity intelligent detection step, a quality problem grading processing step, a processing result closed-loop verification step, and a multi-dimensional dynamic evaluation report generation step. Firstly, a quality rule system is constructed according to technical attributes and business semantic features of source-attached layer data, then a data volume prediction model is constructed based on an LSTM algorithm, a dynamic threshold value is generated, and a dual anomaly detection mechanism for full-amount check of the source-attached layer data is realized in combination with the quality rule system; performing grading processing and closed-loop verification on the abnormal data; an open data quality evaluation system is constructed, definition and association mapping of quality dimensions, quantitative indexes and quality rules are completed in a configuration mode, and then a multi-dimensional dynamic evaluation report is automatically generated. The method can accurately capture data volume fluctuation abnormity, improves the intelligence and efficiency of financial data quality management, and is suitable for quality management and control of financial enterprise data in a full life cycle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of financial industry data governance, in particular to a financial enterprise data quality evaluation method and system based on an LSTM algorithm. BACKGROUND

[0002] With the deepening of the digital transformation of the financial industry, data quality has become a key factor affecting the business operation and risk management of financial institutions. The data quality evaluation method currently widely used by financial enterprises is mainly implemented based on the traditional evaluation method of rules, although it is simple to implement, but there are obvious limitations: 1) The time sequence characteristics of financial enterprise data are not considered, which leads to the inability to avoid timely and accurate evaluation of data quality due to dynamic changes in data; 2) The traditional evaluation method index is relatively fixed, and cannot be flexibly configured according to the business main of the financial enterprise, and it is difficult to adapt to the high-frequency change of the financial business scene. These problems lead to the challenges of response lag and low efficiency in data governance of financial enterprises. SUMMARY

[0003] The present application is aimed at the high value, high sensitivity, and high real-time requirements of financial industry data, but the traditional data quality evaluation method has the problems of fixed evaluation index, limited coverage dimension, and difficulty in adapting to dynamic changes, and proposes a financial enterprise data quality evaluation method based on an LSTM algorithm, which can accurately capture data volume fluctuation anomalies, efficiently activate resources through hierarchical disposal strategies, deeply fit the financial business scene, and realize intelligent governance of the whole life cycle of data quality. The present application also relates to a financial enterprise data quality evaluation system based on an LSTM algorithm.

[0004] The technical scheme of the present application is as follows:

[0005] A financial enterprise data quality evaluation method based on an LSTM algorithm, characterized by the following steps:

[0006] The quality rule modeling and quality anomaly intelligent detection step: according to the technical attributes and business semantic features of the business data of the financial enterprise data warehouse source layer, a quality rule system covering several rule types of field type compliance, value length constraint, logical consistency and data correlation is constructed; the historical data volume time series of each data table of the source layer is collected, a training sample set containing time series periodicity features, lag features and rolling statistical features is constructed, a data volume prediction model is constructed based on the LSTM algorithm, the model is trained using the training sample set, after the model training is completed, the historical data volume time series of the data table to be predicted is input, and the future time period data volume prediction value is output, the prediction value is used as the dynamic threshold of the data table data change rate index; based on the quality rule system and the dynamic threshold, the full-quantity quality inspection of the source layer data is performed, the data without abnormality is taken as the initial effective data into the data warehouse, and the quality abnormal data detected is recorded to the quality problem library;

[0007] The quality problem grading and disposal step: for the quality abnormal data in the quality problem library, a grading disposal is performed according to the corresponding relationship between the preset quality rules and disposal strategies, an automatic repair strategy matched is performed on the abnormal data meeting the automatic disposal condition, the automatic disposal condition is that the quality rule corresponding to the abnormal data has a preset repair logic that can be directly executed; for the abnormal data beyond the automatic disposal range or the automatic disposal fails, a structured problem work order containing abnormal details, associated quality rules and preliminary disposal suggestions is generated, and is distributed to the corresponding disposal node according to the business domain for manual intervention; the disposal scheme formed by the automatic disposal strategy and the manual intervention is stored in the quality problem disposal knowledge base in synchronization;

[0008] The disposal result closed loop verification step: for the abnormal data after the automatic repair strategy or manual intervention, the full-quantity quality inspection process is re-executed and the disposal effect is verified; if the verification passes, the verification state of the corresponding quality abnormal data in the quality problem library is updated to "closed loop" and is taken as the repaired effective data into the data warehouse, and the validity label of the corresponding strategy in the quality problem disposal knowledge base is updated in synchronization; if the verification fails, the verification state of the corresponding quality abnormal data in the quality problem library is set to "to be re-disposed", a structured problem work order containing the historical disposal process, failure reason and optimization suggestion is re-iteratively generated, and is distributed to the corresponding disposal node according to the business domain for manual intervention until the abnormality is solved;

[0009] The multi-dimensional dynamic evaluation report generation step: taking the DAMA standard data quality dimension as the benchmark, an open data quality evaluation system is constructed, and the definition of quality dimension, quantitative index and quality rule and the one-to-many association mapping of quality dimension and quantitative index and the many-to-many association mapping of quantitative index and quality rule are realized through configuration; the evaluation range is set to include business theme division, quality index screening, quality dimension weight assignment and index weight calculation logic, supporting full data quality evaluation and incremental data dynamic evaluation of financial enterprise whole data and sub-business theme data respectively; based on the open data quality evaluation system, the multi-dimensional quality score is calculated for the set of the initial valid data and the repaired valid data in the data warehouse, and the financial enterprise data quality multi-dimensional dynamic evaluation report containing the overall quality score, the quality dimension subdivision score, the quantitative index detail score and the business association information is automatically generated according to the preset structured report template.

[0010] Preferably, in the quality rule modeling and quality anomaly intelligent detection step, the technical attributes of the financial enterprise data warehouse source layer business data include primary key, field type, value length and format specification, the business semantic features include value domain, data correlation, logical consistency and data timeliness, the quality rule system constructed covers field type, value length, duplicate data, number, logical consistency, data correlation, null value, value domain, format specification, data timeliness, primary key and rule violation data checking, and supports custom rule extension; based on the quality rule system, full quality checking including field level, record level and table level is performed on the source layer data, and at the same time, the dynamic threshold value is monitored to detect data quantity anomaly fluctuation, forming a double checking mechanism of rule checking and threshold value monitoring; the data of rule violation anomaly or data quantity fluctuation anomaly detected is recorded as quality anomaly data to the quality problem library, and the quality anomaly data is intercepted to prevent it from entering the data warehouse.

[0011] Preferably, in the quality rule modeling and quality anomaly intelligent detection step, the data quantity prediction model constructed based on the LSTM algorithm adopts a double-layer network architecture including two LSTM units: the first layer LSTM unit has return_sequences=True to extract time sequence features, and the number of neurons is 50, and a Dropout layer with a Dropout probability of 0.2 is set after the first layer LSTM unit; the second layer LSTM unit has return_sequences=False to abstract and integrate the time sequence features, and the number of neurons is 50, and a Dropout layer with a Dropout probability of 0.2 is set after the second layer LSTM unit.

[0012] Preferably, in the quality problem grading and handling step, low-level handling is performed on abnormal data meeting the automatic handling conditions, and a matching automatic repair strategy is executed, the automatic repair strategy including a null value completion strategy, a value range correction strategy, and a format standardization strategy; the null value completion strategy fills in the field with a historical average value or a value derived based on a business rule or a business default value, the value range correction strategy adjusts data exceeding a reasonable range to a threshold boundary, and the format standardization strategy uniformly converts data into a preset format specification; high-level handling is performed on abnormal data exceeding the automatic handling range or failing automatic handling, and the generated structured problem work order further includes abnormal occurrence time, involved data table name, and data belonging to business cycle information, and the allocation by business domain is allocation to corresponding handling nodes according to business domains of retail finance, corporate finance, and financial market.

[0013] Preferably, in the multi-dimensional dynamic evaluation report generation step, the definition of the quality dimension, the quantitative index, and the quality rule is achieved in a configuration manner, specifically including:

[0014] The quality dimension is defined in a configuration manner as completeness, validity, accuracy, and timeliness.

[0015] The quantitative index under each dimension is defined in a configuration manner: the completeness dimension includes a loan application material completeness rate, which requires that the missing rate of necessary attachments of an application form, a credit report, and a flow sheet is less than 0.5%; the validity dimension includes a loan interest rate value, which requires that the interest rate value is within a specified LPR floating range; the accuracy dimension includes a core system total score balance check, which requires a pass rate of 100%; and the timeliness dimension includes a transaction flow sheet warehousing completion rate, which requires a warehousing completion rate of 100% before a preset time on T+1 day of a transaction flow sheet; the configuration-defined quantitative index further includes a data completeness rate, a field accuracy rate, a cross-table consistency matching rate, a data timely update rate, a duplicate record elimination rate, and a compliance check pass rate.

[0016] The quality rule is defined in a configuration manner: including a data correlation rule for quantifying a loan application material completeness rate and a core system total score balance check index; a format compliance rule for quantifying a field accuracy rate; a timeliness constraint rule for quantifying a data timely update rate and a transaction flow sheet warehousing completion rate; and supporting the addition of a self-defined rule.

[0017] Preferably, in the multi-dimensional dynamic evaluation report generation step, the index weight calculation logic adopts an analytic hierarchy process or an entropy weight method, the analytic hierarchy process determines the subjective weight of each index by constructing a judgment matrix, and the entropy weight method calculates the objective weight of each index based on the information entropy of index data; and the business correlation information in the multi-dimensional dynamic evaluation report of the financial enterprise data quality includes a business theme quantity, an associated database table quantity, a total number of valid data records, an abnormal data repair success rate, and a data quality optimization suggestion.

[0018] A financial enterprise data quality evaluation system based on an LSTM algorithm, characterized by sequentially connected quality rule modeling and quality anomaly intelligent detection module, quality problem grading and disposal module, disposal result closed-loop verification module and multi-dimensional dynamic evaluation report generation module,

[0019] The quality rule modeling and quality anomaly intelligent detection module, according to the technical attributes and business semantic features of the business data of the source layer of the financial enterprise data warehouse, constructs a quality rule system covering several rule types of field type compliance, value length constraint, logical consistency and data correlation; collect the historical data volume time series of each data table of the source layer, construct a training sample set containing time series periodicity features, lag features and rolling statistical features, build a data volume prediction model based on the LSTM algorithm, use the training sample set to train the model, after the model training is completed, input the historical data volume time series of the data table to be predicted, output the future time period data volume prediction value, use the prediction value as the dynamic threshold of the data table data change rate index; based on the quality rule system and dynamic threshold, perform full-quantity quality inspection on the source layer data, and the data without abnormality is regarded as the initial effective data entering the data warehouse, and the quality abnormal data is recorded to the quality problem library;

[0020] The quality problem grading and disposal module, for the quality abnormal data in the quality problem library, according to the corresponding relationship between the preset quality rules and disposal strategies, performs grading and disposal, executes the matching automatic repair strategy for the abnormal data meeting the automatic disposal condition, and the automatic disposal condition is that the quality rule corresponding to the abnormal data has a preset repair logic that can be directly executed; for the abnormal data beyond the automatic disposal range or the automatic disposal fails, generate a structured problem work order containing abnormal details, associated quality rules and preliminary disposal suggestions, and distribute it to the corresponding disposal node according to the business domain for manual intervention; the disposal scheme formed by the automatic disposal strategy and manual intervention is stored in the quality problem disposal knowledge base synchronously;

[0021] The disposal result closed-loop verification module, for the abnormal data after automatic repair strategy or manual intervention, re-executes the full-quantity quality inspection process and verifies the disposal effect; if the verification passes, update the verification state of the corresponding quality abnormal data in the quality problem library to "closed loop" and enter the data warehouse as the repaired effective data, and update the validity label of the corresponding strategy in the quality problem disposal knowledge base synchronously; if the verification fails, set the verification state of the corresponding quality abnormal data in the quality problem library to "to be re-disposed", re-iteratively generate a structured problem work order containing the history disposal process, failure reason and optimization suggestion, and distribute it to the corresponding disposal node according to the business domain for manual intervention until the abnormality is solved;

[0022] The multi-dimensional dynamic evaluation report generation module constructs an open data quality evaluation system based on the DAMA standard data quality dimensions, and realizes the definition of quality dimensions, quantitative indicators, and quality rules, and the one-to-many association mapping of quality dimensions and quantitative indicators, and the many-to-many association mapping of quantitative indicators and quality rules through a configuration mode; the evaluation range is set to include business theme division, quality indicator selection, quality dimension weight assignment, and index weight calculation logic, supporting full data quality evaluation and incremental data dynamic evaluation of financial enterprise overall data and sub-business theme data respectively; based on the open data quality evaluation system, multi-dimensional quality scores are calculated for the set of initial valid data and repaired valid data in the data warehouse, and a financial enterprise data quality multi-dimensional dynamic evaluation report containing overall quality score, each quality dimension score, each quantitative indicator score, and business-related information is automatically generated according to a preset structured report template.

[0023] Preferably, in the quality rule modeling and quality anomaly intelligent detection module, the technical attributes of the business data of the source layer of the financial enterprise data warehouse include field mandatory items, primary keys, unique keys, field types, lengths, and formats, and the business semantic features include value ranges, data correlation, and business constraints, and the quality rule system constructed covers field types, value lengths, duplicate data, numbers, logical consistency, data correlation, null values, value ranges, format specifications, data timeliness, primary keys, and rule violation data checking, and supports custom rule extension; based on the quality rule system, full-quality quality checking including field-level, record-level, and table-level checking is performed on the source layer data, and at the same time, the dynamic threshold monitoring data volume anomaly fluctuation is combined to form a dual-checking mechanism of rule checking and threshold monitoring; the data of rule violation anomalies or data volume fluctuation anomalies checked out is recorded as quality anomaly data to the quality problem library, and the quality anomaly data is intercepted to prevent it from entering the data warehouse.

[0024] Preferably, in the quality rule modeling and quality anomaly intelligent detection module, the data volume prediction model constructed based on the LSTM algorithm adopts a double-layer network architecture including two LSTM units: the first layer LSTM unit has return_sequences=True to extract time sequence features, and the number of neurons is 50, and a Dropout layer with a Dropout probability of 0.2 is set after the first layer LSTM unit; the second layer LSTM unit has return_sequences=False to abstract and integrate the time sequence features, and the number of neurons is 50, and a Dropout layer with a Dropout probability of 0.2 is set after the second layer LSTM unit.

[0025] Preferably, in the quality problem grading and handling module, the abnormal data meeting the automatic handling condition is handled at a low level, and a matched automatic repair strategy is executed, the automatic repair strategy including a null value completion strategy, a value range correction strategy and a format standardization strategy; the null value completion strategy fills in a historical mean value of the field or a deduced value based on a business rule or a business default value, the value range correction strategy adjusts data exceeding a reasonable range to a threshold boundary, and the format standardization strategy uniformly converts data into a preset format specification; the abnormal data exceeding the automatic handling range or failing in automatic handling is handled at a high level, and the generated structured problem work order further includes abnormal occurrence time, involved data table name and data belonging business cycle information, and the distribution according to a business domain specifically includes distribution to corresponding handling nodes according to business fields of retail finance, corporate finance and financial market.

[0026] The technical effects of the present application are as follows:

[0027] The application relates to a financial enterprise data quality evaluation method based on an LSTM algorithm, a quality rule modeling and quality anomaly intelligent detection step, a combination of technical attributes and semantic features of financial business data, construction of a precise quality checking framework, a multi-dimensional quality rule system, ensuring comprehensive coverage of core quality problems such as field format, value range and logical association, using the learning ability of the LSTM algorithm on the time sequence of historical data volume to generate dynamic thresholds that conform to the data change rule, breaking through the limitations of traditional fixed thresholds, accurately identifying abnormal conditions such as data volume mutation, realizing dynamic data volume anomaly early warning, improving the timeliness and accuracy of anomaly detection, performing full-quantity quality checking on the source layer data based on the quality rule system and dynamic threshold, forming a double-checking mechanism of rule checking and threshold monitoring, realizing accurate separation of effective data and abnormal data through full-quantity quality checking, ensuring that compliant data is timely warehoused to support business, and at the same time, abnormal data is recorded in the quality problem library, completing data shunting and traceability foundation construction, retaining complete data traces for subsequent disposal and tracing, and guaranteeing the auditability of data governance; the quality problem grading disposal step, based on the corresponding relationship between quality rules and disposal strategies, implements grading disposal, automatically completes repair for simple repairable anomalies, avoids redundant cost of manual intervention; accurately allocates complex anomalies to corresponding business nodes, ensures that professional personnel focus on core problems, realizes optimized allocation of disposal resources, improves anomaly disposal efficiency and resource utilization, integrates anomaly details, associated rules and preliminary suggestions through a structured problem work order, provides clear guidance for manual disposal, reduces information searching and judgment cost, at the same time, standardizes disposal processes, reduces the risk of secondary problems caused by human operation errors, synchronously stores automatic repair strategies and manual disposal schemes into a knowledge base, constructs a standardized disposal template library, forms reusable disposal knowledge precipitation, provides efficient reference for subsequent similar anomalies, promotes continuous iterative upgrading of data quality disposal capability; the disposal result closed-loop verification step, through repeated execution of the full-quantity quality checking process, strictly checks the repaired data, ensures that the abnormal problems are completely solved, guarantees the compliance and reliability of the repaired data, avoids the inflow of unqualified data into the data warehouse to affect business decision-making, builds a data quality defense line, clearly controls the abnormal disposal progress through a state updating mechanism (closed loop / ready for re-disposal), starts the iterative disposal process for the abnormal problems that do not pass the verification until the problems are completely solved, eliminates the situation that abnormal data is pending, forms a complete closed-loop management of 'disposal-verification-re-disposal', synchronously updates the effectiveness label of the disposal strategy in the knowledge base, optimizes the effectiveness of the disposal strategy, provides data support for subsequent strategy optimization, gradually selects efficient and feasible disposal schemes, continuously improves the accuracy and efficiency of the overall abnormal disposal.The multi-dimensional dynamic evaluation report generation step constructs a flexible and adaptive open evaluation system based on the DAMA standard, realizes flexible definition and association of dimensions, indexes and rules through configuration mode, can adjust evaluation dimensions and weights according to different business scenarios and regulatory requirements of financial enterprises, meets the personalized quality evaluation demand, supports multi-dimensional evaluation of enterprise as a whole, sub-business theme and full / incremental data, fully presents the performance of data quality in different dimensions and different business scenarios, breaks the limitation of single evaluation perspective, provides comprehensive reference for data quality optimization, and clearly presents the overall score, dimension score, index details and business association information in the structured report, converts the abstract data quality concept into intuitive quantitative results, facilitates the management to master the overall condition of data quality, provides data support for data governance decision, and promotes continuous improvement of data quality. The application constructs a whole-process intelligent data quality governance system: by integrating four core links of "detection-disposal-verification-evaluation", the whole life cycle automation and standardized management from data access to quality evaluation are realized, the dependence on manual intervention is greatly reduced, and the pain points of traditional financial data governance process, such as dispersion, low efficiency and inaccurate quality control, are solved. The financial data quality and business support capability are significantly improved: through multi-dimensional rule checking, dynamic threshold early warning, closed-loop repair and other technical means, the accuracy, integrity, consistency and compliance of data are fully improved, the high quality of data in the data warehouse is ensured, the reliable data support for core businesses such as financial risk control, customer management and operation decision is provided, and the business risk caused by data quality problems is reduced. The digitalization and intelligentization upgrade of financial data governance is promoted: with the help of LSTM algorithm, knowledge base precipitation and configuration evaluation technology innovation, the transformation from passive response to active early warning and from experience-driven to data-driven is realized, and through knowledge precipitation and iterative optimization, the data governance ability of financial enterprises is continuously improved, and a solid data foundation is laid for digital transformation.

[0028] The application can effectively learn the time sequence characteristics and change rules of financial data by introducing the LSTM algorithm. By predicting the change amount of each type of business data according to the business theme of the financial enterprise data, the timeliness and accuracy of the financial enterprise data quality evaluation are improved. The long short-term memory characteristics of LSTM are particularly suitable for processing long-term dependencies in financial data, and its gating mechanism can adaptively capture the change rules of data quality. This method breaks through the limitations of traditional technology and provides a more intelligent solution for financial data quality governance. Based on the in-depth analysis of the existing technology, the application mainly proposes innovative solutions to the following key technical problems in the field of financial enterprise data quality evaluation: 1) insufficient handling of time sequence dynamics: existing rule-based or statistical methods cannot effectively capture the time sequence dynamic characteristics of financial data, especially for quality anomaly patterns with long-term dependencies; the memory cell design of the LSTM network is used to solve the time sequence dynamic modeling problem in financial data quality change. 2) Lack of abnormal prediction ability: traditional technology can only react to the quality problems that have occurred, and lacks the ability to predict potential quality risks; the application uses the time series prediction characteristics of LSTM to predict the daily change amount of business data, effectively evaluates the change rate of data by business theme, and then realizes the early warning of abnormal data change, realizing the change from "post-detection" to "pre-warning". 3) Lack of quality governance closed loop: the data quality evaluation results of the existing technology are disconnected with the subsequent processing flow, and lack of automatic closed loop feedback mechanism; the application starts from the data quality processing flow, and realizes the closed loop management control of data quality problems by connecting the four links of quality problem discovery, processing, verification and evaluation. 4) Static solidification of evaluation index: traditional data quality evaluation methods use fixed quality dimensions, quality indicators and indicator weights, which cannot adapt to the dynamic change of financial data business scenarios; the application proposes an open data quality evaluation system, which realizes the openness of the evaluation system by flexibly configuring quality dimensions, quality indicators and quality rules, and further realizes the quality evaluation of full data and incremental data based on the data volume change of each table, solving the problem of rigid evaluation index and poor business scenario adaptability of traditional methods, making the quality evaluation more scientific and the results more comparable. The solution of these technical problems will make the financial enterprise data quality management work realize the change from "post-detection" to "pre-warning", the quality evaluation system will be more scientific and effective, and the closed loop management control of data quality problems will be realized, which will significantly improve the quality governance efficiency and risk prevention and control ability.

[0029] Further, by explicitly defining the technical attributes (such as field type, primary key, etc.) and business semantic features (such as domain, data correlation, etc.) of financial enterprise data, a quality rule system covering specific multiple dimensions is constructed and supports custom extension, combining full-quality checks at the field, record, and table levels with a dual mechanism of dynamic threshold monitoring, to achieve comprehensive and accurate identification of data quality abnormalities in the source layer, while preventing abnormal data from entering the warehouse, ensuring the initial data quality of the data warehouse from the source, and solving the problem of insufficient coverage and missed detection of abnormalities in traditional single rule verification.

[0030] Further, a two-layer network architecture including two LSTM units is adopted, the first layer LSTM unit extracts time sequence features and retains sequence output, and the second layer LSTM unit abstracts and integrates the time sequence features, with a Dropout layer to prevent overfitting, significantly improving the ability of the data volume prediction model to capture the periodicity and volatility of financial business data, making the generated dynamic threshold more consistent with the actual data variation, providing a high-precision benchmark for data volume anomaly fluctuation monitoring, and overcoming the defects of poor adaptability and high false alarm rate of traditional fixed thresholds.

[0031] Further, through a hierarchical disposition mechanism of low-level automatic disposition (such as null value completion, value range correction, etc.) and high-level manual intervention, differentiated and efficient processing of quality abnormal data is achieved, with automatic repair strategies quickly solving standardization problems and reducing manual costs; structured work orders and accurate allocation by business domain ensure that complex abnormalities are handled professionally, while rich work order information improves the efficiency of manual intervention, forming a closed-loop disposition mode of automation and manual cooperation, and solving the problems of chaotic traditional disposition process, resource waste, and slow response.

[0032] Further, by defining quality dimensions (such as integrity, validity, etc.), quantitative indicators (such as loan application material completeness rate, interest rate compliance, etc.), and associated quality rules (data correlation, format compliance, etc.) in a configurable manner and supporting custom rule extension, an open evaluation system that flexibly adapts to financial business scenarios is constructed, achieving deep binding of quality evaluation and business logic and meeting individual evaluation needs under different business themes and regulatory requirements, and solving the limitations of fixed dimensions and disconnection from business in traditional evaluation systems.

[0033] Further, the index weight calculation logic combining the analytic hierarchy process (subjective weight) and the entropy weight method (objective weight) allows quality evaluation to consider both business experience and data objective laws, improving the scientificity of the evaluation results; meanwhile, business-related information (such as repair success rate, optimization suggestions, etc.) is supplemented in the multi-dimensional dynamic evaluation report of financial enterprise data quality, converting abstract quality scores into actionable business improvement criteria, providing precise support for financial enterprise data governance decisions, and solving the problem of one-sidedness and limited guidance value of traditional evaluation results.

[0034] The application also relates to a financial enterprise data quality evaluation system based on an LSTM algorithm, which corresponds to the financial enterprise data quality evaluation method based on the LSTM algorithm and can be understood as a system for implementing the financial enterprise data quality evaluation method based on the LSTM algorithm, and comprises sequentially connected quality rule modeling and quality anomaly intelligent detection modules, a quality problem grading and disposal module, a disposal result closed-loop verification module and a multi-dimensional dynamic evaluation report generation module. The modules work cooperatively, and the core advantages mainly lie in four aspects: 1) the time sequence dynamic modeling capacity is significantly improved, the long-term dependence relationship and dynamic characteristics of the financial data quality change are effectively captured by means of the unique memory unit and gate mechanism of the LSTM network, and the problem of insufficient processing of the time sequence characteristics in the prior art is solved; 2) the quality risk prediction capacity is broken through, and the potential data quality problems can be predicted in advance by means of the time sequence prediction capacity of the LSTM, so that the quality problem solution window period is doubled, the financial business risk is significantly reduced, and the quality problem solving efficiency is improved by 98%; 3) the quality management closed loop is completely realized, and a whole-process closed loop system from quality problem discovery, treatment, verification to evaluation is constructed, the average time from the discovery to the solution of the quality problem is shortened from 24 hours to 8 hours, and 90% of the simple quality problems can be automatically repaired; and 4) the business adaptability is enhanced, the decoupling of the quality evaluation index and the specific quality rule is realized by means of the configuration mode on the basis of the DAMA standard data quality dimension, the data warehouse data can be evaluated as a whole and by business, the whole amount and the change amount index evaluation can be supported according to the data table data change amount, the multi-dimension and dynamic of the data quality evaluation are realized, the adaptation cost of the system cross-business deployment is reduced by 80%, and the universality and scalability of the scheme are greatly improved. These beneficial effects make the application have significant technical progress and practical value in the field of financial data quality management, and can effectively solve the quality management pain points faced by the current industry. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 The flowchart of the financial enterprise data quality evaluation method based on the LSTM algorithm.

[0036] Figure 2 The multi-dimensional dynamic evaluation report display diagram of the data quality of a certain financial enterprise. DETAILED DESCRIPTION

[0037] The application will be described below with reference to the drawings.

[0038] The application provides a financial enterprise data quality evaluation method based on an LSTM algorithm. The LSTM algorithm is used to predict the variation trend of the actual business data volume of a financial enterprise, and the predicted value is used as a threshold to evaluate the reasonable variation rate of the data volume of each business table. Meanwhile, the increment and total data volume of each table are recorded, and on this basis, the overall and business-specific data quality of the data warehouse is evaluated. In addition, the data variation of each data table is used to support the evaluation of the total and variation indicators, which realizes the multi-dimensionality and dynamics of data quality evaluation, improves the timeliness and accuracy of the quality evaluation results, makes the quality evaluation more scientific, and makes the results more comparable. The flowchart of the method is shown in Figure 1 The method comprises the following steps:

[0039] Step I: quality rule modeling and quality anomaly intelligent detection. According to the technical attributes and business semantic features of the data in the source layer of the data warehouse of a financial enterprise, a quality rule system covering several rule types such as field type compliance, value length constraint, logical consistency and data correlation is constructed. The historical data volume time series of each data table in the source layer is collected, a training sample set containing time series periodicity features, lag features and rolling statistical features is constructed, a data volume prediction model is constructed based on the LSTM algorithm, the training sample set is used to train the model, after the model training is completed, the historical data volume time series of the data table to be predicted is input, the future data volume prediction value is output, and the prediction value is used as the dynamic threshold of the data table data variation rate indicator. The total quality of the data in the source layer is checked based on the quality rule system and the dynamic threshold. The data that is not detected as abnormal is used as the initial valid data to enter the data warehouse, and the quality abnormal data detected is recorded in the quality problem library.

[0040] This step can be understood as a quality problem discovery, based on each database table preset quality rules, predict data change, execute rule discovery quality problem. Discovery data quality problem record to quality problem library, the data with problems is not allowed to enter the data warehouse, hand over to subsequent quality problem classification disposal step for processing. Quality problem discovery depends on the preset quality rules, quality rules are configured based on the financial enterprise data warehouse model, the technical attributes of the financial enterprise data warehouse source layer business data include field mandatory item, primary key, unique key, field type, length and format, etc., the business semantic features include value range, data correlation, etc., the complex business features can adopt custom rules, such as total score balance check. According to the technical attributes and business semantic features of the financial enterprise data warehouse source layer business data, 14 kinds of rule types can be identified, including: field type, value length, duplicate data, number, logical consistency, data correlation, null value, value range, threshold, format specification, data timeliness, primary key, rule violation data check, and custom. Among them, the quality rules of field type, value length, null value, value range, threshold, and primary key are generally set according to the technical characteristics of the data, and can be preset after the data warehouse model is determined. The quality rules of duplicate data, number, logical consistency, data correlation, format specification, data timeliness, and rule violation data check are generally set according to the business characteristics of the data, so as to ensure that the quality rules can cover most scenarios, and provide custom rules for personalized scenarios, which can be implemented by writing code.

[0041] The daily data volume change of each data table of the data warehouse source layer is recorded, and on this basis, the LSTM algorithm model is used to predict the daily data volume change of each table. The predicted value can be used as the threshold of the data volume change rate index of each table. For example, by predicting the daily data volume change of each table in the past 90 days, the daily increment of the next 10 days is obtained, which is used as the data basis to support the full amount and increment data quality evaluation of the business theme. At the same time, the predicted value can be used as the threshold of the data timeliness index of a certain business theme. If it is lower than the threshold, it is reasonable, and if it is higher than the threshold, a warning should be made in time.

[0042] A data volume prediction model is constructed based on the LSTM algorithm, and the model design and implementation includes four steps: algorithm scene design, data preparation and preprocessing, model construction and training, prediction and effect evaluation.

[0043] 1) Algorithm scene design

[0044] Core target: predict the data volume (or incremental data volume) of each core table in the data warehouse in the next 10 days, and compare it with the actual value, so as to judge the integrity and correctness of daily data extraction (business system -> data warehouse ODS);

[0045] Prediction target: Total_Count, Daily_Count of each table;

[0046] Input: Time series of daily data volume of each table in the past 12 months (about 365 time points);

[0047] Output: Prediction value of daily data volume in the next 10 days;

[0048] 2) Data preparation and preprocessing

[0049] 2.1 Data collection: Extract historical data volume information of each table in the past 12 months from the data warehouse source layer to form a basic data set, such as the LSTM algorithm model basic data set shown in Table 1:

[0050] Table 1

[0051]

[0052] In Table 1, Table_Name represents the name of the data table, which is used to identify the specific table to which the data belongs; EtlDate represents the data extraction date, recording the time node of data acquisition; Total_Count represents the total number of records of the table, reflecting the cumulative data volume of the table on the corresponding date; Daily_Count represents the number of new records on the day, reflecting the daily incremental data volume of the table on the corresponding date; Is_WorkDay represents whether it is a working day, with "0" representing a non-working day and "1" representing a working day, used to distinguish the time attribute of data volume change (such as the difference in data volume fluctuation between working days and non-working days).

[0053] 2.2 Feature engineering: As a key to improve prediction accuracy, features that can reflect the change rule of data volume need to be provided for the model, and the features used in this application can include time series, lag features, and rolling statistics features, as follows:

[0054] Time series features:

[0055] Day_Of_Week (0-6): Day of the week, capturing weekly periodicity (e.g. low data volume on weekends).

[0056] Day_Of_Month (1-31): Day of the month, capturing monthly periodicity (e.g. report generation at the end of the month, data volume surge).

[0057] Month (1-12): Month, capturing annual periodicity.

[0058] Is_WorkDay (0 / 1): Whether it is a working day, business activities are more frequent on working days.

[0059] Is_Month_End (0 / 1): Whether it is the end of the month, very important.

[0060] Is_Holiday (0 / 1): Whether it is a statutory holiday.

[0061] Lag features:

[0062] Data volume in the past 7 days (Lag_1, Lag_2,..., Lag_7).

[0063] Data volume in the same period last week (Lag_7).

[0064] Data volume in the same period last month (about Lag_30).

[0065] Rolling statistics features:

[0066] Average data volume in the past 7 days (Rolling_Mean_7).

[0067] Standard deviation of data volume in the past 7 days (Rolling_Std_7).

[0068] 2.3 Data preprocessing, including handling missing values and normalization. Handle missing values: holidays may have no data, use forward fill (.fillna(method='ffill')); normalization: standardize or normalize log-valued features (such as daily_count, rolling_mean_7) to accelerate LSTM convergence.

[0069] For example, in Python, the example code is as follows:

[0070]

[0071] Build a supervised learning dataset: convert the time series into the (samples, timesteps, features) format required by LSTM. For example, in Python, the example code is as follows:

[0072]

[0073] 3) Model building and training

[0074] 3.1 Model architecture design: The application adopts a two-layer LSTM design, which can better capture complex nonlinear time series dependencies in the data (such as the superposition of end-of-month effects and weekly effects). Although the data volume pattern in the data warehouse is regular, it may also have noise, and Dropout effectively prevents model overfitting. Specifically, the data volume prediction model constructed based on the LSTM algorithm adopts a double-layer network architecture including two LSTM units: the first layer LSTM unit has return_sequences=True to extract time series features, and the number of neurons is 50, and a Dropout layer with a Dropout probability of 0.2 is set after the first layer LSTM unit; the second layer LSTM unit has return_sequences=False to abstract and integrate the time series features, and the number of neurons is 50, and a Dropout layer with a Dropout probability of 0.2 is set after the second layer LSTM unit. Taking Python as an example, the example code is as follows:

[0075]

[0076] 3.2 Model training: The sample data is divided into training set and test set according to the ratio of 8:2, and must be divided in time sequence.

[0077] Taking Python as an example, the example code is as follows:

[0078]

[0079] 4) Prediction and effect evaluation

[0080] 4.1 Prediction: Take the data of the last time_steps (30) days as the initial input window, predict the data of the T+1 day, take the predicted value of T+1 day as a new feature, update the input window (sliding window), and predict T+2 day; repeat the above steps until the future 10 days are predicted.

[0081] Taking Python as an example, the example code is as follows:

[0082]

[0083] 4.2 Effect evaluation: The evaluation index is as follows

[0084] MAE (Mean Absolute Error): (1 / n) *Σ|Y_true - Y_pred|. It intuitively reflects the average error size.

[0085] MAPE (Mean Absolute Percentage Error): (1 / n) *∑|(Y_true - Y_pred) / y_true| Very suitable for volume prediction, can reflect the relative proportion of error. MAPE < 10% is usually considered as an excellent model.

[0086] RMSE (Root Mean Square Error): More sensitive to large errors, and the penalty is greater.

[0087] Business evaluation: Set a dynamic threshold (for example, based on 2 times the standard deviation of historical MAPE). When the predicted value of a day deviates from the actual value by more than this threshold, a data quality alarm is triggered.

[0088] Further, the present application records the data volume, daily increment and daily predicted increment of each data table of the data warehouse source layer, and records the business theme to which each table belongs, as shown in Table 2. Based on this, the data volume change rate index can be set for each type of business theme, and the timeliness of data update can be evaluated by this index.

[0089] Table 2

[0090]

[0091] The present application is based on the preset quality rules of each data table of the data warehouse source layer, and the quality rules must be set to the data table. Based on this, the quality check according to the quality rules is to run the preset quality rules of each data table, and the check result is recorded in the quality problem library. The structure of the quality problem library is shown in Table 3, wherein, ID: is an identification column, used to uniquely mark each quality problem record; BusinessSubject: represents a business theme, clearly indicating the business field to which the problem belongs; DataTable: is a data table column, indicating the specific data table involved in the problem; DataKey: is a data primary key column, locating the primary key information of the problem data; CheckResult: is a check result column, "1" indicating that the check is successful, and "0" indicating that the check fails; CheckInfor: is a check information column, recording the detailed check; CheckTime: is a check time column, marking the time when the check occurs.

[0092] Table 3

[0093]

[0094] That is, data quality inspection is performed on the data of the source layer, and a data inspection scheme needs to be defined before the quality inspection. The inspection scheme is composed of quality rules of the source layer. After the inspection according to the inspection scheme, the quality problems found are automatically recorded in a quality problem library. Further, based on the quality rule system, full-quality inspection including field level, record level, and table level is performed on the data of the source layer, and the dynamic threshold is monitored to detect abnormal fluctuations in the data volume, forming a dual inspection mechanism of rule checking and threshold monitoring. The data with rule violation exceptions or data volume fluctuation exceptions found in the inspection is recorded as quality abnormal data in the quality problem library, and the quality abnormal data is intercepted to prevent it from entering the data warehouse.

[0095] II. Quality problem grading and handling step: For the quality abnormal data in the quality problem library, grading and handling are performed according to the correspondence between the preset quality rules and handling strategies. For abnormal data that meets the automatic handling conditions, a matching automatic repair strategy is executed. The automatic handling conditions are that the quality rules corresponding to the abnormal data have preset repair logic that can be directly executed. For abnormal data that exceeds the automatic handling range or fails in automatic handling, a structured problem ticket containing abnormal details, associated quality rules, and preliminary handling suggestions is generated and distributed to the corresponding handling node according to the business domain for manual intervention. The handling scheme formed by the automatic handling strategy and manual intervention is stored in the quality problem handling knowledge base.

[0096] This step can be understood as quality problem grading and handling. The discovered quality problems are automatically handled based on rules, and the problems that cannot be automatically handled are processed manually. The problem handling method is recorded in the problem handling knowledge base to provide support for subsequent automatic handling. For each type of quality rule, a problem handling method is preset and configured when the quality rule is configured. The preset problem handling method corresponding to each type of quality rule is shown in Table 4.

[0097] Table 4

[0098]

[0099] The quality problems are automatically handled according to the problem handling rules. For example, the enterprise social credit code cannot be missing for customer information. If it is missing, it is automatically updated to "9999". If there is no problem handling rule configured, the problem is manually handled by automatically forming a structured problem ticket. Further, the problem handling process data such as the cause of the problem, whether it can be handled, the handling scheme, and the impact analysis are recorded in the problem handling knowledge base, and the structure is shown in Table 5.

[0100] Table 5

[0101]

[0102] In Table 5, ID: is an identification column for uniquely marking each processing record; ProblemID: is a problem identification column associated with the corresponding problem in the quality problem library; RootCause: is a root cause analysis column recording the root cause of the problem; IsCanDo: represents a processing column indicating whether the problem is feasible for manual processing; Solution: is a processing scheme column recording specific measures for manual processing; ImpactAnalysis: is an impact analysis column evaluating the impact of problem processing on business; AutoProcMethod: is an automatic processing method column indicating the corresponding method if the problem supports automatic processing; IsValidate: represents a validity column indicating the validity of the processing scheme.

[0103] Further, the quality problem grading and handling step performs low-level handling of abnormal data meeting automatic handling conditions, executes a matching automatic repair strategy, which includes a null value completion strategy, a value range correction strategy, and a format standardization strategy; the null value completion strategy fills in the historical mean value of the field or a derived value based on business rules or a business default value, the value range correction strategy adjusts data outside the reasonable range to the threshold boundary, and the format standardization strategy uniformly converts data to a preset format specification; abnormal data that exceeds the automatic handling range or fails automatic handling is handled at a high level, and the generated structured problem work order further includes abnormal occurrence time, involved data table name, and data belonging to business cycle information; the allocation by business domain is allocated to the corresponding handling node according to the business fields of retail finance, corporate finance, and financial markets.

[0104] Three, the handling result closed loop verification step: for abnormal data after automatic repair strategy or manual intervention, re-execute the full quality inspection process and verify the handling effect; if the verification passes, update the verification state of the corresponding quality abnormal data in the quality problem library to "closed loop" and enter the data warehouse as the repaired valid data, and update the validity label of the corresponding strategy in the quality problem handling knowledge base; if the verification fails, set the verification state of the corresponding quality abnormal data in the quality problem library to "to be re-handled", and re-iterate to generate a structured problem work order containing historical handling process, failure reason and optimization suggestion, and allocate it to the corresponding handling node for manual intervention until the problem is solved.

[0105] This step can be understood as quality problem verification, verifying the quality problem after processing, and entering the warehouse if the verification passes, and returning to processing if it fails. That is, re-verify the data whose quality problem library verification result is failed, set the verification result to success if the verification passes, set it to failure if the verification does not pass, update the problem handling knowledge base, update the validity state, form a problem handling work order, and transfer it to the worker for continuous tracking and handling until the problem is solved.

[0106] IV. Multi-dimensional dynamic evaluation report generation step: based on the DAMA standard data quality dimension, an open data quality evaluation system is constructed, and the definition of quality dimension, quantitative index, quality rule and the one-to-many association mapping of quality dimension and quantitative index, the many-to-many association mapping of quantitative index and quality rule are realized through configuration; the evaluation range is set, including business theme division, quality index screening, quality dimension weight assignment and index weight calculation logic, supporting full data quality evaluation and incremental data dynamic evaluation of financial enterprise whole data and sub-business theme data respectively; based on the open data quality evaluation system, multi-dimensional quality score is calculated for the set of initial valid data and repaired valid data in the data warehouse, and a financial enterprise data quality multi-dimensional dynamic evaluation report containing overall quality score, each quality dimension subdivision score, each quantitative index detail score and business related information is automatically generated according to the preset structured report template.

[0107] This step can be understood as a quality evaluation report, which performs multi-dimensional and dynamic data quality evaluation on data quality according to business theme, full amount / increment. It includes main functions such as quality dimension definition, quality index definition, evaluation system setting, index weight setting and evaluation report generation; according to the characteristics of each type of business of a financial enterprise, the quality dimension and quality index can be selected from the DAMA index evaluation system according to the business theme, and then the quality index calculation formula is defined based on the quality rule to form the quality evaluation system. The present application is based on the definition of data quality evaluation by DAMA and the DAMA standard data quality dimension, and proposes an open data quality evaluation system. Specifically, the openness of the evaluation system is realized by flexible configuration of quality dimension, quality index and quality rule, the quality dimension definition function realizes the self-definition of integrity, uniqueness, validity, accuracy, consistency and timeliness, and can be extended to support normative and accessibility dimensions. Quality index is the basis for quantitative quality evaluation of quality dimension. A quality dimension generally corresponds to multiple quality indexes. Quality index must be quantifiable, and has a calculation formula, such as the index calculation formula of the accuracy index (core system total balance check index): check passing subject number / total account subject number*100%. Quality index is based on quality rule, and the total balance check logic is defined by quality rule. The system records the quality check result after executing the check on the quality rule, and then calculates the index score based on the quality check result as the quantitative basis. Finally, the data quality score is calculated according to the evaluation system.

[0108] The application realizes flexible configuration of quality indexes by setting the two-by-two correlation relationship among quality dimensions, quality indexes and quality rules, specifically, the quality dimensions can be defined according to the characteristics of financial enterprises, the quality dimensions and the quality indexes are in a one-to-many relationship, the quality dimensions and the quality indexes designed according to the characteristics of financial enterprises, such as the loan application material completeness rate index representing the integrity dimension, including the missing rate of necessary attachments such as application form, credit report and flow, etc., <0.5%; the loan interest rate value index representing the validity dimension, must be within the LPR floating range specified by the central bank; the core system total score balance check index representing the accuracy dimension, must be 100%; the transaction flow warehousing completion rate index representing the timeliness dimension, the transaction flow T+1 day before 2:00 in the morning is warehoused and completed at 100% and the like; the configured and defined quantitative indexes can also include data completeness rate, field accuracy rate, cross-table consistency matching rate, data timely updating rate, duplicate record elimination rate and compliance check pass rate and the like; the configured and defined quality rules include data correlation rules, format compliance rules, timeliness constraint rules and the like, wherein, the data correlation rules are used to quantify the loan application material completeness rate and the core system total score balance check index; the format compliance rules are used to quantify the field accuracy rate; the timeliness constraint rules are used to quantify the data timely updating rate and the transaction flow warehousing completion rate; and the newly added self-defined rules are supported. The quality indexes and the quality rules are in a many-to-many relationship, suitable quality rules are selected according to the index meaning and the optimization requirements to perform quality quantification, such as the loan application material completeness rate index representing the integrity is defined by using the type of data correlation quality rules, the core system total score balance check index representing the accuracy can also be defined by using the type of data correlation quality rules, and the configuration relationship between the two is shown in Table 6.

[0109] Table 6

[0110]

[0111] The application sets the data quality evaluation range by the open data quality evaluation system, the evaluation range can set the business theme, the quality index, the quality dimension weight, the index weight calculation rule setting, wherein the business theme, the quality index setting and the quality dimension weight setting belong to the basic setting, the index weight calculation logic can adopt the analytic hierarchy process or the entropy weight method, the analytic hierarchy process determines the subjective weight of each index by constructing a judgment matrix, the entropy weight method calculates the objective weight of each index based on the information entropy of the index data; the overall data quality and the data quality of the sub-business theme of the financial enterprise can be flexibly realized according to the open data quality evaluation system, further, based on the data volume change of each table of the data warehouse, the quality evaluation of the full amount data and the incremental data is realized, the full amount data reflects the overall evaluation of the quality status of all data of the financial enterprise, the incremental data reflects the dynamic evaluation of the real-time data quality of the financial enterprise, and the dynamic evaluation of the data quality of the financial enterprise is realized.

[0112] The present application realizes the automatic generation of the financial enterprise data quality evaluation report by presetting the report template, and the specific process is as follows: based on the open data quality evaluation system, the score of each dimension is calculated according to the configured quality indicators and weights, and then according to the preset structured report template, the key information such as the quality rule library, the quality problem library, the quality indicator score, the quality dimension weight, the overall quality score and the like is automatically extracted, a multi-dimensional dynamic evaluation report of the financial enterprise data quality is generated, which contains the overall quality score, the score of each quality dimension, the score of each indicator, the business theme range (involving the business theme and the specific database table quantity, the total number of records, the incremental record number and the like), the business correlation information (the abnormal data repair success rate, the data quality optimization suggestion and the like), and is displayed in the form of intuitive charts. For example, the data quality evaluation report of a certain financial enterprise treasury supervision field as shown in Figure 2 is generated, and the multi-dimensional achievements of the data quality governance are completely presented.

[0113] Specifically, the calculation logic of the quality score calculated by the present application can be as follows: according to the open data quality evaluation system, the quality indicators are calculated according to the theme, and the detailed score of each indicator is obtained, then according to the corresponding relationship between the quality dimension and the quality indicator, the total score of the quality indicator corresponding to each quality dimension is calculated as the quality dimension score of the business theme, then according to the quality dimension score of each theme and the quality dimension weight, the quality score of the business theme is formed, and finally, the final overall quality score, i.e. the financial enterprise data quality score, is calculated by weighting according to the business theme; further, according to the open data quality evaluation system setting, only the incremental data is evaluated for quality, and the incremental data quality score of the financial enterprise as a whole and each type of business theme is obtained.

[0114] The application is based on a financial enterprise data quality evaluation method based on an LSTM algorithm, which starts from a data quality processing flow, in a quality problem discovery stage, defines quality rules based on data warehouse source layer data table, performs quality inspection according to the quality rules, discovers quality problem records in a quality problem library, records the data volume change of each data table of the data warehouse source layer every day, uses the LSTM algorithm for prediction, uses the predicted value as the threshold to monitor the data change rate of each data table, in the quality problem classification processing stage, according to the problem processing rule, first uses automatic processing, and then carries out manual checking processing, the problem processing method records the problem processing knowledge base, in the quality problem verification stage, the processed data is verified, the verified record is recorded to the problem processing knowledge base, and the unverified is converted into manual processing, in the quality evaluation report stage, based on the DAMA standard data quality dimension, the quality evaluation index and the specific quality rule are decoupled through the configuration mode, the quality evaluation system can be flexibly defined, the overall and sub-business data quality of the data warehouse is evaluated based on this, the full amount and change amount index evaluation is supported according to the data volume of each data table, the multidimensionality and dynamic of data quality evaluation are realized, the timeliness and accuracy of the quality evaluation result are improved, and the quality evaluation is more scientific and the result is more comparable.

[0115] The application also relates to a financial enterprise data quality evaluation system based on an LSTM algorithm, which corresponds to the financial enterprise data quality evaluation method based on the LSTM algorithm, and can be understood as a system for realizing the financial enterprise data quality evaluation method based on the LSTM algorithm, and comprises a quality rule modeling and quality anomaly intelligent detection module, a quality problem classification processing module, a disposal result closed-loop verification module and a multi-dimensional dynamic evaluation report generation module connected in sequence, wherein,

[0116] The quality rule modeling and quality anomaly intelligent detection module constructs a quality rule system covering several rule types of field type compliance, value length constraint, logical consistency and data correlation according to the technical attributes and business semantic features of the financial enterprise data warehouse source layer business data; collects the historical data volume time series of each data table of the source layer, constructs a training sample set containing time series periodicity features, lag features and rolling statistical features, constructs a data volume prediction model based on the LSTM algorithm, trains the model using the training sample set, inputs the historical data volume time series of the data table to be predicted after the model training is completed, outputs the future time period data volume prediction value, and uses the prediction value as the dynamic threshold of the data table data change rate index; performs full amount quality inspection on the source layer data based on the quality rule system and the dynamic threshold, and the data without abnormality is used as initial effective data to enter the data warehouse, and the quality abnormal data is recorded to the quality problem library;

[0117] The quality problem hierarchical treatment module, for quality abnormal data in the quality problem library, performs hierarchical treatment according to a preset correspondence relationship between quality rules and treatment strategies, executes a matching automatic repair strategy on abnormal data that meets an automatic treatment condition, and the automatic treatment condition is that a preset directly executable repair logic exists for the quality rule corresponding to the abnormal data; for abnormal data that exceeds the automatic treatment range or fails in automatic treatment, a structured problem work order containing abnormal details, associated quality rules and preliminary treatment suggestions is generated, and is distributed to a corresponding treatment node according to a business domain for manual intervention; the treatment scheme formed by the automatic treatment strategy and manual intervention is synchronously stored in the quality problem treatment knowledge base;

[0118] The treatment result closed loop verification module, for abnormal data after automatic repair strategy or manual intervention, re-executes a full quality inspection process and verifies the treatment effect; if the verification passes, the verification state of the corresponding quality abnormal data in the quality problem library is updated to "closed loop" and enters the data warehouse as valid data after repair, and the validity label of the corresponding strategy in the quality problem treatment knowledge base is synchronously updated; if the verification fails, the verification state of the corresponding quality abnormal data in the quality problem library is set to "to be re-treated", a structured problem work order containing a historical treatment process, a failure reason and an optimization suggestion is re-iteratively generated, and is distributed to a corresponding treatment node according to a business domain for manual intervention until the abnormality is solved;

[0119] The multi-dimensional dynamic evaluation report generation module, taking DAMA standard data quality dimensions as a benchmark, constructs an open data quality evaluation system, and respectively realizes definition of quality dimensions, quantitative indexes and quality rules and one-to-many association mapping of quality dimensions and quantitative indexes, many-to-many association mapping of quantitative indexes and quality rules through a configuration mode; setting the evaluation range includes business theme division, quality index screening, quality dimension weight assignment and index weight calculation logic, supporting full data quality evaluation and incremental data dynamic evaluation on financial enterprise whole data and sub-business theme data respectively; based on the open data quality evaluation system, multi-dimensional quality scores are calculated for the set of the initial valid data and the valid data after repair in the data warehouse, and a financial enterprise data quality multi-dimensional dynamic evaluation report containing overall quality scores, each quality dimension subdivision score, each quantitative index detail score and business association information is automatically generated according to a preset structured report template.

[0120] Preferably, in the quality rule modeling and quality anomaly intelligent detection module, the technical attributes of the business data of the source-layer data warehouse of the financial enterprise include field mandatory items, primary keys, unique keys, field types, lengths, and formats, the business semantic features include value ranges, data correlation, and business constraints, the quality rule system constructed covers field types, value lengths, duplicate data, numbers, logical consistency, data correlation, null values, value ranges, format specifications, data timeliness, primary keys, and rule violation data checking, and supports custom rule extension; based on the quality rule system, full-quality checking including field-level, record-level, and table-level checking is performed on the source-layer data, and at the same time, the dynamic threshold is monitored to detect data volume anomalies, forming a dual-checking mechanism of rule checking and threshold monitoring; the data of rule violation anomalies or data volume fluctuation anomalies detected by checking is recorded as quality anomaly data to the quality problem library, and the quality anomaly data is intercepted to prevent it from entering the data warehouse.

[0121] Preferably, in the quality rule modeling and quality anomaly intelligent detection module, the data volume prediction model constructed based on the LSTM algorithm adopts a double-layer network architecture including two LSTM units: the first layer LSTM unit has return_sequences=True to extract time sequence features, and the number of neurons is 50; a Dropout layer with a Dropout probability of 0.2 is arranged after the first layer LSTM unit; the second layer LSTM unit has return_sequences=False to abstract and integrate the time sequence features, and the number of neurons is 50; a Dropout layer with a Dropout probability of 0.2 is arranged after the second layer LSTM unit.

[0122] Preferably, in the quality problem grading and handling module, the abnormal data meeting the automatic handling conditions are handled at a low level, and a matched automatic repair strategy is executed, the automatic repair strategy including a null value completion strategy, a value range correction strategy, and a format standardization strategy; the null value completion strategy fills the field with a historical average value or a value derived based on business rules or a business default value, the value range correction strategy adjusts data exceeding a reasonable range to a threshold boundary, and the format standardization strategy uniformly converts data to a preset format specification; the abnormal data exceeding the automatic handling range or failing in automatic handling are handled at a high level, and the generated structured problem work order further includes an abnormal occurrence time, a data table name involved, and business cycle information to which the data belongs; the allocation according to business domains is allocation to corresponding handling nodes according to business domains of retail finance, corporate finance, and financial markets.

[0123] It should be noted that the above detailed description of the specific embodiments of the present application is not intended to limit the present application in any way. Thus, while the present application has been described in detail with reference to specific embodiments thereof, it will be apparent to those skilled in the art that various modifications and changes can be made thereto without departing from the spirit and scope of the present application.

Claims

1. A method for evaluating the data quality of financial enterprises based on the LSTM algorithm, characterized in that, Includes the following steps: The steps for quality rule modeling and intelligent detection of quality anomalies are as follows: Based on the technical attributes and business semantic characteristics of the source layer business data in the financial enterprise data warehouse, a quality rule system covering several rule types including field type compliance, value length constraints, logical consistency, and data correlation is constructed. Historical data volume time series of each data table in the source layer are collected, and a training sample set including time-series periodicity, lag characteristics, and rolling statistical characteristics is constructed. A data volume prediction model is built based on the LSTM algorithm. The model is trained using the training sample set. After model training, the historical data volume time series of the data table to be predicted is input, and the predicted data volume value for the future time period is output. The predicted value is used as a dynamic threshold for the data table's data change rate indicator. Based on the quality rule system and dynamic threshold, a full quality check is performed on the source layer data. Data without detected anomalies is entered into the data warehouse as initial valid data, while detected quality anomaly data is recorded in the quality problem database. Quality issue classification and handling steps: For quality anomaly data in the quality issue database, classification and handling are carried out according to the correspondence between preset quality rules and handling strategies. For anomaly data that meets the automatic handling conditions, a matching automatic repair strategy is executed. The automatic handling conditions are that the quality rules corresponding to the anomaly data have preset, directly executable repair logic. For anomaly data that exceeds the scope of automatic handling or fails to be handled automatically, a structured issue work order containing anomaly details, associated quality rules, and preliminary handling suggestions is generated and assigned to the corresponding handling node according to business domain for manual intervention. The automatic handling strategy and the handling plan formed by manual intervention are synchronously stored in the quality issue handling knowledge base. Closed-loop verification steps for handling results: For abnormal data after automatic repair strategies or manual intervention, re-execute the full quality inspection process and verify the handling effect; if the verification is passed, update the verification status of the corresponding quality abnormal data in the quality problem database to "closed loop" and enter it into the data warehouse as valid data after repair, and simultaneously update the validity label of the corresponding strategy in the quality problem handling knowledge base; If the verification fails, the verification status of the corresponding quality anomaly data in the quality issue database will be set to "pending reprocessing". A structured issue work order containing the historical processing process, failure reasons and optimization suggestions will be generated again and assigned to the corresponding processing node according to the business domain for continued manual intervention until the anomaly is resolved. The steps for generating a multi-dimensional dynamic evaluation report are as follows: Based on the DAMA standard data quality dimensions, an open data quality evaluation system is constructed. Through configuration, the definitions of quality dimensions, quantitative indicators, and quality rules are realized, as well as the one-to-many association mapping between quality dimensions and quantitative indicators, and the many-to-many association mapping between quantitative indicators and quality rules. The evaluation scope is set, including business theme division, quality indicator screening, quality dimension weight assignment, and indicator weight calculation logic. It supports full data quality evaluation and incremental dynamic evaluation of financial enterprise overall data and data by business theme. Based on the open data quality evaluation system, a multi-dimensional quality score is calculated for the set of initial valid data and repaired valid data in the data warehouse. According to the preset structured report template, a multi-dimensional dynamic evaluation report on the data quality of financial enterprises is automatically generated, which includes the overall quality score, the subdivided scores of each quality dimension, the detailed scores of each quantitative indicator, and business-related information.

2. The method for evaluating the data quality of financial enterprises based on the LSTM algorithm according to claim 1, characterized in that, In the quality rule modeling and intelligent quality anomaly detection steps, the technical attributes of the source layer business data in the financial enterprise data warehouse include primary keys, field types, value lengths, and format specifications. Business semantic features include value ranges, data correlation, logical consistency, and data timeliness. The constructed quality rule system covers field types, value lengths, duplicate data, number of records, logical consistency, data correlation, null values, value ranges, format specifications, data timeliness, primary keys, and violation data checks, and supports custom rule extensions. Based on the quality rule system, a full-scale quality check is performed on the source layer data, including field-level, record-level, and table-level checks. Simultaneously, combined with dynamic threshold monitoring of abnormal data volume fluctuations, a dual check mechanism of rule verification and threshold monitoring is formed. Data with detected rule violations or abnormal data volume fluctuations is recorded as quality anomaly data in the quality problem database, and the quality anomaly data is intercepted to prevent it from entering the data warehouse.

3. The method for evaluating the data quality of financial enterprises based on the LSTM algorithm according to claim 1 or 2, characterized in that, In the steps of quality rule modeling and intelligent detection of quality anomalies, the data volume prediction model built based on the LSTM algorithm adopts a two-layer network architecture including two LSTM units: the first layer LSTM unit sets return_sequences=True to extract temporal features, with 50 neurons, and a Dropout layer with a Dropout probability of 0.2 is set after the first layer LSTM unit; the second layer LSTM unit sets return_sequences=False to abstract and integrate the temporal features, with 50 neurons, and a Dropout layer with a Dropout probability of 0.2 is set after the second layer LSTM unit.

4. The data quality evaluation method for financial enterprises based on the LSTM algorithm according to claim 1, characterized in that, In the quality problem classification and handling steps, abnormal data that meet the automatic handling conditions are handled at a low level, and a matching automatic repair strategy is executed. The automatic repair strategy includes a null value completion strategy, a value range correction strategy, and a format standardization strategy. The null value completion strategy fills in the field with its historical average, derived value based on business rules, or default business value. The value range correction strategy adjusts data that exceeds a reasonable range to the threshold boundary. The format standardization strategy converts the data into a preset format specification. Abnormal data that exceeds the automatic processing range or fails to be processed automatically is handled at a higher level. The generated structured problem work order also includes the time of the abnormality, the name of the data table involved, and the business cycle information to which the data belongs. The allocation by business domain specifically allocates data to the corresponding processing node according to the business areas of retail finance, corporate finance, and financial markets.

5. The method for evaluating the data quality of financial enterprises based on the LSTM algorithm according to claim 1, characterized in that, In the multidimensional dynamic evaluation report generation step, the configurable method for defining quality dimensions, quantitative indicators, and quality rules specifically includes: The quality dimensions are defined by configuration as completeness, effectiveness, accuracy, and timeliness. The configurable quantitative indicators for each dimension are defined as follows: The completeness dimension includes the loan application material completeness rate, which requires that the missing rate of necessary attachments for the application form, credit report, and bank statements be less than 0.5%; the validity dimension includes the loan interest rate value, which requires that the interest rate value be within the prescribed LPR fluctuation range; the accuracy dimension includes the core system total score balance verification, which requires that the verification pass rate be 100%; the timeliness dimension includes the transaction flow entry completion rate, which requires that the transaction flow entry completion rate be 100% before the preset time on T+1 day; the configurable quantitative indicators also include data integrity rate, field accuracy rate, cross-table consistency matching rate, data timely update rate, duplicate record removal rate, and compliance verification pass rate; Configurable quality rules include: data correlation rules, used to quantify the completeness rate of loan application materials and the overall balance verification index of the core system; format compliance rules, used to quantify the accuracy rate of fields; timeliness constraint rules, used to quantify the timely update rate of data and the completion rate of transaction flow into the warehouse; and support for adding custom rules.

6. The method for evaluating the data quality of financial enterprises based on the LSTM algorithm according to claim 5, characterized in that, In the multidimensional dynamic evaluation report generation step, the indicator weight calculation logic adopts the analytic hierarchy process (AHP) or the entropy weight method. The AHP determines the subjective weight of each indicator by constructing a judgment matrix, while the entropy weight method calculates the objective weight of each indicator based on the information entropy of the indicator data. The business-related information in the multidimensional dynamic evaluation report of financial enterprise data quality includes the number of business topics, the number of related database tables, the total number of valid data records, the success rate of abnormal data repair, and data quality optimization suggestions.

7. A data quality evaluation system for financial enterprises based on the LSTM algorithm, characterized in that, It includes a quality rule modeling and intelligent quality anomaly detection module, a quality problem classification and handling module, a handling result closed-loop verification module, and a multi-dimensional dynamic evaluation report generation module, which are connected in sequence. The quality rule modeling and intelligent quality anomaly detection module constructs a quality rule system covering several rule types, including field type compliance, value length constraints, logical consistency, and data correlation, based on the technical attributes and business semantic characteristics of the source layer business data in the financial enterprise data warehouse. It collects historical data volume time series from each data table in the source layer, constructs a training sample set containing time-series periodicity, lag characteristics, and rolling statistical characteristics, and builds a data volume prediction model based on the LSTM algorithm. The model is trained using the training sample set. After training, the module inputs the historical data volume time series of the data table to be predicted and outputs the predicted data volume for the future time period. This predicted value is used as a dynamic threshold for the data table's data change rate indicator. Based on the quality rule system and dynamic threshold, a full quality check is performed on the source layer data. Data without detected anomalies is entered into the data warehouse as initial valid data, while detected quality anomaly data is recorded in the quality problem database. The quality issue classification and handling module classifies and handles quality anomaly data in the quality issue database according to the pre-set correspondence between quality rules and handling strategies. For anomaly data that meets the automatic handling conditions, a matching automatic repair strategy is executed. The automatic handling conditions are that the quality rules corresponding to the anomaly data have pre-set, directly executable repair logic. For anomaly data that exceeds the scope of automatic handling or fails to be handled automatically, a structured issue work order containing anomaly details, associated quality rules, and preliminary handling suggestions is generated and assigned to the corresponding handling node according to the business domain for manual intervention. The automatic handling strategy and the handling plan formed by manual intervention are synchronously stored in the quality issue handling knowledge base. The closed-loop verification module for handling results re-executes the full quality inspection process and verifies the handling effect for abnormal data after automatic repair strategy or manual intervention. If the verification is successful, the verification status of the corresponding quality abnormal data in the quality problem database is updated to "closed loop" and it is entered into the data warehouse as valid data after repair. The validity label of the corresponding strategy in the quality problem handling knowledge base is updated simultaneously. If the verification fails, the verification status of the corresponding quality anomaly data in the quality issue database will be set to "pending reprocessing". A structured issue work order containing the historical processing process, failure reasons and optimization suggestions will be generated again and assigned to the corresponding processing node according to the business domain for continued manual intervention until the anomaly is resolved. The multi-dimensional dynamic evaluation report generation module uses DAMA standard data quality dimensions as a benchmark to construct an open data quality evaluation system. It defines quality dimensions, quantitative indicators, and quality rules through configuration, as well as one-to-many association mappings between quality dimensions and quantitative indicators, and many-to-many association mappings between quantitative indicators and quality rules. The evaluation scope includes business theme division, quality indicator screening, quality dimension weight assignment, and indicator weight calculation logic. It supports full data quality evaluation and incremental dynamic evaluation of financial enterprise overall data and data by business theme. Based on the open data quality evaluation system, a multi-dimensional quality score is calculated for the set of initial valid data and repaired valid data in the data warehouse. According to the preset structured report template, a multi-dimensional dynamic evaluation report on the data quality of financial enterprises is automatically generated, which includes the overall quality score, the subdivided scores of each quality dimension, the detailed scores of each quantitative indicator, and business-related information.

8. The financial enterprise data quality evaluation system based on the LSTM algorithm according to claim 7, characterized in that, In the quality rule modeling and intelligent quality anomaly detection module, the technical attributes of the source layer business data in the financial enterprise data warehouse include required fields, primary keys, unique keys, field types, lengths, and formats. Business semantic features include value ranges, data correlations, and business constraints. The constructed quality rule system covers field types, value lengths, duplicate data, number of duplicates, logical consistency, data correlations, null values, value ranges, format specifications, data timeliness, primary keys, and checks for violating data, and supports custom rule extensions. Based on the quality rule system, a full-scale quality check is performed on the source layer data, including field-level, record-level, and table-level verifications. Simultaneously, combined with dynamic threshold monitoring of abnormal data volume fluctuations, a dual verification mechanism of rule verification and threshold monitoring is formed. Data with detected rule violations or abnormal data volume fluctuations is recorded as quality anomaly data in the quality problem database, and the quality anomaly data is intercepted to prevent it from entering the data warehouse.

9. The financial enterprise data quality evaluation system based on the LSTM algorithm according to claim 7 or 8, characterized in that, In the quality rule modeling and intelligent quality anomaly detection module, the data volume prediction model built based on the LSTM algorithm adopts a two-layer network architecture including two LSTM units: the first layer LSTM unit sets return_sequences=True to extract temporal features, with 50 neurons, and a Dropout layer with a Dropout probability of 0.2 is set after the first layer LSTM unit; the second layer LSTM unit sets return_sequences=False to abstract and integrate the temporal features, with 50 neurons, and a Dropout layer with a Dropout probability of 0.2 is set after the second layer LSTM unit.

10. The financial enterprise data quality evaluation system based on the LSTM algorithm according to claim 7, characterized in that, In the quality problem classification and handling module, abnormal data that meets the automatic handling conditions are handled at a low level, and a matching automatic repair strategy is executed. The automatic repair strategy includes a null value completion strategy, a value range correction strategy, and a format standardization strategy. The null value completion strategy fills in the field with its historical average, derived value based on business rules, or default business value. The value range correction strategy adjusts data that exceeds a reasonable range to the threshold boundary. The format standardization strategy converts the data into a preset format specification. Abnormal data that exceeds the automatic processing range or fails to be processed automatically is handled at a higher level. The generated structured problem work order also includes the time of the abnormality, the name of the data table involved, and the business cycle information to which the data belongs. The allocation by business domain specifically allocates data to the corresponding processing node according to the business areas of retail finance, corporate finance, and financial markets.

Images