Risk control data mart data quality monitoring method and system based on AI technology

Abstract

The invention discloses a risk control data mart data quality monitoring method and system based on an AI technology. The method comprises the following steps: collecting risk control data from various data sources based on a machine learning algorithm; performing data cleaning on the inaccurate or incomplete risk control data through an anomaly detection model; comparing and verifying the isolated forest model and the support vector machine model with known accurate data samples, and dynamically analyzing whether indexes are normal or not for risk control data; a real-time monitoring system based on micro-service is established, the data quality index in the data mart is continuously monitored, and when the data quality index deviates from a preset threshold value, the system triggers an early warning mechanism and sends a risk early warning signal. The system collects risk control related data from a plurality of data sources, processes inaccurate or incomplete data, dynamically analyzes whether an index is normal or not by using a data analysis algorithm, and sends a risk early warning signal in time according to an analysis result, so as to improve the timeliness and accuracy of risk management.

Description

Technical Field

[0001] The present invention relates to the field of financial risk control, and more specifically, to a method and system for monitoring data quality of a risk control data mart based on AI technology. Background Art

[0002] In the field of financial risk control, data is the core asset. As a platform for centralized storage and management of risk control data, data marts integrate data from multiple channels, including basic customer information, credit records, transaction data, market data, etc. These data are used to build risk assessment models, formulate credit strategies, monitor market risks and many other key business links. For example, when banks assess personal credit risk, they need to comprehensively consider data such as customers' income, assets, credit history, etc. Any data quality issues may lead to incorrect risk assessment and thus cause credit losses. High-quality data is essential for risk assessment and investment decisions. For banks, accurate credit data is a key factor in deciding whether to grant loans to customers and the amount of loans. If the credit data is of low quality and contains incorrect repayment records or inaccurate income information, banks may mistakenly grant loans to high-risk customers, resulting in an increase in the non-performing loan rate and huge economic losses to the bank. Similarly, in the investment field, investors need to rely on high-quality market data, company financial data, etc. to assess the value and risk of investment targets. If the data quality is poor, it may cause investors to misjudge the value of stocks or bonds and make wrong investment decisions.

[0003] In recent years, with the development of AI, it has been widely used in various fields and has demonstrated its powerful technical advantages. AI technology can process large amounts of data and perform complex computing tasks, far beyond human processing capabilities. This allows many tasks to be completed more efficiently, thereby improving overall productivity. AI's machine learning (ML) algorithms are able to identify complex patterns and trends, and as the amount of data increases, the accuracy of predictions will continue to improve. AI can reduce the operating costs of enterprises by automating and optimizing business processes. AI has high-precision execution capabilities, which can reduce human errors and improve overall accuracy during task processing. Summary of the invention

[0004] In view of the technical problems existing in the prior art, the present invention provides a data quality monitoring method and system for a risk control data mart based on AI technology. The present invention brings new opportunities for data quality monitoring and can achieve more efficient, accurate and intelligent monitoring, which helps to timely discover data problems, reduce risk exposure, and ensure the sound operation of financial institutions.

[0005] According to a first aspect of the present invention, a method for monitoring data quality of a risk control data mart based on AI technology is provided, comprising the following steps:

[0006] Collect risk control data from various data sources based on machine learning algorithms;

[0007] Clean inaccurate or incomplete risk control data through anomaly detection models;

[0008] Through the comparison and verification of the isolation forest model and the support vector machine model with known accurate data samples, the risk control data is dynamically analyzed to see whether the indicators are normal;

[0009] Establish a real-time monitoring system based on microservices to continuously monitor the data quality indicators in the data mart. When the data quality indicators deviate from the preset threshold, the system triggers the early warning mechanism and issues a risk warning signal.

[0010] Based on the above technical solution, the present invention can also make the following improvements.

[0011] Optionally, the collecting of risk control data from various data sources based on a machine learning algorithm includes:

[0012] Use AI-driven intelligent data collection tools to extract data from various data sources and conduct preliminary checks on the integrity and consistency of the data. Use machine learning algorithms to identify whether the data format meets predefined specifications and detect whether there are any fields with missing or duplicate data.

[0013] Optionally, the method is carried out by intelligent screening and expansion of risk control data collection sources collected from various data sources based on machine learning algorithms, adaptive data collection interfaces and intelligent web crawlers.

[0014] Optionally, a dynamic data mapping and conversion mechanism is established for the incoming data, and the AI ​​system automatically adjusts the mapping rules based on the characteristics of the data.

[0015] Optionally, the data cleaning includes:

[0016] Adopt anomaly detection models to intelligently identify and process noise, outliers, and erroneous data in the data;

[0017] Data points are grouped based on cluster analysis algorithms, and data points that deviate from normal clusters are considered outliers and corrected or marked according to business rules and data distribution characteristics;

[0018] Use natural language processing to clean and standardize text data, extract key information and convert it into structured data.

[0019] Optionally, the comparison and verification of the isolation forest model and the support vector machine model with known accurate data samples to dynamically analyze whether the risk control data indicators are normal include:

[0020] The accuracy index, completeness index, consistency index and timeliness index are analyzed respectively.

[0021] Optionally, the accuracy indicators include data matching rate and data error rate; the integrity indicators include required field missing rate and data record missing rate; the consistency indicators include data logic consistency ratio and data source data consistency rate; the timeliness indicators include data update delay rate and data validity period ratio.

[0022] Optionally, building a real-time monitoring system based on microservices includes:

[0023] Build an early warning indicator system, including early warning indicators: data accuracy warning, logical contradiction warning, required field missing warning, data relevance missing warning, data consistency warning, internal data consistency warning, data update delay warning and data validity period warning;

[0024] Set up early warning trigger mechanisms, including threshold triggers, model prediction triggers, and anomaly detection triggers.

[0025] Optionally, a real-time monitoring system based on microservices is established to continuously monitor the data quality indicators in the data mart. When the data quality indicators deviate from the preset threshold, the system triggers the early warning mechanism and issues risk warning signals including:

[0026] The early warning system uses machine learning algorithms to prioritize early warning information and push the most urgent and important early warning information to relevant personnel based on the potential impact of data issues on business risks so that timely measures can be taken to deal with it.

[0027] According to a second aspect of the present invention, there is provided a risk control data mart data quality monitoring system based on AI technology, comprising:

[0028] Data collection module, used to collect risk control related data from multiple data sources based on machine learning algorithms;

[0029] The data cleaning module is used to clean inaccurate or incomplete risk control data through anomaly detection models;

[0030] The evaluation indicator module is used to compare and verify the known accurate data samples through the isolation forest model and the support vector machine model, and dynamically analyze the risk control data to see if the indicators are normal;

[0031] The early warning module is used to establish a real-time monitoring system based on microservices to continuously monitor the data quality indicators in the data mart. When the data quality indicators deviate from the preset threshold, the system triggers the early warning mechanism and issues a risk warning signal.

[0032] Technical effects and advantages of the present invention:

[0033] The present invention provides a data quality monitoring method and system for a risk control data mart based on AI technology, which collects risk control related data from multiple data sources, processes inaccurate or incomplete data, uses data analysis algorithms to dynamically analyze data to see if the indicators are normal, and issues risk warning signals in a timely manner based on the analysis results to improve the timeliness and accuracy of risk management. The present invention brings new opportunities for data quality monitoring, can achieve more efficient, accurate and intelligent monitoring, helps to timely discover data problems, reduce risk exposure, and ensure the sound operation of financial institutions.

[0034] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flow chart of a method for monitoring data quality in a risk control data mart based on AI technology provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] In the field of financial risk control, data marts serve as centralized data storage and processing centers, and the quality of their data directly affects the accuracy and effectiveness of risk assessment, decision-making, and business operations. The embodiments of the present invention bring new opportunities for data quality monitoring, can achieve more efficient, accurate, and intelligent monitoring, help to timely discover data problems, reduce risk exposure, and ensure the sound operation of financial institutions. The present invention brings new opportunities for data quality monitoring, can achieve more efficient, accurate, and intelligent monitoring, help to timely discover data problems, reduce risk exposure, and ensure the sound operation of financial institutions.

[0038] It is understandable that based on the defects in the background technology, the present invention provides a data quality monitoring method for risk control data mart based on AI technology, specifically as follows: Figure 1 As shown, the following steps are included:

[0039] Step S1. Collect risk control data from various data sources based on machine learning algorithms;

[0040] In this embodiment, data collection based on machine learning algorithms can be performed by intelligent screening and expansion of data collection sources, adaptive data collection interfaces, and intelligent web crawlers.

[0041] Because AI technology can intelligently analyze and screen massive potential data sources. By learning the characteristics of existing data mart data, such as the structure, format, semantic information of the data, and its relevance to business goals such as risk assessment, AI can automatically identify new data sources that may be valuable to the data mart. For example, natural language processing technology is used to analyze unstructured texts such as industry reports and news information, extract entities related to the data mart theme (such as company names, financial products, etc.) and key information (such as market trends, policy changes, etc.), and include the data sources corresponding to this information in the collection scope.

[0042] At the same time, machine learning algorithms can dynamically adjust the priority and collection frequency of data sources based on historical data quality monitoring results and current business needs. For example, if it is found that the data from a certain data source has significantly improved the accuracy of risk assessment over the past period of time, AI can automatically increase the collection frequency of that data source; conversely, if the data quality of a certain data source continues to decline or its relevance to the business decreases, its collection frequency can be appropriately reduced or suspended, thereby optimizing the resource allocation for data collection and ensuring that the collected data has higher quality and value.

[0043] Adaptive data collection interface: AI-driven collection interfaces are adaptive and can automatically identify different data source types (such as relational databases, non-relational databases, file systems, APIs, etc.) and data formats (such as structured data, semi-structured data, unstructured data). For example, when faced with a new data source, the collection interface can use a deep learning model to analyze the metadata and a small amount of sample data of the data source, quickly determine its data structure and access method, and then automatically generate appropriate collection strategies and code snippets. This greatly reduces the workload and error rate of manual development and configuration of collection interfaces, and improves the efficiency and flexibility of data collection. -For API data sources, AI can automatically learn changes in API calling rules and data return formats. For example, by analyzing API response data over multiple time periods, AI can detect API version updates or data structure adjustments, and automatically adjust the parameter settings of the collection program to ensure the continuity and accuracy of data collection.

[0044] Intelligent web crawlers: When collecting public data on the Internet, AI-enhanced web crawlers can perform intelligent crawling according to the needs of the data mart and data quality requirements. It is no longer a simple crawling according to the preset URL list and rules, but can understand the semantics and content relevance of the web page. For example, the convolutional neural network (CNN) is used to jointly analyze the visual layout and text information of the web page, accurately identify the key data areas in the page (such as tables, text paragraphs in a specific format, etc.), and give priority to crawling these high-value data. At the same time, AI web crawlers can be dynamically scheduled according to the timeliness and importance of the data. For data with high timeliness requirements (such as real-time financial news, market data, etc.), crawling will be prioritized, and a reasonable crawling interval will be set according to the frequency of data updates; for relatively stable but high-importance data (such as industry regulatory documents, etc.), it can be collected in a regular scanning and updating manner to ensure that the data in the data mart always maintains a high quality level.

[0045] Quality pre-assessment and control during data collection;

[0046] Real-time data quality detection: During the data collection process, AI algorithms can perform real-time quality detection on the collected data. For example, using anomaly detection algorithms based on statistical models, the distribution characteristics of numerical data are analyzed to quickly identify data points that are significantly deviated from the normal range (such as abnormal transaction amounts, credit scores, etc.). For text data, text classification and semantic analysis techniques in natural language processing are used to detect whether the data has semantic ambiguity, grammatical errors, or is inconsistent with preset business rules (such as sensitive information in customer reviews, false propaganda, etc.). When quality problems are found, the collection system can immediately take corresponding measures, such as suspending collection, marking problem data, and notifying relevant personnel to handle it.

[0047] Data collection integrity verification: AI can verify the integrity of data in the collection process by learning the existing data patterns and integrity constraints in the data mart. For example, based on the association rule mining algorithm, the intrinsic connections between different data fields are analyzed to determine whether the collected data meets these associations. If it is found that a data record lacks key associated fields (such as the lack of corresponding customer information or product information in customer order data), it will prompt that there is an integrity problem in data collection, and try to supplement the missing information from other relevant data sources or historical data, or mark the data record for further processing, so as to ensure that the collected data has a high degree of integrity before entering the data mart, reducing quality problems caused by missing data. Through the above data collection methods based on AI technology, the data required for the data mart can be obtained more efficiently, intelligently and accurately, and the data quality can be effectively monitored and controlled during the collection process, laying a solid foundation for subsequent data quality monitoring and stable operation of the data mart.

[0048] Step S2. Clean the inaccurate or incomplete risk control data through anomaly detection models;

[0049] The importance of data cleaning In the risk control data mart, the accuracy and completeness of data are crucial. Inaccurate or incomplete data may lead to incorrect risk assessment results, thus affecting the decision-making and business development of the enterprise. Therefore, data cleaning is one of the key steps in building a reliable risk control data mart. The following is about using AI for data cleaning in data marts:

[0050] Applications of AI in data cleaning include:

[0051] Outlier detection: AI algorithms can automatically detect outliers in data. For example, by grouping data points through clustering algorithms, those data points that are far away from the center of the cluster may be considered outliers. Deep learning models can also be used for outlier detection by learning the patterns of normal data and identifying data points that do not conform to the normal pattern. When an outlier is detected, it can be processed according to the specific situation, such as deleted, corrected, or marked as a special case for further analysis.

[0052] Missing value processing: AI can predict missing values. For example, using machine learning algorithms to predict missing numerical data based on other related variables. For missing values ​​of categorical variables, frequency-based methods or similarity-based methods can be used to fill them. Autoencoders in deep learning can also be used to interpolate missing values, reconstructing missing data by learning the potential representation of the data.

[0053] Deduplication: AI can quickly and accurately identify duplicate data. It finds duplicate records by calculating the similarity between data records, such as using hash algorithms or vector space models. When duplicate data is determined, you can choose to keep one of the records or merge them according to specific rules.

[0054] Data standardization and normalization: AI can automatically standardize and normalize data to ensure that data from different sources has a consistent format and range. For example, for numerical data, Z-score standardization or Min-Max normalization methods can be used to convert data into a specific range. For text data, natural language processing techniques can be used for standardization, such as stemming and removing stop words.

[0055] The data cleaning process includes:

[0056] Data evaluation: First, conduct a comprehensive evaluation of the collected data to understand the quality of the data, including checking the completeness, accuracy, consistency, etc. of the data. - Data visualization tools such as histograms and scatter plots can be used to intuitively understand the distribution and characteristics of the data.

[0057] Develop a cleaning strategy: Based on the results of the data assessment, develop a specific data cleaning strategy. Determine the types of problems that need to be dealt with, such as outliers, missing values, duplicate data, etc., and select appropriate AI algorithms and technologies to handle them. At the same time, consider the business meaning and importance of the data to ensure that the cleaning process does not negatively affect the validity of the data.

[0058] Perform cleaning operations: Use the selected AI algorithms and tools to perform data cleaning operations. This may involve steps such as data preprocessing, model training, prediction, and correction. - During the cleaning process, the cleaning effect should be continuously monitored and evaluated to ensure that the data quality is effectively improved.

[0059] Verification and confirmation: Verify and confirm the cleaned data to ensure that the data meets the expected quality standards. Statistical analysis, data visualization and other methods can be used for verification. - If problems are found, adjust the cleaning strategy in time and re-clean the operation until the data quality meets the requirements. Data quality is a dynamic issue that requires continuous monitoring and optimization. Establish a data quality monitoring mechanism, evaluate and clean the data regularly, and ensure that the risk control data mart always maintains high-quality data. In short, using AI for data cleaning of risk control data marts can improve data quality and provide a more reliable basis for risk assessment and decision-making. However, during the application process, attention should be paid to data privacy and security, ensure the interpretability and traceability of the cleaning process, and continuously monitor and optimize data quality.

[0060] Step S3. Compare and verify the known accurate data samples through the isolation forest model and the support vector machine model, and dynamically analyze the risk control data to see if the indicators are normal;

[0061] The following are the details of the data quality assessment indicator setting for the risk control data mart using AI:

[0062] 3.1 Accuracy Indicators

[0063] 3.1.1 Data matching rate:

[0064] Definition: Compare the data in the data mart with the authoritative data source (such as official credit agency data, original data from the enterprise's core business system, etc.), and calculate the proportion of the matching data volume to the total data volume.

[0065] Calculation method: matching data volume / total data volume × 100%. For example, in the customer identity information data set, the ID card number, name, etc. in the data mart are compared with the information provided by the public security system. If there are a total of 1,000 customer identity data, of which 950 are completely matched, the data matching rate is 95%.

[0066] AI application: Use natural language processing (NLP) technology in AI to perform fuzzy matching on text data (such as names, addresses, etc.) to improve matching accuracy and efficiency. Deep learning models can learn the semantic features of data from different data sources and better handle situations where data formats or expressions are slightly different.

[0067] 3.1.2. Data error rate:

[0068] Definition: The ratio of the amount of error between the data in the statistical data mart and the actual true value to the total amount of data, which is used to measure the degree to which the data deviates from the actual situation.

[0069] Calculation method: Error data volume / total data volume × 100%. For example, in the enterprise financial statement data, if there are 50 data errors in the sales volume of a certain product recorded in the data mart compared with the original accurate data in the enterprise financial system, and the total data volume is 5,000, then the data error rate is 1%.

[0070] AI application: By establishing a reasonable range of data for machine learning regression model prediction, data beyond the prediction range is marked as data with possible errors, thereby assisting in calculating the error rate. For example, a model is established based on historical sales data, market trends and other factors to predict a reasonable range for sales. If the sales data in the data mart is not within this range, there may be errors.

[0071] 3.2 Integrity indicators

[0072] 3.2.1. Missing rate of required fields:

[0073] Definition: For the key fields that must be filled in the data set, calculate the ratio of the amount of missing data to the total amount of data that should be filled in.

[0074] Calculation method: number of missing data in required fields / total amount of data to be filled in × 100%. For example, in a loan application dataset, ID number, contact information, loan amount, etc. are required fields. If there are 800 loan application data in total, and 40 of them lack contact information, then the missing rate of required fields is 5%.

[0075] AI applications: AI algorithms can automatically scan data sets, identify required fields, and quickly count missing fields. Image recognition technology based on deep learning can also be applied to data marts that scan paper documents to detect whether there are any required fields that are not filled in, thereby improving the comprehensiveness of data integrity checks.

[0076] 3.2.2 Data record missing rate:

[0077] Definition: Calculates the ratio of the number of missing data records to the expected number of complete data records in a data mart.

[0078] Calculation method: number of missing data records / expected number of complete data records × 100%. For example, according to business rules, 100 new transaction data records should enter the data mart every day. If there are only 90 in reality, the missing data record rate is 10%.

[0079] AI application: Use AI prediction models to predict the number of data records that should be recorded each day based on the rules of historical data records and related business factors, and compare them with the actual number of data records to more accurately calculate the data record missing rate. For example, a model is established based on the time series analysis of historical transaction data and seasonal factors to predict the daily transaction data volume.

[0080] 3.3 Consistency Indicators

[0081] 3.3.1 Data logical consistency ratio:

[0082] Definition: Check whether the logical relationship between data in the data mart is correct, and calculate the proportion of logically consistent data to the total data volume.

[0083] Calculation method: logically consistent data volume / total data volume × 100%. For example, in customer relationship data, if the customer's registration date is later than the date of their first transaction, there is a logical inconsistency. If there are 5,000 customer relationship data in total, of which 4,800 are logically consistent, the data logical consistency ratio is 96%.

[0084] AI application: By building a semantic relationship network between data through the knowledge graph technology in AI, the logical consistency of data can be automatically detected. For example, customer information, transaction information, product information, etc. are built into a knowledge graph, and the logical relationship between nodes and edges is checked through graph algorithms to see if it complies with business rules.

[0085] 3.3.2 Data consistency rate of data source;

[0086] Definition: Compare the consistency of data entering the data mart from different data sources, and calculate the proportion of consistent data to the total amount of compared data.

[0087] Calculation method: consistent data volume across data sources / total data volume for comparison × 100%. For example, if sales data is obtained from the company's online sales platform and offline store sales system and entered into the data mart, and there are a total of 6,000 sales data for comparison, of which 5,500 data are consistent in the two data sources, then the consistency rate of data across data sources is 91.67%.

[0088] AI applications: AI can automatically integrate and compare data from different data sources, use data fusion algorithms and deep learning models to learn the characteristics and differences of data from different data sources, and more accurately determine data consistency. For example, deep learning models can be used to extract and match features of sales data from different database structures and formats to detect data consistency.

[0089] 3.4 Timeliness Index

[0090] 3.4.1 Data update delay rate:

[0091] Definition: Measures the time delay between data update and data source update in the data mart, and calculates the average delay time or the ratio of delayed data volume to the total data volume.

[0092] Calculation method: average delay time = ∑ (update delay time for each data item) / total data amount; delayed data volume percentage = delayed update data volume / total data volume × 100%. For example, a financial data mart should update stock price data in real time. If there are a total of 1,000 stock price data items, 50 of which are updated with a delay of more than 5 minutes, and the average delay time is 3 minutes, the data update delay rate can be measured from these two perspectives.

[0093] AI application: AI can monitor the update status of data sources in real time and compare it with the data in the data mart. It can use time series analysis and prediction models to predict the reasonable time range for data updates and timely discover delayed update data. For example, a model is established based on the time pattern of historical stock price data updates to predict the time window for the next update. If the actual update exceeds this window, it is considered a delayed update.

[0094] 3.4.2 Proportion of data within validity period:

[0095] Definition: The ratio of the amount of data within the valid usage period to the total amount of data in the statistical data mart.

[0096] Calculation method: data volume within validity period / total data volume × 100%. For example, for some market research data, its validity period is 3 months. If there are 2,000 market research data in the data mart, of which 1,800 are within the validity period, then the data within the validity period ratio is 90%.

[0097] AI application: AI can automatically determine the validity period of data based on the data type, source and business rules, and regularly check whether the data in the data mart is expired. The machine learning classification model is used to divide the data into valid and expired data according to the characteristics of the data and business scenarios, which is convenient for calculating the ratio. By setting these data mart data quality assessment indicators using AI, the data quality of the data mart can be comprehensively and accurately measured, providing a reliable data foundation for risk control decisions. At the same time, with the help of AI technology, the accuracy and efficiency of indicator calculation can be improved to better adapt to the complex and changing risk control data environment.

[0098] Step S4. Establish a real-time monitoring system based on microservices to continuously monitor the data quality indicators in the data mart. When the data quality indicators deviate from the preset threshold, the system immediately triggers the early warning mechanism.

[0099] The following is a detailed description of the data quality monitoring and early warning of risk control data mart based on AI technology;

[0100] 4.1 Construction of early warning indicator system;

[0101] 4.1.1 Data Accuracy Warning Indicators

[0102] Numerical deviation warning: By comparing the data in the data mart with the average value, median and other statistics of authoritative data sources or historical data, a reasonable deviation threshold is set. For example, in credit score data, if the credit score of a customer deviates from the average score of customers of the same type by more than a certain range (such as 30%), there may be data errors or anomalies, triggering an early warning.

[0103] 4.1.2 Logical contradiction warning:

[0104] AI models are used to check the logical relationship between data. For example, in insurance business data, there is a certain logical relationship between the age of the insured and the insurance period and the insured amount. If the insured is too old but the insurance period is too long and the insured amount is too high, it violates the normal insurance business logic and the system will issue an early warning.

[0105] 4.1.3 Warning of missing required fields

[0106] Determine the required fields for each record in the data set, and issue an immediate warning when the AI ​​system detects that a large number of required fields are missing (such as the missing ratio exceeds 10%). For example, in customer loan application data, if key required fields such as name, ID number, and income are missing, it will seriously affect the accuracy of risk assessment.

[0107] 4.1.4 Warning of missing data relevance:

[0108] Check the integrity of the associated data between different data tables. For example, in an enterprise financial data mart, there is a specific relationship between the balance sheet, income statement, and cash flow statement. If the data in a table cannot match the corresponding data in other tables or the associated data is missing, the early warning system will prompt a data integrity problem.

[0109] 4.1.5 Data consistency warning indicators

[0110] Cross-data source consistency warning: When the data obtained from multiple data sources have differences in the same data item, a warning is triggered. For example, the credit rating data of the same enterprise obtained from different credit rating agencies are inconsistent, and the difference exceeds the acceptable range (such as a difference of more than two levels), which may lead to uncertainty in the risk assessment results and require a warning prompt for further verification.

[0111] 4.1.6 Internal data consistency warning:

[0112] Within the data mart, the data generated by different modules or business processes should be consistent. For example, in the bank's savings and loan business data, the identity information, account balance and other data of the same customer should match each other in different business links. If there is any inconsistency, the early warning system will detect and report it in time.

[0113] 4.1.7 Data update delay warning:

[0114] Different data update time requirements are set according to the business characteristics and importance of the data. For example, for real-time market data of financial markets, it is required to be updated every few minutes. If it is not updated within the specified time, the early warning system will send a signal to remind data managers to pay attention to whether the data source is normal.

[0115] 4.1.8 Data validity period warning:

[0116] Some data has a certain validity period, such as a customer's credit report data may need to be re-evaluated after a certain period of time. When the data is close to or exceeds the validity period, the early warning system will prompt relevant personnel to update or re-collect the data to ensure that the risk assessment is based on the latest valid data.

[0117] 4.2 Early warning trigger mechanism;

[0118] 4.2.1 Threshold Trigger

[0119] Based on the thresholds set by the above warning indicators, when the data quality indicators reach or exceed the thresholds, the warning is triggered immediately. For example, when the error rate of the data exceeds 5%, the warning system automatically sends an alarm to notify relevant personnel. The setting of the threshold needs to be scientifically determined in combination with the actual business situation and risk tolerance, and can be dynamically adjusted according to the historical performance of the data and business changes.

[0120] 4.2.2 Model prediction trigger

[0121] Use AI prediction models to predict data quality trends. When the prediction model shows that data quality will deteriorate to an unacceptable level in a certain period of time in the future, trigger an early warning. For example, if the accuracy index of the data is predicted by the time series analysis model and it is found that the accuracy index will drop below the warning threshold in the next week, the early warning system will issue a warning message in advance so that the data team has enough time to take preventive measures.

[0122] 4.2.3 Anomaly detection trigger

[0123] AI anomaly detection algorithms (such as cluster analysis, outlier detection, etc.) continuously monitor data. When abnormal patterns or outliers are found in the data, and these abnormalities may affect data quality and risk assessment, an early warning is triggered. For example, in customer transaction data, if there is a transaction with a huge amount and a transaction pattern that is completely different from the customer's historical transaction pattern, the anomaly detection algorithm will identify this anomaly and trigger an early warning, indicating that there may be data errors or fraud risks.

[0124] According to a second aspect of the present invention, there is provided a risk control data mart data quality monitoring system based on AI technology, comprising:

[0125] Data collection module, used to collect risk control related data from multiple data sources based on machine learning algorithms;

[0126] The data cleaning module is used to clean inaccurate or incomplete risk control data through anomaly detection models;

[0127] The evaluation indicator module is used to compare and verify the known accurate data samples through the isolation forest model and the support vector machine model, and dynamically analyze the risk control data to see if the indicators are normal;

[0128] The early warning module is used to establish a real-time monitoring system based on microservices to continuously monitor the data quality indicators in the data mart. When the data quality indicators deviate from the preset threshold, the system triggers the early warning mechanism and issues a risk warning signal.

[0129] In summary, the embodiment of the present invention provides a data quality monitoring method and system for a risk control data mart based on AI technology, which collects risk control related data from multiple data sources, processes inaccurate or incomplete data, uses data analysis algorithms to dynamically analyze data to see if the indicators are normal, and issues risk warning signals in a timely manner based on the analysis results to improve the timeliness and accuracy of risk management. The present invention brings new opportunities for data quality monitoring, can achieve more efficient, accurate and intelligent monitoring, helps to timely discover data problems, reduce risk exposure, and ensure the sound operation of financial institutions.

[0130] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A data quality monitoring method for risk control data mart based on AI technology, characterized in that: The following steps are involved: Collect risk control data from various data sources based on machine learning algorithms; Clean inaccurate or incomplete risk control data through anomaly detection models; Through the comparison and verification of the isolation forest model and the support vector machine model with known accurate data samples, the risk control data is dynamically analyzed to see whether the indicators are normal; Establish a real-time monitoring system based on microservices to continuously monitor the data quality indicators in the data mart. When the data quality indicators deviate from the preset threshold, the system triggers the early warning mechanism and issues a risk warning signal.

2. According to claim 1, a method for monitoring data quality of risk control data mart based on AI technology is characterized in that: The risk control data collected from various data sources based on machine learning algorithms includes: Use AI-driven intelligent data collection tools to extract data from various data sources and conduct preliminary checks on the integrity and consistency of the data. Use machine learning algorithms to identify whether the data format meets predefined specifications and detect whether there are any fields with missing or duplicate data.

3. According to the AI-based risk control data mart data quality monitoring method of claim 1, it is characterized by: The risk control data collected from various data sources based on machine learning algorithms is carried out by means of intelligent screening and expansion of data collection sources, adaptive data collection interfaces and intelligent web crawlers.

4. The method for monitoring data quality of risk control data mart based on AI technology according to claim 3 is characterized in that: For the incoming data, a dynamic data mapping and conversion mechanism is established, and the AI ​​system automatically adjusts the mapping rules according to the characteristics of the data.

5. The method for monitoring data quality of risk control data mart based on AI technology according to claim 1 is characterized in that: The data cleaning includes: Use anomaly detection models to intelligently identify and process noise, outliers, and erroneous data in the data; Data points are grouped based on cluster analysis algorithms, and data points that deviate from normal clusters are considered outliers and corrected or marked according to business rules and data distribution characteristics; Use natural language processing to clean and standardize text data, extract key information and convert it into structured data.

6. The method for monitoring data quality of risk control data mart based on AI technology according to claim 1 is characterized in that: The isolation forest model and the support vector machine model are compared and verified with known accurate data samples to dynamically analyze whether the risk control data indicators are normal, including: The accuracy index, completeness index, consistency index and timeliness index are analyzed respectively.

7. The method for monitoring data quality of risk control data mart based on AI technology according to claim 6 is characterized in that: The accuracy indicators include data matching rate and data error rate; the integrity indicators include required field missing rate and data record missing rate; the consistency indicators include data logic consistency ratio and data source data consistency rate; the timeliness indicators include data update delay rate and data validity period ratio.

8. The method for monitoring data quality of risk control data mart based on AI technology according to claim 1 is characterized in that: The establishment of a real-time monitoring system based on microservices includes: Build an early warning indicator system, including early warning indicators: data accuracy warning, logical contradiction warning, required field missing warning, data relevance missing warning, data consistency warning, internal data consistency warning, data update delay warning and data validity period warning; Set up early warning trigger mechanisms, including threshold triggers, model prediction triggers, and anomaly detection triggers.

9. The method for monitoring data quality of risk control data mart based on AI technology according to claim 1, characterized in that: The real-time monitoring system based on microservices is established to continuously monitor the data quality indicators in the data mart. When the data quality indicators deviate from the preset threshold, the system triggers the early warning mechanism and issues risk early warning signals, including: The early warning system uses machine learning algorithms to prioritize early warning information and push the most urgent and important early warning information to relevant personnel based on the potential impact of data issues on business risks so that timely measures can be taken to deal with it.

10. A data quality monitoring system for risk control data mart based on AI technology, used in the data quality monitoring system method for risk control data mart based on AI technology according to any one of claims 1 to 9, characterized in that: include: Data collection module, used to collect risk control related data from multiple data sources based on machine learning algorithms; The data cleaning module is used to clean inaccurate or incomplete risk control data through anomaly detection models; The evaluation indicator module is used to compare and verify the known accurate data samples through the isolation forest model and the support vector machine model, and dynamically analyze the risk control data to see if the indicators are normal; The early warning module is used to establish a real-time monitoring system based on microservices to continuously monitor the data quality indicators in the data mart. When the data quality indicators deviate from the preset threshold, the system triggers the early warning mechanism and issues a risk warning signal.

Images