Data quality optimization method and device, computer equipment and storage medium

Abstract

The embodiment of the invention belongs to the field of financial science and technology, and relates to a data quality optimization method and device, computer equipment and a storage medium, and the method comprises the following steps: obtaining banking business data; inputting the bank business data into a pre-constructed multi-dimensional data feature extraction model to obtain a business multi-dimensional feature vector; performing clustering analysis on the service multi-dimensional feature vector to obtain a service data quality monitoring result; acquiring a historical business data quality trend and performing causal relationship test to obtain a data quality causal relationship; performing data quality prediction based on the business multi-dimensional feature vector and the data quality causal relationship to obtain a data quality prediction result; and optimizing and adjusting the bank business data based on the data quality prediction result and the data quality causal relationship. According to the method and the device, the business data with poor data quality in the bank business data can be effectively, quickly and accurately identified, and the identified business data quality is effectively optimized.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, specifically to the field of financial technology, and in particular to a data quality optimization method, device, computer equipment and storage medium. Background Art

[0002] In today's complex and ever-changing financial environment, the scale and complexity of banking systems, as the infrastructure supporting financial services, are increasing day by day, with frequent interactions between systems, huge data flows and intricate links. This highly interconnected and dynamically changing environment makes data quality a key factor in ensuring the safe and efficient operation of banking business. However, with the continuous iteration and upgrading of business systems and the explosive growth of data volume, data quality issues have become increasingly prominent, becoming one of the bottlenecks restricting the development of banking business.

[0003] Data quality issues not only arise from data exchange errors, inconsistent formats, missing or redundant data between systems, but may also be caused by human operational errors, system anomalies or quality issues of external data sources. If these problems are not discovered and corrected in a timely manner, they will directly lead to management's decision-making errors based on incorrect information, which may in turn lead to compliance risks, face severe penalties from regulators, damage the bank's reputation and inhibit business growth potential.

[0004] Currently, although the banking industry has tried to monitor data quality through manual inspection and system text inspection, these methods have obvious limitations. Manual inspection is intuitive but inefficient and costly, and it is difficult to cope with the needs of massive data processing; while text inspection technology based on natural language processing can automate some processes, it is limited by the maturity of technology and the complexity of data, and its accuracy and timeliness are still difficult to meet the strict requirements of banking business for data quality monitoring. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose a data quality optimization method, device, computer equipment and storage medium to solve the problem of being unable to accurately and effectively monitor the data quality of banking business data and optimize it in a timely manner.

[0006] In order to solve the above technical problems, the present application embodiment provides a data quality optimization method, which adopts the following technical solutions:

[0007] Obtain banking data;

[0008] Inputting the banking business data into a pre-built multi-dimensional data feature extraction model to obtain a business multi-dimensional feature vector;

[0009] Performing cluster analysis on the business multi-dimensional feature vectors to obtain business data quality monitoring results;

[0010] Obtaining historical business data quality trends, performing causal relationship testing based on the business data quality monitoring results and the historical business data quality trends, and obtaining data quality causal relationships;

[0011] Building a data quality prediction model based on the business multi-dimensional feature vector and the data quality causal relationship, and inputting the banking business data into the data quality prediction model to obtain a data quality prediction result;

[0012] A data quality optimization plan is formulated based on the data quality prediction results and the data quality causal relationship, and the banking business data is optimized and adjusted according to the data quality optimization plan.

[0013] Furthermore, before the step of inputting the banking business data into a pre-built multi-dimensional data feature extraction model to obtain a business multi-dimensional feature vector, the following steps are also included:

[0014] Obtaining multi-dimensional information on banking services, wherein the multi-dimensional information on banking services includes the source of business data, the flow path of business data, the processing process of business data, and the usage of business data;

[0015] Perform data lineage analysis based on the business data source, the business data flow path, the business data processing process, and the business data usage to obtain the business data lineage relationship;

[0016] A business multi-dimensional feature vector is defined, and the business multi-dimensional feature vector is quantified into a feature index, and the multi-dimensional data feature extraction model is constructed according to the business data lineage relationship and the feature index.

[0017] Furthermore, the step of performing cluster analysis on the business multi-dimensional feature vector to obtain the business data quality monitoring result specifically includes:

[0018] Performing dimensionality reduction processing on the multi-dimensional feature vector of the business to obtain key feature dimensions of the business;

[0019] Clustering the business key feature dimensions according to a preset clustering algorithm to obtain business data clusters;

[0020] Performing cluster distribution analysis based on the business data clustering to obtain a business data clustering model;

[0021] Data quality deviation calculation is performed according to the business data clustering and the business data clustering pattern to obtain the business data quality monitoring result.

[0022] Furthermore, the step of performing data quality deviation calculation according to the business data clustering and the business data clustering pattern to obtain the business data quality monitoring result specifically includes:

[0023] Identifying a data cluster center of the business data cluster;

[0024] Obtaining a corresponding clustering distance threshold according to the business data clustering mode;

[0025] Calculating the Euclidean distance between the business multi-dimensional feature vector and the data cluster center to obtain a feature deviation value;

[0026] Determine whether the feature deviation value is greater than or equal to the clustering distance threshold;

[0027] If the feature deviation value is greater than or equal to the cluster distance threshold, business data quality abnormality information is generated;

[0028] If the feature deviation value is less than the cluster distance threshold, generating normal service data quality information;

[0029] The abnormal business data quality information and / or the normal business data quality information are used as the business data quality monitoring result.

[0030] Furthermore, the step of obtaining the historical business data quality trend, performing a causal relationship test based on the business data quality monitoring result and the historical business data quality trend, and obtaining the data quality causal relationship specifically includes:

[0031] Acquire a historical data information extraction identifier, and extract the historical business data quality trend from a database according to the historical data information extraction identifier;

[0032] Constructing a time series according to the business data quality monitoring result and the historical business data quality trend to obtain a time series data set;

[0033] A causal relationship test is performed according to a preset causal test method and the time series data set to obtain the data quality causal relationship.

[0034] Furthermore, the step of constructing a data quality prediction model based on the business multi-dimensional feature vector and the data quality causal relationship specifically includes:

[0035] Identify the change trend of the business multi-dimensional feature vector based on the data quality causal relationship to obtain business data change information;

[0036] Performing data correlation analysis based on the data quality causal relationship and the business multi-dimensional feature vector to obtain business data correlation information;

[0037] The business data change information and the business data correlation information are used as independent variables to construct a model to obtain the data quality prediction model.

[0038] Furthermore, the step of formulating a data quality optimization plan based on the data quality prediction result and the data quality causal relationship specifically includes:

[0039] Determining the business data to be optimized according to the data quality prediction result and the data quality causal relationship;

[0040] Performing correlation analysis and principal component analysis on the business data to be optimized to obtain analysis results of the data to be optimized;

[0041] Obtain a preset optimization solution table, map the analysis results of the data to be optimized to the preset optimization solution table, extract corresponding values, and obtain the data quality optimization solution.

[0042] In order to solve the above technical problems, the embodiment of the present application further provides a data quality optimization device, which adopts the following technical solution:

[0043] Data acquisition module, used to acquire banking business data;

[0044] A feature acquisition module, used to input the banking business data into a pre-built multi-dimensional data feature extraction model to obtain a business multi-dimensional feature vector;

[0045] A cluster analysis module, used to perform cluster analysis on the multi-dimensional feature vectors of the business to obtain business data quality monitoring results;

[0046] A causal verification module is used to obtain the quality trend of historical business data, perform causal relationship verification based on the business data quality monitoring result and the historical business data quality trend, and obtain the data quality causal relationship;

[0047] A data prediction module, used to construct a data quality prediction model based on the business multi-dimensional feature vector and the data quality causal relationship, and input the banking business data into the data quality prediction model to obtain a data quality prediction result;

[0048] The data optimization module is used to formulate a data quality optimization plan based on the data quality prediction results and the data quality causal relationship, and optimize and adjust the banking business data according to the data quality optimization plan.

[0049] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0050] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of any of the above data quality optimization methods when executing the computer-readable instructions.

[0051] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0052] A computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the data quality optimization method described in any one of the above items are implemented.

[0053] Compared with the prior art, the embodiment of the present application has the following beneficial effects: the embodiment obtains banking business data; inputs the banking business data into a pre-built multi-dimensional data feature extraction model to obtain a business multi-dimensional feature vector; performs cluster analysis on the business multi-dimensional feature vector to obtain a business data quality monitoring result; obtains the historical business data quality trend, performs a causal relationship test based on the business data quality monitoring result and the historical business data quality trend to obtain a data quality causal relationship; constructs a data quality prediction model based on the business multi-dimensional feature vector and the data quality causal relationship, and inputs the banking business data into the data quality prediction model to obtain a data quality prediction result; formulates a data quality optimization plan based on the data quality prediction result and the data quality causal relationship, and optimizes and adjusts the banking business data according to the data quality optimization plan. This effectively realizes the rapid and accurate identification of business data with poor data quality in banking business data, and effectively optimizes the quality of the identified business data. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0056] Figure 2 A flowchart of an embodiment of a data quality optimization method according to the present application;

[0057] Figure 3 yes Figure 2 A flowchart of a specific implementation of step S30;

[0058] Figure 4 yes Figure 3 A flowchart of a specific implementation of step S304;

[0059] Figure 5 yes Figure 2 A flowchart of a specific implementation of step S40;

[0060] Figure 6 yes Figure 2 A flowchart of a specific implementation of step S50;

[0061] Figure 7 yes Figure 2 A flowchart of a specific implementation of step S60;

[0062] Figure 8 is a structural schematic diagram of an embodiment of a data quality optimization device according to the present application;

[0063] Fig. 9 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0065] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor does it refer to non-related or alternative embodiments that are mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0066] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0067] like Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0068] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0069] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0070] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0071] It should be noted that the data quality optimization method provided in the embodiment of the present application is generally executed by a server, and accordingly, the data quality optimization device is generally arranged in the server.

[0072] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0073] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for optimizing data quality according to the present application. The data quality optimization method comprises the following steps:

[0074] Step S10, obtaining banking business data;

[0075] In this embodiment, the banking business data includes customer data, transaction data, internal management data, external data, etc., wherein customer data includes personal customer data: basic information, occupational information, family information, account information; corporate customer data: institutional information, financial information, and operating information. Transaction data includes account transaction data, loan transaction data, payment transaction data, financial transaction data, and credit card transaction data. Internal management data includes operational management data, risk management data, and financial data. External data includes macroeconomic data, industry data, external credit data, and social data. Banking business data corresponds to a banking business data extraction identifier, and information is extracted from the database according to the banking business data extraction identifier to obtain banking business data.

[0076] Step S20, inputting the banking business data into a pre-built multi-dimensional data feature extraction model to obtain a business multi-dimensional feature vector;

[0077] In this embodiment, the pre-built multi-dimensional data feature model adopts a random forest model, and the business multi-dimensional feature vector includes customer feature vectors (customer basic information, customer transaction behavior, customer loyalty), product feature vectors (business product types, business product sales status), etc.

[0078] Step S30, performing cluster analysis on the business multi-dimensional feature vector to obtain a business data quality monitoring result;

[0079] In this embodiment, the clustering analysis of the business multi-dimensional feature vector includes data clustering, cluster distribution analysis and data quality deviation calculation, wherein the data clustering is performed based on the K-means clustering algorithm, and the data clustering steps include obtaining the business multi-dimensional feature vector, setting the initial parameters of the K-means algorithm, including the value range of the number of clusters k, the selection method of the initial cluster center, the number of iterations, etc.; for each k value, randomly select k data points as the initial cluster center, and then assign each data point to the nearest cluster center to form an initial clustering result; according to the current initial clustering result, recalculate the center of each cluster, and redistribute the data points to the nearest new cluster center to update the clustering result; repeat the above steps until the clustering result converges or reaches the maximum number of iterations, and obtain the final clustering result under the current k value, and use the final clustering result as the business data clustering. The steps of cluster distribution analysis include counting the data quality distribution information of each business data cluster to obtain data quality distribution information; counting the data occurrence frequency of each business data cluster to obtain data distribution frequency information; associating the data quality distribution information with the data distribution frequency information to obtain the business data clustering pattern. The data quality deviation calculation is based on the business data clustering and the business data clustering pattern, and is obtained by calculating the deviation value of the business data in the corresponding cluster from the data cluster center.

[0080] Step S40, obtaining the quality trend of historical business data, performing a causal relationship test based on the business data quality monitoring result and the quality trend of historical business data, and obtaining a data quality causal relationship;

[0081] In this embodiment, the historical business data quality trend is extracted from the historical business data quality monitoring record. In this embodiment, the historical data corresponding to the historical business data quality trend is within one year before the current time. The corresponding historical business data quality trend is extracted from the historical business data quality monitoring record according to this time period. Causality test refers to the process of determining the possible causal relationship by analyzing the relationship between events, conditions or variables. In this embodiment, the causal correspondence between the business data quality monitoring results of the current banking business data and the historical business data quality trend of the historical banking business data is obtained through causal test to obtain the data quality causal relationship.

[0082] Step S50, constructing a data quality prediction model based on the business multi-dimensional feature vector and the data quality causal relationship, and inputting the banking business data into the data quality prediction model to obtain a data quality prediction result;

[0083] In this embodiment, the data quality prediction model is used to predict the data quality of banking business data in the future. The data quality prediction model adopts a multiple regression model. The multiple regression model is a statistical method used to study the linear relationship between a dependent variable and two or more independent variables. In this embodiment, the multiple regression model is used to analyze the relationship between the multi-dimensional feature vector of the business and the data quality to obtain the factors affecting the quality of banking business data, and predict the data quality of banking business in the future based on the factors to obtain the data quality prediction results.

[0084] Step S60, formulating a data quality optimization plan based on the data quality prediction result and the data quality causal relationship, and optimizing and adjusting the banking business data according to the data quality optimization plan.

[0085] In this embodiment, the data quality optimization plan includes optimizing data sources, adjusting data processing procedures, updating data usage specifications, etc., wherein optimizing data sources includes data source evaluation, data source integration, data source cleaning, and data source monitoring; adjusting data processing procedures includes process standardization, automated processing, exception handling, process monitoring and optimization; and updating data usage specifications includes formulating data usage specifications, regular review, and updating.

[0086] This embodiment obtains banking business data; inputs the banking business data into a pre-built multi-dimensional data feature extraction model to obtain a business multi-dimensional feature vector; performs cluster analysis on the business multi-dimensional feature vector to obtain a business data quality monitoring result; obtains historical business data quality trends, performs causal relationship tests based on the business data quality monitoring results and the historical business data quality trends to obtain data quality causal relationships; constructs a data quality prediction model based on the business multi-dimensional feature vector and the data quality causal relationship, and inputs the banking business data into the data quality prediction model to obtain data quality prediction results; formulates a data quality optimization plan based on the data quality prediction results and the data quality causal relationship, and optimizes and adjusts the banking business data according to the data quality optimization plan. This effectively realizes the rapid and accurate identification of business data with poor data quality in banking business data, and effectively optimizes the quality of the identified business data.

[0087] In some optional implementations of this embodiment, before step S10, the following steps are further included:

[0088] Obtaining multi-dimensional information on banking services, wherein the multi-dimensional information on banking services includes the source of business data, the flow path of business data, the processing process of business data, and the usage of business data;

[0089] In this embodiment, the sources of business data include internal systems: such as core banking systems, customer relationship management systems (CRM), enterprise resource planning systems (ERP), etc., external data sources: market data, credit rating agencies, third-party service providers, etc., and user input: data provided by customers through online banking, mobile banking, counters and other channels. The business data flow path refers to identifying the complete path of data from the source system to the target system or application, including data extraction, transformation, loading (ETL) processes, intermediate storage (such as data warehouses, data lakes), and the final data consumer. The business data processing process refers to a detailed description of the various processing steps that the data undergoes during the flow process, such as data cleaning, deduplication, conversion, aggregation, encryption, etc. The business data usage refers to analyzing which business processes, reports, analysis tools or decision support systems use the data, as well as the frequency and method of use.

[0090] Perform data lineage analysis based on the business data source, the business data flow path, the business data processing process, and the business data usage to obtain the business data lineage relationship;

[0091] In this embodiment, data lineage analysis can establish a relationship diagram between data items based on the above-mentioned business data source, business data flow path, business data processing process, and business data usage, and the relationship diagram shows the source, flow, dependency, and impact range of the data. A visual relationship diagram can be generated through data management tools, ETL tools, or special lineage analysis tools.

[0092] A business multi-dimensional feature vector is defined, and the business multi-dimensional feature vector is quantified into a feature index, and the multi-dimensional data feature extraction model is constructed according to the business data lineage relationship and the feature index.

[0093] In this embodiment, a series of business multi-dimensional feature vectors reflecting data quality are defined according to the characteristics of banking business. Among them, the business multi-dimensional feature vectors include customer feature vectors (basic customer information, customer transaction behavior, customer loyalty), product feature vectors (business product types, business sales), etc. Feature indicators are indicator values ​​quantified according to the business multi-dimensional feature vectors. Feature indicators include customer basic information indicators: age distribution, gender ratio, etc.; customer transaction behavior indicators: number of transactions, average transaction amount, RFM model (last purchase time, purchase frequency, purchase amount), etc.; customer loyalty indicators: repurchase rate, CLV (customer life cycle value), etc.; business product indicators: business product average score, business product praise rate; business sales indicators: monthly / annual sales, market share, etc.

[0094] This embodiment obtains multi-dimensional information of banking business, wherein the multi-dimensional information of banking business includes the source of business data, the flow path of business data, the processing process of business data, and the usage of business data; performs data lineage analysis based on the source of business data, the flow path of business data, the processing process of business data, and the usage of business data to obtain the lineage relationship of business data; defines a multi-dimensional feature vector of business, and quantifies the multi-dimensional feature vector of business into a feature index, and constructs the multi-dimensional data feature extraction model based on the lineage relationship of business data and the feature index. Thus, a multi-dimensional data feature extraction model is effectively constructed based on the multi-dimensional information of banking business, so as to facilitate the subsequent feature extraction of banking business data.

[0095] Continue to refer Figure 3 In some optional implementations of this embodiment, step S30 includes the following steps:

[0096] Step S301, performing dimensionality reduction processing on the multi-dimensional feature vector of the business to obtain key feature dimensions of the business;

[0097] In this embodiment, dimensionality reduction is performed by performing principal component analysis on the business multi-dimensional feature vector, the first few principal components with the largest eigenvalues ​​in the business multi-dimensional feature vector are selected, and the contribution rate of each principal component (eigenvalue divided by the sum of all eigenvalues) is calculated to determine how many principal components to select, and the selected principal components (i.e., the corresponding business multi-dimensional feature vector) are used as the new basis to convert the original data into this new feature space, thereby obtaining the business key feature dimensions.

[0098] Step S302, clustering the business key feature dimensions according to a preset clustering algorithm to obtain business data clusters;

[0099] In this embodiment, the preset clustering algorithm is the K-means clustering algorithm, and the data clustering steps include obtaining a multi-dimensional feature vector of the business, setting the initial parameters of the K-means algorithm, including the value range of the number of clusters k, the selection method of the initial cluster center, the number of iterations, etc.; for each k value, randomly select k data points as the initial cluster center, and then assign each data point to the nearest cluster center to form an initial clustering result; according to the current initial clustering result, recalculate the center of each cluster, and redistribute the data points to the nearest new cluster center, and update the clustering result; repeat the above steps until the clustering result converges or the maximum number of iterations is reached, and obtain the final clustering result under the current k value, and use the final clustering result as the business data clustering.

[0100] Step S303, performing cluster distribution analysis according to the business data clustering to obtain a business data clustering model;

[0101] In this embodiment, the cluster distribution analysis step includes counting the data quality distribution information of each business data cluster to obtain data quality distribution information; counting the data occurrence frequency of each business data cluster to obtain data distribution frequency information; and associating the data quality distribution information with the data distribution frequency information to obtain a business data clustering pattern.

[0102] Step S304: performing data quality deviation calculation according to the business data clustering and the business data clustering pattern to obtain the business data quality monitoring result.

[0103] In this embodiment, data quality deviation calculation is based on calculating the Euclidean distance between the multi-dimensional feature vector of the business and the data clustering center corresponding to the data clustering pattern. The calculated feature deviation value is used to represent the deviation between the business data and the data clustering result, and the feature deviation value is divided according to the clustering distance threshold to obtain the business data quality monitoring result.

[0104] This embodiment obtains business key feature dimensions by performing dimensionality reduction processing on the multi-dimensional feature vector of the business; clusters the business key feature dimensions according to a preset clustering algorithm to obtain business data clusters; performs clustering distribution analysis based on the business data clusters to obtain a business data clustering pattern; performs data quality deviation calculation based on the business data clusters and the business data clustering pattern, thereby effectively obtaining business data quality monitoring results that accurately represent the bank business data, so as to facilitate subsequent causal relationship verification processing.

[0105] Continue to refer Figure 4 In some optional implementations of this embodiment, step S304 includes the following steps:

[0106] Step S3041, identifying the data cluster center of the business data cluster;

[0107] In this embodiment, after the clustering of the business key feature dimensions is completed, each business data cluster obtained will have a data clustering center, which represents the "average" or "center" position of the data within the cluster. The data clustering center is generated during clustering and is automatically calculated by a preset clustering algorithm.

[0108] Step S3042, obtaining a corresponding clustering distance threshold according to the business data clustering mode;

[0109] In this embodiment, the cluster distance threshold refers to a value used to determine whether a data point in a business data cluster deviates too far from the center of the data cluster and generates anomalies. The cluster distance threshold is a preset value and can be set based on the tightness of historical business data clustering, business needs or domain knowledge.

[0110] Step S3043, calculating the Euclidean distance between the business multi-dimensional feature vector and the data cluster center to obtain a feature deviation value.

[0111] In this embodiment, Euclidean distance calculation is a commonly used distance measurement method, which is used to calculate the straight-line distance between two points in a multidimensional space. The dimension difference is obtained by calculating the difference between the business multidimensional feature vector and the data cluster center in each dimension, and then the dimension difference is squared. The squared dimension differences are added to obtain the total dimension difference, and the square root of the total dimension difference is taken to obtain the Euclidean distance between the business multidimensional feature vector and the data cluster center, and the Euclidean distance is used as the feature deviation value.

[0112] Step S3044, determining whether the feature deviation value is greater than or equal to the clustering distance threshold;

[0113] In this embodiment, the determination step is completed by comparing the feature deviation value with the cluster distance threshold.

[0114] Step S3045: if the feature deviation value is greater than or equal to the cluster distance threshold, generating business data quality abnormality information;

[0115] In this embodiment, data quality anomaly information refers to that banking business data does not meet expected data quality requirements, and the data quality anomaly information corresponds to the comparison result in the above judgment step that the feature deviation value is greater than or equal to the cluster distance threshold.

[0116] Step S3046: if the feature deviation value is less than the cluster distance threshold, generating normal service data quality information;

[0117] In this embodiment, the normal data quality information refers to that the banking business data meets the expected data quality requirements, and the normal data quality information corresponds to the comparison result in the above judgment step that the feature deviation value is less than the cluster distance threshold.

[0118] Step S3047: taking the abnormal business data quality information and / or the normal business data quality information as the business data quality monitoring result.

[0119] In this embodiment, for banking business data, part of the judgment results may belong to abnormal data quality information and part to normal data quality information. The abnormal data quality information and the normal data quality information are summarized to obtain the business data quality monitoring results corresponding to the banking business data.

[0120] This embodiment identifies the data cluster center of the business data cluster; obtains the corresponding cluster distance threshold according to the business data clustering mode; calculates the Euclidean distance between the business multi-dimensional feature vector and the data cluster center to obtain a feature deviation value; determines whether the feature deviation value is greater than or equal to the cluster distance threshold; if the feature deviation value is greater than or equal to the cluster distance threshold, generates business data quality abnormality information; if the feature deviation value is less than the cluster distance threshold, generates business data quality normal information; uses the business data quality abnormality information and / or the business data quality normal information as the business data quality monitoring result. Thus, the business data quality monitoring result corresponding to the bank business data is effectively determined through business data clustering and the preset cluster distance threshold, so as to facilitate the subsequent causal relationship verification step.

[0121] Continue to refer Figure 5 In some optional implementations of this embodiment, step S40 includes the following steps:

[0122] Step S401, obtaining a historical data information extraction identifier, and extracting the historical business data quality trend in a database according to the historical data information extraction identifier;

[0123] In this embodiment, the historical data information extraction identifier is a unique data identifier corresponding to the historical business data quality trend, and a matching query can be performed in the database through the historical data information extraction identifier to obtain the historical business data quality trend. The historical business data quality trend is visual information obtained by statistical analysis of the quality corresponding to the historical banking business data, including the change trend of indicators such as data error rate and missing rate.

[0124] Step S402, constructing a time series according to the business data quality monitoring result and the historical business data quality trend to obtain a time series data set;

[0125] In this embodiment, the business data quality monitoring results obtained by real-time monitoring and the historical business data quality trends are sorted in chronological order to ensure that the timestamps of the sorted business data quality monitoring results and the historical business data quality trends are continuous, and the missing time points are appropriately interpolated or marked, thereby ultimately obtaining a time series data set containing timestamps, data quality indicators (such as error rate, missing rate, etc.) and other related variables (such as business type, data source, etc.).

[0126] Step S403, performing a causal relationship test on the time series data set according to a preset causal relationship test method to obtain the data quality causal relationship.

[0127] In this embodiment, the preset causal test method adopts the Granger causality test method, which performs a causal relationship test on the time series data set corresponding to the business data quality monitoring results and the time series data set corresponding to the historical business data quality trend, and then calculates the F statistic and p-value to determine whether the two time series data sets are the Granger causes of the data quality change, and then generates the corresponding data quality causal relationship based on the calculated Granger cause.

[0128] This embodiment obtains a historical data information extraction identifier, extracts the historical business data quality trend in the database according to the historical data information extraction identifier; constructs a time series according to the business data quality monitoring result and the historical business data quality trend to obtain a time series data set; performs a causal relationship test according to a preset causal test method and the time series data set, thereby effectively obtaining the data quality causal relationship between the current business data and the historical business data trend, so as to facilitate the subsequent construction of a data quality prediction model.

[0129] Continue to refer Figure 6 In some optional implementations of this embodiment, step S50 includes the following steps:

[0130] Step S501, identifying the change trend of the business multi-dimensional feature vector based on the data quality causal relationship to obtain business data change information;

[0131] In this embodiment, the changing trend of the multi-dimensional feature vector of the business is identified according to the time series analysis method (such as moving average, exponential smoothing, ARIMA, etc.), and then the data quality causal relationship is combined to analyze which feature vectors' changing trends may be directly or indirectly affected by the data quality problem, so as to obtain the business data change information.

[0132] Step S502, performing data correlation analysis according to the business multi-dimensional feature vector to obtain business data correlation information;

[0133] In this embodiment, the correlation between the multi-dimensional feature vectors of the business is calculated according to the Pearson correlation coefficient, so as to obtain the correlation information of the business data, wherein the Pearson correlation coefficient (Pearson correlation coefficient), also known as the Pearson product-moment correlation coefficient, is an indicator to measure the linear correlation degree of two variables X and Y. Its value is between -1 and 1, where 1 indicates a complete positive correlation, -1 indicates a complete negative correlation, and 0 indicates no linear correlation.

[0134] Step S503: constructing a model using the business data change information and the business data correlation information as independent variables to obtain the data quality prediction model.

[0135] In this embodiment, the ARIMA model is used to build the model, and the steps of model building include: preprocessing the business data change information and the business data correlation information, including missing value processing and outlier processing; according to the preprocessed business data change information and the business data correlation information, the order (p, d, q) of the ARIMA model is determined by using the autocorrelation and partial autocorrelation analysis methods, and the order is realized by observing the autocorrelation diagram (ACF) and the partial autocorrelation diagram (PACF), wherein the ACF reflects the correlation between the sequence and its lagged value, and the PACF reflects the direct correlation between the sequence and the lagged value, and then the maximum likelihood estimation method is used to estimate the parameters of the ARIMA model under different p, d, q combinations, and the model is built with the order and parameters of the above model, thereby obtaining a data quality prediction model.

[0136] This embodiment obtains business data change information by identifying the change trend of the business multi-dimensional feature vector based on the data quality causal relationship; performs data correlation analysis based on the business multi-dimensional feature vector to obtain business data correlation information; and constructs a model using the business data change information and the business data correlation information as independent variables, thereby effectively obtaining a data quality prediction model that can accurately predict changes in banking business data quality in the future, so as to facilitate subsequent data quality optimization processing.

[0137] Continue to refer Figure 7 In some optional implementations of this embodiment, step S60 includes the following steps:

[0138] Step S601, determining the business data to be optimized according to the data quality prediction result and the data quality causal relationship;

[0139] In this embodiment, business data with low predicted data quality is identified based on the data quality prediction results, and then combined with the data quality causal relationship analysis, the specific factors causing the quality problems of the business data are further confirmed, and the determined specific factors are combined with the business data with low predicted data quality to obtain the business data to be optimized.

[0140] Step S602, performing correlation analysis and principal component analysis on the business data to be optimized to obtain analysis results of the data to be optimized;

[0141] In this embodiment, correlation analysis is performed on the determined business data to be optimized to reveal the degree of association between different data items, so as to understand the propagation path and potential impact range of data quality problems. In this embodiment, correlation analysis can be achieved by calculating correlation coefficients (such as Pearson correlation coefficients), and the linear relationship between data items is effectively quantified by calculating correlation coefficients. On the basis of correlation analysis, principal component analysis (PCA) is further applied to reduce the dimension of the data while retaining the main information in the data. PCA converts the original data into a new coordinate system through linear transformation, so that the data variance on the new coordinate axis (i.e., the principal component) is maximized to identify key variables and potential data structures in the data.

[0142] Step S603, obtaining a preset optimization solution table, mapping the analysis results of the data to be optimized to the preset optimization solution table, extracting corresponding values, and obtaining the data quality optimization solution.

[0143] In this embodiment, the preset optimization solution table is a pre-established information table containing a variety of data quality problems and their corresponding optimization measures, and the preset optimization solution table can be formulated based on past experience, industry best practices, and theoretical knowledge of data quality management. By mapping the analysis results of the data to be optimized (including the correlation analysis results and the principal component analysis results) to the preset optimization solution table to match the type, scope of influence, and potential causes of the data quality problem, and extracting the corresponding optimization measures from the table, a data quality optimization solution is obtained.

[0144] This embodiment determines the business data to be optimized based on the data quality prediction results and the data quality causal relationship; performs correlation analysis and principal component analysis on the business data to be optimized to obtain analysis results of the data to be optimized; obtains a preset optimization plan table, maps the analysis results of the data to be optimized to the preset optimization plan table to extract corresponding values, thereby effectively obtaining a data quality optimization plan corresponding to the data quality prediction results and the data quality causal relationship, so as to effectively optimize the current banking business data and its processing flow.

[0145] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0146] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0147] Further references Figure 8 , as a response to the above Figure 1 In order to realize the method shown in the figure, the present application provides an embodiment of a data quality optimization device, and the device embodiment is Figure 1Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0148] like Figure 8 As shown, the data quality optimization device 700 described in this embodiment includes: a data acquisition module 701, a feature acquisition module 702, a cluster analysis module 703, a causal verification module 704, a data prediction module 705, and a data optimization module 706. Among them:

[0149] Data acquisition module 701, used to acquire banking business data;

[0150] A feature acquisition module 702 is used to input the banking business data into a pre-built multi-dimensional data feature extraction model to obtain a business multi-dimensional feature vector;

[0151] The cluster analysis module 703 is used to perform cluster analysis on the multi-dimensional feature vector of the business to obtain the business data quality monitoring result;

[0152] A causal relationship verification module 704 is used to obtain the quality trend of historical business data, perform causal relationship verification based on the business data quality monitoring result and the historical business data quality trend, and obtain the data quality causal relationship;

[0153] A data prediction module 705 is used to construct a data quality prediction model based on the business multi-dimensional feature vector and the data quality causal relationship, and input the banking business data into the data quality prediction model to obtain a data quality prediction result;

[0154] The data optimization module 706 is used to formulate a data quality optimization plan based on the data quality prediction results and the data quality causal relationship, and optimize and adjust the banking business data according to the data quality optimization plan.

[0155] This embodiment can obtain banking business data by adopting the above-mentioned data quality optimization device; input the banking business data into a pre-built multi-dimensional data feature extraction model to obtain a business multi-dimensional feature vector; perform cluster analysis on the business multi-dimensional feature vector to obtain a business data quality monitoring result; obtain historical business data quality trends, perform causal relationship tests based on the business data quality monitoring results and the historical business data quality trends to obtain data quality causal relationships; construct a data quality prediction model based on the business multi-dimensional feature vector and the data quality causal relationship, and input the banking business data into the data quality prediction model to obtain data quality prediction results; formulate a data quality optimization plan based on the data quality prediction results and the data quality causal relationship, and optimize and adjust the banking business data according to the data quality optimization plan. This effectively realizes the rapid and accurate identification of business data with poor data quality in banking business data, and effectively optimizes the quality of the identified business data.

[0156] To solve the above technical problems, the present application also provides a computer device. Fig. 9 , Fig. 9 This is a basic structural block diagram of the computer device in this embodiment.

[0157] The computer device 8 includes a memory 81, a processor 82, and a network interface 83 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 8 with components 81-83, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0158] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0159] The memory 81 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 81 can be an internal storage unit of the computer device 8, such as a hard disk or memory of the computer device 8. In other embodiments, the memory 81 can also be an external storage device of the computer device 8, such as a plug-in hard disk equipped on the computer device 8, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 81 can also include both the internal storage unit of the computer device 8 and its external storage device. In this embodiment, the memory 81 is generally used to store the operating system and various application software installed on the computer device 8, such as computer-readable instructions of the data quality optimization method, etc. In addition, the memory 81 can also be used to temporarily store various types of data that have been output or are to be output.

[0160] The processor 82 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 82 is generally used to control the overall operation of the computer device 8. In this embodiment, the processor 82 is used to run computer-readable instructions stored in the memory 81 or process data, such as computer-readable instructions for running the data quality optimization method.

[0161] The network interface 83 may include a wireless network interface or a wired network interface. The network interface 83 is generally used to establish a communication connection between the computer device 8 and other electronic devices.

[0162] By adopting the above-mentioned computer device, this embodiment can obtain banking business data; input the banking business data into a pre-built multi-dimensional data feature extraction model to obtain a business multi-dimensional feature vector; perform cluster analysis on the business multi-dimensional feature vector to obtain a business data quality monitoring result; obtain the historical business data quality trend, perform a causal relationship test based on the business data quality monitoring result and the historical business data quality trend to obtain a data quality causal relationship; construct a data quality prediction model based on the business multi-dimensional feature vector and the data quality causal relationship, and input the banking business data into the data quality prediction model to obtain a data quality prediction result; formulate a data quality optimization plan based on the data quality prediction result and the data quality causal relationship, and optimize and adjust the banking business data according to the data quality optimization plan. This effectively realizes the rapid and accurate identification of business data with poor data quality in banking business data, and effectively optimizes the quality of the identified business data.

[0163] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the data quality optimization method as described above.

[0164] By adopting the above-mentioned computer-readable storage medium, this embodiment can obtain banking business data; input the banking business data into a pre-built multi-dimensional data feature extraction model to obtain a business multi-dimensional feature vector; perform cluster analysis on the business multi-dimensional feature vector to obtain a business data quality monitoring result; obtain historical business data quality trends, perform causal relationship tests based on the business data quality monitoring results and the historical business data quality trends, and obtain data quality causal relationships; construct a data quality prediction model based on the business multi-dimensional feature vector and the data quality causal relationship, and input the banking business data into the data quality prediction model to obtain data quality prediction results; formulate a data quality optimization plan based on the data quality prediction results and the data quality causal relationship, and optimize and adjust the banking business data according to the data quality optimization plan. This effectively realizes the rapid and accurate identification of business data with poor data quality in banking business data, and effectively optimizes the quality of the identified business data.

[0165] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0166] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A data quality optimization method, characterized in that: The steps include: Obtain banking data; Inputting the banking business data into a pre-built multi-dimensional data feature extraction model to obtain a business multi-dimensional feature vector; Performing cluster analysis on the business multi-dimensional feature vectors to obtain business data quality monitoring results; Obtaining historical business data quality trends, performing causal relationship testing based on the business data quality monitoring results and the historical business data quality trends, and obtaining data quality causal relationships; Building a data quality prediction model based on the business multi-dimensional feature vector and the data quality causal relationship, and inputting the banking business data into the data quality prediction model to obtain a data quality prediction result; A data quality optimization plan is formulated based on the data quality prediction results and the data quality causal relationship, and the banking business data is optimized and adjusted according to the data quality optimization plan.

2. The data quality optimization method according to claim 1, characterized in that: Before the step of inputting the banking business data into a pre-built multi-dimensional data feature extraction model to obtain a business multi-dimensional feature vector, the following steps are also included: Obtaining multi-dimensional information on banking services, wherein the multi-dimensional information on banking services includes the source of business data, the flow path of business data, the processing process of business data, and the usage of business data; Perform data lineage analysis based on the business data source, the business data flow path, the business data processing process, and the business data usage to obtain the business data lineage relationship; A business multi-dimensional feature vector is defined, and the business multi-dimensional feature vector is quantified into a feature index, and the multi-dimensional data feature extraction model is constructed according to the business data lineage relationship and the feature index.

3. The data quality optimization method according to claim 1, characterized in that: The step of performing cluster analysis on the business multi-dimensional feature vector to obtain the business data quality monitoring result specifically includes: Performing dimensionality reduction processing on the multi-dimensional feature vector of the business to obtain key feature dimensions of the business; Clustering the business key feature dimensions according to a preset clustering algorithm to obtain business data clusters; Performing cluster distribution analysis based on the business data clustering to obtain a business data clustering model; Data quality deviation calculation is performed according to the business data clustering and the business data clustering pattern to obtain the business data quality monitoring result.

4. The data quality optimization method according to claim 3, characterized in that: The step of performing data quality deviation calculation according to the business data clustering and the business data clustering pattern to obtain the business data quality monitoring result specifically includes: Identifying a data cluster center of the business data cluster; Obtaining a corresponding clustering distance threshold according to the business data clustering mode; Calculating the Euclidean distance between the business multi-dimensional feature vector and the data cluster center to obtain a feature deviation value; Determine whether the feature deviation value is greater than or equal to the clustering distance threshold; If the feature deviation value is greater than or equal to the cluster distance threshold, business data quality abnormality information is generated; If the feature deviation value is less than the cluster distance threshold, generating normal service data quality information; The abnormal business data quality information and / or the normal business data quality information are used as the business data quality monitoring result.

5. The data quality optimization method according to claim 1, characterized in that: The step of obtaining the historical business data quality trend, performing a causal relationship test based on the business data quality monitoring result and the historical business data quality trend, and obtaining the data quality causal relationship specifically includes: Acquire a historical data information extraction identifier, and extract the historical business data quality trend from a database according to the historical data information extraction identifier; Constructing a time series according to the business data quality monitoring result and the historical business data quality trend to obtain a time series data set; A causal relationship test is performed according to a preset causal test method and the time series data set to obtain the data quality causal relationship.

6. The data quality optimization method according to claim 1, characterized in that: The step of constructing a data quality prediction model based on the business multi-dimensional feature vector and the data quality causal relationship specifically includes: Identify the change trend of the business multi-dimensional feature vector based on the data quality causal relationship to obtain business data change information; Performing data correlation analysis based on the business multi-dimensional feature vector to obtain business data correlation information; The business data change information and the business data correlation information are used as independent variables to construct a model to obtain the data quality prediction model.

7. The data quality optimization method according to claim 1, characterized in that: The step of formulating a data quality optimization plan based on the data quality prediction result and the data quality causal relationship specifically includes: Determining the business data to be optimized according to the data quality prediction result and the data quality causal relationship; Performing correlation analysis and principal component analysis on the business data to be optimized to obtain analysis results of the data to be optimized; Obtain a preset optimization solution table, map the analysis results of the data to be optimized to the preset optimization solution table, extract corresponding values, and obtain the data quality optimization solution.

8. A data quality optimization device, characterized in that: include: Data acquisition module, used to acquire banking business data; A feature acquisition module, used to input the banking business data into a pre-built multi-dimensional data feature extraction model to obtain a business multi-dimensional feature vector; A cluster analysis module, used to perform cluster analysis on the multi-dimensional feature vectors of the business to obtain business data quality monitoring results; A causal verification module is used to obtain the quality trend of historical business data, perform causal relationship verification based on the business data quality monitoring result and the historical business data quality trend, and obtain the data quality causal relationship; A data prediction module, used to construct a data quality prediction model based on the business multi-dimensional feature vector and the data quality causal relationship, and input the banking business data into the data quality prediction model to obtain a data quality prediction result; The data optimization module is used to formulate a data quality optimization plan based on the data quality prediction results and the data quality causal relationship, and optimize and adjust the banking business data according to the data quality optimization plan.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the data quality optimization method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the data quality optimization method according to any one of claims 1 to 7 are implemented.

Images