Data cleaning method and system in bank big data environment

Abstract

The invention discloses a data cleaning method and system in a bank big data environment. The data cleaning method comprises the following steps: collecting original data from each source business system of a bank; preprocessing the original data to obtain preprocessed data; verifying and cleaning the preprocessed data to obtain cleaned data; and performing quality evaluation on the cleaned data. The problems of low bank data cleaning efficiency and poor accuracy in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the field of bank big data cleaning, and in particular to a data cleaning method and system in a bank big data environment. Background Art

[0002] Banking business generates a large amount of data every day, including customer information, account information, transaction information, external data, etc. This data is of great significance to the bank's data management, decision support and customer service. However, due to the many sources of data channels, non-uniform data formats, and uneven quality, there is a large amount of dirty data in bank data, such as duplicate data, erroneous data, and missing data. The existence of these dirty data seriously affects the quality and availability of bank data, so bank data needs to be cleaned.

[0003] Traditional data cleaning methods mainly rely on manual work, which is inefficient and prone to errors. With the continuous growth of bank data, traditional data cleaning methods can no longer meet the needs of banks in the big data environment. Therefore, there is an urgent need for an efficient and accurate data cleaning system and method to solve the data cleaning problem in the big data environment of banks. Summary of the invention

[0004] In view of the above problems, the present invention is proposed to provide a data cleaning method and system in a banking big data environment that overcomes the above problems or at least partially solves the above problems.

[0005] According to one aspect of the present invention, a data cleaning method in a bank big data environment is provided, the data cleaning method comprising:

[0006] Collect original data from various source business systems of the bank;

[0007] Preprocessing the original data to obtain preprocessed data;

[0008] Verifying and cleaning the preprocessed data to obtain cleaned data;

[0009] The cleaned data are subjected to quality assessment.

[0010] Optionally, the data cleaning method further includes: storing the cleaned data in a data warehouse.

[0011] Optionally, the collecting of raw data from various source business systems of the bank specifically includes:

[0012] Using API interfaces and data crawler technology, data is automatically collected from various bank business systems, including customer information, account information and transaction information.

[0013] Optionally, the preprocessing of the original data specifically includes:

[0014] Perform data identification and classification on the preprocessed data;

[0015] Perform preliminary verification of the data to check whether the data meets the predetermined format and type requirements, and use regular expressions, data type verification and other methods to filter out data that does not meet the requirements;

[0016] Use statistics and machine learning methods to automatically identify outliers, and use SQL language or Linux commands to decide whether to retain, delete or correct outliers based on business rules;

[0017] Use data deduplication algorithms to remove duplicate data through SQL language and defined functions.

[0018] Optionally, the checking and cleaning of the preprocessed data specifically includes: correcting erroneous data and completing incomplete data.

[0019] Optionally, the quality assessment of the cleaned data specifically includes:

[0020] Perform quality assessment on cleaned data using SQL logic;

[0021] Design data quality assessment indicators to quantitatively evaluate data quality;

[0022] Establish a data quality monitoring mechanism, regularly evaluate and monitor data quality, promptly identify and resolve data problems, and iteratively upgrade data cleaning algorithms.

[0023] Optionally, storing the cleaned data in a data warehouse specifically includes:

[0024] Design data storage solutions, including relational databases and distributed databases;

[0025] Choose an appropriate storage method based on the characteristics and needs of the data.

[0026] Optionally, the data quality assessment indicators include data missing rate and data error rate.

[0027] The present invention also provides a data cleaning system in a bank big data environment, applying the data cleaning method in a bank big data environment described above, the data cleaning method comprising:

[0028] Data collection module, used to collect raw data from various source business systems of the bank;

[0029] A data preprocessing module, used to preprocess the raw data to obtain preprocessed data;

[0030] A data cleaning module, used to verify and clean the preprocessed data to obtain cleaned data;

[0031] The data quality assessment module is used to perform quality assessment on the cleaned data.

[0032] Optionally, the data cleaning system further includes: a data storage module, used to store the cleaned data in a data warehouse.

[0033] The present invention provides a data cleaning method and system in a bank big data environment, wherein the data cleaning method comprises: collecting original data from various source business systems of the bank; preprocessing the original data to obtain preprocessed data; verifying and cleaning the preprocessed data to obtain cleaned data; and performing quality assessment on the cleaned data. The method solves the problems of low efficiency and poor accuracy of bank data cleaning in the prior art.

[0034] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0036] Figure 1 A flow chart of a data cleaning method in a bank big data environment provided by an embodiment of the present invention;

[0037] Figure 2 A block diagram of the composition of a data cleaning system in a bank big data environment provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0039] The terms "comprises" and "having" and any variations thereof in the description embodiments, claims and drawings of the present invention are intended to cover non-exclusive inclusions, for example, including a series of steps or units.

[0040] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0041] like Figure 1 As shown, a data cleaning method in a bank big data environment includes:

[0042] Collect original data from various source business systems of the bank;

[0043] Preprocessing the original data to obtain preprocessed data;

[0044] Verifying and cleaning the preprocessed data to obtain cleaned data;

[0045] The cleaned data are subjected to quality assessment.

[0046] like Figure 2 As shown, a data cleaning system in a bank big data environment includes:

[0047] Data collection module: using API interface, data crawler and other technologies to automatically collect data from various business systems of the bank, including customer information, account information, transaction information, etc.;

[0048] Design data collection strategies to ensure the comprehensiveness and accuracy of data. Data collection strategies include setting up scheduled extraction tasks, designing extraction frequencies, and designing incremental and full data extraction strategies for different data information types.

[0049] Data preprocessing module:

[0050] Conduct preliminary statistical analysis and quality checks on the collected data, such as data format, data range, etc., to filter out data that meets the requirements.

[0051] Design data preprocessing algorithms, such as data standardization and data normalization, to convert data into a unified format;

[0052] 1. Use the liunx command to check the data format and encoding to confirm whether they meet the specification requirements.

[0053] 2. Use SQL logic to query the data information range to check whether there is redundant data and whether it meets the data standards.

[0054] Data cleaning module: identifies and classifies the preprocessed data, such as structured data (tabular data in the database) and unstructured data (such as text, pictures, etc.);

[0055] Perform preliminary verification of the data to check whether the data meets the predetermined format and type requirements, and use regular expressions, data type verification and other methods to filter out data that does not meet the requirements.

[0056] Use statistics and machine learning methods (decision trees) to automatically identify outliers, and use SQL language or Linux commands to decide whether to retain, delete or correct outliers based on business rules.

[0057] Use data deduplication algorithms, such as hash algorithms, similarity comparison of specific fields, etc., through SQL language and defined functions to remove duplicate data and avoid data redundancy.

[0058] Data quality assessment module: Use SQL logic to assess the quality of cleaned data, including completeness, accuracy, consistency, etc.

[0059] Design data quality assessment indicators, such as data missing rate, data error rate, etc., to quantitatively assess data quality;

[0060] Establish a data quality monitoring mechanism, regularly evaluate and monitor data quality, promptly identify and resolve data problems, and iteratively upgrade the data cleaning algorithm.

[0061] Data storage module:

[0062] Design data storage solutions, such as relational databases, distributed databases, etc.

[0063] Choose the appropriate storage method based on the characteristics and needs of the data to improve the efficiency of data retrieval and access.

[0064] Beneficial effects: Automated cleaning process: The present invention adopts an automated cleaning process to improve the efficiency and accuracy of data cleaning. By designing a modular processing process of data acquisition, preprocessing, cleaning, quality assessment and storage, the automation of data cleaning is achieved, which reduces manual intervention and reduces the error rate.

[0065] Data cleaning algorithms: Efficient data cleaning algorithms, such as rule matching and machine learning, automatically identify and correct erroneous data. These algorithms can process large amounts of data quickly and accurately, improving the efficiency and accuracy of data cleaning.

[0066] Data quality assessment mechanism: A data quality assessment mechanism has been designed to ensure the reliability and consistency of data by quantitatively evaluating the cleaned data. Data quality assessment indicators can detect and solve data problems in a timely manner and improve data availability.

[0067] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data cleaning method in a bank big data environment, characterized in that: The data cleaning method comprises: Collect original data from various source business systems of the bank; Preprocessing the original data to obtain preprocessed data; Verifying and cleaning the preprocessed data to obtain cleaned data; The cleaned data are then subjected to quality assessment.

2. According to the data cleaning method in a bank big data environment according to claim 1, it is characterized in that: The data cleaning method further includes: storing the cleaned data in a data warehouse.

3. According to the data cleaning method in a bank big data environment according to claim 1, it is characterized in that: The collection of original data from various source business systems of the bank specifically includes: Using API interfaces and data crawler technology, data is automatically collected from various bank business systems, including customer information, account information and transaction information.

4. According to the data cleaning method in a bank big data environment according to claim 1, it is characterized in that: The preprocessing of the raw data specifically includes: Perform data identification and classification on the preprocessed data; Perform preliminary verification of the data to check whether the data meets the predetermined format and type requirements, and use regular expressions, data type verification and other methods to filter out data that does not meet the requirements; Use statistics and machine learning methods to automatically identify outliers, and use SQL language or Linux commands to decide whether to retain, delete or correct outliers based on business rules; Use data deduplication algorithms to remove duplicate data through SQL language and defined functions.

5. The data cleaning method in a bank big data environment according to claim 1 is characterized in that: The checking and cleaning of the preprocessed data specifically includes: correcting erroneous data and completing incomplete data.

6. The data cleaning method in a bank big data environment according to claim 1 is characterized in that: The quality assessment of the cleaned data specifically includes: Perform quality assessment on cleaned data using SQL logic; Design data quality assessment indicators to quantitatively evaluate data quality; Establish a data quality monitoring mechanism, regularly evaluate and monitor data quality, promptly identify and resolve data problems, and iteratively upgrade data cleaning algorithms.

7. The data cleaning method in a bank big data environment according to claim 2 is characterized in that: The storing of the cleaned data into the data warehouse specifically includes: Design data storage solutions, including relational databases and distributed databases; Choose an appropriate storage method based on the characteristics and needs of the data.

8. The data cleaning method in a bank big data environment according to claim 6 is characterized in that: The data quality assessment indicators include data missing rate and data error rate.

9. A data cleaning system in a bank big data environment, using a data cleaning method in a bank big data environment as described in any one of claims 1 to 8, characterized in that: The data cleaning method comprises: Data collection module, used to collect raw data from various source business systems of the bank; A data preprocessing module, used to preprocess the raw data to obtain preprocessed data; A data cleaning module, used to verify and clean the preprocessed data to obtain cleaned data; The data quality assessment module is used to perform quality assessment on the cleaned data.

10. A data cleaning system in a bank big data environment according to claim 9, characterized in that: The data cleaning system also includes: a data storage module, which is used to store the cleaned data in a data warehouse.

Images