Bank industry data management system

Abstract

The invention discloses a bank industry data governance system, and the system comprises a data extraction module which is used for extracting and storing business data; the data sampling module is used for sampling the extracted service data and asynchronously reporting the sampled data to the kafak; the governance rule configuration module is used for configuring a data governance rule and a data sampling rule; the index calculation module is used for acquiring the sampled data, verifying the data and writing the non-compliant data back into the relational database; the data query module is used for querying non-compliant data, acquiring permission setting and controlling a visible data range; and the notification module is used for issuing the data governance progress to the corresponding principal. According to the method, data autonomy of each team in the data production process is realized, additional data governance specialists are not needed, human input is reduced, and time and cost overhead caused by repeated analysis of total data is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data management, and in particular to a banking data management system. Background Art

[0002] Data governance is a key strategic area for banks, helping to ensure compliance, manage risks, improve data quality, enhance operational efficiency, ensure data security, enhance competitive advantage, ensure business continuity and promote technological innovation. The current data governance solution requires the establishment of data governance specialists, and monitoring data needs to be completely extracted, cleaned and processed before extracting features and creating reports for analysis, which results in high data governance costs. Therefore, a more cost-effective approach is needed to solve the problem.

[0003] The existing data governance platform functions include five parts: data security control, data integration management, data exploration and auditing, data cleaning and fusion, and data quality monitoring. Data governance specialists are required to extract features after the data is completely extracted, cleaned and processed, and create reports to analyze data compliance.

[0004] Problem 1: There are many data governance rules and the maintenance cost is high. In addition to the cognitive bias in understanding the rules, data governance specialists also need to repeatedly align with the data processing team when applying the rules. Once multiple rules are changed at the same time, a lot of time and energy need to be invested in repeated communication.

[0005] Problem 2: The status of data governance cannot be tracked, and it is difficult to measure the effects and risks. After the data governance rules are issued, due to the lack of a systematic monitoring method, whether the corresponding data has been governed, whether the governance effect meets expectations, and whether there are risks in completing the governance before the regulatory deadline, all these require data governance specialists to repeatedly analyze the entire data, which consumes a lot of manpower. Summary of the invention

[0006] In view of the above problems, the present invention is proposed to provide a banking data governance system that overcomes the above problems or at least partially solves the above problems.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A banking data governance system, the system comprising:

[0009] Data extraction module, used to extract and store business data;

[0010] The data sampling module is used to sample the extracted business data and asynchronously report the sampled data to Kafka;

[0011] Governance rule configuration module, used to configure data governance rules and data sampling rules;

[0012] The indicator calculation module is used to obtain the sampled data, verify the data, and write the non-compliant data back to the relational database;

[0013] Data query module, used to query non-compliant data and obtain permission settings to control the visible data range;

[0014] The notification module is used to send data governance progress to the corresponding person in charge, realizing the closed-loop operation of data processing and data governance.

[0015] Optionally, the system further includes a point-of-use consumption module, and the point-of-use consumption service module is used to write the sampled data into a relational database.

[0016] Optionally, the relational database is an Oceanbase database.

[0017] Optionally, the system further comprises a cache module, and the cache module is used to store frequently accessed data into a cache.

[0018] Optionally, the system also includes a permission middle platform module, which is used to store permission configuration and control data permissions.

[0019] Optionally, the system also includes a platform front-end module, which is used to display non-compliant data status, governance progress and detailed data status.

[0020] Optionally, the business data includes customer information, account information, transaction information and product information.

[0021] Optionally, use ETL technology to extract business data.

[0022] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0023] The present invention realizes data autonomy of each team in the data production process, without the need to set up additional data governance specialists, reducing manpower investment, and solving the time and cost overhead caused by repeated analysis of the full amount of data; through the platform front-end module, the governance progress and governance results of the project can be clearly understood, and automatic monitoring and tracking of the data governance process can be realized; data governance rules are automatically updated to solve the pain point of repeated alignment of related parties. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of a banking data governance system flow provided for an embodiment of the present application;

[0025] Figure 2A schematic diagram of a banking data governance system architecture flow chart provided for an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0027] Embodiment 1:

[0028] See also Figure 1 This embodiment provides a banking data governance system, the system comprising:

[0029] The data extraction module is used to extract and store business data. It uses ETL technology to extract business data and is an important part of data processing and data application.

[0030] The business data includes customer information, account information, transaction information and product information.

[0031] The data sampling module is used to sample the extracted business data and asynchronously report the sampled data to Kafka; at the same time, it monitors the sampling rules of the governance rule configuration center.

[0032] The tracking consumption module is used to write the sampled data into the relational database, which is the Oceanbase database.

[0033] The governance rule configuration module is used to configure data governance rules and data sampling rules.

[0034] The indicator calculation module is used to obtain the sampled data. It also obtains the rules to be executed from the configuration center, verifies the data, and writes non-compliant data back to the relational database.

[0035] The cache module is used to store frequently accessed data in the cache, reduce data access latency, reduce I / O operations, and improve system processing capabilities.

[0036] The data query module is used to query non-compliant data in accordance with the front-end data governance display requirements and return it to the platform front end. At the same time, it obtains permission settings to control the data scope visible to each team.

[0037] The permission middle platform module is used to store permission configuration and control data permissions.

[0038] The platform front-end module is used to display non-compliant data status, governance progress and detailed data status, etc.

[0039] The notification module is used to send data governance progress to the corresponding person in charge, realizing the closed-loop operation of data processing and data governance.

[0040] This embodiment realizes the data autonomy of each team in the data production process, without the need to set up additional data governance specialists, reducing manpower investment, and solving the time and cost overhead caused by repeated analysis of the full amount of data; through the platform front-end module, you can clearly understand the governance progress and governance results of the project, and realize automatic monitoring and tracking of the data governance process; data governance rules are automatically updated to solve the pain point of repeated alignment of related parties

[0041] Embodiment 2:

[0042] See also Figure 2 When banks do business, they will generate corresponding business data: customer information, account information, transaction information, product information, etc. However, due to various reasons, the data quality is poor, which can neither meet regulatory requirements nor be conducive to the bank's own data analysis and decision-making, and cannot bring into play the potential value of data. The following is a specific example of a data governance system in the banking field. The architecture flow chart is as follows: Figure 2 shown.

[0043] First, using ETL technology, the data extraction system extracts data from the bank's various business systems and stores the source data in an operational manner.

[0044] Then set the sampling rules and data verification rules in the rule configuration center. Set the sampling rate SamplingRate to 0.1, and the sampling method SamplingMethod to random sampling random; data verification rules, set different verification rules according to specific business information, such as: customer number, account number non-empty verification, customer number, account number, document number uniqueness verification, phone number integrity verification, transaction remarks can be empty, verification of various storage business data types, etc.

[0045] The data governance SDK reads the sampling rules of the rule configuration center, samples the data obtained by the data extraction system, and writes it to Kafka asynchronously to avoid blocking the normal sampling process. At the same time, Kafka is used as the middleware to solve the efficiency problem of direct writing to the database.

[0046] The tracking consumption service consumes the sampled data in Kafka and writes the data to the Oceanbase database, which can not only realize the subsequent rule calculation, but also retain the sampling history. When the rule change requires data backtracking, the sampled data can be directly obtained from the database. The various systems in the data production process no longer need to repeat the production process starting from the original data.

[0047] After the data verification rule configuration and sampling data of the rule configuration center are stored in the Oceanbase database, the system calls the indicator calculation service to perform rule verification on the data. If data that does not comply with the data governance rules is found, it is written back to the database. The work of sampling, modeling, and analysis that originally required data governance specialists to perform is converted into automated execution.

[0048] Add Redis cache to the system to reduce data access delays, reduce I / O operations, and improve system processing capabilities. Configure functional permissions and data permissions in the permission center: system administrators, data governance senior management, and data governance specialists can view all functions and data, and personnel in each business system can only view data related to their own business system.

[0049] After the permission configuration is completed, the data query service will query the non-compliant data according to the front-end data governance display requirements, return to the platform front-end, and obtain the permission settings from the permission middle-end to control the data range visible to each team. After the platform front-end obtains the queried data, it is displayed in the form of data dashboards and reports. The data dashboard displays the governance, non-governance, exemptions, governance item distribution, exemption types, governance task lists, and data governance quantity trend charts for the past week for each business system; the report form can display detailed data on data governance.

[0050] Finally, configure the email system notification to send the data governance progress to the data rectification person in charge of each business system and the senior person in charge of data governance.

[0051] After each business system completes a data governance task, the data system will re-extract the data after T (the same day) + 1 day, and perform the above-mentioned repeated sampling, verification, platform display, email notification and other processes to achieve closed-loop operations of data processing and data governance, reducing the manpower input of data governance specialists.

[0052] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A banking data governance system, characterized in that: The system comprises: Data extraction module, used to extract and store business data; The data sampling module is used to sample the extracted business data and asynchronously report the sampled data to Kafka; Governance rule configuration module, used to configure data governance rules and data sampling rules; The indicator calculation module is used to obtain the sampled data, verify the data, and write the non-compliant data back to the relational database; Data query module, used to query non-compliant data and obtain permission settings to control the visible data range; The notification module is used to send data governance progress to the corresponding person in charge, realizing the closed-loop operation of data processing and data governance.

2. A banking data management system as claimed in claim 1, characterized in that: The system also includes a point-of-use consumption module, and the point-of-use consumption service module is used to write the sampled data into a relational database.

3. A banking data management system as claimed in claim 2, characterized in that: The relational database is the Oceanbase database.

4. A banking data management system as claimed in claim 1, characterized in that: The system further comprises a cache module, and the cache module is used to store frequently accessed data into a cache.

5. A banking data management system as claimed in claim 1, characterized in that: The system also includes a permission middle platform module, which is used to store permission configuration and control data permissions.

6. A banking data management system as claimed in claim 1, characterized in that: The system also includes a platform front-end module, which is used to display non-compliant data status, governance progress and detailed data status.

7. A banking data management system as claimed in claim 1, characterized in that: The business data includes customer information, account information, transaction information and product information.

8. A banking data management system as claimed in claim 1, characterized in that: Use ETL technology to extract business data.

Images