A data warehouse governance method, device, equipment and storage medium thereof

Abstract

The embodiment of the application belongs to the technical field of financial science and technology, is applied to a financial data warehouse management scene, and relates to a data warehouse management method, device, equipment and storage medium thereof, comprising the following steps: obtaining characteristic indexes of all data source tables in a target data warehouse by analyzing metadata of the target data warehouse; identifying a target data source table according to the characteristic indexes and a preset evaluation strategy, and managing the target data source table according to a preset management strategy; and managing the target data warehouse in a timing task mode. According to the characteristic indexes and the preset evaluation strategy, the target data source table is identified, and the target data source table is managed according to the preset management strategy; a timing task is adopted, the target data warehouse can be managed regularly, the automatic and regular processing of the target data warehouse is ensured, the situation that a large amount of CPU resources are wasted due to chimney processing is avoided, and the standardization and scientific arrangement of financial business data in the data warehouse are facilitated.

Description

Technical Field

[0001] This application relates to the field of financial technology and is applied to the scenario of financial data warehouse governance, and in particular to a data warehouse governance method, apparatus, equipment and its storage medium. Background Technology

[0002] With the rapid development of the financial industry, financial companies are involved in an increasingly wide range of businesses. With the development of big data, more and more financial companies have established their own unique big data processing platforms, which integrate source databases such as data warehouses, data lakes, and business databases.

[0003] Within financial companies today, the scale of data with tags and model features is growing exponentially over time, consuming increasingly larger amounts of storage resources. This growth in data volume increases the difficulty of data governance. Data governance for big data platforms differs significantly from that of traditional databases in areas such as data cleaning, archiving, compression, and task optimization. Currently, many data warehouses or data marts are built on big data to support rapid data retrieval and utilization by businesses. However, after iterations across multiple business departments or versions, these data warehouses or data marts often develop similar models and processing logics. This type of similar model and processing logic is known in the industry as siloed processing. Siloed processing leads to significant waste of CPU and storage resources and hinders the standardization and scientific organization of financial business data within the data warehouse. Summary of the Invention

[0004] The purpose of this application is to propose a data warehouse governance method, apparatus, equipment and its storage medium to solve the problem that the existing technology of siloed processing in financial business data warehouses will cause a lot of CPU resources to be wasted and storage resources to be wasted, which is not conducive to the standardization and scientific organization of financial business data in the data warehouse.

[0005] To address the aforementioned technical problems, this application provides a data warehouse governance method, which employs the following technical solution:

[0006] A data warehouse governance method includes the following steps:

[0007] Step 201: Parse the metadata of the target data warehouse to obtain the characteristic indicators of all data source tables in the target data warehouse;

[0008] Step 202: Identify the target data source table based on the feature indicators and the preset evaluation strategy, and govern the target data source table according to the preset governance strategy;

[0009] Step 203: The target data warehouse is managed through scheduled tasks.

[0010] Furthermore, the characteristic indicators include the number of times the data source table is referenced by downstream applications, the reference duplication rate of the data source table, the table structure of the data source table, and the processing script corresponding to the data source table. The preset evaluation strategy includes a first evaluation strategy, a second evaluation strategy, a third evaluation strategy, and a fourth evaluation strategy. The step of identifying the target data source table based on the characteristic indicators and the preset evaluation strategy, and governing the target data source table according to the preset governance strategy, specifically includes:

[0011] Based on the number of times the data source table is referenced by downstream applications, statistical analysis is performed to obtain statistical analysis results, and the target data source table of the first category is identified based on the statistical analysis results and the first evaluation strategy.

[0012] The target data source tables of the second category are identified based on the degree of reference duplication between the data source tables and the second evaluation strategy;

[0013] The target data source table of the third category is identified based on the table structure of the data source table and the third evaluation strategy;

[0014] Based on the processing script corresponding to the data source table and the fourth evaluation strategy, the target data source table of the fourth category is identified;

[0015] According to the governance strategy, the target data source tables of the first category, the second category, the third category, and the fourth category are governed respectively, wherein the governance strategy includes reconstructing the target data source tables of the first category, the second category, the third category, and the fourth category respectively.

[0016] Furthermore, the step of performing statistical analysis based on the number of times the data source table is referenced by downstream applications, obtaining statistical analysis results, and identifying the target data source table of the first category based on the statistical analysis results and the first evaluation strategy specifically includes:

[0017] Count the number of all data source tables in the target data warehouse;

[0018] The total number of times each data source table in the target data warehouse is referenced by downstream applications is obtained by summing the data.

[0019] Based on the number of all data source tables and the total number of times all data source tables are referenced by downstream applications, the average number of times all data source tables are referenced by downstream applications is calculated to obtain the average number of times all data source tables are referenced by downstream applications.

[0020] Based on the number of times each of the data source tables is referenced by downstream applications and the average number of times, a comparison method is used to filter out data source tables that are referenced by downstream applications less than the average number of times, which are the target data source tables of the first category.

[0021] Furthermore, the step of identifying the target data source table of the second category based on the reference duplication rate between the data source tables and the second evaluation strategy specifically includes:

[0022] Step 501: Identify all tasks that perform data referencing from the target data warehouse;

[0023] Step 502: Calculate the data source tables referenced by each of the tasks;

[0024] Step 503: Randomly select two tasks from all the tasks as the comparison tasks for the current group;

[0025] Step 504: Count the number of data source tables referenced by the current group comparison task, and the number of identical data source tables referenced by the current group comparison task;

[0026] Step 505, according to the preset first ratio algorithm formula: Calculate the ratio between the number of identical data source tables referenced by the current group comparison task and the total number of data source tables referenced by the current group comparison task, and use this ratio as the actual ratio value. Here, A1 represents the number of identical data source tables referenced by the current group comparison task, and B1 represents the total number of data source tables referenced by the current group comparison task.

[0027] Step 506: By comparison and identification, filter out the same data source tables referenced by the current group comparison task when the number of data source tables referenced by the current group comparison task is greater than the preset threshold number and the actual ratio value is greater than the preset standard ratio value, i.e. the target data source tables of the second category.

[0028] Step 507: Repeat steps 503 to 506 in a loop to filter out all target data source tables of the second category in the target data warehouse.

[0029] Furthermore, the step of identifying the target data source table of the third category based on the table structure of the data source table and the third evaluation strategy specifically includes:

[0030] Step 601: Identify the table structure of all data source tables in the target data warehouse;

[0031] Step 602: Randomly select two data source tables from all the data source tables as the comparison source tables for the current group, and compare the table structure similarity of the comparison source tables for the current group.

[0032] Step 603: By comparing and identifying, the comparison source tables with a table structure similarity greater than a preset similarity threshold are selected, which are the target data source tables of the third category.

[0033] Step 604: Repeat steps 602 to 603 in a loop to filter out all third-category target data source tables in the target data warehouse.

[0034] Furthermore, the step of identifying the target data source table of the fourth category based on the processing script corresponding to the data source table and the fourth evaluation strategy specifically includes:

[0035] Step 701: Randomly select two data source tables from all the data source tables as the comparison source tables for the current group, and obtain the processing scripts corresponding to the comparison source tables for the current group respectively;

[0036] Step 702: Parse the processing scripts corresponding to the current group comparison source table respectively, and parse out the same processing code contained in the processing scripts corresponding to the current group comparison source table respectively;

[0037] Step 703: Calculate the amount of data with the same processing code, and the total amount of data of the processing scripts corresponding to the source tables of the current group;

[0038] Step 704, according to the preset second ratio algorithm formula: Calculate the proportion of the same processing code in the processing scripts corresponding to the current group comparison source table, as the script duplication rate, where A2 represents the amount of data of the same processing code and B2 represents the total amount of data of the processing scripts corresponding to the current group comparison source table.

[0039] Step 705: By comparing and identifying scripts, the comparison source table when the script repetition is greater than the preset repetition threshold is selected, which is the target data source table of the fourth category.

[0040] Step 706: Repeat steps 701 to 705 in a loop to filter out all target data source tables of the fourth category in the target data warehouse.

[0041] Furthermore, the step of governing the target data warehouse through scheduled tasks specifically includes:

[0042] Steps 201 to 202 are encapsulated into a preset timed processing method, and the encapsulated timed processing method is set as the timed management task.

[0043] Obtain the current system time and identify whether the time interval between the current system time and the end of the last governance reaches a preset time threshold.

[0044] If the time interval threshold does not reach the preset time interval threshold, the timed governance task will continue to be started and monitored according to the preset monitoring thread.

[0045] If the time interval threshold reaches the preset time interval threshold, the timed governance task is initiated, and the timed processing method is invoked to govern the target data warehouse.

[0046] To address the aforementioned technical problems, this application also provides a data warehouse governance device, which employs the following technical solution:

[0047] A data warehouse governance device, comprising:

[0048] The data source table feature indicator acquisition module is used to parse the metadata of the target data warehouse and obtain the feature indicators of all data source tables in the target data warehouse.

[0049] The target data source table identification and management module is used to identify the target data source table based on the feature indicators and the preset evaluation strategy, and to manage the target data source table according to the preset management strategy.

[0050] The target data warehouse scheduled governance module is used to govern the target data warehouse through scheduled tasks.

[0051] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0052] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the data warehouse governance method described above.

[0053] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0054] A computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the data warehouse governance method described above.

[0055] Compared with the prior art, the embodiments of this application have the following main advantages:

[0056] The data warehouse governance method described in this application involves parsing the metadata of the target data warehouse to obtain characteristic indicators of all data source tables within the target data warehouse; identifying target data source tables based on the characteristic indicators and a preset evaluation strategy; and governing the target data source tables according to a preset governance strategy. The governance of the target data warehouse is performed through scheduled tasks. Identifying target data source tables based on the characteristic indicators and the preset evaluation strategy, and governing them according to the preset governance strategy, using scheduled tasks, ensures that the target data warehouse is governed periodically, guaranteeing automated and regular processing. This avoids the waste of significant CPU resources caused by siloed processing and facilitates the standardization and scientific organization of financial business data within the data warehouse. Attached Figure Description

[0057] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0059] Figure 2 This is a flowchart of an embodiment of the data warehouse governance method according to this application;

[0060] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 202 shown;

[0061] Figure 4 yes Figure 3 A flowchart of a specific embodiment of step 301 shown;

[0062] Figure 5 yes Figure 3 A flowchart of a specific embodiment of step 302 shown;

[0063] Figure 6 yes Figure 3 A flowchart of a specific embodiment of step 303 shown;

[0064] Figure 7 yes Figure 3 A flowchart of a specific embodiment of step 304 shown;

[0065] Figure 8 yes Figure 2 A flowchart of a specific embodiment of step 203 shown;

[0066] Figure 9 This is a schematic diagram of a structure of an embodiment of the data warehouse governance apparatus according to this application;

[0067] Figure 10 yes Figure 9 A schematic diagram of a specific embodiment of the target data source table identification and management module 902 described above;

[0068] Figure 11 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0070] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0071] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0072] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0073] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0074] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0075] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0076] It should be noted that the data warehouse governance method provided in this application embodiment is generally executed by a server / terminal device, and correspondingly, the data warehouse governance device is generally set in the server / terminal device.

[0077] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0078] Continue to refer to Figure 2 A flowchart of an embodiment of the data warehouse governance method according to this application is shown. The data warehouse governance method includes the following steps:

[0079] Step 201: Parse the metadata of the target data warehouse to obtain the characteristic indicators of all data source tables in the target data warehouse.

[0080] In this embodiment, the target data warehouse includes a financial data warehouse or a financial data mart that stores financial business data. The metadata refers to data that describes the data within the target data warehouse. Specifically, the metadata records the repository where the data source table storing the data in the target data warehouse is located, the file type of the data source table, the storage path of the data source table, the data attribute field information contained in all data source tables, the number of times each data source table is accessed and used, the number of times the data source table is referenced by downstream applications, the reference duplication rate of the data source table, the table structure of the data source table, and the processing script corresponding to the data source table.

[0081] In this embodiment, the feature indicators include the number of times the data source table is referenced by downstream applications, the reference duplication rate of the data source table, the table structure of the data source table, and the processing script corresponding to the data source table.

[0082] Step 202: Identify the target data source table based on the feature indicators and the preset evaluation strategy, and govern the target data source table according to the preset governance strategy.

[0083] In this embodiment, the preset evaluation strategies include a first evaluation strategy, a second evaluation strategy, a third evaluation strategy, and a fourth evaluation strategy. The purpose is to identify the target data source tables to be addressed based on different evaluation strategies.

[0084] In this embodiment, the preset governance strategy includes reconstructing the target data source table. This reconstruction includes merging data source tables, adjusting attribute fields, and partitioning data source tables. Reconstruction refers to all optimization and governance operations performed on the data source table. These will not be elaborated upon further here.

[0085] Continue to refer to Figure 3 , Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 202 shown includes:

[0086] Step 301: Perform statistical analysis based on the number of times the data source table is referenced by downstream applications, obtain the statistical analysis results, and identify the target data source table of the first category based on the statistical analysis results and the first evaluation strategy;

[0087] Continue to refer to Figure 4 , Figure 4 yes Figure 3 A flowchart of a specific embodiment of step 301 shown includes:

[0088] Step 401: Count the number of all data source tables in the target data warehouse;

[0089] Step 402: Count the number of times each data source table in the target data warehouse is referenced by downstream applications, and use an accumulation method to obtain the total number of times each data source table is referenced by downstream applications;

[0090] Step 403: Calculate the average number of times each of the data source tables is referenced by downstream applications based on the number of all data source tables and the total number of times each of the data source tables is referenced by downstream applications.

[0091] Step 404: Based on the number of times each of the data source tables is referenced by downstream applications and the average number of times, a comparison method is used to filter out the data source tables that are referenced by downstream applications less than the average number of times, which are the target data source tables of the first category.

[0092] Because of the existing siloed processing method, a data source table often only serves a specific application and does not fully consider business sharing scenarios. The number of times the data source table is referenced by downstream is often very low. Therefore, selecting such data source tables in the financial data warehouse as the target data source tables of the first category can initially discover the target data source tables of the first category of siloed processing, which is convenient for subsequent merging and reconstruction of the data source tables of the first category, or for secondary target data source table mining to discover the target data source tables of the second category.

[0093] Step 302: Identify the target data source tables of the second category based on the reference duplication rate between the data source tables and the second evaluation strategy;

[0094] Continue to refer to Figure 5 , Figure 5 yes Figure 3 A flowchart of a specific embodiment of step 302 shown includes:

[0095] Step 501: Identify all tasks that perform data referencing from the target data warehouse;

[0096] Step 502: Calculate the data source tables referenced by each of the tasks;

[0097] In this embodiment, the step of statistically analyzing the data source tables referenced by all tasks can be performed using the entire target data warehouse as the search domain. Alternatively, it can use all the target data source tables of the first category obtained in step 404 as the search domain, filtering out the target data source tables of the first category referenced by the tasks executing the data references. The former uses data source tables throughout the entire target data warehouse as the data source tables to be managed, while the latter uses all target data source tables of the first category as the data source tables to be managed. Both methods have different advantages. The former's advantage lies in using data source tables throughout the entire target data warehouse as the data source tables to be managed, reflecting comprehensive optimization. The latter's advantage lies in using all target data source tables of the first category as the data source tables to be managed, and in addition to the target data warehouse management based on the characteristic indicator of the number of times the data source table is referenced by downstream applications, a second management is performed based on the characteristic indicator of the reference duplication rate between the data source tables.

[0098] Step 503: Randomly select two tasks from all the tasks as the comparison tasks for the current group;

[0099] Step 504: Count the number of data source tables referenced by the current group comparison task, and the number of identical data source tables referenced by the current group comparison task;

[0100] Step 505, according to the preset first ratio algorithm formula: Calculate the ratio between the number of identical data source tables referenced by the current group comparison task and the total number of data source tables referenced by the current group comparison task, and use this ratio as the actual ratio value. Here, A1 represents the number of identical data source tables referenced by the current group comparison task, and B1 represents the total number of data source tables referenced by the current group comparison task.

[0101] Step 506: By comparison and identification, filter out the same data source tables referenced by the current group comparison task when the number of data source tables referenced by the current group comparison task is greater than the preset threshold number and the actual ratio value is greater than the preset standard ratio value, i.e. the target data source tables of the second category.

[0102] In this embodiment, the number of preset thresholds can be set freely, such as 2 or 3, and the standard ratio can also be set freely, such as 70%. Assuming the number of preset thresholds is 2 and the standard ratio is 70%, when the number of data source tables referenced by the current group comparison task is greater than 2 through comparison identification, and the ratio between the number of identical data source tables referenced by the current group comparison task and the number of data source tables referenced by the current group comparison task is greater than 70%, it means that there are two tasks that reference multiple data source tables and reference multiple identical data source tables. This indicates that the construction of the identical data source tables referenced in the target data warehouse is not scientific enough and can be optimized and reconstructed.

[0103] Step 507: Repeat steps 503 to 506 in a loop to filter out all target data source tables of the second category in the target data warehouse.

[0104] Step 303: Identify the target data source table of the third category based on the table structure of the data source table and the third evaluation strategy;

[0105] Continue to refer to Figure 6 , Figure 6 yes Figure 3 A flowchart of a specific embodiment of step 303 shown includes:

[0106] Step 601: Identify the table structure of all data source tables in the target data warehouse;

[0107] In this embodiment, the table structure of all data source tables in the target data warehouse can be identified, or the identification scope can be further narrowed. For example, the table structure of all target data source tables of the first category obtained in step 404 can be identified, or the table structure of all target data source tables of the second category obtained in step 507 can be identified. The difference is that identifying the table structure of all data source tables in the target data warehouse ensures the comprehensiveness of the target data warehouse governance, while identifying the table structure of all target data source tables of the first category obtained in step 404, or identifying the table structure of all target data source tables of the second category obtained in step 507, is for further optimization of governance based on the previous governance. In actual governance implementation, the choice can be made freely according to governance needs.

[0108] Step 602: Randomly select two data source tables from all the data source tables as the comparison source tables for the current group, and compare the table structure similarity of the comparison source tables for the current group.

[0109] Step 603: By comparing and identifying, the comparison source tables with a table structure similarity greater than a preset similarity threshold are selected, which are the target data source tables of the third category.

[0110] In this embodiment, the comparison source tables with a table structure similarity greater than a preset similarity threshold are selected as the target data source tables of the third category. Since the table structures of the data source tables are too similar, it means that the data source tables with too similar table structures can be merged and reconstructed.

[0111] Step 604: Repeat steps 602 to 603 in a loop to filter out all third-category target data source tables in the target data warehouse.

[0112] Step 304: Identify the target data source table of the fourth category based on the processing script corresponding to the data source table and the fourth evaluation strategy;

[0113] Continue to refer to Figure 7 , Figure 7 yes Figure 3 A flowchart of a specific embodiment of step 304 shown includes:

[0114] Step 701: Randomly select two data source tables from all the data source tables as the comparison source tables for the current group, and obtain the processing scripts corresponding to the comparison source tables for the current group respectively;

[0115] Accordingly, in this embodiment, the step of arbitrarily selecting two data source tables from all data source tables as the comparison source tables for the current group can be further narrowed. Specifically, it can be done by arbitrarily selecting two data source tables from all target data source tables of the first category, or arbitrarily selecting two data source tables from all target data source tables of the second category, or arbitrarily selecting two data source tables from all target data source tables of the third category. The difference lies in that arbitrarily selecting two data source tables from all data source tables ensures the comprehensiveness of the target data warehouse governance, while narrowing the selection scope to select two data source tables is for further optimization based on the previous governance step. In actual governance implementation, the choice can be made freely according to governance needs.

[0116] Step 702: Parse the processing scripts corresponding to the current group comparison source table respectively, and parse out the same processing code contained in the processing scripts corresponding to the current group comparison source table respectively;

[0117] Step 703: Calculate the amount of data with the same processing code, and the total amount of data of the processing scripts corresponding to the source tables of the current group;

[0118] Step 704, according to the preset second ratio algorithm formula: Calculate the proportion of the same processing code in the processing scripts corresponding to the current group comparison source table, as the script duplication rate, where A2 represents the amount of data of the same processing code and B2 represents the total amount of data of the processing scripts corresponding to the current group comparison source table.

[0119] Step 705: By comparing and identifying scripts, the comparison source table when the script repetition is greater than the preset repetition threshold is selected, which is the target data source table of the fourth category.

[0120] In this embodiment, if the script duplication of the processing scripts corresponding to the current group comparison source tables reaches the preset duplication threshold, that is, the processing scripts corresponding to the current group comparison source tables are the same or similar, it indicates that the current group comparison source tables have a large commonality and can be merged and reconstructed.

[0121] Step 706: Repeat steps 701 to 705 in a loop to filter out all target data source tables of the fourth category in the target data warehouse.

[0122] Step 305: Perform governance on the target data source tables of the first category, the second category, the third category, and the fourth category according to the governance strategy, wherein the governance strategy includes reconstructing the target data source tables of the first category, the second category, the third category, and the fourth category respectively.

[0123] In this embodiment, the step of reconstructing the target data source tables for the first category, the second category, the third category, and the fourth category respectively specifically includes: obtaining the target data source table for the first category obtained in step 404, and merging the target data source tables for the first category; obtaining the target data source table for the second category obtained in step 507, and merging the target data source tables for the second category; obtaining the target data source table for the third category obtained in step 604, and merging the target data source tables for the third category; and obtaining the target data source table for the fourth category obtained in step 706, and merging the target data source tables for the fourth category.

[0124] Step 203: The target data warehouse is managed through scheduled tasks.

[0125] Continue to refer to Figure 8 , Figure 8 yes Figure 2 A flowchart of a specific embodiment of step 203 shown includes:

[0126] Step 801: Encapsulate steps 201 to 202 into a preset timed processing method, and set the encapsulated timed processing method as the timed management task.

[0127] Step 802: Obtain the current system time and identify whether the time interval between the current system time and the end of the last treatment reaches a preset time threshold.

[0128] Step 803: If the time interval threshold does not reach the preset time interval threshold, the timed governance task will continue to be started and monitored according to the preset monitoring thread.

[0129] Step 804: If the time interval threshold reaches the preset time interval threshold, then the timed governance task is started, and the timed processing method is called to govern the target data warehouse.

[0130] By encapsulating steps 201 to 202 into a preset timed processing method and setting the encapsulated timed processing method as the timed governance task, the target data warehouse can be governed periodically, ensuring automated and periodic processing of the target data warehouse. This avoids the waste of CPU resources caused by siloed processing and facilitates the standardized organization of financial business data in the data warehouse.

[0131] This application obtains characteristic indicators of all data source tables within the target data warehouse by parsing its metadata; identifies target data source tables based on these characteristic indicators and a preset evaluation strategy; and governs these target data source tables according to a preset governance strategy. The target data warehouse is then governed through scheduled tasks. Identifying target data source tables based on the characteristic indicators and the preset evaluation strategy, and governing them according to the preset governance strategy, using scheduled tasks, ensures the periodic and automated processing of the target data warehouse. This avoids the waste of CPU resources caused by siloed processing and facilitates the standardization and scientific organization of financial business data within the data warehouse.

[0132] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0133] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0134] In this embodiment, by parsing the metadata of the target data warehouse, characteristic indicators of all data source tables within the target data warehouse are obtained; target data source tables are identified based on the characteristic indicators and a preset evaluation strategy, and the target data source tables are governed according to a preset governance strategy; the target data warehouse is governed through scheduled tasks. Identifying target data source tables based on the characteristic indicators and the preset evaluation strategy, and governing them according to the preset governance strategy, using scheduled tasks, allows for periodic governance of the target data warehouse, ensuring automated and regular processing of the target data warehouse. This avoids the waste of significant CPU resources caused by siloed processing and facilitates the standardization and scientific organization of financial business data in the data warehouse.

[0135] Further reference Figure 9 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of a data warehouse governance device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0136] like Figure 9 As shown, the data warehouse governance device 900 described in this embodiment includes: a data source table feature indicator acquisition module 901, a target data source table identification and governance module 902, and a target data warehouse timed governance module 903. Wherein:

[0137] The data source table feature indicator acquisition module 901 is used to parse the metadata of the target data warehouse and obtain the feature indicators of all data source tables in the target data warehouse.

[0138] The target data source table identification and management module 902 is used to identify the target data source table according to the feature indicators and the preset evaluation strategy, and to manage the target data source table according to the preset management strategy.

[0139] The target data warehouse scheduled governance module 903 is used to govern the target data warehouse through scheduled tasks.

[0140] Continue to refer to Figure 10 , Figure 10 yes Figure 9This is a schematic diagram of a specific embodiment of the target data source table identification and management module 902 described herein. The target data source table identification and management module 902 includes a first identification submodule 10a, a second identification submodule 10b, a third identification submodule 10c, a fourth identification submodule 10d, and a management submodule 10e. Wherein:

[0141] The first identification submodule 10a is used to perform statistical analysis based on the number of times the data source table is referenced by downstream applications, obtain statistical analysis results, and identify the target data source table of the first category based on the statistical analysis results and the first evaluation strategy.

[0142] The second identification submodule 10b is used to identify the target data source table of the second category based on the reference duplication between the data source tables and the second evaluation strategy.

[0143] The third identification submodule 10c is used to identify the target data source table of the third category based on the table structure of the data source table and the third evaluation strategy;

[0144] The fourth identification submodule 10d is used to identify the target data source table of the fourth category based on the processing script corresponding to the data source table and the fourth evaluation strategy;

[0145] The governance submodule 10e is used to govern the target data source tables of the first category, the second category, the third category, and the fourth category according to the governance strategy, wherein the governance strategy includes reconstructing the target data source tables of the first category, the second category, the third category, and the fourth category respectively.

[0146] This application obtains characteristic indicators of all data source tables within the target data warehouse by parsing its metadata; identifies target data source tables based on these characteristic indicators and a preset evaluation strategy; and governs these target data source tables according to a preset governance strategy. The target data warehouse is then governed through scheduled tasks. Identifying target data source tables based on the characteristic indicators and the preset evaluation strategy, and governing them according to the preset governance strategy, using scheduled tasks, ensures the periodic and automated processing of the target data warehouse. This avoids the waste of CPU resources caused by siloed processing and facilitates the standardization and scientific organization of financial business data within the data warehouse.

[0147] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0148] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0149] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 11 , Figure 11 This is a basic structural block diagram of the computer device in this embodiment.

[0150] The computer device 11 includes a memory 11a, a processor 11b, and a network interface 11c that are interconnected via a system bus. It should be noted that only the computer device 11 with components 11a-11c is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0151] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0152] The memory 11a includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11a may be an internal storage unit of the computer device 11, such as the hard disk or memory of the computer device 11. In other embodiments, the memory 11a may also be an external storage device of the computer device 11, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 11. Of course, the memory 11a may include both the internal storage unit and its external storage device of the computer device 11. In this embodiment, the memory 11a is typically used to store the operating system and various application software installed on the computer device 11, such as computer-readable instructions for a data warehouse governance method. In addition, the memory 11a can also be used to temporarily store various types of data that have been output or will be output.

[0153] In some embodiments, the processor 11b may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 11b is typically used to control the overall operation of the computer device 11. In this embodiment, the processor 11b is used to execute computer-readable instructions stored in the memory 11a or to process data, such as executing computer-readable instructions for the data warehouse governance method.

[0154] The network interface 11c may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 11 and other electronic devices.

[0155] The computer equipment proposed in this embodiment belongs to the field of financial technology and is applied in the scenario of financial data warehouse governance. This application obtains characteristic indicators of all data source tables within the target data warehouse by parsing the metadata of the target data warehouse; identifies the target data source tables based on the characteristic indicators and a preset evaluation strategy, and governs the target data source tables according to a preset governance strategy; and governs the target data warehouse through scheduled tasks. Identifying the target data source tables based on the characteristic indicators and the preset evaluation strategy, and governing them according to the preset governance strategy, using scheduled tasks, enables periodic governance of the target data warehouse, ensuring automated and regular processing of the target data warehouse, avoiding the waste of significant CPU resources caused by siloed processing, and facilitating the standardization and scientific organization of financial business data in the data warehouse.

[0156] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by a processor to cause the processor to perform the steps of the data warehouse governance method described above.

[0157] The computer-readable storage medium proposed in this embodiment belongs to the field of financial technology and is applied in the scenario of financial data warehouse governance. This application obtains characteristic indicators of all data source tables within the target data warehouse by parsing the metadata of the target data warehouse; identifies the target data source tables based on the characteristic indicators and a preset evaluation strategy, and governs the target data source tables according to a preset governance strategy; and governs the target data warehouse through scheduled tasks. Identifying the target data source tables based on the characteristic indicators and the preset evaluation strategy, and governing the target data source tables according to the preset governance strategy, using scheduled tasks, enables periodic governance of the target data warehouse, ensuring automated and regular processing of the target data warehouse, avoiding the waste of significant CPU resources caused by siloed processing, and facilitating the standardization and scientific organization of financial business data in the data warehouse.

[0158] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0159] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A data warehouse governance method, characterized in that, Includes the following steps: Step 201: Parse the metadata of the target data warehouse to obtain the characteristic indicators of all data source tables in the target data warehouse; Step 202: Identify the target data source table based on the characteristic indicators and the preset evaluation strategy, and govern the target data source table according to the preset governance strategy. The characteristic indicators include the number of times the data source table is referenced by downstream applications, the reference duplication rate of the data source table, the table structure of the data source table, and the corresponding processing script for the data source table. The preset evaluation strategy includes a first evaluation strategy, a second evaluation strategy, a third evaluation strategy, and a fourth evaluation strategy. The step of identifying the target data source table based on the characteristic indicators and the preset evaluation strategy, and governing the target data source table according to the preset governance strategy, specifically includes: Based on the number of times the data source table is referenced by downstream applications, statistical analysis is performed to obtain statistical analysis results, and the target data source table of the first category is identified based on the statistical analysis results and the first evaluation strategy. The target data source tables of the second category are identified based on the degree of reference duplication between the data source tables and the second evaluation strategy; The target data source table of the third category is identified based on the table structure of the data source table and the third evaluation strategy; Based on the processing script corresponding to the data source table and the fourth evaluation strategy, the target data source table of the fourth category is identified; According to the governance strategy, the target data source tables of the first category, the second category, the third category, and the fourth category are governed respectively, wherein the governance strategy includes reconstructing the target data source tables of the first category, the second category, the third category, and the fourth category respectively; Step 203 involves managing the target data warehouse through scheduled tasks, specifically including: Steps 201 to 202 are encapsulated into a preset timed processing method, and the encapsulated timed processing method is set as a timed management task. Obtain the current system time and identify whether the time interval between the current system time and the end of the last governance reaches a preset time threshold. If the time interval value does not reach the preset time threshold, the timed governance task will continue to be started and monitored according to the preset monitoring thread. If the time interval value reaches the preset time threshold, the timed governance task is initiated, and the timed processing method is invoked to govern the target data warehouse.

2. The data warehouse governance method according to claim 1, characterized in that, The step of performing statistical analysis based on the number of times the data source table is referenced by downstream applications, obtaining statistical analysis results, and identifying the target data source table of the first category based on the statistical analysis results and the first evaluation strategy specifically includes: Count the number of all data source tables in the target data warehouse; The total number of times each data source table in the target data warehouse is referenced by downstream applications is obtained by summing the data. Based on the number of all data source tables and the total number of times all data source tables are referenced by downstream applications, the average number of times all data source tables are referenced by downstream applications is calculated to obtain the average number of times all data source tables are referenced by downstream applications. Based on the number of times each of the data source tables is referenced by downstream applications and the average number of times, a comparison method is used to filter out data source tables that are referenced by downstream applications less than the average number of times, which are the target data source tables of the first category.

3. The data warehouse governance method according to claim 1, characterized in that, The step of identifying the target data source table of the second category based on the reference duplication rate between the data source tables and the second evaluation strategy specifically includes: Step 501: Identify all tasks that perform data referencing from the target data warehouse; Step 502: Calculate the data source tables referenced by each of the tasks; Step 503: Randomly select two tasks from all the tasks as the comparison tasks for the current group; Step 504: Count the number of data source tables referenced by the current group comparison task, and the number of identical data source tables referenced by the current group comparison task; Step 505, according to the preset first ratio algorithm formula: Calculate the ratio between the number of identical data source tables referenced by the current group's comparison tasks and the total number of data source tables referenced by the current group's comparison tasks, and use this ratio as the actual ratio value. This indicates the number of identical data source tables referenced by the current group's comparison tasks. This indicates the number of data source tables referenced by the current group's comparison task; Step 506: By comparison and identification, filter out the same data source tables referenced by the current group comparison task when the number of data source tables referenced by the current group comparison task is greater than the preset threshold number and the actual ratio value is greater than the preset standard ratio value, i.e. the target data source tables of the second category. Step 507: Repeat steps 503 to 506 in a loop to filter out all target data source tables of the second category in the target data warehouse.

4. The data warehouse governance method according to claim 1, characterized in that, The step of identifying the target data source table of the third category based on the table structure of the data source table and the third evaluation strategy specifically includes: Step 601: Identify the table structure of all data source tables in the target data warehouse; Step 602: Randomly select two data source tables from all the data source tables as the comparison source tables for the current group, and compare the table structure similarity of the comparison source tables for the current group. Step 603: By comparing and identifying, the comparison source tables with a table structure similarity greater than a preset similarity threshold are selected, which are the target data source tables of the third category. Step 604: Repeat steps 602 to 603 in a loop to filter out all third-category target data source tables in the target data warehouse.

5. The data warehouse governance method according to claim 1, characterized in that, The step of identifying the target data source table of the fourth category based on the processing script corresponding to the data source table and the fourth evaluation strategy specifically includes: Step 701: Randomly select two data source tables from all the data source tables as the comparison source tables for the current group, and obtain the processing scripts corresponding to the comparison source tables for the current group respectively; Step 702: Parse the processing scripts corresponding to the current group comparison source table respectively, and parse out the same processing code contained in the processing scripts corresponding to the current group comparison source table respectively; Step 703: Calculate the amount of data with the same processing code, and the total amount of data of the processing scripts corresponding to the source tables of the current group; Step 704, according to the preset second ratio algorithm formula: The proportion of the same processing code in the processing scripts corresponding to the source table of the current group is calculated as the script duplication rate. This indicates the amount of data containing the same processing code. This indicates the total amount of data in the processing scripts corresponding to the source tables of the current group; Step 705: By comparing and identifying scripts, the comparison source table when the script repetition is greater than the preset repetition threshold is selected, which is the target data source table of the fourth category. Step 706: Repeat steps 701 to 705 in a loop to filter out all target data source tables of the fourth category in the target data warehouse.

6. A data warehouse governance device, characterized in that, The data warehouse governance device implements the steps of the data warehouse governance method as described in any one of claims 1 to 5, and the data warehouse governance device includes: The data source table feature indicator acquisition module is used to parse the metadata of the target data warehouse and obtain the feature indicators of all data source tables in the target data warehouse. The target data source table identification and management module is used to identify the target data source table based on the feature indicators and the preset evaluation strategy, and to manage the target data source table according to the preset management strategy. The target data warehouse scheduled governance module is used to govern the target data warehouse through scheduled tasks, specifically including: The processing steps executed by the data source table feature index acquisition module and the target data source table identification and management module are encapsulated into a preset timed processing method, and the encapsulated timed processing method is set as a timed management task. Obtain the current system time and identify whether the time interval between the current system time and the end of the last governance reaches a preset time threshold. If the time interval value does not reach the preset time threshold, the timed governance task will continue to be started and monitored according to the preset monitoring thread. If the time interval value reaches the preset time threshold, the timed governance task is initiated, and the timed processing method is invoked to govern the target data warehouse.

7. A computer device, characterized in that, The system includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the data warehouse governance method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the data warehouse governance method as described in any one of claims 1 to 5.

Images