A table dimension statistical caliber calculation method and system

Abstract

The present invention relates to the technical field of data governance, and provides a method and system for calculating the statistical caliber of table dimensions, including: regularly collecting basic statistical information of a table, obtaining the blood relationship information of the table and the number of task execution times of the table according to the collected basic statistical information of the table; constructing a first data attribute table according to the obtained basic data of the statistical caliber; calculating the pre-prepared data of the statistical caliber of the table dimension according to the attribute data in the first data attribute table, and constructing a second data attribute table according to the pre-prepared data of the statistical caliber of the table dimension obtained by calculation; looping through the first data attribute table and the second data attribute table, calculating the score of the statistical caliber of each dimension of the table for each table; adding labels to the statistical caliber of each dimension of the table, saving the corresponding table dimension statistical caliber of the table with the label added to the database and displaying it on the page. The present invention can analyze the value of the table from each caliber dimension, which is conducive to improving the efficiency and accuracy of data governance.

Description

Technical Field

[0001] The present invention relates to the field of data governance technology, and in particular to a method and system for calculating table dimension statistical caliber. Background Art

[0002] As data applications continue to deepen, all major traditional user entities are undergoing digital transformation. In the process of digital transformation, data extraction, governance, management and other data processing play a vital role in the progress of digital transformation. In the process of data governance, a large number of data tables will be generated. Calculating, scoring and labeling the dimensional statistical calibers of these tables, such as access popularity, growth, dependence concentration, revenue cost, influence, attention and data timeliness, will provide important help for data governance and governance effectiveness.

[0003] At present, there is no calculation rule for the statistical caliber of table dimensions. How to provide an efficient, accurate and highly applicable calculation method for the statistical caliber of table dimensions has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, in order to overcome the deficiencies of the prior art, the present invention aims to provide a method and system for calculating the statistical caliber of a table dimension.

[0005] According to a first aspect of the present invention, a method for calculating table dimension statistical caliber is provided, the method comprising:

[0006] Collect basic statistical information of tables regularly, and obtain the table's lineage information and the number of task executions in which the table participates based on the collected basic statistical information of the table;

[0007] According to the basic statistical information of the table, the blood relationship information and the number of task executions in which the table participates, the basic statistical data of the table are obtained, and the first data attribute table is constructed according to the obtained basic statistical data;

[0008] Calculate pre-prepared data of the table dimension statistical caliber according to the attribute data in the first data attribute table, and construct a second data attribute table according to the pre-prepared data of the table dimension statistical caliber obtained by calculation;

[0009] Loop through the first data attribute table and the second data attribute table, and calculate the score of each dimension of the table for each table;

[0010] Add labels to the statistical caliber of each dimension of the table, save the statistical caliber of the corresponding table dimension of the labeled table to the database and display it on the page.

[0011] Preferably, in the table dimension statistical caliber calculation method of the present invention, the basic statistical information of the table includes the storage size occupied by the table, the number of data items in the table, the last update time of the table data, the number of times the table is browsed in the system, and the number of times the table is collected.

[0012] Preferably, in the table dimension statistical caliber calculation method of the present invention, the attributes of the first data attribute table include the number of table collections in the last week, the calculation engine Id, the data partition Id, the table Id, the total storage capacity of the table, the total number of table data items, the last update time of the table data, the table storage growth in the last week, the growth of the number of table data items in the last week, the number of table executions in the last week, the number of upstream direct blood relationship tables, the number of downstream direct blood relationship tables, the number of downstream blood relationship tables and the number of table browsing records in the last week.

[0013] Preferably, in the table dimension statistical caliber calculation method of the present invention, the pre-prepared data of the table dimension statistical caliber include: the maximum number of times the table participated in the task execution in the past week, the minimum number of times the table participated in the task execution in the past week, the maximum storage growth of all tables in the past week, the minimum storage growth of all tables in the past week, the maximum storage growth ratio, the minimum storage growth ratio, the maximum number of data items growth of all tables in the past week, the minimum number of data items growth of all tables in the past week, the maximum number of data items growth ratio, the minimum number of data items growth ratio, the maximum dependency concentration, the minimum dependency concentration, the maximum storage occupancy of all tables, the minimum storage occupancy of all tables, the maximum number of downstream and downstream bloodline tables in all tables, the minimum number of downstream and downstream bloodline tables in all tables, the maximum number of views of all tables in the past week, the minimum number of views of all tables in the past week, the maximum number of collections of all tables in the past week, and the minimum number of collections of all tables in the past week.

[0014] Preferably, in the table dimension statistical caliber calculation method of the present invention, the first data attribute table and the second data attribute table are looped through to calculate the score of each dimension statistical caliber of the table for each table, including:

[0015] The access popularity score of the table is calculated based on the maximum number of times the table has participated in task execution in the past week, the minimum number of times the table has participated in task execution in the past week, and the number of times the table has been executed in the past week;

[0016] The dependency concentration score of the table is calculated based on the maximum dependency concentration, the minimum dependency concentration, the number of upstream direct blood relationship tables, and the number of downstream direct blood relationship tables;

[0017] The table storage score is calculated based on the total storage capacity of the table, the maximum storage capacity of all tables, and the minimum storage capacity of all tables. The table benefit cost score is calculated based on the table access popularity score and the table storage score.

[0018] The influence score of a table is calculated based on the number of downstream kinship tables, the maximum number of downstream kinship tables among all tables, and the minimum number of downstream kinship tables among all tables.

[0019] Calculate the data timeliness score of the table based on the last modification time of the table data and the current time.

[0020] Preferably, in the table dimension statistical caliber calculation method of the present invention, the first data attribute table and the second data attribute table are looped through to calculate the score of each dimension statistical caliber of the table for each table, including:

[0021] The growth score of the table is calculated based on the storage growth of the table in the past week, the maximum storage growth of all tables in the past week, and the minimum storage growth of all tables in the past week.

[0022] Calculate the table's growth score based on the table's storage usage, maximum storage growth ratio, and minimum storage growth ratio in the most recent week.

[0023] The data score of the table is calculated based on the growth of the number of data items in the last week, the maximum growth of the number of data items in all tables in the last week, and the minimum growth of the number of data items in all tables in the last week;

[0024] Calculate the data proportion score of the table based on the total number of data items in the table in the past week, the growth rate of the largest number of data items, and the growth rate of the smallest number of data items;

[0025] A weighted sum value is obtained by taking a weighted sum of the growth score of the table, the growth proportion score of the table, the data score of the table and the data proportion score of the table, and the weighted sum value is used as the growth score of the table.

[0026] Preferably, in the table dimension statistical caliber calculation method of the present invention, the first data attribute table and the second data attribute table are looped through to calculate the score of each dimension statistical caliber of the table for each table, including:

[0027] The collection score of a table is calculated based on the number of collections of the table in the last week, the maximum number of collections of all tables in the last week, and the minimum number of collections of all tables in the last week;

[0028] The table browsing score is calculated based on the number of table browsing records in the past week, the maximum number of browsing records of all tables in the past week, and the minimum number of browsing records of all tables in the past week;

[0029] A weighted sum value is obtained by performing a weighted summation on the collection score and the browsing score of the table, and the weighted sum value is used as the attention score of the table.

[0030] Preferably, in the table dimension statistical caliber calculation method of the present invention, a label is added to the statistical caliber of each dimension of the table, and the corresponding table dimension statistical caliber of the labeled table is saved to the database and displayed on the page, including:

[0031] According to the table ID and data partition ID, the scores of each dimension of the table are saved to the database and displayed on the page;

[0032] Sort the relevant dimension statistical calibers of all tables according to the scores of the statistical calibers of each dimension, and obtain the binary values ​​of the relevant dimension statistical calibers;

[0033] Compare the score of the corresponding table dimension statistical caliber of the table with the corresponding binary value. When the score of the corresponding table dimension statistical caliber of the table is not less than the corresponding binary value, add a high index label to the corresponding table dimension statistical caliber of the table. When the score of the corresponding table dimension statistical caliber of the table is less than the corresponding binary value, add a low index label to the corresponding table dimension statistical caliber of the table.

[0034] The corresponding table dimension statistical caliber of the tagged table is saved to the database and displayed on the page.

[0035] According to a second aspect of the present invention, a table dimension statistical caliber calculation system is provided, the system comprising a table dimension statistical caliber calculation server, the table dimension statistical caliber calculation server is used to:

[0036] Collect basic statistical information of tables regularly, and obtain the table's lineage information and the number of task executions in which the table participates based on the collected basic statistical information of the table;

[0037] According to the basic statistical information of the table, the blood relationship information and the number of task executions in which the table participates, the basic statistical data of the table are obtained, and the first data attribute table is constructed according to the obtained basic statistical data;

[0038] Calculate pre-prepared data of the table dimension statistical caliber according to the attribute data in the first data attribute table, and construct a second data attribute table according to the pre-prepared data of the table dimension statistical caliber obtained by calculation;

[0039] Loop through the first data attribute table and the second data attribute table, and calculate the score of each dimension of the table for each table;

[0040] Add labels to the statistical caliber of each dimension of the table, save the statistical caliber of the corresponding table dimension of the labeled table to the database and display it on the page.

[0041] Preferably, in the table dimension statistical caliber calculation system of the present invention, the table dimension statistical caliber calculation server includes:

[0042] The information collection module is used to collect the basic statistical information of the table at regular intervals, and obtain the table's blood relationship information and the number of task executions in which the table participates based on the collected basic statistical information of the table;

[0043] A first data attribute table construction module is used to obtain the basic statistical data of the table according to the basic statistical information of the table, the blood relationship information and the number of task executions in which the table participates, and to construct the first data attribute table according to the obtained basic statistical data;

[0044] A second data attribute table construction module, used to calculate the pre-prepared data of the table dimension statistical caliber according to the attribute data in the first data attribute table, and construct the second data attribute table according to the pre-prepared data of the table dimension statistical caliber obtained by calculation;

[0045] A dimension statistical caliber score calculation module is used to loop through the first data attribute table and the second data attribute table, and calculate the score of each dimension statistical caliber of each table for each table;

[0046] The dimension statistical caliber labeling and saving module is used to add labels to the statistical caliber of each dimension of the table, save the statistical caliber of the corresponding table dimension of the labeled table to the database and display it on the page.

[0047] According to a third aspect of the present invention, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect of the present invention when executing the program.

[0048] The table dimension statistical caliber calculation method and system of the present invention can directly display the scores of each statistical caliber of the table and the high and low levels of each caliber on the page by calculating the scores of the table statistical caliber and marking the statistical caliber, so that data governance personnel can analyze the value of the table from each caliber dimension and analyze the governance effectiveness of the table, which is conducive to improving the efficiency and accuracy of data governance. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 It is a schematic diagram of a system applicable to the table dimension statistical caliber calculation method of the embodiment of the present application;

[0051] Figure 2This is an example diagram of the architecture of a table dimension statistical caliber calculation server in a table dimension statistical caliber calculation system according to an embodiment of the present invention;

[0052] Figure 3 A flowchart of a method for calculating table dimension statistical caliber according to an embodiment of the present invention;

[0053] Figure 4 A schematic diagram of the structure of the device provided by the present invention. DETAILED DESCRIPTION

[0054] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0055] It should be noted that the following embodiments and features in the embodiments may be combined with each other in the absence of conflict; and, based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making any creative work are within the scope of protection of the present disclosure.

[0056] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0057] Figure 1 An exemplary system for calculating the table dimension statistical caliber applicable to the embodiment of the present application is shown. Figure 1 As shown, the system may include a table dimension statistical caliber calculation server 101, a communication network 102 and / or one or more table dimension statistical caliber calculation clients 103. Figure 1 The example in the figure is a client 103 for calculating the statistical caliber of multiple table dimensions.

[0058] The table dimension statistical caliber calculation server 101 can be any appropriate server for storing information, data, programs and / or any other suitable type of content. In some embodiments, the table dimension statistical caliber calculation server 101 can perform appropriate functions. For example, in some embodiments, the table dimension statistical caliber calculation server 101 can be used for table dimension statistical caliber calculation. As an optional example, in some embodiments, the table dimension statistical caliber calculation server 101 can be used to implement table dimension statistical caliber calculation by constructing a first data attribute table and a second data attribute table. For example, the table dimension statistical caliber calculation server 101 can be used to:

[0059] Collect basic statistical information of tables regularly, and obtain the table's lineage information and the number of task executions in which the table participates based on the collected basic statistical information of the table;

[0060] According to the basic statistical information of the table, the blood relationship information and the number of task executions in which the table participates, the basic statistical data of the table are obtained, and the first data attribute table is constructed according to the obtained basic statistical data;

[0061] Calculate pre-prepared data of the table dimension statistical caliber according to the attribute data in the first data attribute table, and construct a second data attribute table according to the pre-prepared data of the table dimension statistical caliber obtained by calculation;

[0062] Loop through the first data attribute table and the second data attribute table, and calculate the score of each dimension of the table for each table;

[0063] Add labels to the statistical caliber of each dimension of the table, save the statistical caliber of the corresponding table dimension of the labeled table to the database and display it on the page.

[0064] Figure 2 FIG. 1 is a diagram showing an example of an architecture of a table dimension statistical caliber calculation server in a table dimension statistical caliber calculation system according to an embodiment of the present invention. Figure 2 As shown, the table dimension statistical caliber calculation server of this embodiment includes:

[0065] The information collection module is used to collect the basic statistical information of the table at regular intervals, and obtain the table's blood relationship information and the number of task executions in which the table participates based on the collected basic statistical information of the table;

[0066] A first data attribute table construction module is used to obtain the basic statistical data of the table according to the basic statistical information of the table, the blood relationship information and the number of task executions in which the table participates, and to construct the first data attribute table according to the obtained basic statistical data;

[0067] A second data attribute table construction module, used to calculate the pre-prepared data of the table dimension statistical caliber according to the attribute data in the first data attribute table, and construct the second data attribute table according to the pre-prepared data of the table dimension statistical caliber obtained by calculation;

[0068] A dimension statistical caliber score calculation module is used to loop through the first data attribute table and the second data attribute table, and calculate the score of each dimension statistical caliber of each table for each table;

[0069] The dimension statistical caliber labeling and saving module is used to add labels to the statistical caliber of each dimension of the table, save the statistical caliber of the corresponding table dimension of the labeled table to the database and display it on the page.

[0070] As another example, in some embodiments, the table dimension statistical caliber calculation server 101 may send the table dimension statistical caliber calculation method to the table dimension statistical caliber calculation client 103 for user use based on the request of the table dimension statistical caliber calculation client 103 .

[0071] As an optional example, in some embodiments, the table dimension statistical caliber calculation client 103 is used to provide a visual calculation interface, which is used to receive a user's selection input operation for table dimension statistical caliber calculation, and, in response to the selection input operation, obtain a calculation interface corresponding to the option selected by the selection input operation from the table dimension statistical caliber calculation server 101 and display the calculation interface, wherein the calculation interface at least displays information on the table dimension statistical caliber calculation and operation options for the information on the table dimension statistical caliber calculation.

[0072] In some embodiments, the communication network 102 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 102 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The table dimension statistics caliber calculation client 103 can be connected to the communication network 102 via one or more communication links (e.g., communication link 104), and the communication network 102 can be linked to the table dimension statistics caliber calculation server 101 via one or more communication links (e.g., communication link 105). The communication link can be any communication link suitable for transmitting data between the table dimension statistics caliber calculation client 103 and the table dimension statistics caliber calculation server 101, such as a network link, a dial-up link, a wireless link, a hard-wired link, any other suitable communication link, or any suitable combination of such links.

[0073] The table dimension statistical caliber calculation client 103 may include any one or more clients that present an interface related to the table dimension statistical caliber calculation in an appropriate form for use and operation by a user. In some embodiments, the table dimension statistical caliber calculation client 103 may include any suitable type of device. For example, in some embodiments, the table dimension statistical caliber calculation client 103 may include a mobile device, a tablet computer, a laptop computer, a desktop computer, and / or any other suitable type of client device.

[0074] Although the table dimension statistical caliber calculation server 101 is illustrated as one device, in some embodiments, any appropriate number of devices may be used to perform the functions performed by the table dimension statistical caliber calculation server 101. For example, in some embodiments, multiple devices may be used to implement the functions performed by the table dimension statistical caliber calculation server 101. Alternatively, a cloud service may be used to implement the functions of the table dimension statistical caliber calculation server 101.

[0075] Based on the above system, an embodiment of the present application provides a method for calculating the statistical caliber of a table dimension, which is illustrated by the following embodiment.

[0076] Figure 3 1 is a flowchart of a method for calculating the statistical caliber of a table dimension according to an embodiment of the present invention. The method for calculating the statistical caliber of a table dimension in this embodiment can be executed on a table dimension statistical caliber calculation server, such as Figure 3 As shown, the method for calculating the statistical caliber of the table dimension includes the following steps:

[0077] Step S201: regularly collect basic statistical information of the table, and obtain the table's blood relationship information and the number of task executions in which the table participates based on the collected basic statistical information of the table.

[0078] As an optional example, in this embodiment, the basic statistical information of the table includes the storage size occupied by the table, the number of data items in the table, the last update time of the table data, the number of times the table is browsed in the system, and the number of times the table is collected. For example, in this embodiment, the basic statistical information of the table, such as the storage size occupied by the table, the number of data items in the table, the last update time of the table data, the number of times the table is browsed in the system, and the number of times the table is collected, is uniformly managed and regularly counted. The blood relationship of the table and the number of times the table is involved in the execution of tasks are analyzed and saved.

[0079] In this embodiment, the blood relationship means that the table structure or table data of one table is wholly or partially derived from another table. For example, the syntax of insert overwrite..select or create table..as select or insert into..select can generate the blood relationship between two tables. There are direct blood relationship and indirect blood relationship.

[0080] In this embodiment, the number of task executions indicates the number of times a table is used in a data governance task.

[0081] Step S202: Obtain the basic statistical data of the table based on the basic statistical information, blood relationship information and the number of task executions in which the table participates, and construct a first data attribute table based on the obtained basic statistical data.

[0082] As an optional example, in this embodiment, the attributes of the first data attribute table include the number of table collections in the last week, the computing engine Id, the data partition Id, the table Id, the total storage capacity of the table, the total number of table data items, the last update time of the table data, the table storage growth in the last week, the growth in the number of table data items in the last week, the number of table executions in the last week, the number of upstream direct blood relationship tables, the number of downstream direct blood relationship tables, the number of downstream blood relationship tables, and the number of table browse records in the last week.

[0083] For example, in actual application, this embodiment obtains the basic statistical data of all tables based on the basic statistical information of the table, the blood relationship information and the number of task executions in which the table participates, and assembles them into a List <tablemetadatabo>The properties of the TableMetadataBO entity class are shown in Table 1 below.

[0084] Table 1

[0085]

[0086]

[0087] Step S203: Calculate the pre-prepared data of the table dimension statistical caliber according to the attribute data in the first data attribute table, and construct the second data attribute table according to the pre-prepared data of the table dimension statistical caliber obtained by calculation.

[0088] As an optional example, in this embodiment, the pre-prepared data of the table dimension statistical caliber include: the maximum number of times the table participated in the execution of the task in the past week, the minimum number of times the table participated in the execution of the task in the past week, the maximum storage growth of all tables in the past week, the minimum storage growth of all tables in the past week, the maximum storage growth ratio, the minimum storage growth ratio, the maximum number of data items in all tables in the past week, the minimum number of data items in all tables in the past week, the maximum number of data items, the minimum number of data items, the maximum dependency concentration, the minimum dependency concentration, the maximum storage occupancy of all tables, the minimum storage occupancy of all tables, the maximum number of downstream and downstream bloodline tables in all tables, the minimum number of downstream and downstream bloodline tables in all tables, the maximum number of views of all tables in the past week, the minimum number of views of all tables in the past week, the maximum number of collections of all tables in the past week, and the minimum number of collections of all tables in the past week.

[0089] For example, in actual application, this embodiment calculates the pre-preparation data of the table dimension statistical caliber according to the attribute data in the first data attribute table, and constructs the second data attribute table according to the pre-preparation data of the table dimension statistical caliber obtained by calculation, as shown in Table 2 below.

[0090] Table 2

[0091]

[0092]

[0093] Step S204: loop through the first data attribute table and the second data attribute table, and calculate the score of each dimension statistical caliber of each table.

[0094] This embodiment calculates the access heat score of the table based on the maximum number of times the table has participated in task execution in the last week, the minimum number of times the table has participated in task execution in the last week, and the number of times the table has been executed in the last week. As an optional example, in the method of this embodiment, the access heat score of the table is calculated as follows:

[0095] rddf (access popularity score): If rdmax-rdmin=0 or the number of table executions in the last week-rdmin=0, then rddf=1; otherwise, rddf=1+(99*(number of table executions in the last week-rdmin) / (rdmax-rdmin)).

[0096] This embodiment calculates the dependency concentration score of the table based on the maximum dependency concentration, the minimum dependency concentration, the number of upstream direct blood relationship tables, and the number of downstream direct blood relationship tables. As an optional example, in the method of this embodiment, the dependency concentration score of the table is calculated in the following manner:

[0097] yljzddf (dependency concentration score of the table): If yljzdmax-yljzdmin=0, then yljzddf=0; otherwise, yljzddf=1+(99*(the number of upstream direct blood relationship tables+the number of downstream direct blood relationship tables-yljzdmin) / (yljzdmax-yljzdmin)).

[0098] In this embodiment, the table storage score is calculated based on the total storage capacity of the table, the maximum storage occupancy of all tables, and the minimum storage occupancy of all tables, and the benefit cost score of the table is calculated based on the access popularity score of the table and the table storage score. As an optional example, in the method of this embodiment, the benefit cost score of the table is calculated as follows:

[0099] bccdf (table storage score): If the total table storage is 0 or sycbmax-sycbmin=0, bccdf=1; otherwise, bccdf=1+(99*(total table storage-sycbmin) / (sycbmax-sycbmin));

[0100] sydf (benefit cost score of the table) = rddf*0.5+(100-bccdf)*0.5.

[0101] In this embodiment, the influence score of a table is calculated based on the number of downstream kinship tables, the maximum number of downstream kinship tables in all tables, and the minimum number of downstream kinship tables in all tables. As an optional example, in the method of this embodiment, the influence score of a table is calculated as follows:

[0102] Yxldf (influence score): If the number of downstream bloodline tables is 0 or yxlmax-yxlmin=0, then yxldf=1; otherwise, yxldf=1+(99*(number of downstream bloodline tables-yxlmin) / (yxlmax-yxlmin)).

[0103] In this embodiment, the data timeliness score of the table is calculated based on the last modification time of the table data and the current time. As an optional example, in the method of this embodiment, the data timeliness score of the table is calculated in the following manner:

[0104] Sjjsxdf (data timeliness score) = 100 - the number of days between the last change time of the table data and the current time.

[0105] In this embodiment, the growth score of the table is calculated based on the storage growth of the table in the most recent week, the maximum storage growth of all tables in the most recent week, and the minimum storage growth of all tables in the most recent week; the growth proportion score of the table is calculated based on the storage occupancy of the table in the most recent week, the maximum storage growth ratio, and the minimum storage growth ratio; the data score of the table is calculated based on the growth of the number of data items in the most recent week, the maximum growth of the number of data items in all tables in the most recent week, and the minimum growth of the number of data items in all tables in the most recent week; the data proportion score of the table is calculated based on the total number of data items in the table in the most recent week, the maximum growth ratio of the number of data items, and the minimum growth ratio of the number of data items; a weighted sum value is obtained by weighted summing the growth score of the table, the growth proportion score of the table, the data score of the table, and the data proportion score of the table, and the weighted sum value is used as the growth score of the table. As an optional example, in the method of this embodiment, the growth score of the table is calculated as follows:

[0106] When calculating zzxdf (growth score), the following scores need to be calculated first:

[0107] Zzdf (growth score): If the table storage growth in the most recent week is 0 or ccmax-ccmin=0, then zzdf=1; otherwise, zzdf=1+(99*(table storage growth in the most recent week-ccmin) / (ccmax-ccmin));

[0108] Zzzbdf (growth ratio score): If the table storage usage in the most recent week is 0 or cczbmax-cczbmin=0, then zzzbdf=1; otherwise, zzzbdf=1+(99 / (cczbmax-cczbmin)*(table storage growth in the most recent week / total table storage)-cczbmin);

[0109] sjdf (data score): if the growth of the number of table data items in the most recent week is 0, then sjdf = 1; otherwise, sjdf = 1 + (99 / (sjmax-sjmin)*(the growth of the number of table data items in the most recent week - sjmin));

[0110] sjzbdf (data proportion score): If the total number of table data in the most recent week is 0 or sjzbmax-sjzbmin=0, then sjzbdf=1; otherwise, sjzbdf=1+(99 / (sjzbmax-sjzbmin)*((the increase in the number of table data in the most recent week / the total number of table data)-sjzbmin);

[0111] After calculating zdf, zzzbdf, sjdf and sjzbdf, zzxdf is calculated according to zzdf, zzzbdf, sjdf and sjzbdf, zzxdf=zzdf*0.15+sjdf*0.15+zzzbdf*0.35+sjzbdf*0.35.

[0112] In this embodiment, the collection score of a table is calculated based on the number of collections of the table in the last week, the maximum number of collections of all tables in the last week, and the minimum number of collections of all tables in the last week; the browsing score of the table is calculated based on the number of browse records of the table in the last week, the maximum number of browse records of all tables in the last week, and the minimum number of browse records of all tables in the last week; the weighted sum value is obtained by weighted summing the collection score of the table and the browsing score of the table, and the weighted sum value is used as the attention score of the table. As an optional example, in the method of this embodiment, the attention score of the table is calculated in the following way:

[0113] When calculating the popularity score of a table, the collection score and the browse score of the table are calculated first.

[0114] scdf (collection score): If the number of collections in the most recent week is 0 or bscmax-bscmin=0, then scdf=1; otherwise, scdf=1+(99*(number of collections in the most recent week-bscmin) / (bscmax-bscmin));

[0115] blldf (table browsing score): If the number of table browsing records in the last week is 0 or bllmax-bllmin=0, then blldf=1; otherwise blldf=1+(99*(number of table browsing records in the last week-bllmin) / (bllmax-bllmin));

[0116] Calculate gzddf based on scdf and blldf, gzddf=scdf*0.5+blldf*0.5.

[0117] Step S205: adding labels to the statistical caliber of each dimension of the table, saving the statistical caliber of the corresponding table dimension of the tagged table to the database and displaying it on the page.

[0118] As an optional example, this embodiment saves the scores of the statistical caliber of each dimension of the table to the database according to the table ID and the data partition ID and displays them on the page; sorts the statistical caliber of the relevant dimensions of all tables according to the scores of the statistical caliber of each dimension, and obtains the binary value of the statistical caliber of the relevant dimension; compares the score of the statistical caliber of the corresponding table dimension of the table with the corresponding binary value, and when the score of the statistical caliber of the corresponding table dimension of the table is not less than the corresponding binary value, adds a high index label to the statistical caliber of the corresponding table dimension of the table; when the score of the statistical caliber of the corresponding table dimension of the table is less than the corresponding binary value, adds a low index label to the statistical caliber of the corresponding table dimension of the table; saves the statistical caliber of the corresponding table dimension of the table with the label to the database and displays it on the page.

[0119] For example, this embodiment saves the calculated statistical caliber indicator results of each table dimension in combination with information such as table ID and data partition ID to the database, and provides a display of statistical caliber indicators of each table dimension on the page; all tables are labeled to distinguish high and low indicator labels of each statistical caliber of the table. The labeling process is as follows, taking the table heat as an example: the heat of all tables is sorted in ascending order, and the heat binary value is taken. If the current table heat is greater than the heat binary value, it is defined as a high heat label, otherwise it is defined as a low heat label. After all indicators of all tables are labeled, they are stored in the database to provide an indicator label display for the page table.

[0120] The table dimension statistical caliber calculation method and system of this embodiment can directly display the scores of each statistical caliber of the table and the high and low levels of each caliber on the page by calculating the scores of the table statistical caliber and marking the statistical caliber, so that data governance personnel can analyze the value of the table from various caliber dimensions and analyze the governance effectiveness of the table, which is conducive to improving the efficiency and accuracy of data governance.

[0121] like Figure 4 As shown, the present invention also provides a device, including a processor 310, a communication interface 320, a memory 330 for storing a computer program executable by the processor, and a communication bus 340. The processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 implements the above-mentioned table dimension statistical caliber calculation method by running the executable computer program.

[0122] Among them, the computer program in the memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0123] The system embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected based on actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0124] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0125] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.< / tablemetadatabo>

Claims

1. A method for calculating the statistical caliber of table dimensions, characterized in that: The method comprises: Collect basic statistical information of tables regularly, and obtain the table's lineage information and the number of task executions in which the table participates based on the collected basic statistical information of the table; According to the basic statistical information of the table, the blood relationship information and the number of task executions in which the table participates, the basic statistical data of the table are obtained, and the first data attribute table is constructed according to the obtained basic statistical data; Calculate pre-prepared data of the table dimension statistical caliber according to the attribute data in the first data attribute table, and construct a second data attribute table according to the pre-prepared data of the table dimension statistical caliber obtained by calculation; Loop through the first data attribute table and the second data attribute table, and calculate the score of each dimension of the table for each table; Add labels to the statistical caliber of each dimension of the table, save the statistical caliber of the corresponding table dimension of the labeled table to the database and display it on the page; Basic statistics of the table, including the storage size occupied by the table, the number of data entries in the table, the last update time of the table data, the number of times the table has been browsed in the system, and the number of times the table has been added to favorites; The attributes of the first data attribute table include the number of table collections in the last week, calculation engine ID, data partition ID, table ID, total table storage capacity, total number of table data items, last update time of table data, table storage growth in the last week, growth in the number of table data items in the last week, number of table executions in the last week, number of upstream direct lineage relationship tables, number of downstream direct lineage relationship tables, number of downstream lineage tables, and number of table browse records in the last week; Preliminary data for table dimension statistics include: the maximum number of times a table has participated in task execution in the past week, the minimum number of times a table has participated in task execution in the past week, the maximum storage growth of all tables in the past week, the minimum storage growth of all tables in the past week, the maximum storage growth ratio, the minimum storage growth ratio, the maximum number of data items in all tables in the past week, the minimum number of data items in all tables in the past week, the maximum number of data items in the past week, the minimum number of data items in the past week, the maximum dependency concentration, the minimum dependency concentration, the maximum storage usage of all tables, the minimum storage usage of all tables, the maximum number of downstream and downstream lineage tables in all tables, the minimum number of downstream and downstream lineage tables in all tables, the maximum number of views of all tables in the past week, the minimum number of views of all tables in the past week, the maximum number of favorites for all tables in the past week, and the minimum number of favorites for all tables in the past week.

2. The method for calculating the statistical caliber of table dimension according to claim 1, characterized in that: Loop through the first data attribute table and the second data attribute table, and calculate the scores of each dimension of each table, including: The access popularity score of the table is calculated based on the maximum number of times the table has participated in task execution in the past week, the minimum number of times the table has participated in task execution in the past week, and the number of times the table has been executed in the past week; The dependency concentration score of the table is calculated based on the maximum dependency concentration, the minimum dependency concentration, the number of upstream direct blood relationship tables, and the number of downstream direct blood relationship tables; The table storage score is calculated based on the total storage capacity of the table, the maximum storage capacity of all tables, and the minimum storage capacity of all tables. The table benefit cost score is calculated based on the table access popularity score and the table storage score. The influence score of a table is calculated based on the number of downstream kinship tables, the maximum number of downstream kinship tables among all tables, and the minimum number of downstream kinship tables among all tables. Calculate the data timeliness score of the table based on the last modification time of the table data and the current time.

3. The method for calculating the statistical caliber of table dimension according to claim 1, characterized in that: Loop through the first data attribute table and the second data attribute table, and calculate the scores of each dimension of each table, including: The growth score of the table is calculated based on the storage growth of the table in the past week, the maximum storage growth of all tables in the past week, and the minimum storage growth of all tables in the past week. Calculate the table's growth score based on the table's storage usage, maximum storage growth ratio, and minimum storage growth ratio in the most recent week. The data score of the table is calculated based on the growth of the number of data items in the last week, the maximum growth of the number of data items in all tables in the last week, and the minimum growth of the number of data items in all tables in the last week; Calculate the data proportion score of the table based on the total number of data items in the table in the past week, the growth rate of the largest number of data items, and the growth rate of the smallest number of data items; A weighted sum value is obtained by taking a weighted sum of the growth score of the table, the growth proportion score of the table, the data score of the table and the data proportion score of the table, and the weighted sum value is used as the growth score of the table.

4. The method for calculating the statistical caliber of table dimension according to claim 1, characterized in that: Loop through the first data attribute table and the second data attribute table, and calculate the scores of each dimension of each table, including: The collection score of a table is calculated based on the number of collections of the table in the last week, the maximum number of collections of all tables in the last week, and the minimum number of collections of all tables in the last week; The table browsing score is calculated based on the number of table browsing records in the past week, the maximum number of browsing records of all tables in the past week, and the minimum number of browsing records of all tables in the past week; A weighted sum value is obtained by performing a weighted summation on the collection score and the browsing score of the table, and the weighted sum value is used as the attention score of the table.

5. The method for calculating table dimension statistical caliber according to claim 1, characterized in that: Add labels to the statistical caliber of each dimension of the table, save the statistical caliber of the corresponding table dimension of the tagged table to the database and display it on the page, including: According to the table ID and data partition ID, the scores of each dimension of the table are saved to the database and displayed on the page; Sort the relevant dimension statistical calibers of all tables according to the scores of the statistical calibers of each dimension, and obtain the binary values ​​of the relevant dimension statistical calibers; Compare the score of the corresponding table dimension statistical caliber of the table with the corresponding binary value. When the score of the corresponding table dimension statistical caliber of the table is not less than the corresponding binary value, add a high index label to the corresponding table dimension statistical caliber of the table. When the score of the corresponding table dimension statistical caliber of the table is less than the corresponding binary value, add a low index label to the corresponding table dimension statistical caliber of the table. The corresponding table dimension statistical caliber of the tagged table is saved to the database and displayed on the page.

6. A table dimension statistical caliber calculation system, characterized in that: The system includes a table dimension statistical caliber calculation server, and the table dimension statistical caliber calculation server is used to: Collect basic statistical information of tables regularly, and obtain the table's lineage information and the number of task executions in which the table participates based on the collected basic statistical information of the table; According to the basic statistical information of the table, the blood relationship information and the number of task executions in which the table participates, the basic statistical data of the table are obtained, and the first data attribute table is constructed according to the obtained basic statistical data; Calculate pre-prepared data of the table dimension statistical caliber according to the attribute data in the first data attribute table, and construct a second data attribute table according to the pre-prepared data of the table dimension statistical caliber obtained by calculation; Loop through the first data attribute table and the second data attribute table, and calculate the score of each dimension of the table for each table; Add labels to the statistical caliber of each dimension of the table, save the statistical caliber of the corresponding table dimension of the labeled table to the database and display it on the page; Basic statistics of the table, including the storage size occupied by the table, the number of data entries in the table, the last update time of the table data, the number of times the table has been browsed in the system, and the number of times the table has been added to favorites; The attributes of the first data attribute table include the number of table collections in the last week, calculation engine ID, data partition ID, table ID, total table storage capacity, total number of table data items, last update time of table data, table storage growth in the last week, growth in the number of table data items in the last week, number of table executions in the last week, number of upstream direct lineage relationship tables, number of downstream direct lineage relationship tables, number of downstream lineage tables, and number of table browse records in the last week; Preliminary data for table dimension statistics include: the maximum number of times a table has participated in task execution in the past week, the minimum number of times a table has participated in task execution in the past week, the maximum storage growth of all tables in the past week, the minimum storage growth of all tables in the past week, the maximum storage growth ratio, the minimum storage growth ratio, the maximum number of data items in all tables in the past week, the minimum number of data items in all tables in the past week, the maximum number of data items in the past week, the minimum number of data items in the past week, the maximum dependency concentration, the minimum dependency concentration, the maximum storage usage of all tables, the minimum storage usage of all tables, the maximum number of downstream and downstream lineage tables in all tables, the minimum number of downstream and downstream lineage tables in all tables, the maximum number of views of all tables in the past week, the minimum number of views of all tables in the past week, the maximum number of favorites for all tables in the past week, and the minimum number of favorites for all tables in the past week.

7. The table dimension statistical caliber calculation system according to claim 6, characterized in that: The table dimension statistical caliber calculation server includes: The information collection module is used to collect the basic statistical information of the table at regular intervals, and obtain the table's blood relationship information and the number of task executions in which the table participates based on the collected basic statistical information of the table; A first data attribute table construction module is used to obtain the basic statistical data of the table according to the basic statistical information of the table, the blood relationship information and the number of task executions in which the table participates, and to construct the first data attribute table according to the obtained basic statistical data; A second data attribute table construction module, used to calculate the pre-prepared data of the table dimension statistical caliber according to the attribute data in the first data attribute table, and construct the second data attribute table according to the pre-prepared data of the table dimension statistical caliber obtained by calculation; A dimension statistical caliber score calculation module is used to loop through the first data attribute table and the second data attribute table, and calculate the score of each dimension statistical caliber of each table for each table; The dimension statistical caliber labeling and saving module is used to add labels to the statistical caliber of each dimension of the table, save the statistical caliber of the corresponding table dimension of the labeled table to the database and display it on the page.

Images